Online speaker diarization system and method for training a model for use in online speaker diarization

Abstract

To improve precision of on-line speaker diarization in the beginning of utterance.SOLUTION: For on-line speaker diarization, a neural network-based model is used. Learning of the model includes two stages. The first stage includes: inputting the model to a first chunk (speech data) to acquire a first permutation data segment (data representing relation between a speaker and an utterance probability in each time section) from the model; and then calculating permutation loss. The first chunk is divided into a plurality of second chunks. The second stage includes: inputting the model to a plurality of second chunks to acquire a plurality of second permutation data segments; and then utilizing the plurality of second permutation data and a result of the first stage to calculate binary cross entropy.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates generally to online speaker diarization. [Background technology]

[0002] As a technology related to speaker distinction, for example, there is a technology disclosed in Patent Document 1. According to Patent Document 1, simultaneous speech by multiple speakers is not assumed, but papers written by some of the inventors of the present invention (Non-Patent Documents 1 and 2) disclose a technology related to speaker diarization that assumes simultaneous speech by multiple speakers. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-109712 [Non-patent literature]

[0004] [Non-Patent Document 1] ONLINE END-TO-END NEURALDIARIZATION WITH SPEAKER-TRACING BUFFER(https: / / arxiv.org / pdf / 2006.02616.pdf) [Non-patent document 2] Online Streaming End-to-EndNeural Diarization Handling Overlapping Speech and Flexible Numbers of Speakers(https: / / arxiv.org / pdf / 2101.08473.pdf) Summary of the Invention [Problem to be solved by the invention]

[0005] According to Non-Patent Documents 1 and 2, speaker diarization is performed using a machine-learned neural network-based learning model (hereinafter referred to as a speaker diarization model). Specifically, for example, Non-Patent Documents 1 and 2 disclose chunk-wise speaker diarization that divides speech data into multiple chunks, and also disclose a speaker tracking buffer mechanism for the speaker diarization. The speaker tracking buffer mechanism selects multiple input frames representing speaker sequence information from the previous chunk and stores the multiple input frames in a speaker tracking buffer (STB).

[0006] The online speaker diarization disclosed in Non-Patent Documents 1 and 2 does not provide high accuracy in speaker diarization (identifying who spoke and when) in the early stage of an utterance (for example, within about 30 seconds from the start of the utterance). The reasons for this are considered to be as follows, for example. The learning model applied to the STB is an offline learning model trained using chunks with a fixed large chunk size. · Unsupervised covariance is used in learning. [Means for solving the problem]

[0007] Online speaker diarization involves sequentially inputting chunks into a speaker diarization model (a neural network-based model for speaker diarization) and sequentially identifying which speaker spoke at what time based on permutation data segments sequentially obtained from the speaker diarization model. A "chunk" is speech data for a certain period of time from speech by N speakers (N is a natural number) or data for a certain period of time based on the speech data. A "permutation data segment" is data representing the relationship between M speakers (M is a natural number and M≦N) and speech probabilities for each of multiple time intervals in the certain period of time.

[0008] The training of the speaker diarization model is done in two stages.

[0009] The first stage of learning includes inputting a first chunk (a chunk of a first time duration) into a speaker diarization model to obtain a first permutation data segment from the speaker diarization model, and calculating a permutation loss of the first permutation data segment.

[0010] The first chunk is divided into a plurality of second chunks, each chunk having a second time duration that is shorter than the first time duration.

[0011] The second-stage training includes inputting the second chunks into the speaker diarization model to obtain second permutation data segments, obtaining permutation results using the training permutation data segments for the second permutation data segments, and calculating binary cross-entropy for each of the permutation results. Note that the "training permutation data segments" are permutation data segments for the first time period based on the permutation loss and the first permutation data segment. [Effects of the Invention]

[0012] According to the present invention, it is possible to improve the accuracy of online speaker diarization at the beginning of an utterance. [Brief explanation of the drawings]

[0013] [Figure 1] 1 illustrates an example of the hardware configuration of a computer system according to an embodiment. [Figure 2] 1 shows an example of the configuration of an online speaker diarization system. [Figure 3] The learning flow of the FS-EEND model is shown below. [Figure 4] An example in which the consistency of permutation is not maintained is shown below. [Figure 5]An example in which the consistency of permutation is maintained is shown. [Figure 6] Here is the reasoning flow for the first chunk: [Figure 7] 1 shows an example of chunks and permutation data segments. [Figure 8] An example of a combined chunk is shown below. [Figure 9] An example of inference processing will be described below. [Figure 10] An example of inference processing will be described below. DETAILED DESCRIPTION OF THE INVENTION

[0014] In the following description, an "interface apparatus" may refer to one or more interface devices, which may be at least one of the following: One or more I / O (Input / Output) interface devices. The I / O (Input / Output) interface devices are interface devices for at least one of the I / O device and a remote display computer. The I / O interface device for the display computer may be a communications interface device. The at least one I / O device may be a user interface device, for example, either an input device such as a keyboard and a pointing device, or an output device such as a display device. One or more communication interface devices. The one or more communication interface devices may be one or more homogeneous communication interface devices (e.g., one or more NICs (Network Interface Cards)) or two or more heterogeneous communication interface devices (e.g., an NIC and an HBA (Host Bus Adapter)).

[0015] In the following description, "memory" refers to one or more memory devices, which are an example of one or more storage devices, and may typically be a primary storage device. At least one memory device in the memory may be a volatile memory device or a non-volatile memory device.

[0016] In the following description, a "persistent storage device" may refer to one or more persistent storage devices, which are an example of one or more storage devices. A persistent storage device may typically be a non-volatile storage device (e.g., an auxiliary storage device), and more specifically, may be, for example, a hard disk drive (HDD), a solid state drive (SSD), a non-volatile memory express (NVMe) drive, or a storage class memory (SCM).

[0017] In the following description, the term "storage device" may refer to at least one of memory and persistent storage device.

[0018] Furthermore, in the following description, a "processor" may refer to one or more processor devices. The at least one processor device may typically be a microprocessor device such as a CPU (Central Processing Unit), but may also be another type of processor device such as a GPU (Graphics Processing Unit). The at least one processor device may be a single-core or multi-core. The at least one processor device may also be a processor core. The at least one processor device may also be a processor device in a broader sense, such as a circuit that is a collection of gate arrays written in a hardware description language that performs some or all of the processing (for example, an FPGA (Field-Programmable Gate Array), a CPLD (Complex Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit)).

[0019] Furthermore, in the following description, functions may be described using the expression "yyy unit." However, the functions may be realized by one or more computer programs executed by a processor, by one or more hardware circuits (e.g., FPGAs or ASICs), or by a combination thereof. When a function is realized by a program executed by a processor, the specified processing is performed using a storage device and / or an interface device, etc., as appropriate, and therefore the function may be considered to be at least a part of the processor. Processing described using a function as the subject may be processing performed by a processor or a device having the processor. A program may be installed from a program source. The program source may be, for example, a computer from which the program is distributed or a computer-readable recording medium (e.g., a non-transitory recording medium). The description of each function is merely an example; multiple functions may be combined into one function, or one function may be divided into multiple functions.

[0020] In the following description, when elements of the same type are described without distinction, common reference symbols are used, and when elements of the same type are described with distinction, reference symbols are used.

[0021] FIG. 1 is a diagram showing an example of the hardware configuration of a computer system according to an embodiment of the present invention.

[0022] The computer system 300 may be a computer such as a desktop computer, a mobile computer, or a smartphone. The computer system 300 includes a processor 302 (e.g., multiple CPUs 302A and 302B), a memory 304, a display system 324, a terminal interface 312, a storage interface 314, an I / O (input / output) device interface 316, and a network interface 318. These components may be interconnected via a memory bus 306, an I / O bus 308, a bus interface 309, and an I / O bus interface 310. The computer system 300 may be a physical computer system (one or more physical computers (e.g., a cloud infrastructure)) or a logical computer system based on a physical computer system (e.g., a cloud computing service system).

[0023] The memory 304 may store a speaker diarization application 350 (computer program). Execution of the speaker diarization application 350 on the processor 302 may cause the computer system 300 to function as an online speaker diarization system 380 (see FIG. 2).

[0024] Computer system 300 may include a bus interface 309 that provides communication between processor 302, memory 304, display system 324, and I / O bus interface 310. I / O bus interface 310 may couple to I / O bus 308 for transferring data to and from various I / O units. I / O bus interface 310 may communicate via I / O bus 308 with multiple I / O interface units 312, 314, 316, and 318, also known as I / O processors (IOPs) or I / O adapters (IOAs).

[0025] The display system 324 is connected to the display device 326 and may include a display controller for controlling the display, a display memory for storing data to be displayed, or both. The display memory may be part of the memory 304.

[0026] The I / O interface unit has the function of communicating with various storage or I / O devices. For example, the terminal interface unit 312 is connected to a user I / O device 320 (e.g., a user output device such as a speaker, and a user input device such as a microphone). The storage interface 314 is connected to a persistent storage device 322. The I / O device interface 316 may be an interface to a printer or the like. The network interface 318 may be an interface to a network 330. The network 330 may be the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), or the like.

[0027] The interfaces 312, 314, 316, and 318 and the display system 324 may be examples of interface devices. Furthermore, the computer system 300 may not include all of the components illustrated in Fig. 1. The computer system 300 may include, for example, an interface device, a storage device, and a processor connected thereto. The storage device may include an STB.

[0028] FIG. 2 is a diagram showing an example of the functional configuration of an online speaker diarization system 380 according to this embodiment.

[0029] The online speaker diarization system 380 is connected to a client terminal 375 and / or a voice data acquisition device 365 via the network 330 (or without the network 330). At least some of the functions of the client terminal 375 and the voice data acquisition device 365 may be included in the online speaker diarization system 380.

[0030] The audio data acquisition device 365 is a device for acquiring audio data. This audio data acquisition device 365 may be, for example, a computing device equipped with a microphone, such as a smartphone or a personal computer, a recording device, or the like, or may be a microphone itself.

[0031] The client terminal 375 is an information processing terminal that transmits the voice data acquired by the voice data acquisition device 365 and the speaker number data indicating the number of possible speakers to the online speaker diarization system 380 via the network 330. Input data such as the voice data and the speaker number data may be input from the user I / O device 320 illustrated in FIG.

[0032] The storage area 372 is an area based on a storage device (e.g., memory 304 and persistent storage 322) of the computer system 300. The storage area 372 includes, for example, an STB. The STB may be an area provided in the memory 304. At least a part of the storage area 372 may be an area based on a storage device external to the online speaker diarization system 380. Input data such as speech data and speaker count data is stored in the storage area 372.

[0033] The online speaker diarization system 380 includes a memory area 372, an input unit 382, a dual-mode EEND model 22, a training unit 360, and an inference unit 342. The dual-mode EEND model 22 may be stored in the memory area 372.

[0034] The storage area 372 is an area based on at least a part of the memory 304 and the persistent storage device 322, and includes an STB (speaker tracking buffer) 374. As will be described later, the STB 374 is a buffer in which chunks past the chunk to be inferred and permutation data segments obtained for the past chunks are stored during inference. Using such an STB 374, online speaker diarization with a flexible number of speakers (without a limit on the number of speakers) is performed.

[0035] The input unit 382 receives voice data and speaker number data input from the client terminal 375 (or the voice data acquisition device 365 ), and stores the voice data and speaker number data in the storage area 372 .

[0036] The dual-mode EEND model 22 is a neural network-based chunk-wise model trained by the training unit 360 and used in inference (online speaker diarization) by the inference unit 342. The dual-mode EEND model 22 is an example of a speaker diarization model. The term "dual-mode EEND model" means that the EEND model is used in inference in a selected mode from two modes. "EEND" stands for End-to-End Neural Diarization. The EEND may be, for example, an EEND that uses a self-attention method and has a flexible (unlimited) number of speakers. The EEND model is a neural network-based model configured to determine the probability that a speaker is speaking in a given time interval. EEND can accurately identify when a speaker spoke, even when multiple speakers' speech overlaps in input speech data. In the description of this embodiment, the term "end-to-end" is used to mean that a series of processes from input speech data to outputting speaker diarization results can be performed using a neural network-based model.

[0037] The learning unit 360 performs two-stage learning of the dual-mode EEND model 22. Specifically, the learning unit 360 has a first-stage learning unit 361 that performs the first-stage learning, a second-stage learning unit 362 that performs the second-stage learning, and a learning control unit 363 that controls the learning.

[0038] The inference unit 342 performs inference using the trained dual-mode EEND model 22. Online speaker diarization as inference involves sequentially inputting chunks into the dual-mode EEND model 22 and sequentially identifying which speaker spoke and when based on permutation data segments sequentially obtained from the dual-mode EEND model 22.

[0039] Here, a "chunk" is a certain amount of speech data of N speakers (N is a natural number) for a certain period of time, or data based on the speech data. In this embodiment, a chunk is data based on the speech data for a certain period of time, specifically, data representing an acoustic feature in each of a plurality of time intervals that make up the certain period of time.

[0040] A "permutation data segment" is data that represents the relationship between M speakers (M is a natural number and M≦N) and the utterance probability for each of a plurality of time intervals in the given time. In other words, the permutation data segment represents the utterance probability for each of the M speakers for each time interval.

[0041] FIG. 3 shows the flow of learning the FS-EEND model according to the embodiment.

[0042] It is assumed that speech data and speaker number data are stored in memory area 372 for learning purposes. The speech data is data on speech uttered by N speakers (N is a natural number). The number of speakers represented by the speaker number data is K. The speech data and speaker number data for learning may be used for inference. In other words, the speech data and speaker number data used in inference may be used in learning.

[0043] The learning control unit 363 acquires one or more first chunks from the audio data. Two-stage learning is performed for each of the one or more first chunks. In the following description of FIG. 3, one first chunk 27 will be taken as an example. The first chunk 27 is a chunk of a first time duration (audio data of a first time duration). In the description of learning, the first chunk may be referred to as a "whole chunk."

[0044] The first-stage learning unit 361 performs the first stage of learning. Specifically, the first-stage learning unit 361 inputs the first chunk 27 to the dual-mode EEND model 22 to obtain a first permutation data segment 29 from the dual-mode EEND model 22. The first-stage learning unit 361 calculates a permutation loss 30 of the first permutation data segment 29. The permutation loss 30 is calculated using, for example, PIT (Permutation-Invariant Training). Note that in FIG. 3, for each time interval, "x α " means the feature value in the time interval α, and "y α ” means the probability of each speaker speaking in the time interval α, and “y^ α " is x α means the output from the dual-mode EEND model 22 with input y^´ α " is y^ α The data obtained by applying a predetermined process to α ) In FIG. 3, the symbol "^" is located directly above the character, but in this specification it is written next to the character for convenience.

[0045] The learning control unit 363 divides the first chunk 27 into multiple second chunks 21 (e.g., 21A to 21C), each of which is a chunk of a second time period shorter than the first time period. In the explanation of learning, the second chunks may be called "sub-chunks." Furthermore, the first time period may be an integer multiple of the second time period.

[0046] The second-stage learning unit 362 performs second-stage learning. Specifically, the second-stage learning unit 362 inputs the plurality of second chunks 21A to 21C into the dual-mode EEND model 22 (the same model as the model used in the first-stage learning) to obtain a plurality of second permutation data segments 23A to 23C. The second-stage learning unit 362 obtains a plurality of permutation results 24A to 24C in which the permutation data segment 31 for learning is used for the plurality of second permutation data segments 23A to 23C. The second-stage learning unit 362 calculates the binary cross-entropy 25 for each of the plurality of permutation results 24A to 24C. As a result, a plurality of second permutation data segments 26A to 26C are obtained as a final result. Note that in the second permutation data segment 26, "y~ α ” is the utterance probability for each speaker in the time interval α, and is y^´ after the binary cross-entropy calculation. α The training permutation data segment 31 is a permutation data segment for a first time based on the permutation loss 30 and the first permutation data segment 29. The first permutation data segment 29 is data representing the permutation result using, for example, the speaker order of the ground truth labels.

[0047] According to the description of FIG. 3 , the first-stage learning is learning using a first chunk (a large-sized chunk). The second-stage learning is learning using a plurality of second chunks (a plurality of small-sized chunks (a plurality of sub-chunks)) that constitute the first chunk. In the second-stage learning, the permutation data segment 31 for learning obtained as a result of the first-stage learning is used as a kind of training data (i.e., the learning in this embodiment is so-called supervised learning). In online speaker diarization as inference, small-sized chunks (e.g., chunks of 1 millisecond, 10 milliseconds, or 1 second) are sequentially input. Therefore, the first-stage learning can be called offline learning, and the second-stage learning can be called online learning that uses the results of offline learning. The dual-mode EEND model 22 trained in this way is used in inference (applied to the STB 374), thereby improving the accuracy of online speaker diarization at the beginning of utterance.

[0048] For example, the size (second time length) of the second chunk (sub-junction) in online learning can be the same as the size (the same time length) of the chunks sequentially input in inference. This is expected to further improve the accuracy of online speaker diarization.

[0049] In the first stage of learning, the dual-mode EEND model 22 may be trained to reduce (e.g., minimize) the permutation loss, and in the second stage of learning, the dual-mode EEND model 22 may be trained to reduce (e.g., minimize) the binary cross entropy.

[0050] Maintaining consistency of permutation (speaker order) is important for the accuracy of online speaker diarization. Specifically, if permutation consistency is not maintained, as shown in Figure 4, when the order of speakers A and B in chunk A is changed from that of the next chunk B, the online speaker diarization result for chunk B will be incorrect even if the online speaker diarization result for chunk A (e.g., the labels indicating when each speaker spoke) is correct. On the other hand, if permutation consistency is maintained, as shown in Figure 5, the online speaker diarization results for both chunks A and B will be correct even if the order of speakers A and B in chunk A is changed from that of the next chunk B.

[0051] In this embodiment, inference is performed for each of a plurality of consecutive chunks in the voice data (e.g., voice data for inference) stored in the memory area 372. The size of the chunk in inference is smaller than at least the first chunk (whole chunk) in learning.

[0052] The inference performed in this embodiment will be described below with reference to Figures 6 to 10. For the inference, it is assumed that voice data and speaker number data are stored in memory area 372. It is assumed that the speaker number data represents the number of speakers N (N is a natural number). In the following description, the relationship between the configuration of reference symbols and the elements to which the reference symbols are assigned is as follows: 60X p :pth chunk 60X p(buf) :p-th chunk stored in STB374 70Y p :pth permutation data segment (permutation data segment corresponding to pth chunk) 70Y p(buf) :Chunk 60X p(buf) The p-th permutation data segment corresponding to 70Y^ p: p-th permutation data segment output from the dual-mode EEND model 22 in response to input to the dual-mode EEND model 22 of the p-th chunk 70Y^ p(buf) :Chunk 60X p(buf) Permutation data segment 70Y^ corresponding to p

[0053] When the chunk to be inferred is the first chunk 60X1, the first chunk 60X1 is processed as shown in FIG. 6, for example. That is, the inference unit 342 inputs the first chunk 60X1 to the dual-mode EEND model 22 to obtain the first permutation data segment 70Y^1 corresponding to the first chunk 60X1. The inference unit 342 stores the first chunk 60X1 and the first permutation data segment 70Y^1 in the STB 374. As a result, two matrices, i.e., the chunk 60X1 and the permutation data segment 70Y^1, exist in the STB 374.

[0054] 6, the number of cells constituting a chunk means the number of time intervals (time positions) corresponding to the chunk. The shading of each cell represents a feature. When the chunk to be inferred is the pth chunk 60Xp (p is a natural number) and p is equal to or less than a predetermined threshold Th, a first mode of the two modes related to the dual mode EEND model 22 may be used. The first mode may be an inference mode that uses predetermined parameters of the dual mode EEND model 22 instead of the trained parameters of the dual mode EEND model 22.

[0055] 6, the permutation data segment has two rows, which means that the number of speakers is two. In other words, the number of rows in the permutation data segment indicates the number of speakers. The shading of each cell represents the utterance probability.

[0056] Furthermore, in this embodiment, the chunk size in inference is smaller than the maximum frame size of the STB 374. The "maximum frame size" is the product of the size of one frame of the STB 374 and the maximum number of frames of the STB 374.

[0057] The chunk to be inferred is the i-th chunk 60X i (i is an integer greater than or equal to 2), the i-th chunk 60X i is processed as follows, for example: That is, the inference unit 342 processes the i-th chunk 60X i and the (i-1)th chunk 60X in STB374 i-1(buf) In the join process, the (i-1)th chunk 60X i-1(buf) and the i-th chunk 60X, which is at least a part (or may be the whole) of the (i-1)-th chunk part. i The combining process is performed along the time axis of the chunks. In other words, the (i-1)th chunk part is the (i-1)th chunk 60X i-1(buf) The data may be data corresponding to at least a part of the time interval of the entire time interval constituting the time corresponding to the i-th chunk 60X i may be coupled to

[0058] For this reason, the inference unit 342 may determine whether i is equal to or less than the predetermined threshold Th. Depending on the result of this determination, it may be determined whether the first mode or the second mode of the two modes related to the dual-mode EEND model 22 is to be used. Specifically, if i is equal to or less than the predetermined threshold Th, the first mode may be used. If i exceeds the predetermined threshold Th, the second mode may be used. The second mode may be an inference mode that uses trained parameters of the dual-mode EEND model 22. This is expected to further improve the accuracy of online speaker diarization at the beginning of utterances.

[0059] The inference unit 342 inputs the combined chunk into the dual-mode EEND model 22 to generate the (i-1)th permutation data segment 70Y^ corresponding to the (i-1)th chunk portion. i-1(buf) and the i-th chunk in the combined chunk is 60X i The i-th permutation data segment 70Y^ corresponding to i and get.

[0060] Thereafter, the inference unit 342 calculates the (i-1)th permutation data segment 70Y stored in the STB 374. i-1(buf) and the obtained (i-1)th permutation data segment 70Y^ i-1(buf) and the obtained i-th permutation data segment 70Y^ i For each of the (NM) speakers, if the number of speakers M represented by the permutation data segment is smaller than N, zero padding is performed. "Zero padding" is an example of padding processing, and involves adding a row (data) to the permutation data segment in which the speech probability for each time interval constituting the time corresponding to chunk 60X is set to "0" (an example of a given probability) for each of the (NM) speakers. Zero-padded permutation data segment 70Y i-1(buf) , 70Y^ i-1(buf) and 70Y^ i , permutation data segment 70Z i-1(buf) , 70Z^ i-1(buf) and 70Z^ i (not shown) (For the permutation data segment 70Y where M=N, zero padding is not required, but for convenience, it is written as 70Z i-1(buf) , 70Z^ i-1(buf) and 70Z^ i (The symbol will be used.)

[0061] The inference unit 342 determines the permutation data segment 70Z i-1(buf) and 70Z^ i-1(buf)The inference unit 342 generates a permutation matrix Pφ, which is a matrix based on the correlation (e.g., the highest correlation) between the i-th permutation data segment 70Ẑ. i and the generated permutation matrix Pφ, the i-th chunk 60X i The i-th permutation data segment 70Y corresponding to i This maintains the consistency of the permutation. For example, the i-th permutation data segment 70Y i is the i-th permutation data segment 70Z^ i and the permutation matrix Pφ.

[0062] The inference unit 342 calculates the (i-1)th chunk 60X i-1(buf) The (i-1)th permutation data segment 70Z after zero padding is stored in the STB 374. i-1(buf) Permutation data segment 70Y i-1(buf) The inference unit 342 also stores the i-th permutation data segment 70Y i and the i-th chunk 60X i and are stored in the STB 374. In this way, chunk 60X i The STB 374 is updated every time the chunk 60X is inferred. i At the start of inference, STB374 contains the (i-1)th chunk 60X i-1(buf) and permutation data segment 70Y i-1(buf) and (i-2)th chunk 60X i-2(buf) and permutation data segment 70Y i-2(buf) The extent to which past data sets are stored in the STB 374 may be determined according to the maximum number of frames in the STB, etc.

[0063] Hereinafter, the following explanations may be applied to the inferences as appropriate.

[0064] The inference unit 342 may input the chunk 60X to the dual-mode EEND model 22, and may acquire a permutation data segment 70Y^ corresponding to the chunk 60X from the dual-mode EEND model 22 based on the number of speakers represented by the speaker count data. As illustrated in FIG. 7, the chunk 60X may be composed of ten features corresponding to ten time intervals. The permutation data segment 70Y^ may have a row for each speaker, and each row may contain a value representing the probability (e.g., a probability between 0 and 1) that the speaker spoke in each of the ten time intervals. A plurality of time intervals constituting a time period may be referred to as a "time interval set."

[0065] As illustrated in FIG. 8, the combined chunk 610 includes the (i-1)th chunk part 60XP i-1(buf) and the i-th chunk 60X i The (i-1)th chunk part is 60 XP. i-1(buf) may be a chunk composed of features corresponding to some of the time intervals in the time interval set (for example, 10 time intervals). When such a combined chunk 610 is input to the dual-mode EEND model 22, the permutation data segment 70YP^ i-1(buf) and 70Y^ i The dual-mode EEND model 22 may be configured to use, for each time interval, the probability corresponding to the time interval, based on the probability corresponding to at least one of the time intervals before and after the time interval.

[0066] In the inference, the process illustrated in Fig. 9 may be performed. i-1 By inputting this into the dual mode EEND model 22, the permutation data segment 70Y^ i-1 The inference unit 342 may obtain the chunk 60X. i-1 60 XP for chunks that are part of i-1 And chunk part 60XP i-1Permutation data segment 70YP^ corresponding to i-1 and may be stored in the STB 374. The inference unit 342 may also store the permutation data segment 70Y^ i-1 The permutation data segment 70Y^ is generated by converting each probability in i-1 may be used as the final speaker identification result 770 (the final permutation data segment indicating whether or not each speaker spoke in which time interval).

[0067] In the inference, the process illustrated in Fig. 10 may be performed. That is, the inference unit 342 i-1(buf) and chunk 60X i (combined with) into the dual mode EEND model 22 to permutate the data segment 70YP^ i-1(buf) and permutation data segment 70P^ i You may obtain the following.

[0068] The i-th permutation data segment set 630 (permutation data segment 70YP^ i-1(buf) and 70P^ i ) is, for example, the (i-1)th permutation data segment set (permutation data segment 70YP^ i-2(buf) and 70P^ i-1 ) may be generated independently. Therefore, the i-th chunk 60X i Speakers c3 and c4 and the (i-1)th chunk 60X in their inferences about i-1 The correspondence between speakers c1 and c2 in the inference about is unknown. This is known as the "permutation problem" in neural networks.

[0069] Therefore, to generate a speaker identification result consistent with the speaker identification result 770 described above, the speakers involved in the combined chunk 610 are identified as chunks 60X i-1Therefore, the inference unit 342 may associate the speakers by comparing the probability values in the past permutation data segments stored in the STB 374 with the probability values in the permutation data segment set 630 corresponding to the combined chunk 610. For example, after calculating the permutation data segment set 630, the inference unit 342 may compare the permutation data segments 70P^ calculated for speakers c3 and c4 with the probability values in the past permutation data segments stored in the STB 374 with the probability values in the permutation data segment set 630 corresponding to the combined chunk 610. i and the permutation data segment 70YP^ stored in STB374 i-1(buf) You may generate permutations of (c1×c3, c1×c4, c2×c3, c2×c4).

[0070] Then, for each generated permutation, the inference unit 342 calculates the permutation data segment 70YP^ stored in the STB 374. i-1(buf) and the permutation data segment 70YP^ in the permutation data segment set 630. i-1(buf) The correlation score calculation here may be performed by, for example, an existing correlation coefficient or similarity calculation method, and is not particularly limited.

[0071] After the correlation scores for all permutations are calculated, the inference unit 342 may consider the speakers of the permutations that satisfy a predetermined correlation score criterion (e.g., the highest correlation score) to be the same person. For example, the probability value calculated for speaker c3 (permutation data segment 70YP^ in the permutation data segment set 630) is i-1(buf) If the probability value calculated for speaker c1 matches the probability value stored in STB 374 for speaker c1, and the probability value calculated for speaker c4 matches the probability value stored in STB 374 for speaker c2, then speakers c1 and c3 may be considered to be the same person, and speakers c2 and c4 may be considered to be the same person.

[0072] In this way, the speakers involved in the combined chunk 610 are divided into chunks 60Xi-1 The permutation data segment 70Y^ in the permutation data segment set 630 is then associated with the speaker associated with i may be used to generate a speaker identification result 870 that is consistent with speaker identification result 770 and indicates which speaker spoke.

[0073] After the permutation data segment set 630 is calculated, the inference unit 342 may generate a chunk portion 852, which is at least a part of the combined chunk 610, and a permutation data segment 854 corresponding to the chunk portion, and store the chunk portion 852 and the permutation data segment 854 in the STB 374. i The previous chunk in the inference of 60X i-1(buf) Similarly, the permutation data segment 854 may be the immediately preceding chunk 60X. i-1(buf) Permutation data segment 70Y corresponding to i-1(buf) It can be said that.

[0074] Although one embodiment has been described above, this is merely an example for explaining the present invention, and the scope of the present invention is not limited to this embodiment. The present invention can be implemented in various other forms. For example, in the embodiment, the speaker diarization model is a dual-mode EEND model, but the model adopted as the speaker diarization model may be a neural network-based model other than the dual-mode EEND model (e.g., a single-mode EEND model). [Explanation of symbols]

[0075] 380: Online Speaker Diarization System

Claims

1. a learning unit that learns a speaker diarization model used in online speaker diarization; the speaker diarization model (22) is a neural network-based chunk-wise EEND (End-to-End Neural Diarization) model of speaker diarization; The online speaker diarization is to sequentially input chunks into the speaker diarization model (22) and sequentially identify which speaker spoke at what time based on permutation data segments sequentially obtained from the speaker diarization model (22); The chunk is speech data for a certain period of time of speech of N speakers (N is a natural number) or data for a certain period of time based on the speech data, the permutation data segment is data representing a relationship between M speakers (M is a natural number and M≦N) and utterance probabilities for each of a plurality of time intervals in the fixed time; The learning unit: a first chunk (27) that is a chunk of a first time duration is input to the speaker diarization model (22) to obtain a first permutation data segment (29) from the speaker diarization model (22), and a permutation loss (30) of the first permutation data segment (29) is calculated; Dividing the first chunk (27) into a plurality of second chunks (21A to 21C), each of which is a chunk for a second time period shorter than the first time period; a plurality of pre-processed second permutation data segments (23A to 23C) are obtained as a plurality of second permutation data segments by inputting the plurality of second chunks (21A to 21C) into the speaker diarization model (22); a plurality of permutation results (24A to 24C) are obtained by using a permutation data segment (31) for learning on the plurality of pre-processed second permutation data segments (23A to 23C); and a plurality of post-processed second permutation data segments (26A to 26C) are obtained as a plurality of second permutation data segments by calculating a binary cross entropy for each of the plurality of permutation results (24A to 24C); the first time period is an integer multiple of the second time period; The training permutation data segment (31) is a permutation data segment for the first time based on the permutation loss (30) and the first permutation data segment (29), and is data (y^ α ) represents the speaker's utterance probability based on for each of the plurality of raw second permutation data segments, the raw second permutation data segment includes, for each of the M speakers, output data that is data output from the speaker diarization model for a second time corresponding to the raw second permutation data segment; For each of the plurality of permutation results, the permutation result (24A) is determined to be a result of the permutation represented by the pre-processing second permutation data segment (23A) corresponding to the permutation result (24A) being the utterance probability (y α ) and the data is aligned with the permutation according to For each of the plurality of post-processing second permutation data segments, the post-processing second permutation data segment represents, for each of the M speakers, an utterance probability obtained from the speaker diarization model trained to reduce the binary cross entropy using post-processing output data in the permutation result corresponding to the second time corresponding to the pre-processing second permutation data segment. An online speaker diarization system.

2. an inference unit that performs the online speaker diarization as an inference using the speaker diarization model trained by the training unit; a speaker tracking buffer in which chunks past the chunk to be inferred and permutation data segments acquired for the past chunks are stored during the inference; The online speaker diarization system of claim 1 further comprising:

3. the second time period is the same length as the time period corresponding to the chunk to be inferred; The online speaker diarization system of claim 2.

4. When the chunk to be inferred is the p-th chunk (p is a natural number), the inference unit If p is equal to or less than a predetermined threshold, perform the inference in a first mode; if p exceeds the predetermined threshold, performing the inference in a second mode; the inference in the first mode is an inference using predetermined parameters of the speaker diarization model; the inference in the second mode is inference using parameters learned in training the speaker diarization model; The online speaker diarization system of claim 2.

5. training a speaker diarization model for use in online speaker diarization; the speaker diarization model (22) is a neural network-based chunk-wise EEND (End-to-End Neural Diarization) model of speaker diarization; The online speaker diarization is to sequentially input chunks into the speaker diarization model, and sequentially identify which speaker spoke at what time based on permutation data segments sequentially obtained from the speaker diarization model; The chunk is speech data for a certain period of time of speech of N speakers (N is a natural number) or data for a certain period of time based on the speech data, the permutation data segment is data representing a relationship between M speakers (M is a natural number and M≦N) and utterance probabilities for each of a plurality of time intervals in the fixed time; The learning inputting a first chunk (27) that is a chunk of a first time duration into the speaker diarization model (22) to obtain a first permutation data segment (29) from the speaker diarization model (22), and calculating a permutation loss (30) of the first permutation data segment (29); Dividing the first chunk (27) into a plurality of second chunks (21A to 21C), each of which is a chunk of a second time period shorter than the first time period; and inputting the plurality of second chunks (21A to 21C) into the speaker diarization model (22) to obtain a plurality of pre-processed second permutation data segments (23A to 23C) as a plurality of second permutation data segments, obtaining a plurality of permutation results (24A to 24C) in which a training permutation data segment (31) is used for the plurality of pre-processed second permutation data segments (23A to 23C), and calculating a binary cross entropy for each of the plurality of permutation results (24A to 24C) to obtain a plurality of post-processed second permutation data segments (26A to 26C) as a plurality of second permutation data segments; Including, the first time period is an integer multiple of the second time period; The training permutation data segment (31) is a permutation data segment for the first time based on the permutation loss (30) and the first permutation data segment (29), and is data (y^ α ) represents the speaker's utterance probability based on for each of the plurality of raw second permutation data segments, the raw second permutation data segment includes, for each of the M speakers, output data that is data output from the speaker diarization model for a second time corresponding to the raw second permutation data segment; For each of the plurality of permutation results, the permutation result (24A) is determined to be a result of the permutation represented by the pre-processing second permutation data segment (23A) corresponding to the permutation result (24A) being the utterance probability (y α ) and the data is aligned with the permutation according to For each of the plurality of post-processing second permutation data segments, the post-processing second permutation data segment represents, for each of the M speakers, an utterance probability obtained from the speaker diarization model trained to reduce the binary cross entropy using post-processing output data in the permutation result corresponding to the second time corresponding to the pre-processing second permutation data segment. method.

Images