Main speaker voice segment discriminator

Abstract

To provide a technique for discriminating a main speaker voice segment in sound information output from one microphone of a plurality of microphones arranged corresponding to each of a plurality of speakers, which contains voice information of the main speaker corresponding to the one microphone.SOLUTION: Microphones 310(1) to 310(n) are arranged corresponding to each of speakers (1) to (n). Main speaker voice segment discriminating means 200 divides sound information output from any one microphone into a plurality of frames along a time axis, and outputs a main speaker voice segment discrimination result indicating a main speaker voice segment formed by the frames determined to contain voice information of the main speaker corresponding to the one microphone among the plurality of frames. The main speaker voice segment discriminating means 200 consist of feature amount extraction means 210, time series feature amount conversion means 220, first discriminating means 230, second discriminating means 240, and integration means 250.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for determining a voice section of a specific speaker from the voices of multiple speakers collected by a microphone. [Background technology]

[0002] As a method for distinguishing the contents of speech of multiple speakers in a conference or the like, a method is used in which speech information contained in sound information output from a microphone is recognized and converted into text information. Usually, the sound information output from a microphone is analog sound information converted into digital sound information. When multiple speakers are conversing, the sound information may contain a mixture of voice information from multiple speakers who are close to the microphone. For this reason, a process for distinguishing "who spoke and when" is required as a preprocessing step for speech recognition. The method for distinguishing speaker voice sections containing the voice information of each speaker and speaker labels indicating each speaker from sound information containing a mixture of voice information from multiple speakers is collectively called speaker diarization. In recent years, end-to-end neural diarization (EEND) has been proposed as a method of speaker diarization, as disclosed in Non-Patent Document 1. EEND disclosed in Non-Patent Document 1 is composed of a multi-layer neural network, and uses a single discrimination model to discriminate speaker voice segments and speaker labels from input sound information. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] "END-TO-END NEURAL SPEAKER DIARIZATION WITH SELF-ATTENTION", 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), ISBN 978-1-7281-0306-8, Yusuke Fujita, Naoyuki Kanda, Shota Horiguchi, Yawen Xue, Kenji Nagamatsu, Shinji Watanabe) Summary of the Invention [Problem to be solved by the invention]

[0004] The conventional speaker diarization algorithms proposed, including EEND disclosed in Non-Patent Document 1, target the discrimination of speech information of multiple speakers mixed in sound information (single-channel sound information) output from one microphone (monaural microphone). However, it is not easy to distinguish the speaker voice segments and speaker labels of multiple speakers from the sound information output from a single microphone, even if the number of speakers is known. The present invention has been devised in view of such problems, and aims to provide a technique that can easily and accurately distinguish speaker voice intervals and speaker labels. Specifically, multiple microphones are arranged corresponding to multiple speakers, respectively, and the speaker corresponding to each microphone is designated as a "main speaker." Then, from the sound information output from any one of the microphones (sound information of any one single channel) input to the speaker diarization, only the sound information of the main speaker (speaker label) corresponding to that one microphone is selectively extracted to distinguish the main speaker voice interval. [Means for solving the problem]

[0005] The first invention relates to a main speaker voice section discrimination device that discriminates the voice section of a main speaker corresponding to one microphone from sound information output from one of multiple microphones arranged corresponding to each of multiple speakers. The main speaker voice section discrimination device of the first aspect of the invention includes a storage means, a feature extraction means, a time-series feature conversion means, a first discrimination means, a second discrimination means, and a integrating means. The storage means stores sound information output from at least one of the multiple microphones along a time axis. Any appropriate method can be used to store sound information in the storage means. For example, a method can be used in which sound information input directly from a microphone or via a communication line is stored in the storage means. Alternatively, a method can be used in which sound information output from a microphone is stored in a storage medium, and the sound information stored in the storage medium is input via a reading means and stored in the storage means. The feature extraction means extracts feature amounts for each of a plurality of frames obtained by dividing the sound information output from any one of at least one microphone stored in the storage means along a time axis. As a method for extracting feature amounts, various known methods can be used. The time-series feature quantity conversion means converts the feature quantities of the plurality of frames extracted by the feature quantity extraction means into time-series feature quantities and outputs the time-series feature quantities. As a method for converting the feature quantities of the plurality of frames into time-series feature quantities, various known methods can be used. The first discrimination means generates a feature vector for the entire interval of the time-series feature quantity output from the time-series feature quantity conversion means. Based on the generated feature vector for the entire interval, it discriminates whether or not each frame contains voice information of the main speaker corresponding to any one of the microphones. Then, it outputs a first discrimination result indicating a frame that has been determined to contain voice information of the main speaker corresponding to any one of the microphones. The first discrimination result may be in various formats. For example, it may be possible to use the first discrimination result including a score indicating the probability that each frame contains voice information of the main speaker. The second discrimination means generates a feature vector in a partial section of the time-series feature output from the time-series feature conversion means. Based on the generated feature vector in the partial section, it discriminates whether or not each frame contains voice information of the main speaker corresponding to any one of the microphones. Then, it outputs a second discrimination result indicating a frame that is discriminated as containing voice information of the main speaker corresponding to any one of the microphones. As for the format of the second discrimination result, various formats can be used, as with the first discrimination result. The integration means weights and integrates the first discrimination result output from the first discrimination means and the second discrimination result output from the second discrimination means, and outputs a main speaker voice section discrimination result indicating a main speaker voice section formed by frames that are determined to contain voice information of the main speaker corresponding to any one of the microphones. The main speaker voice section discrimination device of the present invention discriminates main speaker voice sections that contain voice information of one main speaker who has a one-to-one correspondence with a microphone from the sound information output from the microphone, making it easy to discriminate the main speaker voice section. In a different embodiment of the main speaker voice segment discrimination device of the first aspect of the present invention, the storage means stores sound information output from at least two of the plurality of microphones in a time-synchronized manner. Then, the feature extraction means performs a process of extracting features, the time-series feature conversion means performs a process of outputting time-series features, the first discrimination means performs a process of outputting the first discrimination result, the second discrimination means performs a process of outputting the second discrimination result, and the integration means performs a process of outputting the main speaker voice zone discrimination result for each of the sound information output from the at least two microphones stored in the storage means. The integration means outputs the main speaker voice zone discrimination results for the main speakers corresponding to each of the at least two microphones. In this embodiment, the main speaker voice segments of all the main speakers corresponding to each microphone can be easily determined. In a different embodiment of the main speaker voice section discrimination device of the first invention, the integration means discriminates whether or not any of the main speaker voice sections indicated in the main speaker voice section discrimination result corresponding to any one of the at least two microphones overlaps along the time axis with any of the main speaker voice sections indicated in the main speaker voice section discrimination result corresponding to at least another microphone. If there is overlap, the degree of overlap is determined for each pair of overlapping main speaker voice sections. Any appropriate method can be used to determine the degree of overlap between two overlapping main speaker voice sections. If any of the overlapping degrees satisfies a predetermined condition, the overlapping main speaker voice sections except for the longest main speaker voice section are deleted from the main speaker voice section discrimination result. The speech of a main speaker corresponding to one microphone may be collected by another microphone, and the voice information of the main speaker corresponding to that one microphone may be included (emerge) in the sound information output from the other microphone (sound information of another channel). In this case, the voice information of the main speaker corresponding to that one microphone may be erroneously determined as the voice information of the main speaker corresponding to the other microphone. As the predetermined condition for the degree of overlap, various conditions are set that can prevent such erroneous determination of the voice section of the main speaker. In this embodiment, it is possible to prevent erroneous determination of the main speaker's voice section. In a different embodiment of the main speaker voice segment discrimination device of the first invention, the time-series feature quantity transformation means, the first discrimination means, the second discrimination means and the integration means are configured with a multi-layer neural network. In this embodiment, the main speaker voice segment discrimination device can be configured simply. The second invention relates to a storage medium. The storage medium of the second invention stores a program for executing the above-mentioned feature extraction process by the feature extraction means, the time-series feature output process by the time-series feature conversion means, the output process of the first discrimination result by the first discrimination means, the output process of the second discrimination result by the second discrimination means, and the output process of the main speaker voice section discrimination result by the integrating means. By using the storage medium of the second aspect of the present invention, a main speaker voice segment discrimination device can be easily configured. Effect of the Invention

[0006] By using the main speaker voice section discrimination device and storage medium of the present invention, the main speaker voice section of the main speaker corresponding to the microphone can be easily discriminated. [Brief description of the drawings]

[0007] [Figure 1] 1 is a block diagram showing a schematic configuration of a main speaker voice segment discrimination device according to an embodiment; [Diagram 2] FIG. 2 is a block diagram of an example of a main speaker voice zone discriminating means constituting a main speaker voice zone discriminating device of an embodiment in an inference mode. [Diagram 3] FIG. 3 is a block diagram of an example of the encoder shown in FIG. 2. [Figure 4] FIG. 2 is a block diagram of an example of a main speaker voice zone discriminating means constituting a main speaker voice zone discriminating device of an embodiment in a learning mode. [Diagram 5] FIG. 1 is a diagram showing examples of utterances from a plurality of speakers. [Figure 6] 11A and 11B are diagrams for explaining the operation when main speaker voice segments overlap along the time axis. [Figure 7] 11A and 11B are diagrams for explaining the operation when main speaker voice segments overlap along the time axis. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] An embodiment of a main speaker voice section discrimination device of the present invention will be described below with reference to Figs. 1 to 4. Fig. 1 is a block diagram showing a schematic configuration of a main speaker voice section discrimination device 10 of an embodiment. Figs. 2 and 4 are block diagrams showing a schematic configuration of a main speaker voice section discrimination means 200, and Fig. 3 is a block diagram showing a schematic configuration of an encoder 232. Fig. 2 shows the configuration when a main speaker voice section is discriminated (estimated), and Fig. 4 shows the configuration when learning.

[0009] The main speaker voice section discrimination device 10 of this embodiment is based on the end-to-end neural diarization (EEND) algorithm, and discriminates a speaker voice section (referred to as the "main speaker voice section") that contains voice information of the speaker (referred to as the "main speaker") corresponding to one of multiple microphones arranged corresponding to each of multiple speakers, within the sound information output from that one microphone. That is, in the main speaker voice section discrimination device 10 of this embodiment, the number of speakers ("main speaker") for discriminating speaker voice sections ("main speaker voice sections") from sound information output from one microphone is one.

[0010] The outline of the EEND algorithm is as follows: The time series feature amount (feature amount series) of the speech information in units of frames is denoted as X, and the speaker label (speaker label series) corresponding to the frame in time series is denoted as Y. The feature amount series X of the speech information can be represented by logarithmic mel filter bank coefficients that indicate the envelope of the short-time spectrum. The estimate Y of the speaker label sequence Y under the observation of the feature sequence X of the speech information is expressed by (Equation 1) by maximizing the posterior probability under the condition that the feature sequence X is observed.

number

number

[0011] The main speaker voice segment discrimination device 10 comprises a processing means 100 , an input means 300 , a storage means 400 , and an output means 500 .

[0012] A microphone 310 that converts the voice of a speaker into sound information is used as the input means 300. Any of various known microphones having different configurations can be used as the microphone 310. In this embodiment, a monaural microphone is used. The main speaker voice section discrimination device 10 of this embodiment is based on the premise that the microphones 310 are arranged in such a manner that the number of microphones 310(1) to 310(n) is equal to the number of speakers n and corresponds to speaker (1) to speaker (n), respectively. The phrase "disposed corresponding to a speaker" means "disposed (worn) near the speaker (for example, near the mouth, neck, chest, etc.)." In this embodiment, microphones 310(1) to 310(n) are worn near the mouths of speakers (1) to (n). In this specification, the term "sound information output from a microphone" refers to analog voice input to a microphone, which has been converted into digital voice by, for example, Fourier transform, etc. The process of converting analog voice into digital voice may be performed on the microphone side, or on the processing means 100 (subject speaker voice zone determination means 200) side.

[0013] As the input means 300, a keyboard 320, a mouse 330, a touch panel 340, etc., for inputting character information, etc. may be used. One of the keyboard 320, the mouse 330, and the touch panel 340 may be used, or an appropriately selected plurality of them may be used. Of course, the input means for inputting character information, etc. is not limited to these. Also, reading means 350 for reading information stored in a storage medium or the like can be used as input means 300. For example, sound information output from microphones 310(1)-310(n) is stored in a storage medium, and the sound information stored in the storage medium is read and input by reading means 350. As reading means 350, a reading means compatible with the storage method of the storage medium is used. Although not shown in the figure, the input means 300 may also be a receiving means for receiving sound information transmitted from a transmitting means located at a distance via a communication line (Internet line, LAN, etc.).

[0014] The storage means 400 temporarily or semi-permanently stores sound information output from the microphones 310(1) to 310(n), programs for executing the processing of the processing means 100, operation information, etc. As the storage means 400, storage means of various known configurations can be used. The sound information output from each of the microphones 310(1) to 310(n) is stored along a time axis in the storage means 400. The sound information output from the microphones 310(1) to 310(n) is stored in time synchronization. The output means 500 outputs the main speaker voice section discrimination result discriminated by the main speaker voice section discrimination means 200 (described later) and various other information. As the output means 500, various known output means such as a display means or a printer can be used. Although not shown in the figure, the output means 500 may be a communication means (transmission means) for transmitting the main speaker voice section discrimination result and the like via a communication line (Internet line, LAN, etc.).

[0015] The processing means 100 includes a main speaker voice section discrimination means 200 for discriminating a main speaker voice section of a main speaker corresponding to one of the microphones 310(1) to 310(n) from sound information output from the one of the microphones. In this embodiment, the sound information output from the microphones 310(1)-310(n) is configured to be input to the processing means 100 directly or via a communication line. Of course, as described above, the sound information output from the microphones can also be stored in a storage medium, and the sound information stored in the storage medium can be input to the processing means 100 via a reading means.

[0016] The main speaker voice section discrimination means 200 includes a feature extraction means 210 , a time-series feature conversion means 220 , a first discrimination means 230 , a second discrimination means 240 , and an integration means 250 .

[0017] The feature extraction means 210 divides the sound information output from each of the microphones 310(1) to 310(n) into frames of a predetermined time width, and extracts a feature (feature vector) from the sound information in each frame. Various known methods can be used to extract features within a frame of sound information. In this embodiment, a Mel filter bank having a frequency scale (Mel Frequency) corresponding to a scale based on human hearing (Mel scale) is assigned to the frequency data, the output is logarithmically transformed, and an envelope is extracted from the logarithmic power spectrum, and Mel Frequency Cepstral Coefficients (MFCCs) are extracted. The time-series feature conversion means 220 converts the feature of each frame extracted by the feature extraction means 210 into a time-series feature of each frame. A known method can be used to convert the feature of each frame into a time-series feature of each frame. In this embodiment, a Time Dilated Convolutional Network (TDCN) is used to convert the feature of each frame into a time-series feature of each frame.

[0018] The first discrimination means 230 outputs a first discrimination result based on the time-series feature amount output from the time-series feature amount conversion means 220 . The first determination means 230 is based on an encoder-decoder model, and uses a Transformer, which is an information conversion model including a Multi-head Attention Mechanism, as an encoder. The first discrimination means 230 targets all frames of the entire section (for example, the entire speech section) and generates a feature vector for each frame based on position information, correlation, etc. of the feature amount for each frame. Then, based on the generated feature vector for each frame, it discriminates whether or not each frame contains voice information of the main speaker. Whether or not each frame contains voice information of the main speaker is determined by calculating a score for each frame using a binary discrimination. A real value between "0" and "1" is used as the score. The first discrimination result includes the score for each frame.

[0019] The second discrimination means 240 outputs a second discrimination result based on the time-series feature amount output from the time-series feature amount conversion means 220. The second discrimination means 240 does not have a transformer encoder. The second discrimination means 240 targets local frames in a partial section (for example, a short section of one or several utterances) and generates a feature vector for each frame. Then, based on the generated feature vector for each frame, it discriminates whether or not each frame contains voice information of the main speaker. Whether or not each frame contains voice information of the main speaker is determined by calculating a score for each frame using a binary discrimination. A real value between "0" and "1" is used as the score. The second discrimination result includes the score for each frame.

[0020] The integrating means 250 weights and integrates the scores of each frame included in the first discrimination result output from the first discrimination means 230 and the scores of each frame included in the second discrimination result output from the second discrimination means 240, and outputs an integrated discrimination result. In this embodiment, the integrating means 250 calculates the integrated discrimination result by adding the second discrimination result multiplied by the weighting coefficient λ and the first discrimination result.

[0021] In this embodiment, the time series feature quantity conversion means 220, the first discrimination means 230, the second discrimination means 240 and the integration means 250 are configured as a multi-layer neural network. That is, the first discrimination means 230, the second discrimination means 240 and the integration means 250 are set to a learning mode and an inference mode. In the following, the configurations of the first discrimination means 230, the second discrimination means 240, and the integration means 250 will be explained separately for the learning mode and the inference mode.

[0022] First, the configuration in the learning mode will be described with reference to FIG. The first discrimination means 230 is composed of linear conversion means 231, encoders 232(1) to 232(m) with an arbitrary number of layers m, layer normalization means 233, score calculation means 234, and evaluation means 235. The linear transformation means 231 linearly transforms the time-series feature quantity output from the time-series feature quantity transformation means 220 . The encoders 232(1) to 232(m) generate global feature vectors that take into account the correlations of the entire speech information sequence. The encoders 232(1) to 232(m) have the same configuration as the encoder 232 shown in FIG. The encoder 232 is composed of a layer normalization means 232A, a multi-head self-attention means 232B, an addition means 232C, a layer normalization means 232D, a position encoding means 232E, and an addition means 232F. The layer normalization means 232A normalizes the input. The multi-head self-attention 232B dynamically assigns importance weights to the features of the vector representation generated by the encoder 232. The adding means 232C adds the input and the output of the multi-head self-attention 232B. The layer normalization means 232D normalizes the output of the addition means 232C. The positional encoding means 232E embeds position information within the entire input into the vector representation generated by the encoder 232. The adding means 232F adds the output of the layer normalizing means 232D and the output of the position encoding means 232E.

[0023] The layer normalization means 233 normalizes the output of the addition means 232F so that the values ​​of the feature expressions with different sequence lengths do not become too large. The score calculation means 234 calculates a score Sa indicating whether or not each frame contains voice information of the main speaker based on the output of the layer normalization means 233. In this embodiment, the score calculation means 234 calculates the score Sa of each frame by a linear conversion process and a binary discrimination using a sigmoid function. A real value between "0" and "1" is used as the score Sa. The score calculation means 234 outputs a first discrimination result including the score Sa of each frame. The evaluation means 235 compares the first discrimination result with the correct answer (teaching data) and numerically evaluates the error of the comparison result of each frame. In this embodiment, the evaluation means 235 calculates an evaluation value La of a loss function indicating the error of the comparison result of each frame.

[0024] The second discrimination means 240 is composed of a layer normalization means 243 , a score calculation means 244 and an evaluation means 245 . The layer normalization means 243 , the score calculation means 244 , and the evaluation means 245 have the same configurations as the layer normalization means 233 , the score calculation means 234 , and the evaluation means 235 of the first discrimination means 230 . The second discrimination means 240 does not have the linear conversion means 231 and the encoders 232(1) to 232(m) of the first discrimination means 230. The score calculation means 244 calculates a score Sb indicating whether or not each frame contains voice information of the main speaker based on the output of the layer normalization means 243. In this embodiment, the score calculation means 244 calculates the score Sb of each frame by a linear conversion process and a binary discrimination using a sigmoid function. A real value between "0" and "1" is used as the score Sb. The score calculation means 244 outputs a second discrimination result including the score Sb of each frame. The evaluation means 245 compares the second discrimination result with the correct answer (teaching data) and numerically evaluates the error of the comparison result of each frame. In this embodiment, the evaluation means 245 calculates an evaluation value Lb of a loss function indicating the error of the comparison result of each frame.

[0025] The integrating means 250 integrates and evaluates the comparison error of each frame. In this embodiment, the integrating means 250 weights and adds the evaluation value La of the loss function calculated by the evaluation means 235 of the first discrimination means 230 and the evaluation value Lb of the loss function calculated by the evaluation means 245 of the second discrimination means 240 to calculate the integrated loss function evaluation value Lt. Specifically, the integrating means 250 adds the evaluation value Lb of the loss function multiplied by the weighting coefficient λ and the evaluation value La of the loss function to calculate the integrated loss function evaluation value Lt (=La+λ×Lb). Then, the parameters of the first discriminating means 230 and the second discriminating means 240 and the weighting factor λ of the integrating means 250 are set by error propagation learning so that the evaluation value Lt of the integrated loss function is minimized (the inference model is optimized).

[0026] Next, the configuration in the inference mode will be described with reference to FIG. The first discrimination means 230 is composed of a linear conversion means 231, encoders 232(1) to 232(m) with an arbitrary number of layers m, a layer normalization means 233, a score calculation means 234, and an evaluation means 235. In the inference mode, the first discrimination means 230 does not use the evaluation means 235. That is, in the inference mode, the first discrimination means 230 outputs a first discrimination result including the score Sa of each frame. In this embodiment, the first discrimination means 230 calculates the score Sa of each frame by a linear conversion process and a binary discrimination using a sigmoid function. A real value between "0" and "1" is used as the score Sa. The second discrimination means 240 is composed of a layer normalization means 243 and a score calculation means 244 . In the inference mode, the second discrimination means 240 does not use the evaluation means 245. That is, in the inference mode, the second discrimination means 240 outputs a second discrimination result including a score Sb of each frame. In this embodiment, the second discrimination means 240 calculates the score Sb of each frame by a linear conversion process and a binary discrimination using a sigmoid function. A real value between "0" and "1" is used as the score Sb. The integrating means 250 outputs an integrated discrimination result including an integrated score St for each frame. In this embodiment, the integrating means 250 weights and adds the first discrimination result output from the first discrimination means 230 and the second discrimination result output from the second discrimination means 240 to calculate an integrated discrimination result. Specifically, the integrating means 250 calculates the integrated score St (=Sa+λ×Sb) by adding the score Sb included in the second discrimination result multiplied by the weighting coefficient λ to the score Sa included in the first discrimination result. The integrated discrimination result output from the integration means 250 corresponds to the "main speaker voice section discrimination result" of the present invention.

[0027] In this embodiment, the second discrimination means 240 and the integration means 250 may be omitted, and the main speaker voice section discrimination means 200 may be configured by the first discrimination means 230. That is, a plurality of microphones 310(1) to 310(n) are arranged corresponding to a plurality of speakers (1) to (n), respectively. Then, on the premise that each microphone corresponds to one speaker, the first discrimination means 230 discriminates a main speaker voice section that includes voice information of the main speaker corresponding to one microphone within the sound information output from the one microphone. Even when the main speaker voice section discrimination means 200 is constituted by the first discrimination means 230, the main speaker voice section can be easily discriminated. In this case, however, it is difficult to improve the accuracy of determining the voice section of the main speaker. As in the above-described embodiment, the main speaker voice section discrimination means 200 is composed of the first discrimination means 230, the second discrimination means 240, and the integration means 250, so that the main speaker voice section can be discriminated based on the characteristics of the entire utterance section determined by the first discrimination means 230 and the characteristics of the partial utterance section determined by the second discrimination means 240. This makes it possible to improve the accuracy of discriminating the main speaker voice section.

[0028] FIG. 5 shows an example of speech times and speech contents of multiple speakers. Fig. 5 shows the speech times and speech contents of three speakers, speaker (1) to speaker (3). Note that microphones (1) to (3) are attached to the mouths or the like of speakers (1) to (3), respectively. The main speaker voice section discrimination means 200 discriminates main speaker voice sections that contain voice information of the main speaker (1) corresponding to the microphone (1) in the sound information output from the microphone (1). In Fig. 5, it discriminates that the main speaker (1) spoke in the sections from time t11 to t12, from time t13 to t14, and from time t19 to t20 (main speaker voice sections). Furthermore, the main speaker voice section discrimination means 200 discriminates a main speaker voice section that contains voice information of the main speaker (2) corresponding to the microphone (2) in the sound information output from the microphone (2). In Fig. 5, it discriminates that the main speaker (2) spoke in the section from time t15 to t16 (main speaker voice section). Furthermore, the main speaker voice section discrimination means 200 discriminates main speaker voice sections that contain voice information of the main speaker (3) corresponding to the microphone (3) in the sound information output from the microphone (3). In Fig. 5, it discriminates that the main speaker (3) spoke in the section from time t17 to t18 and the section from time t21 to t24 (main speaker voice section).

[0029] One microphone may pick up not only the voice of the main speaker corresponding to that microphone, but also the voices of speakers corresponding to other microphones who are close to that microphone. Therefore, the main speaker voice section discrimination means 200 may erroneously discriminate a main speaker voice section that includes voice information of a main speaker corresponding to another microphone in the sound information output from one microphone as the main speaker voice section of the main speaker corresponding to that one microphone. For example, as shown in FIG. 5, when the voice "Okay, continue" uttered by the main speaker (3) corresponding to microphone (3) from time t21 to t24 is collected by microphones (1) and (2), the main speaker voice section discrimination means 200 may discriminate the main speaker voice section C1 (t21 to t24) of the main speaker (3) corresponding to microphone (3) from the sound information output from microphone (3), discriminate the main speaker voice section B1 (t22 to t25) of the main speaker (2) corresponding to microphone (2) from the sound information output from microphone (2), and discriminate the main speaker voice section A1 (t23 to t26) of the main speaker (1) corresponding to microphone (1) from the sound information output from microphone (1). In this case, the main speaker voice section B1 is misclassified as the main speaker voice section A1.

[0030] In this embodiment, the main speaker voice section discrimination means 200 (the integration means 250) is configured to prevent such erroneous discrimination of the main speaker voice section by post-processing. The operation of preventing misclassification of the main speaker's voice section will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is an enlarged view of the main part of Fig. 5, showing the state where the main speaker's voice section has been misclassified. Fig. 7 shows the state where the misclassified main speaker's voice section has been deleted.

[0031] When the speech of a main speaker (3) corresponding to microphone (3) is picked up by microphone (1) and the main speaker voice section A1 is misclassified, the correctly classified main speaker voice section C1 and the misclassified main speaker voice section A1 overlap along the time axis. Similarly, when the speech of the main speaker (3) corresponding to microphone (3) is picked up by microphone (2) and the main speaker voice section B1 is misclassified, the correctly classified main speaker voice section C1 and the misclassified main speaker voice section B1 overlap along the time axis. In FIG. 6, a correctly classified main speaker voice section C1 overlaps with two incorrectly classified main speaker voice sections A1 and B1. The degree of overlap is determined for each pair of two overlapping main speaker voice sections, and if any of the overlap degrees satisfies a predetermined condition, it can be determined that the main speaker voice section has been misclassified.

[0032] As a method for determining the degree of overlap between two overlapping main speaker voice segments, various methods capable of determining erroneous determination of main speaker voice segments can be used. In this embodiment, the overlap rate Z is calculated to determine the degree of overlap. The overlap rate Z is calculated based on the length of the two overlapping main speaker voice segments and the start and end times of the overlapping segments in which the two main speaker voice segments overlap, using the following formula: [Overlap ratio Z = (end time of overlap section - start time of overlap section) / minimum length of main speaker's voice section] Here, the "shortest main speaker voice interval" refers to the shorter main speaker voice interval of the two overlapping main speaker voice intervals.

[0033] The overlap rate Z of the main speaker voice section C1 and the main speaker voice section A1 shown in FIG. CA is calculated as follows: The main speaker voice section C1 has a length of Tc, overlaps with the main speaker voice section A1 in an overlap section Tx1, and does not overlap with the main speaker voice section A1 in a non-overlapping section Tx2. The main speaker voice section A1 has a length Ta, overlaps with the main speaker voice section C1 in an overlap section Tx1, and does not overlap with the main speaker voice section C1 in a non-overlapping section Tx3. The start time of the overlap section Tx1 is t23 (=the start time of the main speaker voice section A1), and the end time of the overlap section Tx1 is t24 (=the end time of the main speaker voice section C1). The length Ta (=t26-t23) of the main speaker voice section A1 is shorter than the length Tc (=t24-t21) of the main speaker voice section C1. [Overlap rate Z CA = (t24-t23) / Ta] In addition, the overlap rate Z CB is calculated as follows: The main speaker voice section C1 has a length of Tc, overlaps with the main speaker voice section B1 in an overlap section Ty1, and does not overlap with the main speaker voice section C1 in a non-overlapping section Ty2. The main speaker voice section B1 has a length of Tb, overlaps with the main speaker voice section C1 in an overlap section Ty1, and does not overlap with the main speaker voice section C1 in a non-overlapping section Ty3. The start time of the overlap section Ty1 is t22 (=the start time of the main speaker voice section B1), and the end time of the overlap section Ty1 is t24 (=the end time of the main speaker voice section C1). The length Tb (=t25-t22) of the main speaker voice section B1 is shorter than the length Tc (=t24-t21) of the main speaker voice section C1. [Overlap rate Z CB =(t24-t22) / Tb] The method of expressing the overlap rate can be appropriately selected. For example, it can be expressed as a percentage or as a real number within the range of "0" to "1".

[0034] Then, it is determined whether any of the calculated overlap rates satisfies a predetermined condition. As the predetermined condition, any suitable condition that can determine whether or not the main speaker's voice section has been erroneously determined can be set. In this embodiment, it is determined whether any of the overlap rates exceeds a threshold value τ. In FIG. 6, the overlap rate Z CB exceeds the threshold τ [Z CB >τ]. In this case, the main speaker voice section discrimination means 200 (the integration means 250) uses the overlap rate Z CB exceeds the threshold τ, it is determined that the voice section of the main speaker has been erroneously identified. Next, the longest main speaker voice section is determined from among the overlapping main speaker voice sections. In FIG. 6, among the length Tc of the main speaker voice section C1, the length Ta of the main speaker voice section A1 overlapping with the main speaker voice section C1, and the length Tb of the main speaker voice section B1 overlapping with the main speaker voice section C1, the main speaker voice section C1 with the longest length Tc is identified. Then, of the overlapping main speaker voice intervals, the longest main speaker voice interval is retained and the other main speaker voice intervals are deleted. Then, of the overlapping main speaker voice intervals, the longest main speaker voice interval is retained and the other main speaker voice intervals are deleted. In the example shown in FIG. 6, as shown in FIG. 7, the main speaker voice section C1 of the main speaker (3) is left, and the main speaker voice section A1 of the main speaker (1) and the main speaker voice section B1 of the main speaker (2) are deleted. In addition, when there is only one pair of overlapping main speaker voice intervals, an overlap rate Z is calculated for the pair of main speaker voice intervals, and misclassification of the main speaker discrimination interval is prevented based on the calculated overlap rate Z in the same manner as described above.

[0035] The present invention can also be configured as follows. (Aspect 1) A main speaker voice section discrimination device that discriminates a voice section of a main speaker corresponding to one microphone in sound information output from one microphone among a plurality of microphones arranged corresponding to each of a plurality of speakers, comprising: The apparatus includes a storage means and a main speaker voice section determination means, The storage means stores sound information output from at least one of the plurality of microphones along a time axis, The main speaker voice section discrimination means discriminates whether or not a plurality of frames obtained by dividing along a time axis the sound information output from any one of the at least one microphone stored in the storage means contain voice information of the main speaker corresponding to the one microphone, and outputs a main speaker voice section discrimination result indicating a main speaker voice section formed by a frame determined to contain the voice information of the main speaker corresponding to the one microphone. (Aspect 2) The subject speaker voice activity discrimination device of aspect 1, The storage means stores sound information output from at least two of the plurality of microphones in a time-synchronized manner, The main speaker voice section discrimination means performs a process of dividing sound information output from one microphone into a plurality of frames along a time axis, and outputting a main speaker voice section discrimination result indicating a main speaker voice section formed by a frame that is determined to contain voice information of the main speaker corresponding to one of the microphones among the divided plurality of frames, for each of the sound information output from the at least two microphones stored in the storage means, and outputs a main speaker voice section discrimination result of the main speaker corresponding to each of the at least two microphones. (Aspect 3) The subject speaker voice activity discrimination device of aspect 2, the main speaker voice section discrimination means, when any of the main speaker voice sections indicated in the main speaker voice section discrimination result corresponding to any one of the at least two microphones overlaps with any of the main speaker voice sections indicated in the main speaker voice section discrimination result corresponding to the other at least one microphone, discriminates a degree of overlap for each pair of two overlapping main speaker voice sections, and, when any of the degrees of overlap satisfy a predetermined condition, deletes the main speaker voice sections except for the longest main speaker voice section from the main speaker voice section discrimination result. (Aspect 4) A subject speaker voice segment discrimination device according to any one of aspects 1 to 3, The main speaker voice section discrimination device is characterized in that the main speaker voice section discrimination means is composed of a multi-layer neural network. (Aspect 5) A computer-readable storage medium storing a program for causing a computer to execute an output process of a main speaker voice section discrimination result by the main speaker voice section discrimination means described in any one of aspects 1 to 4. (Aspect 6) The subject speaker voice activity discrimination device of aspect 1, The main speaker voice section discrimination means includes a feature extraction means, a time-series feature conversion means, a first discrimination means, a second discrimination means, and an integration means, The feature extraction means extracts a feature of each of a plurality of frames obtained by dividing the sound information output from any one of the at least one microphone stored in the storage means along a time axis, and the time-series feature quantity conversion means converts the feature quantities of the plurality of frames extracted by the feature quantity extraction means into time-series feature quantities and outputs the time-series feature quantities; the first discrimination means generates a feature vector for an entire interval of the time-series feature quantity output from the time-series feature quantity transform means, discriminates whether or not each of the frames contains voice information of a main speaker corresponding to any one of the microphones based on the generated feature vector for the entire interval, and outputs a first discrimination result indicating a frame that is determined to contain voice information of the main speaker corresponding to any one of the microphones; the second discrimination means generates a feature vector in a partial section of the time-series feature output from the time-series feature conversion means, discriminates whether or not each of the frames contains voice information of the main speaker corresponding to any one of the microphones based on the generated feature vector in the partial section, and outputs a second discrimination result indicating a frame that is determined to contain voice information of the main speaker corresponding to any one of the microphones; the integrating means weights and integrates the first discrimination result output from the first discrimination means and the second discrimination result output from the second discrimination means, and outputs a main speaker voice section discrimination result indicating a main speaker voice section formed by frames determined to contain voice information of the main speaker corresponding to any one of the microphones. (Aspect 7) The subject speaker voice activity discrimination device of aspect 6, The storage means stores sound information output from at least two of the plurality of microphones in a time-synchronized manner, a feature extraction process by the feature extraction means, a time-series feature output process by the time-series feature conversion means, a first discrimination result output process by the first discrimination means, a second discrimination result output process by the second discrimination means, and a main speaker voice section discrimination result output process by the integrating means, for each piece of sound information output from the at least two microphones stored in the storage means, and outputting a main speaker voice section discrimination result for the main speaker corresponding to each of the at least two microphones. (Aspect 8) The subject speaker voice activity discrimination device of aspect 7, the integrating means, when any of the main speaker voice sections indicated in the main speaker voice section discrimination result corresponding to any one of the at least two microphones overlaps along the time axis with any of the main speaker voice sections indicated in the main speaker voice section discrimination result corresponding to the other microphone, determines a degree of overlap for each pair of two overlapping main speaker voice sections, and, when any of the degrees of overlap satisfy a predetermined condition, deletes the main speaker voice sections except for the longest main speaker voice section from the main speaker voice section discrimination result among the overlapping main speaker voice sections. (Aspect 9) A subject speaker voice segment discrimination device according to any one of aspects 6 to 8, The main speaker voice segment discrimination device, wherein the time series feature conversion means, the first discrimination means, the second discrimination means and the integration means are configured with a multi-layer neural network. (Aspect 10) A computer-readable storage medium storing a program for causing a computer to execute the feature extraction process by the feature extraction means, the time series feature output process by the time series feature conversion means, the output process of a first discrimination result by the first discrimination means, the output process of a second discrimination result by the second discrimination means, and the output process of a main speaker voice section discrimination result by the integrating means, as described in any of aspects 6 to 9.

[0036] The present invention is not limited to the configurations described in the embodiments, and various modifications, additions, and deletions are possible. Various methods can be used to input the sound information output from the microphone to the main speaker voice section discrimination means 200. For example, a direct input method, an input method via a communication line, a method of reading sound information stored in a storage medium by a reading means and inputting the information, etc. can be used. The method of determining the first determination result by the first determination means is not limited to the method described in the embodiment, and various known methods can be used alone or in appropriate combination. The method of determining the second determination result by the second determination means is not limited to the method described in the embodiment, and various known methods can be used alone or in appropriate combination. The method of integrating the first and second discrimination results by the integrating means is not limited to the method described in the embodiment, and various known methods can be used. The method for preventing erroneous determination of the main speaker's voice section based on the overlap rate is not limited to the method described in the embodiment, and various methods can be used. Moreover, the method for preventing erroneous determination of the main speaker's voice section may be omitted. In the embodiment, the main speaker voice section discrimination means is configured by the first discrimination means, the second discrimination means, and the integrating means, but the second discrimination means and the integrating means may be omitted and the means may be configured by only the first discrimination means. Each of the configurations described in the embodiments can be used alone, or multiple appropriately selected configurations can be used in combination. [Explanation of symbols]

[0037] 10. Main speaker voice activity discrimination device 100 Processing means 200 Main speaker voice section discrimination means 210 Feature Extraction Method 220 Time series feature transformation method 230 First determination means 231 Linear Transformation Means 232, 232(1)~232(m) Encoder 232A, 232D layer normalization means 232B Multi-head self-attention 232C, 232F Addition means 232E Position encoding means 233 layer normalization means 234 Score calculation method 235 Evaluation Methods 240 Second determination means 243 layer normalization means 244 Score calculation method 245 Evaluation Methods 250 Integration Methods 300 Input means 310, 310(1)~310(n) Microphones 320 Keyboard 330 Mouse 340 Touch Panel 350 Reading means 400 Memory means 500 Output Method

Claims

1. A main speaker voice section discrimination device that discriminates a voice section of a main speaker corresponding to one microphone in sound information output from one microphone among a plurality of microphones arranged corresponding to each of a plurality of speakers, comprising: The apparatus includes a storage means, a feature extraction means, a time series feature conversion means, a first discrimination means, a second discrimination means, and an integration means, The storage means stores sound information output from at least one of the plurality of microphones along a time axis, The feature extraction means extracts feature amounts for each of a plurality of frames obtained by dividing the sound information output from any one of the at least one microphone, which is stored in the storage means, along a time axis; the time-series feature quantity conversion means converts the feature quantities of the plurality of frames extracted by the feature quantity extraction means into time-series feature quantities and outputs the time-series feature quantities; the first discrimination means generates a feature vector for an entire interval of the time-series feature quantity output from the time-series feature quantity transform means, discriminates whether or not each of the frames contains voice information of a main speaker corresponding to any one of the microphones based on the generated feature vector for the entire interval, and outputs a first discrimination result indicating a frame that is determined to contain voice information of the main speaker corresponding to any one of the microphones; the second discrimination means generates a feature vector in a partial section of the time-series feature quantity output from the time-series feature quantity transform means, discriminates whether or not each of the frames contains voice information of the main speaker corresponding to any one of the microphones based on the generated feature vector in the partial section, and outputs a second discrimination result indicating a frame that is determined to contain voice information of the main speaker corresponding to any one of the microphones; the integrating means weights and integrates the first discrimination result output from the first discrimination means and the second discrimination result output from the second discrimination means, and outputs a main speaker voice section discrimination result indicating a main speaker voice section formed by a frame determined to contain voice information of the main speaker corresponding to any one of the microphones.

2. 2. The main speaker voice segment discrimination device according to claim 1, The storage means stores sound information output from at least two of the plurality of microphones in a time-synchronized manner, a feature extraction process by the feature extraction means, a time series feature output process by the time series feature conversion means, a first discrimination result output process by the first discrimination means, a second discrimination result output process by the second discrimination means, and a main speaker voice section discrimination result output process by the integrating means, for each piece of sound information output from the at least two microphones stored in the storage means, and outputting a main speaker voice section discrimination result for the main speaker corresponding to each of the at least two microphones.

3. 3. The main speaker voice segment discrimination device according to claim 2, the integrating means, when any of the main speaker voice sections indicated in the main speaker voice section discrimination result corresponding to any one of the at least two microphones overlaps along the time axis with any of the main speaker voice sections indicated in the main speaker voice section discrimination result corresponding to the other at least one microphone, determines a degree of overlap for each pair of two overlapping main speaker voice sections, and, when any of the degrees of overlap satisfy a predetermined condition, deletes from the main speaker voice section discrimination result the main speaker voice sections except for the longest main speaker voice section among the overlapping main speaker voice sections.

4. 2. The main speaker voice segment discrimination device according to claim 1, 2. A main speaker voice segment discrimination device, wherein the time series feature conversion means, the first discrimination means, the second discrimination means and the integration means are configured with a multi-layer neural network.

5. 2. A computer-readable storage medium storing a program for causing a computer to execute the process of extracting features by the feature extraction means, the process of outputting time-series features by the time-series feature conversion means, the process of outputting a first discrimination result by the first discrimination means, the process of outputting a second discrimination result by the second discrimination means, and the process of outputting a main speaker voice section discrimination result by the integrating means, as described in claim 1.

Images