Voice recognition inference device, voice recognition system, voice recognition inference method, voice recognition model learning device, voice recognition model learning method, and program

Abstract

Provided is a voice recognition inference device that simultaneously recognizes the voices of all speakers from mixed voice that includes the speech of a plurality of speakers in real time without any greater delay than conventional voice recognition systems. This voice recognition inference device comprises a voice conversion unit and an inference unit. The voice conversion unit converts inputted voice information to intermediate voice feature quantities treating speakers of speech included in the inputted voice information as an Nth speaker by speech order. The inference unit generates Nth voice recognition results that represent the details of the speech of the Nth speaker from the intermediate voice feature quantities on the basis of Nth identification information that corresponds to the position of the Nth speaker in the speech order. The processing performed by the voice conversion unit and the inference unit is based on a pre-trained model.

Description

Speech recognition estimation device, speech recognition system, speech recognition estimation method, speech recognition model training device, speech recognition model training method, and program

[0001] The present disclosure relates to a speech recognition technology that directly outputs an arbitrary character string (phoneme, character, subword, word) representing the speech content of a target speaker from among speeches of multiple people. The present disclosure also relates to a speech recognition technology that directly outputs an arbitrary character string (phoneme, character, subword, word) representing the speech content of each speaker from among speeches of multiple people.

[0002] Recent neural network-based speech recognition systems are capable of directly outputting word sequences from speech features. Figure 1 shows the training method for an end-to-end speech recognition system that directly outputs any character sequence from acoustic features.

[0003] A method for training a neural network for speech recognition using this Recurrent Neural Network Transducer (RNN-T) training method is described in the "Recurrent Neural Network Transducer" section of Non-Patent Document 1. By introducing a "blank" symbol (referred to as "null output" in Non-Patent Document 1) that represents redundancy in training the RNN-T model, it is possible to dynamically learn the correspondence between speech and output sequences from training data as long as only phoneme / character / subword / word sequences (≠ frame-by-frame) that correspond to the content of the speech are prepared.

[0004] In other words, it is possible to learn using features and labels in a mismatched relationship (generally T >> U) between input length T and output length U. In addition, since it can operate frame-by-frame during inference, it has attracted a lot of attention as a technology capable of streaming operation.

[0005] Alex Graves, “Sequence Transduction with Recurrent Neural Networks,” in Proc. of International Conference on Machine Learning (ICML), 2012.

[0006] However, the technology of Non-Patent Document 1 described above could not recognize the speech of each of multiple speakers in a situation such as a conference. Therefore, the present disclosure has been made to solve the above problem, and aims to provide a technology that can recognize the speech of each speaker in real time from a mixed speech containing speeches from multiple speakers while suppressing an increase in the amount of delay.

[0007] A speech recognition and estimation device that solves the above problem includes a speech conversion unit and an estimation unit. The speech conversion unit converts the input speech information into intermediate speech features, assuming that the speaker of an utterance included in the input speech information is the Nth speaker in speaking order. The estimation unit generates an Nth speech recognition result indicating the content of the Nth speaker's utterance from the intermediate speech features, based on the Nth identification information corresponding to the speaking order of the Nth speaker. The speech conversion unit and the estimation unit perform processing based on a pre-trained model.

[0008] Next, we will describe an apparatus for training a speech recognition model used in the above-mentioned speech recognition estimation device. The speech recognition model training device includes a speech conversion unit, a language information conversion unit, an estimation unit, and a model update unit. The speech conversion unit converts training input speech information containing speech from multiple speakers into intermediate speech features. The speaker of an utterance included in the training input speech information is designated as the Nth speaker in speaking order, and the language information conversion unit converts the Nth correct speech recognition result, which indicates the content of the Nth speaker's utterance, into the Nth intermediate language features. The estimation unit receives as input the Nth identification information corresponding to the Nth speaker's speaking order, the intermediate speech features, and the Nth intermediate language features, and generates the Nth speech recognition result. The model update unit receives as input the Nth speech recognition result and the Nth correct speech recognition result, and updates the model parameters of the speech conversion unit, language information conversion unit, and estimation unit, which perform processing based on a predetermined model.

[0009] According to the present disclosure, it is possible to recognize the voice of each speaker in real time from a mixed voice containing speech from multiple speakers while maintaining the amount of delay equivalent to that of conventional voice recognition systems.

[0010] FIG. 1 is a diagram for explaining Technique 1. FIG. 2 is a diagram for explaining Technique 2. FIG. 3 is a diagram for explaining Technique 3. FIG. 4 is a diagram showing an example of the functional configuration of a speech recognition model training device according to the first embodiment. FIG. 5 is a diagram showing an example of the processing flow of a speech recognition model training method according to the first embodiment. FIG. 6 is a diagram showing a mixed sound feature X ＭFIG. 7 is a conceptual diagram illustrating the relationship between the first symbol sequence c'1 and the second symbol sequence c'2. FIG. 7 is a conceptual diagram illustrating the maximum likelihood path in an RNN-T. FIG. 8 is a diagram illustrating an example of the functional configuration of a speech recognition model training device according to a modification of the first embodiment. FIG. 9 is a diagram illustrating an example of the processing flow of a speech recognition model training method according to a modification of the first embodiment. FIG. 10 is a diagram illustrating an example of the functional configuration of a speech recognition estimation device according to a second embodiment. FIG. 11 is a diagram illustrating an example of the processing flow of a speech recognition estimation device according to the second embodiment. FIG. 12 is a diagram illustrating an example of the functional configuration of a speech recognition estimation device according to a first modification of the second embodiment. FIG. 13 is a diagram illustrating an example of the processing flow of a speech recognition estimation device according to the first modification of the second embodiment. FIG. 14 is a diagram illustrating an example of the functional configuration of a speech recognition estimation device according to a second modification of the second embodiment. FIG. 15 is a diagram illustrating an example of the processing flow of a speech recognition estimation device according to the second modification of the second embodiment. FIG. 16 is a diagram illustrating an example of the functional configuration of a speech recognition estimation device according to a third modification of the second embodiment. FIG. 17 is a diagram showing an example of a processing flow of a speech recognition estimation device according to a third modified example of the second embodiment. FIG. 18 is a conceptual diagram for explaining timestamp assignment. FIG. 19 is a diagram showing an example of a functional configuration of a speech recognition estimation device according to a fourth modified example of the second embodiment. FIG. 20 is a diagram showing an example of a processing flow of a speech recognition estimation device according to a fourth modified example of the second embodiment. A functional block diagram showing an example of a configuration of a speech recognition model training device according to a third embodiment. A flowchart explaining the operation of a speech recognition model training device according to the third embodiment. A functional block diagram showing an example of a configuration of a speech recognition estimation device according to the third embodiment. A flowchart explaining the operation of a speech recognition estimation device according to the third embodiment. A functional block diagram showing an example of a configuration of a speech recognition estimation device according to a first modified example of the third embodiment. A flowchart explaining the operation of a speech recognition estimation device according to the first modified example of the third embodiment. A functional block diagram showing an example of a configuration of a speech recognition system according to a second modified example of the third embodiment. A flowchart explaining the operation of a speech recognition system according to a second modified example of the third embodiment. A functional block diagram showing an example of a configuration of a speech recognition system according to a third modified example of the third embodiment.32 is a functional block diagram showing an example of the configuration of a speech recognition system according to Modification 4 of the third embodiment. FIG. 33 is a flowchart illustrating the operation of a speech recognition system according to Modification 4 of the third embodiment. FIG. 34 is a diagram illustrating an example of the functional configuration of a computer.

[0011] <Character notation> The symbol "^" (superscript hat) used in text should normally be written directly above the character immediately following it, but due to limitations in text notation, it is written immediately before the character in question. In mathematical formulas, these symbols are written in their proper position, i.e., directly above the character. For example, "^S" is expressed in a mathematical formula as follows:

[0012] Hereinafter, components having the same functions are assigned the same numbers, and duplicated explanations will be omitted.

[0013] An embodiment of the present disclosure is a technology that enables real-time recognition of the target speaker's voice from mixed voice containing speech from multiple speakers by providing a function within a speech recognition model to convert a distributed representation sequence of voice corresponding to target speaker extraction (fourth variant of the first to second embodiments).

[0014] Another embodiment of the present disclosure is a technology that enables simultaneous recognition of the speech of all speakers in real time from a mixed speech containing speech from multiple speakers by introducing a framework for training in which a tag is added to the beginning of each correct label in order to distinguish the speech content of each speaker (third embodiment to fourth modified example of the third embodiment). Before describing the details of the embodiment of the present disclosure, we will first explain a neural network training method for speech recognition and a target speaker speech extraction method in the prior art.

[0015] As a method for training an acoustic model using a general neural network training method, "Recurrent Neural Network Transducer (RNN-T)" described in Non-Patent Document 1 is known (hereinafter, this method will also be referred to as "Technology 1"). Figure 1 shows a functional configuration diagram of a speech recognition model training device using this method.

[0016] Acoustic features X, which are a sequence of speech features, are converted into a distributed representation sequence via a speech conversion unit 101 having a multi-layer neural network function, to become intermediate speech features H, which are a sequence of acoustic features used for estimation in speech recognition. Furthermore, a symbol sequence c of length U, which is a sequence of symbols corresponding to the acoustic features X, is converted into a distributed representation sequence via a symbol conversion unit 102 having a multi-layer neural network function, to become intermediate character features C, which are a sequence of corresponding continuous-value features. Note that the symbol sequence c can also be said to correspond to the text of the speech recognition result.

[0017] The intermediate speech feature H and the intermediate character feature C are input to an estimation unit 103 having a neural network function, and an output probability distribution Y corresponding to label estimation, which is speech recognition, is calculated.

[0018] The calculated output probability distribution Y is input to the loss calculation unit 104 together with the correct symbol c of length U or T, which is a sequence of correct symbols, and the loss L is calculated using a predetermined calculation formula. ＲＮＮ－Ｔ The calculated loss L ＲＮＮ－Ｔ is used to update the model parameters of the speech conversion unit 101, the symbol conversion unit 102, and the estimation unit 103. By repeating the above-described updating of the model parameters, learning is performed to enable more accurate speech recognition.

[0019] "SpeakerBeam" in Reference 1 is known as a method for extracting the speech of a target speaker from a mixed sound that is the speech of multiple speakers (hereinafter, this method will also be referred to as "Technology 2"). Figure 2 shows the functional configuration of a target speaker speech extraction training device that uses this method.

[0020] Reference 1: K. Zmolikova et. al., “SpeakerBeam: Speaker Aware Neural Network for Target Speaker Extraction in Speech Mixtures,” IEEE Journal of Selected Topics in Signal Processing, vol. 13, no. 4, pp. 800-814, 2019.

[0021] Auxiliary speech A, which is a speech waveform of a pre-recorded target speaker's speech and is used as speech that serves as a clue for extracting the target speaker, is input to an auxiliary feature extraction unit 201 having a multi-layer neural network function and converted into auxiliary intermediate feature A', which is an acoustic feature used to extract the target speaker.

[0022] The mixed speech M, which is a speech waveform composed of speeches from multiple people, and the auxiliary intermediate feature A' are input to a target speaker extraction unit 202 having a multi-layer neural network function, and the target speaker extraction unit 202 extracts a target speaker speech ^S, which is the speech of the target speaker, from the mixed speech M using the auxiliary intermediate feature A' as a clue.

[0023] The extracted target speaker speech ^S is input to the loss calculation unit 203 together with the target speaker speech S, which is the correct speech waveform of the target speaker, and the loss L is calculated using these from a predetermined calculation formula. ＴＳＥ The calculated loss L ＴＳＥ is used to update the model parameters of the auxiliary feature extraction unit 201 and the target speaker extraction unit 202. By repeating the above-described updating of the model parameters, learning is performed so as to more accurately extract the target speaker's voice from the mixed voice.

[0024] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Components having the same functions are designated by the same numbers, and redundant explanations will be omitted.

[0025] <First embodiment> Another method for extracting a target speaker's voice from a mixed sound that is the voices of multiple speakers is disclosed in the above-mentioned Non-Patent Document 1 (hereinafter, this method will also be referred to as "Technique 3"). Figure 3 shows a functional configuration diagram of a target speaker's voice extraction and training device that uses this method.

[0026] This technology makes it possible to recognize the target speaker's voice in real time from a mixture of speeches from multiple speakers. The first speech conversion unit 301 uses a first multilayer neural network to convert the auxiliary feature X Ａ The auxiliary intermediate feature H Ａ The second speech conversion unit 302 converts the auxiliary intermediate feature HＡ and the mixed sound feature X Ｍ and the target speaker intermediate feature H Ｓ The symbol conversion unit 102 converts the symbol sequence c into intermediate character features C using a third multi-layer neural network. The estimation unit 103 converts the target speaker intermediate features H Ｓ The loss calculation unit 104 receives the intermediate character feature C, which is the correct symbol, and the output probability distribution Y, and calculates the loss L ＲＮＮ－Ｔ Furthermore, the loss calculation unit 104 calculates the loss L ＲＮＮ－Ｔ The model parameters of the first speech conversion unit 301, the second speech conversion unit 302, the symbol conversion unit 102, and the estimation unit 103 are updated using the above.

[0027] In the inference of speech recognition using a trained model, auxiliary features X are used as clues before speech recognition. Ａ is input, and the first speech conversion unit 301 converts the auxiliary intermediate feature H Ａ The extracted auxiliary intermediate feature H Ａ is embedded as a part of the parameters of the second speech conversion unit 302. However, the auxiliary intermediate feature H Ａ is a vector of the same size as the output dimension of the encoder, and has almost no effect on the calculation speed of the second speech conversion unit 302 during inference. Therefore, it is possible to quickly recognize only the speech of the target speaker, who is a speaker registered in advance, just like Technology 1 (RNN-T), which is a normal speech recognition system.

[0028] When two models (Technology 2 in the first stage and Technology 1 in the second stage) are cascaded to recognize only the target speaker's voice, the large amount of processing required makes real-time speech recognition difficult. However, Technology 3 uses a single model to recognize the target speaker's voice from a mixed sound. That is, the process for extracting the target speaker's voice from a mixed sound and the speech recognition process are combined into a single model. In other words, an end-to-end system is realized, making it possible to implement a lighter model than before within a speech recognition device. This enables high-speed, real-time speech recognition.

[0029] Furthermore, in the inference of speech recognition (speech recognition estimation device A) using a trained model shown in the second embodiment described below, by registering the user's voice in advance, it becomes possible to recognize the voice of a pre-registered speaker (target speaker) from a mixed sound that includes interfering speakers and background noise at a speed equivalent to that of speech recognition when recognizing voice containing the speech of a normal single speaker (RNN-T of Technology 1 shown in FIG. 1 ).

[0030] As shown in Fig. 4, the speech recognition model training device 1A according to this embodiment includes a first speech conversion unit 301, a speech recognition unit 10, and a model update unit 20. The speech recognition model training device 1A as a whole configures a multi-stage and multi-layered neural network. The speech recognition unit 10 generates auxiliary intermediate features H Ａ and the mixed sound feature X Ｍ The speech recognition unit 10 receives as input a first symbol sequence c'1, which is a symbol sequence with identification information of a target speaker, and a second symbol sequence c'2, which is a symbol sequence with identification information of a non-target speaker, and outputs a first output probability distribution Y1, which is an output probability distribution of the target speaker, and a second output probability distribution Y2, which is an output probability distribution of a non-target speaker. The speech recognition unit 10 includes, for example, a second speech conversion unit 302, a language information conversion unit 102A, and an estimation unit 103A. The model update unit 20 receives as input the first symbol sequence c'1, which is correct data, the second symbol sequence c'2, which is correct data, the first output probability distribution Y1, and the second output probability distribution Y2, and updates each model parameter used in the first speech conversion unit 301, the second speech conversion unit 302, the language information conversion unit 102A, and the estimation unit 103A. Each model parameter is stored, for example, in a model parameter storage unit D2. The model updating unit 20 includes, for example, a first loss calculation unit 104A, a second loss calculation unit 402, and a loss integration unit 403. The speech recognition model training device 1A performs the speech recognition model training method of this embodiment by implementing the processing flow illustrated in FIG.

[0031] (First speech conversion unit 301) The first speech conversion unit 301 is a speech distributed representation sequence conversion unit of target speaker information extraction type. The first speech conversion unit 301 has the same function as the first speech conversion unit 301 in Fig. 3. That is, the first speech conversion unit 301 uses a multi-layer neural network (first multi-layer neural network) to convert auxiliary features X Ａ is used as the auxiliary intermediate feature H Ａ (Step S301). Ａ is a sequence of acoustic features extracted from a pre-recorded utterance of a target speaker, and is a sequence of acoustic features of speech used as a clue for extracting the target speaker (this speech is also referred to as "target speaker information"). That is, unlike the auxiliary feature extraction unit 201 that inputs a speech waveform in Technique 2, the first speech conversion unit 301 plays the role of an encoder that inputs the sequence of acoustic features of the target speaker extracted for speech recognition into a multi-layer neural network and converts them into intermediate acoustic features of the target speaker information.

[0032] The first audio conversion unit 301 performs the conversion using a mathematical formula equivalent to the following formula: Here, H ｔａｒｇｅｔ’ is the auxiliary intermediate feature H Ａ is the auxiliary intermediate feature sequence of length T that is the source of f Ｓｐｋ－Ｅｎｃ’ (·) is the loudspeaker encoder (the first multilayer neural network mentioned above), and f ＦＥ (·) is the feature extraction function, and A ｃｌｕｅ is the auxiliary sound A explained in Technology 2, and θ Ｓｐｋ－Ｅｎｃ’ is a trainable (updatable) parameter in the first speech conversion unit 301, and h ｔａｒｇｅｔ’ is the auxiliary intermediate feature H Ａ and h ｔ ｔａｒｇｅｔ’ is the auxiliary intermediate feature at time t.

[0033] Step S301 may be configured to acquire information about the target speaker and acquire a speech embedding sequence of the target speaker, for example, by repeatedly pronouncing a specific term (word) a predetermined number of times on an arbitrary mobile terminal.

[0034] During learning, the auxiliary feature X becomes the learning data. Ａ The first speech conversion unit 301 receives a speech waveform as an input and converts the speech waveform into an auxiliary feature X Ａ and this auxiliary feature X Ａ Using the auxiliary intermediate feature H Ａ Therefore, the first speech conversion unit 301 may be configured to convert the speech information of the target speaker into the auxiliary feature X Ａ It can also be said that it converts into

[0035] The converted auxiliary intermediate feature H Ａ is transmitted to the second speech conversion unit 302.

[0036] (Second speech conversion unit 302) The second speech conversion unit 302 is a speech distributed representation series conversion unit of target speaker speech extraction type. That is, the second speech conversion unit 302 uses a multi-layer neural network (second multi-layer neural network) to convert the auxiliary intermediate features H Ａ and a mixed sound feature X, which is a feature sequence of speech from a target speaker and a non-target speaker who is a speaker different from the target speaker. Ｍ The mixed sound feature X is converted into intermediate speech feature Hs′, which is a series of features commonly used by the target speaker and non-target speakers (step S302). Ｍ may be a mixture of speeches from multiple speakers, may not include speech from the target speaker, or may include only speech from the target speaker. However, for the convenience of explaining the processing of the present disclosure, hereinafter, the mixed speech feature X Ｍ Unless otherwise specified, the following description will be given assuming that the feature quantity is related to a mixed sound of two speakers, a target speaker and a non-target speaker.

[0037] Unlike the target speaker extraction unit 202 that receives the speech waveform, the second speech conversion unit 302 receives the mixed sound feature X , which is a series of acoustic features of mixed speech including multiple speakers extracted for voice recognition. Ｍis converted into intermediate speech feature Hs′ using a multi-layer neural network separate from the first speech conversion unit 301.

[0038] Therefore, as a subsequent process, a speech recognition training function can be provided that estimates the symbol sequence of the target speaker using the intermediate speech feature Hs′, similar to the processes of the symbol conversion unit 102, the estimation unit 103, and the loss calculation unit 104 described in Technique 1.

[0039] The second speech conversion unit 302 performs the conversion using a mathematical formula equivalent to the following formula: Here, h ｔ ＡＳＲ’ is the intermediate speech feature Hs', and f ＡＳＲ－Ｅｎｃ’ is the encoder of the second speech conversion unit 302 (the second multilayer network described above), and f ＦＥ (·) is the feature extraction function, and x ｔ’ is the mixed voice at time t' (corresponding to the mixed voice M in Technique 2), and h ｔａｒｇｅｔ’ is the auxiliary intermediate feature H Ａ and θ ＡＳＲ－Ｅｎｃ’ is a trainable (updatable) parameter in the second speech conversion unit 302.

[0040] The second speech conversion unit 302 converts the input mixed sound feature X Ｍ is processed as a feature sequence of a mixed voice in which voices of multiple speakers are mixed. Ｍ is simply treated as a feature sequence of mixed sounds and processed. Ｍ may be a mixture of speeches from multiple speakers, may not include the speech of the target speaker, or may include only the speech of the target speaker. Ｍ and convert this mixed sound feature X Ｍ Hereinafter, the speech waveform and the mixed sound feature X Ｍ Thus, the audio information that the second audio conversion unit 302 accepts as input will also be collectively referred to as "input audio information."

[0041] The converted intermediate speech feature Hs' is sent to the estimation unit 103A.

[0042] (Language Information Conversion Unit 102A) The language information conversion unit 102A uses a multi-layer neural network (third multi-layer neural network) to convert a symbol sequence of length U (hereinafter also referred to as the "first symbol sequence c'1"), which is a symbol sequence including first identification information L1 for identifying a target speaker and a first symbol sequence c1 indicating the speech content of the target speaker, into a first intermediate language feature C1, which is a corresponding sequence of continuous-value features. Also, the language information conversion unit 102A uses a multi-layer neural network (third multi-layer neural network) to convert a symbol sequence of length U (hereinafter also referred to as the "second symbol sequence c'2"), which is a symbol sequence including second identification information L2 for identifying a non-target speaker and a second symbol sequence c2 indicating the speech content of the non-target speaker, into a second intermediate language feature C2, which is a corresponding sequence of continuous-value features (step S102A). That is, the language information conversion unit 102A plays the role of an encoder, converting the input into a one-hot vector once and then converting it into a first intermediate language feature C1 and a second intermediate language feature C2 by a multi-layer neural network. Note that the processing by the language information conversion unit 102A processes the target speaker and the non-target speaker in parallel.

[0043] The symbol conversion unit 102 in Techniques 1 and 3 converts data into intermediate character features assuming only the target speaker. The language information conversion unit 102A has basic functions equivalent to those of the symbol conversion unit 102, but converts data into intermediate language features assuming not only the target speaker but also non-target speakers.

[0044] The first symbol sequence c'1 is, for example, a symbol " <ts>The second symbol sequence c'2 is obtained by adding " " to the beginning of the first symbol sequence c1. The second symbol sequence c'2 is obtained by adding " ", which is the initials of the non-target speaker, to the second symbol sequence c2, which is the second utterance content, as the second identification information L2. <nts>" is added to the beginning of the second symbol sequence c2. That is, when the first symbol sequence c1 is "hello", the first symbol sequence c'1 is " <ts>When the second symbol sequence c2 is "good bye", the second symbol sequence c'2 becomes " <nts>"Good bye."

[0045] Therefore, the mixed sound feature X Ｍ An example of the relationship between the mixed sound feature X Ｍ is the speech feature of the target speaker X Ｍ-1 and the speech features of the non-target speaker X Ｍ-２ As shown in Figure 6, Ｍ-1 is "how are you", the first symbol sequence c'1 is " <ts>How are you?' and X Ｍ-2 is "I am fine", the second symbol sequence c'2 is " <nts>If there are multiple non-target speakers, it is sufficient to prepare identification information that allows them to distinguish each other. For example, if there are N non-target speakers (N>0), <nts1> , <nts2> ,...， <ntsn>For example, X Ｍ-1 "Hello," X Ｍ-2 "Goodbye," X Ｍ-3 "Good evening," X Ｍ-4 "Good morning,"...X Ｍ-Ｎ＋１ If the first symbol sequence c'1 is "yoroshiku onegaishimasu", then <ts>Hello," and the second symbol sequence c'2 is " <nts1>"Goodbye," and the third symbol sequence c'3 is " <nts2>"Good evening," and the fourth symbol sequence c'4 is " <nts3>"Good morning", ..., the N+1th symbol sequence c'N+1 is " <ntsn>Thank you in advance."

[0046] The upper limit of the number of non-target speakers that can be processed is determined by the number of non-target speakers that are simultaneously trained at the time of training. <nts3>If learning is performed using the identification tags and speech contents up to this point, the total number of non-target people will be three.

[0047] Furthermore, in both the learning phase (i.e., speech recognition model training device 1A, and speech recognition model training device 1B, which will be described later) and the inference phase (i.e., speech recognition model training device 1B, and speech recognition estimation devices 2A, 2B, 2C, 2D, and 2E, which will be described later), if there are multiple non-target speakers, the processing from feature extraction to speech recognition result output is performed multiple times in parallel for each speaker.

[0048] The order of steps S302 and S102A may be reversed, or they may be performed in parallel.

[0049] The converted first intermediate language feature C1 and second intermediate language feature C2 are transmitted to the estimation unit 103A.

[0050] (Estimation Unit 103A) The estimation unit 103A uses a neural network to calculate a first output probability distribution Y1 (hereinafter also referred to as the "first speech recognition result") and a second output probability distribution Y2 (hereinafter also referred to as the "second speech recognition result") of a two-dimensional matrix corresponding to label estimation, using the intermediate speech feature Hs', the first intermediate language feature C1, and the second intermediate language feature C2 as input (step S103A). The processing by the estimation unit 103A performs processing related to the target speaker and processing related to non-target speakers in parallel. The estimation unit 103 of Technology 1 calculates the output probability distribution of the target speaker. The estimation unit 103A of the present disclosure has basic functions equivalent to those of the estimation unit 103, but calculates the output probability distributions of non-target speakers in addition to the output probability distribution of the target speaker.

[0051] The first output probability distribution Y1 and the second output probability distribution Y2 are calculated using the following equations: Here, y ｔ，ｕ is the intermediate speech feature Hs' (H t ) and the u-th intermediate language feature C ｕ is the output probability distribution when W is input. １ is the input H ｔ is the weight of the hidden layer for ２ is C ｕ are the hidden layer weights for ３ is the input tanh(W １ H ｔ +W ２ C ｕ +b) and Softmax is the activation function.

[0052] In addition, in the above equation, since the lengths of t and u are different, there is also the dimension of the number of elements in the neural network in addition to t and u, so it becomes three-dimensional. １ H copies the same values ​​in the dimension direction of U and expands it into a 3-dimensional tensor. ２ C copies the same value in the dimensional direction of T and expands it into a three-dimensional tensor. Since three-dimensional tensors are added together, the output is also a three-dimensional tensor.

[0053] Generally, RNN-T learning is performed using RNN-T loss, assuming that the result will be a three-dimensional tensor. However, during inference, which is the processing of the estimation unit 103A, the output will be a two-dimensional matrix because there is no expansion operation.

[0054] The calculated first output probability distribution Y1 and second output probability distribution Y2 are transmitted to the model update unit 20.

[0055] (First Loss Calculation Unit 104A) The first loss calculation unit 104A receives as input a first symbol sequence c′1 (having length U or length T) that is a target speaker's symbol sequence corresponding to correct data (hereinafter also referred to as a “first correct speech recognition result”) and a first output probability distribution Y1 that is a three-dimensional tensor, and calculates a first loss L ＲＮＮ－Ｔ－ＴＳ (Step S104A). The first loss calculation unit 104A corresponds to a function equivalent to the processing function of loss calculation carried out by the loss calculation unit 104 of Technique 1.

[0056] Loss L ＲＮＮ－Ｔ－ＴＳ As shown in FIG. 7, the calculation of is performed by creating a tensor with the vertical axis representing the symbol sequence length U, the horizontal axis representing the input sequence length T, and the depth representing the number of classes, i.e., the number of symbol entries K, and then calculating the path of the optimal transition probability on the U×T plane based on a forward-backward algorithm. When moving to the horizontal axis, a blank is output, and when moving to the vertical axis, the correct symbol is output. FIG. 7 is a conceptual diagram of an example of output. Details of the calculation are described, for example, in Chapter 2 "2. Recurrent Neural Network Transducer" of the above-mentioned Non-Patent Document 1. The calculated first loss L ＲＮＮ－Ｔ－ＴＳ is sent to the loss consolidation unit 403.

[0057] (Second loss calculation unit 402) The second loss calculation unit 402 receives as input a second symbol sequence c'2 (hereinafter also referred to as a "second correct speech recognition result"), which is a symbol sequence of a non-target speaker that corresponds to correct data, and a second output probability distribution Y2, which is a three-dimensional tensor, and calculates a second loss L ＲＮＮ－Ｔ－ＮＴＳ (Step S402). The second loss calculation unit 402 has the same function as the first loss calculation unit 104A, and therefore a description of the processing method will be omitted. The calculated second loss L ＲＮＮ－Ｔ－ＮＴＳ is sent to the loss consolidation unit 403.

[0058] The processing order of step S104 and step S402 may be reversed, or they may be performed in parallel.

[0059] (Loss Integration Unit 403) The loss integration unit 403 integrates the losses of the target speaker and the non-target speaker using a hyperparameter λ (λ>0). ＲＮＮ-Ｔ Then, the model parameters are updated (step S403). That is, the loss integration unit 403 calculates the loss L ＲＮＮ－Ｔ Calculate. Next, the loss integration unit 403 calculates the loss L ＲＮＮ－Ｔ The model parameters of the first speech conversion unit 301, the second speech conversion unit 302, the language information conversion unit 102A, and the estimation unit 103A are updated using the above. These model parameters may be stored in the storage area of ​​each component, or a model parameter D2 storing each model parameter may be provided as shown in FIG.

[0060] The speech recognition model training device 1A repeats the above-described updating of the model parameters to perform training so as to perform correct speech recognition.

[0061] The speech recognition model training device 1A according to this embodiment is expected to achieve the effects described in Non-Patent Document 1. That is, the amount of computational processing required for a speech recognition model trained by the speech recognition model training device 1A is considered to be equivalent to that of, for example, the speech recognition device of Non-Patent Document 1. Furthermore, unlike Non-Patent Document 1, the speech recognition model can simultaneously recognize speech from a target speaker and non-target speakers in a single recognition run. Therefore, while maintaining the amount of delay equivalent to that of conventional speech recognition systems, it is possible to recognize speech from target speakers and non-target speakers in real time from a mixed speech containing speech from multiple speakers, demonstrating an improvement in the field of speech recognition model training technology.

[0062] <Modification of the First Embodiment> The above-described speech recognition model training device 1A may be configured as a speech recognition model training device 1B in Fig. 8. The speech recognition model training device 1B differs from the speech recognition model training device 1A in that the speech recognition model training device 1B includes a correct speech recognition result generation unit 401. Accordingly, the flow diagram in Fig. 5 is changed as shown in Fig. 9. That is, step S401 is added before step S301.

[0063] (Correct speech recognition result generating unit 401) The correct speech recognition result generating unit 401 generates a first symbol sequence c'1 by adding first identification information L1 to the beginning of a first symbol sequence c1, which is the first utterance content. Also, it generates a second symbol sequence c'2 by adding second identification information L2 to the beginning of a second symbol sequence c2, which is the second utterance content (step S401). The generated first symbol sequence c'1 and second symbol sequence c'2 are sent to the language information converting unit 102A.

[0064] The speech recognition model training device 1B according to this modification can train a speech recognition model after assigning identification information for identifying target speakers and non-target speakers to symbols that have not been assigned identification information in advance, such as the first symbol sequence c1 and the second symbol sequence c2. Therefore, the estimation unit 103A can be instructed as to whether the target speaker or non-target speaker is the subject of estimation, and then the speech recognition results to be estimated for each speaker can be trained. Furthermore, model training can be performed more efficiently than with the speech recognition model training device 1A. These points represent an improvement in the field of speech recognition model training.

[0065] Second Embodiment A speech recognition estimation device 2A according to this embodiment will now be described. As shown in FIG. 10 , the speech recognition estimation device 2A includes a first speech conversion unit 301 and a speech recognition unit 10A. Each component of the speech recognition estimation device 2A uses trained model parameters D2 obtained by training using the speech recognition model training device 1A or the speech recognition model training device 1B. The model parameters D2 may be provided within the speech recognition estimation device 2A, or may be provided separately from the speech recognition estimation device 2A. Alternatively, each component of the speech recognition estimation device 2A may be configured to store the model parameters it uses.

[0066] (Speech Recognition Unit 10A) The speech recognition unit 10A recognizes at least the mixed sound feature X Ｍ The speech recognition unit 10A uses the input of the first estimated label C1F corresponding to the label estimation result of the target speaker and the second estimated label C2F corresponding to the label estimation result of the non-target speaker, by using a multi-layer neural network. When generating the first estimated label C1F and the second estimated label C2F, the speech recognition unit 10A uses the auxiliary intermediate feature H Ａ The speech recognition unit 10A includes, for example, a second speech conversion unit 302 that has been trained by the speech recognition model training device 1A or 1B, a language information conversion unit 102A (language information conversion unit 102B in FIG. 10), and an estimation unit 103A (estimation unit 103B in FIG. 10). As described above, the mixed sound feature X Ｍ may be a mixture of speech from multiple speakers, may not include speech from the target speaker, or may include only speech from the target speaker.

[0067] The speech recognition estimation device 2A performs the speech recognition estimation method of this embodiment by carrying out the processing flow illustrated in FIG.

[0068] (First speech conversion unit 301) The first speech conversion unit 301 has the same function as the above-mentioned first speech conversion unit 301. That is, the first speech conversion unit 301 converts auxiliary features X Ａ is used as the auxiliary intermediate feature H Ａ (Step S301). Note that, unlike the training of the speech recognition model training device 1A, this step S301 only needs to be performed once for the speaker corresponding to the target speaker. Ａ is transmitted to the second speech conversion unit 302.

[0069] (Second speech conversion unit 302) The second speech conversion unit 302 has the same function as the above-described second speech conversion unit 302. That is, the second speech conversion unit 302 uses a multilayer neural network (second multilayer neural network) to convert the auxiliary intermediate feature H Ａ and the mixed sound feature X Ｍ The input is converted into intermediate speech features Hs' (step S302). The converted intermediate speech features Hs' are sent to the estimation unit 103B.

[0070] (Language Information Conversion Unit 102B) The language information conversion unit 102B basically has the same functions as the above-described language information conversion unit 102A. The language information conversion unit 102B converts the first symbol sequence c'1 into first intermediate language features C1 using a multi-layer neural network (third multi-layer neural network). It also converts the second symbol sequence c'2 into second intermediate language features C2. To perform RNN-T inference, the language information conversion unit 102B receives the first estimated label C1F and the second estimated label C2F most recently estimated by the estimation unit 103B and uses these labels to perform conversion (step S102B). The processing by the language information conversion unit 102B involves processing related to the target speaker and processing related to the non-target speaker in parallel. The converted first intermediate language features C1 and second intermediate language features C2 are transmitted to the estimation unit 103B.

[0071] In order for the estimation unit 103B to start estimation, it is necessary to input the first intermediate character feature C1 and the second intermediate character feature C2 from the language information conversion unit 102B. In order for the language information conversion unit 102B to generate the first intermediate character feature C1 and the second intermediate character feature C2, it is necessary to input the first symbol sequence c'1 and the second symbol sequence c'2. In the speech recognition and estimation device 2A, the first symbols of the first symbol sequence c'1 and the second symbol sequence c'2 correspond to identification information. That is, when the first identification information L1 is " <ts>", and the second identification information L2 is " <nts>In the case of ", the speech recognition unit 10A first receives " as the first symbol sequence c'1. <ts>" is first entered as the second symbol sequence c'2 as " <nts>" will be entered.

[0072] (Estimation Unit 103B) The estimation unit 103B performs label estimation using a neural network with the intermediate speech feature Hs', the first intermediate language feature C1, and the second intermediate language feature C2 as input, and generates a first estimated label C1F that is an estimated label of the target speaker and a second estimated label C2F that is an estimated label of the non-target speaker (step S103B). The processing by the estimation unit 103B performs processing related to the target speaker and processing related to the non-target speakers in parallel. The first estimated label C1F and the second estimated label C2F are output as estimation results from the speech recognition / estimation device 2A and are also transmitted to the language information conversion unit 102B. Note that the estimation results may include, in addition to estimated labels representing the content of the speech utterance, timestamps corresponding to the estimated labels for each predetermined unit.

[0073] As described above, the speech recognition unit 10A in the speech recognition estimation device 2A uses the auxiliary intermediate feature H Ａ and input speech information into intermediate speech features Hs', which are a series of intermediate features, and a first speech recognition result is generated from the intermediate speech features Hs' in response to the input of first identification information L1 for identifying a target speaker, and a second speech recognition result is generated from the intermediate speech features Hs' in response to the input of second identification information L2 for identifying a non-target speaker.

[0074] By registering a user's voice in advance, the speech recognition estimation device 2A according to this embodiment can perform speech recognition of the pre-registered speaker (target speaker) from among mixed sounds including interfering speakers and background noise at a speed equivalent to that of speech recognition when recognizing speech including the speech of a normal single speaker (RNN-T of Technology 1 shown in FIG. 1).

[0075] For example, during inference, the number of speech recognition results (search width / beam width) per time t is set to the same for the target speaker and non-target speakers. In this case, the processing from the inference section onwards is performed in parallel for the target speaker and non-target speakers, doubling the processing volume. On the other hand, because the output is obtained simultaneously for each sequence, the processing time is essentially equivalent to that of a speech recognition device that recognizes only the target speaker. Furthermore, by narrowing the number of speech recognition results per time t (search width / beam width), the processing volume per speaker can be reduced accordingly. For example, if the search width is set to A in conventional speech recognition for only the target speaker, setting the search width for the target speaker and non-target speaker to A / 2 will result in essentially the same processing volume as that of a device that recognizes speech from a single speaker or only the target speaker.

[0076] Furthermore, it is possible to simultaneously recognize the speech of both the target and non-target speakers in a single recognition run. This means that while maintaining the same amount of delay as conventional speech recognition systems, it is possible to recognize the speech of both the target and non-target speakers in real time from a mixed speech containing speech from multiple speakers, demonstrating an improvement in the field of speech recognition and estimation devices.

[0077] 11 is capable of performing the processing described in Fig. 11, there is no particular limitation on the type of terminal of the speech recognition estimation device 2A, and the speech recognition estimation device 2A may be configured to perform processing on a smartphone, a smartwatch, or a personal computer. Also, the function of the first speech conversion unit 301 may be separated from the speech recognition estimation device 2A and processed in another device.

[0078] <First Modification of Second Embodiment> The above-described speech recognition estimation device 2A may be configured as a speech recognition estimation device 2B shown in Fig. 12. The speech recognition estimation device 2B differs from the speech recognition estimation device 2A in that it includes an identification information transmission unit 401B. Accordingly, the flow diagram of Fig. 11 is changed as shown in Fig. 13. That is, step S401B is added before step S301.

[0079] (Identification Information Transmitting Unit 401B) The identification information transmitting unit 401B transmits the first symbol sequence c'1 and the second symbol sequence c'2 to the language information conversion unit 102B (step S401B). As described above, in order for the estimation unit 103B to start estimation, it is necessary to input the first intermediate character feature C1 and the second intermediate character feature C2 from the language information conversion unit 102B. In order for the language information conversion unit 102B to generate the first intermediate character feature C1 and the second intermediate character feature C2, it is necessary to input the first symbol sequence c'1 and the second symbol sequence c'2. In other words, the identification information transmitting unit 401B plays a role in inputting the first symbol sequence c'1 and the second symbol sequence c'2 required for the initial processing of the language information conversion unit 102B to the speech recognition unit 10A. Specifically, the identification information transmitting unit 401B transmits the first identification information L1 as the first symbol sequence c'1 and the second identification information L2 as the second symbol sequence c'2. <ts>", and the second identification information L2 is " <nts>", the identification information transmitting unit 401B transmits " as the first symbol sequence c'1. <ts>", and the second symbol sequence c'2 is " <nts>" to the language information conversion unit 102B.

[0080] This modification also enables efficient speech recognition. That is, by registering a user's speech in advance, the speech recognition estimation device 2B according to this modification also enables speech recognition of the pre-registered speaker (target speaker) from a mixture of sounds containing interfering speakers and background noise at a speed equivalent to that of speech recognition (RNN-T, Technology 1 shown in FIG. 1 ) when recognizing speech containing a normal single-speaker speech. Furthermore, it is possible to simultaneously recognize the speech of the target speaker and non-target speakers in a single recognition run. Therefore, while maintaining the same amount of delay as conventional speech recognition systems, it is possible to recognize the speech of the target speaker and non-target speakers in real time from a mixture of sounds containing multiple speakers' speech, demonstrating an improvement in the field of speech recognition estimation devices.

[0081] <Second Modification of Second Embodiment> The above-described speech recognition estimation device 2A may be configured as a speech recognition model training device 2C in Fig. 14. The speech recognition estimation device 2C differs from the speech recognition estimation device 2A in that the auxiliary intermediate feature H Ａ The difference is that a memory unit D3 for storing the above, a recognition control unit 50, and a saving and output unit 60 are newly provided. Accordingly, the flow chart of Fig. 11 is changed as shown in Fig. 15. That is, step S50 is added before step S302, and step S60 is added after step S103B.

[0082] (Recognition control unit 50) Upon receiving a predetermined signal, the recognition control unit 50 detects the auxiliary intermediate feature H Ａ to the second speech conversion unit 302. That is, when the start signal R1 is received, the recognition control unit 50 sends (transmits) the registered auxiliary intermediate feature H Ａ from the storage unit D3 where the auxiliary intermediate feature H is stored in advance, and transmits it to the second speech conversion unit 302 (step S50). Ａ Since the speech recognition unit 10 cannot start processing without the input of the start signal R1, the recognition control unit 50 can essentially adjust the timing of the start of processing by the speech recognition unit 10. In other words, the processing of the recognition control unit 50 can be said to be processing that controls the execution of recognition. Note that the recognition control unit 50 may be configured to obtain the setting information R2 upon receiving the start signal R1 by registering in advance in a database (not shown) setting information R2 that includes identification information of the speaker to be assigned and information specifying the speaker to be saved in a memory or storage device.

[0083] (Storage and Output Unit 60) The storage and output unit 60 stores (stores) or outputs the first estimated label C1F and the second estimated label C2F, which are the output results of the estimation unit 103B (step S60). The storage and output unit 60 is a part that has a storage device or a display unit such as a display, or both a storage device and a display unit. The storage and output unit 60 may be configured as one of the components of the speech recognition estimation device 2C, or may be provided separately from the speech recognition estimation device 2C, for example, via a communication network such as a network.

[0084] An example of a usage form of the speech recognition estimation device 2C is as follows. In customer service counter operations such as at a store counter, by registering the operator's voice in advance in the storage unit D3, it is possible to recognize the voices of two people by separating them from the customer's voice. For example, in the setting information R2, the target speaker label is set to "operator" and the non-target speaker label is set to "customer." This makes it possible for the estimation unit 103B to output, for example, "operator" at the beginning of the first estimated label C1F and "customer" at the beginning of the second estimated label C2F. If the names of the target speaker and non-target speakers can be known in advance, this information may be registered as setting information R2, and the names of the target speaker and non-target speakers, or either one of the names, may be output.

[0085] Another example of the use of this embodiment is as follows: For example, when one translator is assigned to each important client to participate in a conference (this combination is counted as one group). This is useful in both cases: (1) when the group participates in the conference online from the same location using the same terminal (sharing one speech recognition estimation device), and (2) when multiple groups participate in the conference online from their own locations. By registering the voice of each interpreter in advance, speech recognition for all speakers becomes possible, with the interpreter as the target speaker and the important client as a non-target speaker. When multiple languages ​​are used, multilingual speech recognition may be required. The language of each speaker may be registered in advance, or the language may be automatically determined.

[0086] In addition, if the name of the non-target speaker cannot be determined, a name may be assigned to the speech recognition result of the non-target speaker (second estimated label C2F) after the fact, as long as the output result of the estimation unit 103B is stored in the saving / output unit 60.

[0087] Like the speech recognition estimation device 2A, the speech recognition estimation device 2C according to this modification can recognize speech of target and non-target speakers in real time from a mixed speech containing speech from multiple speakers while maintaining the same amount of delay as conventional speech recognition systems. It can also adjust the timing at which the speech recognition unit starts processing. These points represent an improvement in the field of speech recognition estimation devices.

[0088] <Third Modification of Second Embodiment> The above-described speech recognition estimation device 2C may be configured as a speech recognition model training device 2D shown in Fig. 16. The speech recognition estimation device 2D differs from the speech recognition estimation device 2C in that a speech extraction unit 70 is newly provided. Accordingly, the flow diagram of Fig. 15 is changed as shown in Fig. 17. That is, in addition to step S60, step S70 is added as the process following step S103B.

[0089] (Speech Extraction Unit 70) The speech extraction unit 70 extracts the mixed sound feature X Ｍ , the first estimated label C1F, and the second estimated label C2F are input, and acoustic features of the target speaker and acoustic features of the non-target speaker are extracted (step S70). That is, the speech extraction unit 70 extracts the mixed sound feature X Ｍ From the speech information, speech information about the target speaker (first speech information) or speech information about the non-target speaker (second speech information), or speech information about both, is extracted. Here, the case where acoustic features are extracted as speech information will be described as an example. Here, the acoustic features of the target speaker are acoustic features of the part of the speech that the target speaker is speaking, and are represented by X shown in FIG. Ｍ－１ The acoustic features of the non-target speakers are the acoustic features of the parts of the speech that are spoken by the non-target speakers, and are represented by the X Ｍ－２ Therefore, the speech extraction unit 70 generates acoustic features by separating the parts spoken by the target speaker or the parts spoken by the non-target speaker from the other parts. As an example of a separation method, for example, the mixed sound feature X Ｍ Mixed sound feature X Ｍ In this method, the speech of the target speaker and the speech of the non-target speaker can be distinguished by using the time stamp of the output content of the estimation unit 103B corresponding to the mixed sound feature X Ｍ In this case, the overlapping portions of the acoustic features of the target speaker and non-target speakers cannot be clearly separated. Therefore, for example, when extracting speech information of the target speaker, (1) extraction is performed based on the timestamp of the speech recognition result of the target speaker. In this case, the speech information will be extracted including the portions overlapping with the speech of the non-target speakers. Also, (2) extraction is performed based on the timestamps of the speech recognition results of both the target speaker and non-target speakers. In this case, the speech information can be extracted by removing the portions overlapping with the speech of the non-target speakers. The above method (2) can extract information that does not include the speech information of the non-target speakers, and therefore can take personal information into consideration.

[0090] FIG. 18 is a conceptual diagram for explaining the assignment of a timestamp. In FIG. 18, as in FIG. 6, X Ｍ-1 "How are you", X Ｍ-2 Assume that "I am fine" is entered. X Ｍ-1 and X Ｍ-2 The utterance time T is assumed to be from T1 to T4. In this case, the processing of the speech recognition estimation device 2D outputs a first estimated label C1F and a second estimated label C2F, as shown on the right side of FIG.

[0091] Since the processing of the estimation unit 103B is performed frame-by-frame, when "how" is output at a certain time, the timestamp "T1" of that time is automatically determined. Just as a time index is attached to the probability (the first output probability distribution Y1 or the second output probability distribution Y2 is attached with t), estimation by the estimation unit 103B is performed in time units, so it is possible to recognize which word was uttered at what timing. Note that in actual processing, timestamps are assigned in subword units (or character units), but FIG. 18 explains the processing in word units as an example to facilitate understanding.

[0092] The generated acoustic features are sent to the storage and output unit 60 .

[0093] Note that a speaker to be saved (stored) in the saving / output unit 60 may be specified as setting information R2, so that, of the acoustic feature information extracted by the speech extraction unit 70, only the acoustic feature of the speaker specified by setting information R2 is saved in the saving / output unit 60. As an example, the first speech recognition result and the first speech information related to the target speaker may be kept stored, and the input speech information and other information generated during the processing may be deleted from the device. As another example, the first speech recognition result and the first speech information related to the target speaker may be kept stored, and the input speech information and other information generated during the processing may be deleted from the device.

[0094] For example, after performing speech recognition for multiple speakers (the non-target speaker may be one or more), (1) the recognition results are presented to the user, and a designation of information to be retained or deleted from the terminal regarding acoustic features and speech recognition results is accepted. Possible means for presenting the results to the user in text form may be to display the recognition results on a display included in or connected to the terminal. The designation of information to be retained or deleted may be made for each speech recognition target, or may be made for each piece of information, such as acoustic features or speech recognition results. Information is deleted based on the accepted designation.

[0095] It is possible to promote the protection of personal information by not storing (memorizing) unnecessary acoustic features in the storage and output unit 60. In other words, this technology can contribute to suppressing user concerns about privacy and security and the resulting decline in user motivation.

[0096] Like the speech recognition estimation device 2A, the speech recognition estimation device 2D according to this modification can recognize speech of a target speaker and speech of a non-target speaker in real time from a mixed speech containing speeches from multiple speakers while maintaining the amount of delay equivalent to that of a conventional speech recognition system. Ｍ From this, acoustic features for the target speaker, non-target speakers, or both can be extracted. In this respect, this represents an advancement in the field of speech recognition and estimation devices.

[0097] <Fourth Modification of Second Embodiment> The above-described speech recognition estimation device 2D may be configured as a speech recognition model training device 2E in Fig. 19. The speech recognition estimation device 2E differs from the speech recognition estimation device 2D in that a transmission unit 80 is newly provided. Accordingly, the flow diagram in Fig. 17 is changed as shown in Fig. 20. That is, in addition to step S60, step S80 is added as the process following step S70.

[0098] (Transmitting unit 80) The transmitting unit 80 receives the acoustic feature extraction results (including at least one of the target speaker's speech information and the non-target speaker's speech information) from the speech extracting unit 70 and the estimation results (including at least one of the first estimated label C1F and the second estimated label C2F) from the estimating unit 103B, and transmits them to a pre-specified device (step S80). In this case, if information about the destination and the type of results to be transmitted are registered as setting information R2, the transmitting unit 80 may transmit the acoustic feature of the non-target speaker to the non-target speaker's terminal. For example, if the setting information R2 specifies the destination device as the non-target speaker's terminal and the type of results to be transmitted as speech information of only the non-target speaker, the transmitting unit 80 transmits the acoustic feature of the non-target speaker to the non-target speaker's terminal. As another example, if the type of results to be transmitted is specified as speech information of only the non-target speaker and the estimation result, the transmitting unit 80 transmits the estimation result and acoustic feature of the non-target speaker to the non-target speaker's terminal. The speech recognition estimation device 2E can further increase the reliability of personal information protection in that it can send only information about a predetermined speaker (in this case, a non-target speaker) to a specified device.

[0099] Like the speech recognition estimation device 2A, the speech recognition estimation device 2E according to this modification can recognize the speech of target and non-target speakers in real time from a mixed speech containing speech from multiple speakers, while maintaining the same amount of delay as conventional speech recognition systems. Furthermore, the speech recognition estimation device 2E can further promote the protection of personal information by transmitting only information about a specific speaker to a specified device. These points represent an improvement in the field of speech recognition estimation devices.

[0100] Third Embodiment In the above embodiment, following Technique 3, acoustic features (auxiliary intermediate features) H A Using this, the mixed sound feature X M The acoustic features (intermediate speech features) Hs' for speech recognition were generated from the input speech. This made it possible to identify which of the speeches of multiple speakers recognized from the mixed speech is the speech of the target speaker. If it is not necessary to identify the target speaker, it is not necessary to use the acoustic features of the target speaker to generate Hs'. This will be described as the third embodiment.

[0101] 21 is a functional block diagram showing an example of the configuration of a speech recognition model training device 21 according to the third embodiment. The difference from the speech recognition model training device 1A is that the first speech conversion unit 301 is omitted, and the second speech conversion unit 302 is replaced by X M The speech recognition model training device 21 has the following features: a third speech conversion unit 2101 that generates intermediate speech feature Hs' using only the input; a speaker information addition unit 2102; and N loss calculation units. For ease of understanding, an example with N loss calculation units is described; however, N calculations may be performed with a single loss calculation unit. Figure 22 is a flowchart illustrating an example of the operation of the speech recognition model training device 21. The following description will be made with reference to Figures 21 and 22.

[0102] For example, the storage unit D1 stores mixed sound features X obtained by converting a sound (mixed sound X) that simulates a conversation by mixing utterances from multiple speakers (assumed to be N speakers) into acoustic features on a frame-by-frame basis. M In addition, N symbol sequences that represent the contents of each speaker's speech in text form are stored. M and X M The symbol sequence c1, c2, ..., cN of the speech interval corresponding to each speaker included in the symbol sequence c1, c2, ..., cN is input. This becomes the correct answer data for learning the neural network. The speaker information adding unit 2102 adds the speaker ID to the symbol sequence.<spk i> (i=1,2,...,N) to generate a symbol sequence c'i (i=1,2,...,N) with a speaker ID. For example, for the following speaker-indeterminate symbol sequence, c1: hello c2: goodbye, the symbol sequences c'1 and c'2 with speaker IDs are as follows: c'1: <spk1>hello c'2: <spk2>goodbye Speaker IDs are assigned in the order of speakers appearing in the sound mixture. In other words, speaker IDs represent or correspond to the order of utterances in the speech to be recognized. However, the labels used as speaker IDs do not necessarily have to represent the order as is. For example, during training, speaker IDs can be assigned in the order of first appearance. , , <c> , <d>..., the inference explained in the [Speech Recognition and Estimation Device] below will also be performed in the order of utterance.< / d> < / c> , , <c> , <d>In other words, speaker ID can be said to identify the target of speech recognition by the order of utterances.

[0103] (Language Information Conversion Unit) The language information conversion unit 102A converts the speaker ID-attached symbol sequence c'1, c'2, ..., c'N into a distributed representation to generate intermediate character features C1, C2, ..., CN (step S2202). As described above, the language information conversion unit 102A is configured as a neural network, and its model parameters are determined by learning.

[0104] (Third Speech Conversion Unit) The third speech conversion unit 2101 converts the mixed sound feature X M The third speech conversion unit 2101 converts the mixed speech feature X into a distributed representation to generate intermediate speech feature Hs' to be used for speech estimation (step S2201). The distributed representation is generated based on the model parameters of the neural network that constitutes the third speech conversion unit 2101. M Similar to the second speech conversion unit, the third speech conversion unit 2101 is configured with a neural network, and its model parameters are determined by learning.

[0105] (Estimation Unit) The estimation unit 103A according to the third embodiment includes a first processing unit p-1, a second processing unit p-2, ..., an Nth processing unit pN, each configured as a neural network with the same model parameters. The i-th processing unit p(i=1, 2, ..., N) performs label estimation using the intermediate speech feature Hs' and the intermediate character feature Ci(i=1, 2, ..., N) and generates an output probability distribution Y(i=1, 2, ..., N) of the symbols included in the recognition results for each speaker (step S2203). As described above, the estimation unit 103A is configured as a neural network, and its model parameters are determined by learning.

[0106] (Initial Operation) Based on the above, the initial operation of learning will be explained. For simplicity of explanation, it is assumed that the corresponding speakers appear in the mixed sound in the order of c1, c2, ..., cN. When the first frame (feature) of the first speaker is input, the speaker information adding unit 2102 generates a symbol sequence with speaker ID without symbols. <spk-1>The language information conversion unit 102A outputs the result as c'1. <spk-1>is converted to C1 and sent to the first processing unit p-1. The first processing unit p1 outputs Y1 using Hs' and C1 output by the third speech conversion unit. After that, a symbol sequence with a speaker ID including the symbol is supplied to the first processing unit. When the first frame (feature) of the second speaker is input, the speaker information addition unit 2102 outputs <spk-2>The language information conversion unit 102A outputs <spk-2>is converted to C2 and sent to the second processing unit p-2. The second processing unit p-2 outputs Y2 using Hs' output by the third speech conversion unit and C2. Thereafter, a symbol sequence with a speaker ID including the symbol is supplied to the second processing unit. Thereafter, the first frame of each speaker is processed in the same manner.

[0107] (Model Update Unit) The model update unit 20 repeatedly updates and optimizes the model parameters of the language information conversion unit 102A, the third speech conversion unit 2101, and the estimation unit 103A using the output (estimation result) of the speech recognition unit 10 and the correct answer data stored in the recording unit. The i-th loss calculation unit 402-i (i=1, 2, ..., N) calculates the i-th loss LRNN-T that evaluates the error of Yi from the output probability distribution Yi (i=1, 2, ..., N) and the correct answer data included in the symbol sequence with speaker ID. (i) (Step S2204). The loss integration unit 403 calculates the LRNN-T (i) (i=1,2,…,N) are integrated as follows: λ i is a non-negative hyperparameter for adjusting the loss of multiple speakers. The model update unit 20 updates the model parameters based on the LRNN-T obtained by integration (step S2205).

[0108] The above process is repeated until a predetermined termination condition is satisfied, and the finally obtained model parameters are stored in storage unit D2.

[0109] This concludes the description of the speech recognition model training device. In the above description, the speaker information adding unit 2102 generates a symbol sequence with a speaker ID, and the language information converting unit 102A converts the symbol sequence with a speaker ID into intermediate character features. However, the language information converting unit may generate and convert a symbol sequence with a speaker ID. In this case, the speaker information adding unit 2102 can be omitted.

[0110] 23 is a functional block diagram showing an example of the configuration of a speech recognition estimation device 23 according to the third embodiment. The speech recognition estimation device 23 receives a mixed sound feature X M is input, and symbols C1F, C2F, ..., CNF of the speech recognition result for each speaker are output. The speech recognition estimation device 23 comprises the speech recognition unit 10 and memory unit D2 of the speech recognition model training device 21. For ease of understanding, the estimation unit 103A is described as having N processing units (p-1, p-2, ..., pN), but one processing unit may perform parallel processing of N sequences of speech recognition. Figure 24 is a flowchart illustrating an example of the operation of the speech recognition estimation device 23. The following description will be made using Figures 23 and 24.

[0111] The speech recognition estimation device 23 is assumed to have learned N speakers. That is, it is assumed to estimate, store, and update a maximum of N symbol sequences c1, c2, ..., cN. The third speech conversion unit 2101 estimates, stores, and updates the acoustic features (mixed sound features) X M (step S2401). The third speech conversion unit 2101 acquires the mixed sound feature X M is converted into a distributed representation to generate intermediate speech features Hs′ to be used for speech estimation (step S2402).

[0112] The speaker information adding unit 2102 converts c'i (i=1, 2, . . . , N) into symbol-free information. <spk-i>(i=1, 2, ..., N) (step S2403) and transmits it to the language information conversion unit 102A (step S2404). The language information conversion unit processes c'i (i=1, 2, ..., N) and transmits Ci (i=1, 2, ..., N) to the i-th processing unit (i=1, 2, ..., N) (step S2405). As a result of learning, the estimation unit 103A estimates the voice of the speaker (first speaker) who first appears in the mixed sound (conversation). <spk-1>As a result, C1F is output from the first processing unit as the label estimation result for the first speaker (step S2406). The other processing units do not output estimation results. C1F is also sent to the speaker information addition unit, which adds C1F to the symbol sequence c'1 (updates c'1). Thereafter, the speaker information addition unit sends c'1 including the symbols to the language information conversion unit (step S2404).

[0113] As a result of learning, when a voice different from the first speaker appears in the mixed sound (conversation), the estimation unit 103A estimates the voice as <spk-2>As a result, C2F is output from the second processing unit as the label estimation result for the second speaker (step S2406). C2F is also sent to the speaker information adding unit, which adds C2F to the symbol sequence c'2 (updates c'2). Thereafter, the speaker information adding unit sends c'2 including the symbol to the language information conversion unit (step S2404). Similarly, when a voice different from that of a recognized speaker (up to the (i-1)th speaker) appears, the estimation unit 103A recognizes the voice as <spk-i>and outputs CiF from the i-th processing unit. CiF is also sent to the speaker information adding unit, which adds CiF to the symbol sequence c'i. Thereafter, the speaker information adding unit sends c'i including the symbols to the language information conversion unit.

[0114] When the speech recognition / estimation device 23 completes the label estimation process, it returns to step S2401 and processes the next frame.

[0115] As explained above, the disclosed technology enables speech recognition for multiple speakers without operating multiple single-speaker speech recognition engines. This concludes the explanation of the third embodiment. In the above explanation, the speaker information addition unit 2102 generates a symbol sequence with a speaker ID, and the language information conversion unit 102A converts the symbol sequence with a speaker ID into intermediate character features. However, the language information conversion unit may generate and convert the symbol sequence with a speaker ID. In this case, the speaker information addition unit 2102 can be omitted. Also, in the above explanation, the symbol CiF estimated by the ith processing unit (i=1, 2, ..., N) is sent to the speaker information addition unit. However, it is also possible to send CiF to the language information conversion unit 102A, have the language information conversion unit 102A convert it into intermediate character features and send them to the ith processing unit, and convert the existing converted CiF into intermediate character features by the ith processing unit. <spk-i>In this case, the speaker information adding unit 2102 can be omitted.

[0116] In the above explanation, all N ith processing units <spk-i>However, it is not necessary to set it to all N (the maximum number of speakers at the time of learning). For example, if you set the number of participants at the start of a meeting, you can set the number (but not more than N) <spk-i>Alternatively, at the start of the conference, participants can be asked to speak in turn, and the number of users who have instructed the speech can be set as the initial value. <spk-i>Furthermore, when the speech recognition starts, the target speaker of the speech recognition speaks, and the estimation unit <spk-1>By initially setting only this, it can also be used as a "speech recognition engine that recognizes only the voice of the target speaker (first speaker)."

[0117] (Experimental Results) The disclosed technology (third embodiment) is compared with existing technologies. The conventional technology (Technology 1) is a technology for recognizing the speech of a single speaker from a sound containing only the speech of that speaker. When a speech recognition estimation device trained using Technology 1 attempted to recognize the speech of each speaker from a mixed sound containing speech from multiple speakers, the error rate was 64.5%. Note that if the recognition result for each speaker included a recognition result for another speaker, it was deemed an error even if the recognition result was correct.

[0118] Reference 2 (Kanda et al., "Streaming Multi-Talker ASR with Token-Level Serialized Output Training", arXiv:2202.00842v5, July 14th, 2022) is a technology for recognizing the speech of all speakers from a sound that includes speech from multiple speakers. When a speech recognition estimation device trained using Reference 2 recognized the speech of all speakers from a mixed sound containing speech from multiple speakers, the error rate was 4.4%.

[0119] When a speech recognition estimation device trained using the disclosed technology (third embodiment) recognized the speech of all speakers from a mixed sound containing speech from multiple speakers, the error rate was 4.2%.

[0120] Reference 2 first outputs the recognition results (e.g., words) of all speakers from a mixed sound containing speech from multiple speakers as a single time series. Inserting a predetermined break in the time series at the point where the speaker changes facilitates the process of breaking down the results into individual speakers later. As described above, a speech recognition estimation device trained using the technology in Reference 2 is comparable to a speech recognition estimation device trained using the disclosed technology. However, the technology in Reference 2 requires an enormous amount of time to prepare training data. While training speech recognition typically requires millions of hours of training data, it is not easy to align words, assign timestamps, and create time-series training data (ground truth data) from these millions of hours of training data. In contrast, the disclosed technology can create training data simply by assigning speaker IDs to the training data. No alignment is required. The disclosed technology can be said to have superior cost-effectiveness compared to Reference 2.

[0121] <Variation 1 of the Third Embodiment (Speech Recognition Estimation Apparatus)> In the speech recognition estimation apparatus according to the third embodiment, speaker IDs are assigned in the order in which they appear in a conversation (mixed sounds). Utilizing this feature, speaker information (such as name) can be added to the recognition result. This will be explained using FIGS. 25 and 26 .

[0122] The speech recognition estimation device 23 in Fig. 25 is obtained by adding a name adding unit 2501 to the speech recognition estimation device 23 in Fig. 23. Prior to speech recognition, the name adding unit 2501 acquires the names of speakers (Name-1, Name-2, ..., Name-N) and the order of speech (step S2601). For ease of explanation, it is assumed that the order of speech is Name-1, Name-2, ..., Name-N. The name adding unit 2501 assigns speaker IDs based on the acquired order of speech. <spk-i>and the name Name-i (i = 1, 2, ..., N) are linked (step S2602). At the beginning of the conversation, speakers speak in the specified order. The utterances are converted into mixed sound features and acquired by the speech recognition estimation device 23 (step S2603). The speech recognition estimation device 23 assigns speaker IDs in the order in which they appear in the conversation (mixed sounds), thereby enabling the utterances to be linked to their speaker IDs. The name addition unit 2501 outputs the output of the ith processing unit as the speech recognition result for Name-i (step S2604).

[0123] The above is a description of Variation 1 of the third embodiment. In the above description, the names and speaking orders of speakers are acquired prior to speech recognition. However, this processing may be performed after speech recognition is completed. For example, conference participants may (individually) remember the order in which they spoke at the beginning of a conversation, and after the conference (speech recognition is completed), each participant (speaker) may input information linking the speaker's name with the speaking order i corresponding to the speech recognition result into the device. Alternatively, after speech recognition is completed, at least one of the speech recognition results of the first utterance of each recognized speaker or an extracted audio information frame corresponding to the speech recognition result may be presented to the conference participants, and they may be asked to enter a name corresponding to the presented content. Furthermore, in the above description, the names and speaking orders of N speakers are acquired. However, if the number of conference participants is n, which is less than N, it is sufficient to acquire the names and speaking orders of n speakers. Furthermore, instead of acquiring the names of all conference participants, it is also possible to acquire the name of, for example, only the third speaker.

[0124] <Modification 2 of the Third Embodiment (Speech Recognition Estimation Apparatus)> Conversations in a remote conference system may be speech-recognized and provided to conference participants. This will be described with reference to Figs. 27 and 28.

[0125] In the teleconference system, a server hosting the conference (teleconference server 2703) and the terminals of the conference participants (terminals 2704-1, 2704-2, ..., 2704-N') are connected via a communication network 2705. The number of speakers participating from each terminal is not limited to one, but the number of conference participants T does not exceed N (the maximum number of speeches that can be recognized). The teleconference server 2703 acquires speech from each terminal, mixes it, and transmits it to each terminal as conference speech. A speech recognition server 2701 including a speech recognition estimation device 23 and a speech acquisition unit 2702 is located at the remote site 27.

[0126] The voice acquisition unit 2702 acquires mixed voice (conference voice) from the remote conference server 2703 and calculates the mixed voice feature X M and transmits it to the speech recognition estimation device 23 (step S2801). M (step S2802), and the speech recognition server 2701 transmits the recognition result to all terminals (step S2803).

[0127] The above is the explanation of the second modification of the third embodiment. In the above explanation, the speech recognition results are transmitted to all terminals. Prior to the remote conference, the speech recognition server 2701 may acquire the number of the speech order (called the speech number) and the ID of the terminal to which the speech number belongs, and the conference participants may speak in the order of the speech numbers at the beginning of the conference. Then, the speech number i and the speaker ID <spk-i>The speech recognition server 2701 corresponds to <spk-i>By transmitting the recognition results for utterance number i only to the terminal to which it belongs, the speech recognition results can be transmitted only to the terminal to which the speaker belongs. In this case, assuming that a conference participant does not consent to having their speech recorded, the system may be configured to delete all speech information, recognition results, intermediate features, etc. related to the conference from the server once transmission of the speech recognition results is complete. In the above description, the speech recognition server 2701 is located at the same remote site 27 as the teleconference server 2703. However, the speech recognition server 2701 may also be located at the same site as terminal 1, for example, or terminal 1 may be configured to have the functions of the teleconference server 2701. Since the teleconference server 2703 mixes the speech from all terminals and transmits it to all terminals, the teleconference server 2701 located at the site of terminal 1 simply receives the mixed speech (conference speech) from terminal 1.

[0128] <Modification 3 of the Third Embodiment (Speech Recognition Estimation Device)> In the above modification 1, the speech recognition server 2701 acquires the speech mixed by the teleconferencing server 2703. Alternatively, the speech recognition server 2701 may acquire and process the speech before mixing that is received by the teleconferencing server 2703 from each terminal. This will be described with reference to FIG. 29 .

[0129] As in the second modification, the teleconference system is assumed to have N' terminals and a teleconference server connected via a communication network. The speech recognition server 2901 differs from the speech recognition server 2701 in that it includes N' pairs of speech acquisition units 2702-i (i=1, 2, ..., N') and speech recognition estimation devices 23-i (i=1, 2, ..., N'). However, the function of the speech acquisition unit 2702-i is the same as that of the speech acquisition unit 2702, and the function of the speech recognition estimation device 23-i is the same as that of the speech recognition estimation device 23. The speech acquisition unit 2702-i acquires mixed speech (including the case of a single speech) received from the terminal 2704-i from the teleconference server 2703, and calculates mixed sound feature X M The speech recognition / estimation device 23-i converts the mixed sound feature X M -i is used to perform speech recognition, and the speech recognition server 2901 transmits the recognition results to all terminals.

[0130] This concludes the description of the third modification of the third embodiment. In the above description, the speech recognition results are transmitted to all terminals. The speech acquisition unit 2702-i may acquire the ID of the terminal 2704-i along with the mixed speech of the terminal 2704-i. Then, the speech recognition server 2901 acquires the mixed sound feature X M Alternatively, the recognition result using terminal 2704-i may be transmitted only to terminal 2704-i. In the above description, a pair of speech acquisition unit and speech recognition estimation device is used for each speech of one terminal, but a pair of speech acquisition unit and speech recognition estimation device may be used to process speech of multiple terminals.

[0131] <Variation 4 of the Third Embodiment (Speech Recognition and Estimation Device)> In Variation 3 above, the speech recognition results were transmitted to each terminal. That is, the generation of intermediate speech features, the generation of intermediate character features, and the estimation of speech symbols were all performed by the speech recognition server 2701. The generation of intermediate speech features may be performed by the server, while the generation of intermediate character features and the estimation of speech symbols may be performed by the terminal. Note that the terminal is not limited to a device for a specific purpose such as a smartphone or speakerphone, and may be realized using a general-purpose computer. This will be described using Figures 30 and 31 .

[0132] A speech recognition server 3001 is located at the remote site 27. The speech recognition server 3001 includes a third speech conversion unit 2101 and a speech acquisition unit 2702. The terminal 2704-i (i=1, 2, ..., N') includes a language information conversion unit 102A-i and an estimation unit 103A-i. The function of the language information conversion unit 102A-i is the same as that of the language information conversion unit 102A, and uses the same model parameters. The function of the estimation unit 103A-i is the same as that of the estimation unit 103A, and uses the same model parameters. However, as described at the end of the description of the third embodiment, the symbol sequence with speaker ID is generated by the language information conversion unit 102A-i. The speech acquisition unit 2702 acquires mixed speech sounds (conference speech) from the remote conference server 2703, and acquires mixed speech feature quantities X M and transmits it to the third speech conversion unit 2101 (step S3101). M into a distributed representation to generate intermediate speech features Hs' to be used for speech estimation (step S3102). The speech recognition server 3001 transmits the intermediate speech features Hs' to each terminal 2704-i (i=1, 2, ..., N') (step S3103).

[0133] The language information conversion unit 102A-i of the terminal 2704-i generates intermediate character features Ci (i = 1, 2, ..., N) (step S3104). The estimation unit 103A-i performs speech recognition using the intermediate speech feature Hs' and the intermediate character features Ci (i = 1, 2, ..., N) (step S3105). The output of each estimation unit 103A-i is the speech recognition result for all conference participants.

[0134] This concludes the description of the fourth modification of the third embodiment. In the above description, each estimation unit 103A-i performed speech recognition for all conference participants. The speech recognition server 3001 may generate the speech order of the conference participants and notify each terminal of the user's speech order. For example, if the speech order of the conference participants using terminal 2704-i is second and fifth, the language information conversion unit 102A-i will notify the estimation unit 103A-i <spk-2> ， <spk-5>As a result, the estimation unit 103A-i can recognize only the voice of the participant using the terminal 2704-i.

[0135] <Supplementary Note> Variation 1 of the third embodiment (linking the recognition result with speaker information) may be applied to Variation 2 or 3, and the speech recognition server may be used as a minutes-creation system. This concludes the description of the variation of the third embodiment. The speech recognition estimation device 23 according to the variation of the third embodiment improves an end-to-end speech recognition system that outputs a character sequence directly from acoustic features, which previously targeted speech containing only a single speaker, so that it can recognize speech from multiple speakers from speech containing multiple speakers. Therefore, the disclosed technology brings about an improvement in speech recognition technology.

[0136] The above describes the embodiments and modifications of the present disclosure. The various processes in the above embodiments and modifications may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capabilities of the devices that execute the processes or as needed. Needless to say, other appropriate modifications are possible without departing from the spirit of the present disclosure.

[0137] The present disclosure may further include a device (terminal) for using the device of the present disclosure or the method of the present disclosure via a network (telecommunications line). The "device (terminal) for use" may be provided with functions (e.g., control function, decoding function, restoration function, input / output function, etc.) necessary to obtain the effects of implementing the device of the present disclosure or the method of the present disclosure.

[0138] [Processor, Program, Recording Medium] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), GPUs (Graphics Processing Units), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes a program stored in a memory.

[0139] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0140] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0141] The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 3220 of the computer 3200 shown in Figure 32, and operating the control unit 3210, input unit 3230, output unit 3240, display unit 3250, etc.

[0142] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium.

[0143] The program may be distributed by, for example, selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to other computers via a network, thereby distributing the program.

[0144] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with the program each time a program is transferred from a server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. Furthermore, the server computer may execute the process at the terminal using a so-called SaaS (Software as a Service) service, which allows users to use part of a server computer along with the program. In this embodiment, the program includes information used for processing by an electronic computer that is equivalent to a program (such as data that is not a direct instruction to a computer but has properties that dictate computer processing).

[0145] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

[0146] The following supplementary notes are disclosed regarding the above-described embodiment and modifications.

[0147] (Supplementary Item 1) A speech recognition estimation device including: a memory; and at least one processor connected to the memory, wherein the memory stores pre-trained parameters of a first model and a second model; and the processor converts the input speech information into intermediate speech features based on the first model, assuming that a speaker of an utterance included in input speech information is an Nth speaker in speaking order; and generates an Nth speech recognition result indicating the content of the utterance of the Nth speaker from the intermediate speech features based on the second model and Nth identification information corresponding to the speaking order of the Nth speaker.

[0148] (Supplementary Item 2) A speech recognition estimation device including: a memory; and at least one processor connected to the memory, wherein the memory stores model parameters; and the processor performs the following processes in parallel for each speaker: converting the input speech information into intermediate speech features based on a first model trained in advance, where the speaker of an utterance included in the input speech information is designated as the Nth speaker in order of utterance; and generating an Nth speech recognition result indicating the content of the utterance of the Nth speaker from the intermediate speech features based on a second model trained in advance.

[0149] (Supplementary Item 3) The speech recognition estimation device according to Supplementary Item 1, wherein the processor further acquires a name and an utterance order corresponding to the speaker, and uses the utterance order to link the Nth speech recognition result corresponding to the Nth identification information with the name and output the result.

[0150] (Supplementary Item 4) A speech recognition estimation device including: a memory; and at least one processor connected to the memory, wherein the memory stores model parameters; and the processor receives input speech information including, at the beginning, speech uttered in sequence by N dialogue participants and Nth identification information of the Nth speaker who spoke Nth at the beginning, generates an Nth speech recognition result indicating the content of the speech of the Nth speaker based on a trained model, and transmits the Nth speech recognition result to a predetermined terminal based on the Nth identification information.

[0151] (Supplementary Item 5) A speech recognition system including: a server including a first memory and a first processor connected to the first memory; and a terminal including a second memory and a second processor connected to the second memory, wherein the first memory stores first model parameters, and the second memory stores second model parameters, the first processor acquires information about the terminal corresponding to a dialogue participant and input speech information including speeches of a plurality of people, converts the input speech information into intermediate speech features using a pre-trained first model, and transmits the intermediate speech features to the terminal, and the second processor receives as input an order of utterances of the dialogue participant corresponding to the terminal and the intermediate speech features, and generates a speech recognition result for a speaker corresponding to the order of utterances from the intermediate speech features using a pre-trained second model.

[0152] (Supplementary Item 6) A speech recognition system including a server and one or more terminals, wherein the terminals perform the following processes: the terminals transmit to the server an utterance order at the beginning of a dialogue of a speaker to be speech-recognized, and obtain from the server a speech recognition result for the utterance order.

[0153] (Supplementary Item 7) A speech recognition model training device comprising: a memory; and at least one processor connected to the memory, wherein the memory stores training input speech information, correct speech recognition results, and parameters of a first model, a second model, and a third model; wherein the processor: converts the training input speech information including speech of a plurality of speakers into intermediate speech features based on the first model; designates a speaker of an utterance included in the training input speech information as the Nth speaker in speaking order; converts the Nth correct speech recognition result indicating the speech content of the Nth speaker into Nth intermediate language features based on the second model; generates an Nth speech recognition result based on the third model using as input Nth identification information corresponding to the speaking order of the Nth speaker, the intermediate speech features, and the Nth intermediate language features; and updates the parameters of the first model, the second model, and the third model using as input the Nth speech recognition result and the Nth correct speech recognition result.

[0154] (Supplementary Item 8) A computer-readable non-transitory recording medium having recorded thereon a program for causing a computer to function as the speech recognition estimation device described in any one of Supplementary Items 1 to 4, the speech recognition system described in any one of Supplementary Items 5 or 6, or the speech recognition model training device described in Supplementary Item 7. < / spk-2> < / d> < / c> < / nts> < / ts> < / nts> < / ts> < / nts> < / ts> < / nts> < / ts> < / ntsn> < / ts> < / ntsn> < / nts2> < / nts1> < / nts> < / ts> < / nts> < / ts> < / nts> < / ts>

Claims

1. A speech recognition and estimation device comprising: a speech conversion unit that converts input speech information into intermediate speech features, with a speaker of an utterance included in input speech information designated as the Nth speaker in speaking order; and an estimation unit that generates an Nth speech recognition result indicating the content of the utterance of the Nth speaker from the intermediate speech features based on Nth identification information corresponding to the speaking order of the Nth speaker, wherein the speech conversion unit and the estimation unit perform processing based on a pre-trained model.

2. A speech recognition and estimation device comprising: a speech conversion unit that converts input speech information into intermediate speech features, with a speaker of an utterance included in input speech information designated as the Nth speaker in speaking order; and an estimation unit that performs processing in parallel for each speaker to generate an Nth speech recognition result indicating the speech content of the Nth speaker from the intermediate speech features, wherein the speech conversion unit and the estimation unit perform processing based on a pre-trained model.

3. A speech recognition estimation device according to claim 1, further comprising a name addition unit that acquires a name and speaking order corresponding to the speaker, and uses the speaking order to link the Nth speech recognition result corresponding to the Nth identification information with the name and outputs the linked result.

4. A speech recognition estimation device having: an estimation unit that receives input speech information including, at the beginning, speech uttered in sequence by N dialogue participants and N-th identification information of the N-th speaker who spoke Nth at the beginning, and generates, based on a trained model, an N-th speech recognition result indicating the content of the N-th speaker's utterance; and a transmission unit that transmits, based on the N-th identification information, the N-th speech recognition result to a predetermined terminal.

5. A speech recognition system comprising a server and a terminal, wherein the server acquires information on the terminal corresponding to a dialogue participant and input speech information including speech from multiple people, converts the input speech information into intermediate speech features using a pre-trained first model, and transmits the intermediate speech features to the terminal, and the terminal receives as input the speaking order of the dialogue participant corresponding to the terminal and the intermediate speech features, and generates a speech recognition result for the speaker corresponding to the speaking order from the intermediate speech features using a pre-trained second model.

6. A speech recognition system comprising a server and a terminal, wherein the terminal transmits to the server an utterance order at the beginning of a dialogue of a speaker to be recognized by speech recognition, and obtains speech recognition results for the utterance order from the server.

7. A speech recognition estimation method, in which a speaker of an utterance included in input speech information is designated as an Nth speaker in speaking order, a speech conversion unit converts the input speech information into intermediate speech features, and an estimation unit generates an Nth speech recognition result indicating the content of the utterance of the Nth speaker from the intermediate speech features based on Nth identification information corresponding to the speaking order of the Nth speaker, and the speech conversion unit and the estimation unit perform processing based on a pre-trained model.

8. A speech recognition estimation method, in which a speaker of an utterance included in input speech information is designated as the Nth speaker in the order of utterance, a speech conversion unit converts the input speech information into intermediate speech features, and an estimation unit performs processing in parallel for each speaker to generate an Nth speech recognition result indicating the content of the utterance of the Nth speaker from the intermediate speech features, and the speech conversion unit and the estimation unit perform processing based on a pre-trained model.

9. A speech recognition estimation method, comprising: acquiring input speech information including, at the beginning, speech uttered in sequence by N dialogue participants; and Nth identification information of the Nth speaker who spoke Nth at the beginning; an estimation unit generating an Nth speech recognition result indicating the content of the speech of the Nth speaker based on a trained model; and a transmission unit transmitting the Nth speech recognition result to a predetermined terminal based on the Nth identification information.

10. A speech recognition estimation method, in which a server acquires information about a terminal corresponding to a dialogue participant and input speech information including speech from multiple people, converts the input speech information into intermediate speech features using a pre-trained first model, and transmits the intermediate speech features to the terminal, and the terminal receives as input the speaking order of the dialogue participant corresponding to the terminal and the intermediate speech features, and generates a speech recognition result for a speaker corresponding to the speaking order from the intermediate speech features using a pre-trained second model.

11. A speech recognition model training device comprising: a speech conversion unit that converts training input speech information including speech from multiple speakers into intermediate speech features; a language information conversion unit that converts an N-th correct speech recognition result indicating the content of an utterance of a speaker included in the training input speech information into N-th intermediate language features, where the speaker is the N-th speaker in speaking order; an estimation unit that receives as input N-th identification information corresponding to the speaking order of the N-th speaker, the intermediate speech features, and the N-th intermediate language features, and generates an N-th speech recognition result; and a model update unit that receives as input the N-th speech recognition result and the N-th correct speech recognition result, and performs processing based on a predetermined model, the speech conversion unit, the language information conversion unit, and the estimation unit, updating parameters of the model.

12. A speech recognition model training method, in which a speech conversion unit converts training input speech information including speech from multiple speakers into intermediate speech features, a speaker of an utterance included in the training input speech information is designated as the Nth speaker in speaking order, a language information conversion unit converts an Nth correct speech recognition result indicating the speech content of the Nth speaker into Nth intermediate language features, an estimation unit receives as input Nth identification information corresponding to the speaking order of the Nth speaker, the intermediate speech features, and the Nth intermediate language features, and generates an Nth speech recognition result, and a model update unit receives as input the Nth speech recognition result and the Nth correct speech recognition result, and updates parameters of the model in the speech conversion unit, the language information conversion unit, and the estimation unit, which perform processing based on a predetermined model.

13. A program for causing a computer to function as the speech recognition estimation device according to any one of claims 1 to 4, the speech recognition system according to any one of claims 5 or 6, or the speech recognition model training device according to claim 11.

Images