Learning device, inference device, inference method, and inference program

Abstract

An inference device (20) comprises an inference unit (23) that receives, as input, audio information which contains superimposed speeches of a plurality of speakers and infers the content and a time period of speech which are contained in the audio information, on the basis of a machine learning model which has learned model parameters so as to receive the audio information as input and output the content and a time period of speech of each of the plurality of speakers which are correct answer information.

Description

Learning device, estimation device, estimation method, and estimation program The present invention relates to a learning device, an estimation device, an estimation method, and an estimation program. As a multi-speaker speech recognition technology for recognizing the speech of multiple speakers, for example, there is a technology that writes down the contents of speech for each utterance from speech information containing the overlapping speeches of multiple speakers. As an example, there is a technology that introduces a speaker turn label, which is a label related to the speaker turn, such as a special token that indicates the speaker turn such as [sep], and outputs the utterance contents, which are correct answer information, in order of the earliest start time of the utterance (for example, see Non-Patent Document 1). As a specific example, when a speaker utters "Hello, it's a nice day," when the speaker is interrupted by the utterance of "It's been a while" by another speaker, the output information, which is the correct answer information of the above technology, is "Hello [sep] It's been a while [sep] It's a nice day." Furthermore, as a single speaker speech recognition technology for recognizing the speech of a single speaker without taking into consideration the overlapping of speech, for example, there is a technology that estimates the speech time and speech content of a single speaker using the same machine learning model (for example, see Non-Patent Document 2). As an example, in a single speaker speech recognition technology, when "Hello" is spoken between 0 and 2 seconds and discretized every 0.1 seconds, the correct answer information is [0] Hello.

[0020] Based on a machine learning model that outputs the above, the speech content and speech duration are estimated using audio information as input. Naoyuki Kanda, Yashesh Gaur, Xiaofei Wang, Zhong Meng, Takuya Yoshioka, “Serialized Output Training for End-to-End Overlapped Speech Recognition” in Proc. Interspeech, 2020, pp. 2797-2801Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever, “Robust Speech Recognition via Large-Scale Weak Supervision” arXiv preprint arXiv:2212.04356, 2022 However, the conventional technology may not be able to estimate detailed information in speech. For example, the conventional technology may be able to output speech content for voice information, but may not be able to estimate the speech time of each of multiple speakers. For example, from an output result such as "Hello [sep] It's been a while [sep] It's nice weather, isn't it?" by the conventional technology, it is unclear whether the speech was spoken in order or whether the voices were overlapped. In order to solve the above-mentioned problems, the present invention has an estimation unit that receives audio information including superimposed voices of multiple speakers as input and estimates the speech content and speech time contained in the audio information based on a machine learning model in which model parameters have been trained so as to output correct answer information, which is the speech content and speech time of each of the multiple speakers. According to the present invention, detailed information in an utterance can be estimated. FIG. 1 is a diagram for explaining an example of an outline of an estimation system. FIG. 2 is a diagram showing an example of the configuration of a learning device. FIG. 3 is a diagram showing an example of the configuration of a learning unit. FIG. 4 is a diagram showing an example of the configuration of an estimation device. FIG. 5 is a diagram showing an example of the configuration of an estimation unit. FIG. 6 is a diagram showing an example of the configuration of a decoder unit. FIG. 7 is a flowchart showing an example of the flow of processing executed by the learning device. FIG. 8 is a flowchart showing an example of the flow of processing executed by the estimation device. FIG. 9 is a diagram showing an example of the configuration of a computer that executes an estimation program. Hereinafter, an embodiment of the present invention will be described with reference to the drawings, but the present invention is not limited to the following embodiment. In addition, in the description of the drawings, the same parts are given the same reference numerals, and duplicated explanations are omitted, and explanations of members having similar functions and similar processes are also omitted. (Example of an overview) An example of the outline of the estimation system 1 will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining an example of the outline of the estimation system. In the example shown in Fig. 1, the estimation system 1 first learns model parameters of a machine learning model using a learning device 10. Here, the machine learning model refers to a model in which model parameters are learned so that, for example, speech information including superimposed speeches of multiple speakers is input, and the speech contents and speech times of each of the multiple speakers, which are correct answer information, are output. For example, the estimation system 1 uses a learning device 10 to create learning data D that uses as input audio information in which a single-speaker voice, which is the voice of each of a plurality of speakers, is superimposed based on a predetermined setting, and outputs the speech content, which is correct answer information, as well as the speech start time and the speech end time determined based on the predetermined setting, and learns a model parameter θ of a machine learning model using the learning data D. Next, the estimation system 1 estimates the speech content and the speech time using the speech information as an input based on a machine learning model, for example, by the estimation device 20. As an example, the estimation system 1 estimates the speech content and the speech time Z using the observed signal x, which is an observed signal including speech information, as an input, based on a machine learning model trained by the estimation device 20 to output, together with a label related to the speech content recognized by voice recognition, which is correct answer information, and also a label related to a time token, which is correct answer information represented for each discrete width discretized by a predetermined discrete width. Here, although speech recognition generally refers to the conversion of speech content into text, in this embodiment, for convenience, the conversion of speech content into text and the estimation of speech time may be referred to as "speech recognition". Furthermore, a label refers to, for example, information associated with the speech content and speech time of each of multiple speakers. As a specific example, a label refers to a token or tag associated with the speech content and speech time. The discrete width is not particularly limited, and is determined in advance by the learning device 10 of the estimation system 1 based on, for example, the number of time tokens and the speech time of utterances uttered within a predetermined period. By shortening the discrete width, the estimation system 1 can estimate the speech time in fine units, thereby improving the estimation accuracy of the speech time. Furthermore, in the case where the number of time tokens is 100, for example, if the discrete width is 0.5 seconds, the estimation system 1 can accommodate speech times up to 50 seconds with 100 time tokens, and if the discrete width is 0.1 seconds, the estimation system 1 can accommodate speech times up to 10 seconds with 100 time tokens. In this way, for the same number of time tokens, the longer the discrete width, the longer the available speech time. Therefore, for example, for the same number of time tokens, the estimation system 1 can estimate speech with a longer speech time by increasing the discrete width. In this way, the estimation system 1 according to the present embodiment can estimate the speech duration while maintaining the model structure of a conventional machine learning model for speech recognition by modeling a machine learning model so that the mechanism for estimating the speech duration of each utterance in speech information by multiple speakers is performed together with a mechanism for estimating the speech content of each utterance and performing speech recognition. As a result, the estimation system 1 according to the present embodiment can simultaneously estimate the speech time and the speech content of each utterance for the speech information using a single machine learning model without significantly changing the structure of the conventional machine learning model. As a result, the estimation system 1 according to the present embodiment provides a specific improvement over the conventional technology related to time-label-free multi-speaker speech recognition technology such as Non-Patent Document 1, and can estimate detailed information in the utterance, such as detailed situations in the utterance, thereby indicating an improvement in the technical field related to the speech recognition technology. In addition, the estimation system 1 can use the detailed information in the estimated utterance to visualize, for example, the excitement of the meeting or the frequency of interruptions in the conversation, or to create minutes in which the utterance time and the utterance content are linked. (Example of Estimation System Configuration) Next, an example of the configuration of an estimation system 1 will be described with reference to Figs. 1 to 6. Fig. 2 is a diagram showing an example of the configuration of a learning device. Fig. 3 is a diagram showing an example of the configuration of a learning unit. Fig. 4 is a diagram showing an example of the configuration of an estimation device. Fig. 5 is a diagram showing an example of the configuration of an estimation unit. Fig. 6 is a diagram showing an example of the configuration of a decoder unit. In the example shown in Fig. 1, the estimation system 1 has a learning device 10 and an estimation device 20. (Example of the configuration of a learning device) In the example shown in Fig. 2, the learning device 10 learns model parameters of a machine learning model. The learning device 10 is realized by, for example, a voice recognition device for recognizing voice information uttered by multiple speakers. In the example shown in Fig. 2, the learning device 10 has an acquisition unit 11, a control unit 12, an output unit 14, and a storage unit 15. (Acquisition Department) The acquisition unit 11 acquires learning data. As an example, the acquisition unit 11 acquires learning data D that receives speech information including the superimposed voices of multiple speakers as input and outputs labels related to speech content and speech time, which are correct answer information. For example, the acquisition unit 11 may acquire learning data created by the creation unit 131 described below. As an example, the acquisition unit 11 acquires learning data D that receives speech information including a single speaker voice superimposed thereon, which is generated by the creation unit 131 described below, as input, and outputs labels related to speech content and speech time, which are correct answer information. As a result, even when speech information including the pre-superimposed voices of multiple speakers cannot be obtained, the acquisition unit 11 can acquire learning data that receives speech information as input and outputs labels related to speech content and speech time, which are correct answer information. The acquisition unit 11 may be any device that can acquire the above-mentioned speech information and speech content and speech time. For example, when acquiring speech information including the pre-superimposed voices of multiple speakers, the acquisition unit 11 is realized by a microphone or the like. (Control Unit) The control unit 12 controls the entire learning device 10. For example, the control unit 12 is configured with one or more processors having programs that define the flow of various processes and internal memory that stores control information, and the processor executes each process using the programs and internal memory. The control unit 12 is realized by, for example, electronic circuits such as a central processing unit (CPU), a micro processing unit (MPU), or a graphics processing unit (GPU), or integrated circuits such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). In the example shown in FIG. 2, the control unit 12 has a learning unit 13. (Learning Department) The learning unit 13 learns model parameters of the machine learning model so that speech information including the superimposed voices of the multiple speakers is input and the speech content and speech time of each of the multiple speakers, which is correct answer information, is output. The learning unit 13 may learn the model parameters using, for example, learning data. As an example, the learning unit 13 learns model parameters θ of the machine learning model using learning data D that uses speech information as input and labels related to the speech content and speech time, which are correct answer information, as output. As a result, the learning unit 13 can train the machine learning model so that the speech time of each of the multiple speakers, which was not originally an estimation target in the conventional technology, can be estimated simultaneously with the speech content using the same machine learning model. Therefore, the learning unit 13 can train the machine learning model so that detailed information in the speech can be estimated. In the example shown in FIG. 3, the learning unit 13 has a creation unit 131 and an optimization unit 132. (Creation Department) The creation unit 131 receives as input audio information in which the voices of multiple speakers are superimposed based on predetermined settings, and creates learning data that outputs the speech content, which is correct answer information, as well as the speech start time and speech end time determined based on the predetermined settings. For example, the creation unit 131 creates speech information X^1, ..., X^|D| in which the speeches of multiple speakers are pseudo-mixed by superimposing single speaker speeches X1, ..., X|D|, which are the speeches of each of the multiple speakers, based on a predetermined setting, as input information when performing supervised machine learning of the learning data. In addition, the creation unit 131 further creates labels related to the speech content and the speech time as correct answer information (output information) when performing supervised machine learning of the learning data. As an example, the creation unit 131 determines the speech start time and the speech end time of the single speaker speeches X1, ..., X|D| used to create the speech information based on a predetermined setting. Next, the creation unit 131 creates labels Y^1, ..., Y^|D| for the single speaker utterance content Y1, ..., Y|D| and the single speaker voice X1, ..., X|D| as correct answer information from, for example, the single speaker utterance content Y1, ..., Y|D|, which is the utterance content of each of the multiple speakers, and the utterance start time and the utterance end time of the single speaker voice X1, ..., X|D| used to create the voice information, which are determined based on a predetermined setting. As a result, the creation unit 131 can create labels for learning model parameters as correct answer information so as to become a machine learning model that can estimate which utterance segment contains the target utterance voice during voice recognition by multiple speakers and shows higher voice recognition performance than the conventional techniques such as Non-Patent Document 1 and Non-Patent Document 2. Note that the above-mentioned predetermined setting is not particularly limited and may be any setting. As an example, the above-mentioned predetermined setting may be a setting in which a single speaker voice is superimposed at a predetermined time interval, or a setting in which a speech start time and a speech end time are determined so as to achieve a predetermined speech time. As another example, the above-mentioned predetermined setting may be a setting in which a single speaker voice is superimposed at a random time interval, or a setting in which a speech start time and a speech end time are determined randomly. Hereinafter, an example will be described in which the speech content of the first speaker is A, the speech time is 5 seconds (the time interval or length from 0 seconds, which is the speech start time, to 5 seconds, which is the speech end time, when the speech start time of the first speaker is set to 0 seconds), the speech content of the second speaker is B, the speech time is 4 seconds (the time interval or length from 3 seconds, which is the speech start time, to 7 seconds, which is the speech end time, when the speech start time of the first speaker is set to 0 seconds), and the speech time is discretized every 0.1 seconds. In this case, the creation unit 131 creates labels expressed as time tokens related to time, such as [0], which is the speech start time label related to the speech start time of A,

[0050] , which is the speech end time label related to the speech end time of A,

[0030] , which is the speech start time label related to the speech start time of B, and

[0070] , which is the speech end time label related to the speech end time of B. In this case, the order of the utterance start time, the utterance end time, and the utterance content as the label is not particularly limited and may be arbitrary. For example, the creation unit 131 creates the label in any one of the following orders. - Utterance start time, utterance end time, utterance content - Content of speech, speech start time, speech end time - Content of speech, end time of speech, start time of speech - End time of speech, content of speech, start time of speech - End time of speech, start time of speech, content of speech As an example, the creation unit 131 generates labels such that the order of the labels is the order of the earliest speech start time among the speech contents of each of the multiple speakers included in the speech information. Also, as an example, the creation unit 131 generates labels such that the order of the labels is the speech start time, the speech end time, and the speech contents. To give a specific example, the creation unit 131 creates a label [0]

[0050] A [sep]

[0030]

[0070] B expressed as tokens including [0], which is the speech start time label of A,

[0050] , which is the speech end time label of A, [sep], which is a speaker turn label related to the speaker turn,

[0030] , which is the speech start time label of B, and

[0070] , which is the speech end time label of B. Here, the conventional technology related to the time-labeled single speaker speech recognition technology such as Non-Patent Document 2 creates labels such that the order of the labels is the speech start time, the speech contents, and the speech end time. In response to this, the creation unit 131 can create the next label by referring to the information of the previous label, for example, by creating labels such that the order of the labels is in ascending order of utterance start time, or by creating labels such that the order of the labels is utterance start time, utterance end time, and utterance content. As a result, the creation unit 131 can create labels capable of training a machine learning model to provide a machine learning model that exhibits higher speech recognition performance, as will be apparent from the experimental results described below. For example, the creation unit 131 may create an additional label related to the end of an utterance so that the label is the last in the order of the labels. An example will be described below in which the utterance content of the first speaker is A, the utterance time is 5 seconds (the time interval or length from 0 seconds, which is the utterance start time, to 5 seconds, which is the utterance end time, when the utterance start time of the first speaker is set to 0 seconds), the utterance content of the second speaker is B, the utterance time is 4 seconds (the time interval or length from 3 seconds, which is the utterance start time, to 7 seconds, which is the utterance end time, when the utterance start time of the first speaker is set to 0 seconds), the utterance time is discretized every 0.1 seconds, and the continuous utterances uttered within a predetermined period end after the utterance content B of the second speaker. In this case, the creation unit 131 creates labels in the following order: [0], which is the speech start time label of A;

[0050] , which is the speech end time label of A; A, which is the speech content label of the first speaker; [sep], which is the speaker change label;

[0030] , which is the speech start time label of B;

[0070] , which is the speech end time label of B; B, which is the speech content of the second speaker; and [eos], which is the speech end label. That is, the creation unit 131 creates labels such as, for example, [0]

[0050] A [sep]

[0030]

[0070] B [eos]. As a result, the creation unit 131 can create labels that can train a machine learning model so that the machine learning model can further estimate the end of the utterance, and can create labels that can train a machine learning model so that the machine learning model exhibits even higher speech recognition performance. (Optimization section) The optimization unit 132 learns model parameters of the machine learning model based on the learning data created by the creation unit 131. As an example, the optimization unit 132 optimizes the model parameters θ of the machine learning model using learning data that receives speech information X^1, ..., X^|D| as input and outputs labels related to speech content and speech time, which are correct answer information. This allows the optimization unit 132 to create learning data that can train the machine learning model to become a machine learning model that exhibits high speech recognition performance. The method of learning the model parameters of the machine learning model by the optimization unit 132 is similar to that of the conventional technology. (Output section) Returning to the description of Fig. 2, the output unit 14 outputs a learning result of the machine learning model in which the model parameters have been learned by the learning unit 13. For example, the output unit 14 transmits the machine learning model in which the model parameters have been learned by the learning unit 13 to a storage device such as an external medium. When the learning device 10 and the estimation device 20 are not integrated, the output unit 14 is realized by a communication unit such as a NIC (Network Interface Card). (Storage part) The storage unit 15 stores various information. For example, the storage unit 15 stores various information such as single speaker voice, voice information, labels, model parameters, a machine learning model, an OS (Operating System) and various programs executed by the learning device 10, etc. The storage unit 15 is realized by, for example, a storage device such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, etc., or a semiconductor memory in which information can be rewritten such as a random access memory (RAM), a flash memory, or a non-volatile static random access memory (NVSRAM), etc. (Example of the configuration of an estimation device) In the example shown in FIG. 4, the estimation device 20 estimates the speech content and speech time using voice information as input based on a machine learning model in which model parameters have been trained to receive voice information as input and output the speech content and speech time of each of multiple speakers, which is correct answer information. For example, the estimation device 20 estimates the speech content and speech time using voice information as input based on a trained machine learning model that has been trained in advance by supervised learning in which voice information is input and the speech content and speech time of each of multiple speakers, which is correct answer information, are output. The estimation device 20 is realized, for example, by a voice recognition device or the like for recognizing voice information uttered by multiple speakers. In the example shown in FIG. 4, the estimation device 20 has an acquisition unit 21, a control unit 22, an output unit 24, and a storage unit 25. (Acquisition Department) The acquisition unit 21 acquires voice information including the superimposed voices of multiple speakers. As an example, the acquisition unit 21 acquires voice information including the superimposed voices of multiple speakers by interrupting the speech of one speaker in the middle of the speech of another speaker. In addition to the voice information, the acquisition unit 21 may further acquire, for example, a machine learning model. As an example, the acquisition unit 21 reads the machine learning model transmitted from the output unit 14 of the learning device 10 to a storage device such as an external medium and stored in the storage device. The acquisition unit 21 may be any device capable of acquiring the above-mentioned voice information, and may be realized by a microphone or the like when acquiring voice information including the voices of multiple speakers superimposed in advance. In addition, when the learning device 10 and the estimation device 20 are not integrated, the acquisition unit 21 may further function as, for example, a reading unit that reads the machine learning model from an external storage device. In the above example, a case where the machine learning model is acquired by the acquisition unit 21 is described. However, if the machine learning model is not acquired by the acquisition unit 21, for example, the estimation device 20 may hold (store) the machine learning model in advance in a storage unit 25 such as a memory. (Control Unit) The control unit 22 controls the entire estimation device 20. For example, the control unit 22 is configured with one or more processors having an internal memory in which programs defining the flow of various processes and control information are stored, and the processor executes each process using the programs and the internal memory. The control unit 22 is realized by, for example, electronic circuits such as a CPU, an MPU, and a GPU, or integrated circuits such as an ASIC and an FPGA. In the example shown in FIG. 4, the control unit 22 has an estimation unit 23. (Estimation Department) The estimation unit 23 estimates the speech content and speech time included in the speech information based on a machine learning model in which model parameters are trained so that speech information including the superimposed voices of multiple speakers is input and the speech content and speech time of each of the multiple speakers, which are correct answer information, are output. As an example, when the machine learning model is a neural network model, the estimation unit 23 estimates the speech content and speech time based on a machine learning model in which labels such as tokens related to the speech content and speech time in which the speech time is discretized are output from one output layer (the final layer of the neural network model) based on the speech information as input. In this way, the estimation unit 23 can simultaneously estimate the speech content and speech time using the same machine learning model, and estimate the speech time of each of the multiple speakers, which was not originally an estimation target in the conventional technology. Therefore, the estimation unit 23 can estimate detailed information in the speech. In addition, since the estimation unit 23 simultaneously converts the speech content into text and estimates the speech time, it can demonstrate higher speech recognition performance than conventional techniques, as shown in the experimental results of Table 1 in the following reference document. Reference: Joint Autoregressive Modeling of End-to-End Multi-Talker Overlapped Speech Recognition and Utterance-level Timestamp Prediction, Anonymous submission to INTERSPEECH 2023 As an example, in conventional single-talker ASR

[0020] , which is a conventional method corresponding to the conventional technology related to time-labeled single-speaker speech recognition technology such as Non-Patent Document 2, as shown in the row of Number of speakers in test dataset 1, the CER (Character Error Rate) related to the percentage of errors in the speech content estimated from the voice is 8.31%, and the DER (Diarization Error Rate) related to the percentage of errors in the speech time estimated from the voice is 0.23%. In contrast, in the proposed method w / o transcription proposed method, which is a proposed method corresponding to the estimation unit 23 of the estimation device 20 according to this embodiment, as shown in the row of Number of speakers in test dataset 1, the CER is 7.33% and the DER is 0.07%, which are lower error rates than the conventional method. As is clear from this, the estimation unit 23 can improve both the function of estimating the speech content from the voice and the function of estimating the speech time compared to the conventional technology related to time-labeled single-speaker speech recognition technology such as Non-Patent Document 2. As another example, in the conventional multi-talker ASR

[0011] , which is a conventional method corresponding to the conventional technology related to the time-label-free multi-speaker speech recognition technology such as Non-Patent Document 1, when the number of speakers is 1, 2, and 3, that is, when the Number of speakers in test dataset is 1, 2, and 3, the CER is 7.67%, 8.92%, and 12.95%, respectively. In contrast, in the proposed method w / o transcription proposed method, which is a proposed method corresponding to the estimation unit 23 of the estimation device 20 according to the present embodiment, when the Number of speakers in test dataset is 1, 2, and 3, the CER is 7.33%, 8.56%, and 12.28%, respectively, which is a lower error rate than the conventional method. As is clear from this, the estimation unit 23 can improve the speech recognition function of estimating the speech content from the voice compared to the conventional technology related to the time-label-free multi-speaker speech recognition technology such as Non-Patent Document 1. For example, the estimation unit 23 may estimate the speaker change in addition to the speech content and speech time included in the speech information as input based on a machine learning model. Hereinafter, an example will be described in which the speech content of the first speaker is A, the speech time is 5 seconds (the interval or length from the speech start time of 0 seconds to the speech end time of 5 seconds when the speech start time of the first speaker is 0 seconds), the speech content of the second speaker is B, the speech time is 4 seconds (the interval or length from the speech start time of 3 seconds to the speech end time of 7 seconds when the speech start time of the first speaker is 0 seconds), and the speech time is discretized every 0.1 seconds. In this case, the estimation unit 23 sequentially estimates the speech content and speech time in the order of labels trained by the machine learning model, for example, (50) A [sep] (40) B, which is the speech time (speech interval) label of A related to the time interval (length) of A's speech, A related to the speech content of the first speaker, [sep] which is the speaker turn label, (40) B related to the time interval of B's ​​speech, and B related to the speech content label of the second speaker. This allows the estimation unit 23 to further estimate speaker turns, and therefore estimate more detailed information in the speech. Note that, as in the above example, for the label related to the time interval of the speech, () is used for convenience to distinguish it from [ ]. For example, the estimation unit 23 may estimate the speech time including the speech start time and speech end time and the speech content using the voice information as input based on a machine learning model in which model parameters are trained so as to output the speech time including the speech start time when the speech starts and the speech end time when the speech ends, which are correct answer information, and the speech content. That is, the speech time may include the speech start time and the speech end time. Hereinafter, an example will be described in which the speech content of the first speaker is A, the speech time is 5 seconds (the time interval or length from 0 seconds, which is the speech start time, to 5 seconds, which is the speech end time, when the speech start time of the first speaker is 0 seconds), the speech content of the second speaker is B, the speech time is 4 seconds (the time interval or length from 3 seconds, which is the speech start time, to 7 seconds, which is the speech end time, when the speech start time of the first speaker is 0 seconds), and the speech time is discretized every 0.1 seconds. In this case, the estimation unit 23 sequentially estimates the speech content and speech time in the order of, for example, [0], which is the speech start time of A,

[0050] , which is the speech end time label of A, A, which is the speech content label of the first speaker,

[0030] , which is the speech start time label of B,

[0070] , which is the speech end time label of B, and B, which is the speech content label of the second speaker, that is, in the order of the labels trained by the machine learning model, that is, [0]

[0050] A

[0030]

[0070] B. As a result, the estimation unit 23 can estimate which speech segment contains the speech of the target speech when recognizing speech by multiple speakers, and therefore the accuracy of the speech recognition itself can be improved so that the speech recognition performance is higher than that of conventional techniques such as Non-Patent Document 1 and Non-Patent Document 2. In the above example, the estimation unit 23 estimates the speech start time, speech end time, etc., using voice information as input, based on a machine learning model in which model parameters have been trained to output labels including the speech start time and speech end time, which are correct answer information. However, as a modified example of the above example in which a label including the speech end time is output as correct answer information, the estimation unit 23 may estimate the speech start time, speech end time, etc., using voice information as input, based on a machine learning model in which model parameters have been trained to output labels including a speech interval, which is related to the time interval of the speech from the speech start time, instead of a label related to the speech end time. Hereinafter, an example will be described in which the speech content of the first speaker is A, the speech time is 5 seconds (the interval or length from the speech start time of 0 seconds to the speech end time of 5 seconds when the speech start time of the first speaker is 0 seconds), the speech content of the second speaker is B, the speech time is 4 seconds (the interval or length from the speech start time of 3 seconds to the speech end time of 7 seconds when the speech start time of the first speaker is 0 seconds), and the speech time is discretized every 0.1 seconds. In this case, the estimation unit 23 estimates the speech start time of A to be 0 seconds indicated by the label [0] based on, for example, the label [0] (50) A

[0030] (40) B output from the machine learning model. The estimation unit 23 estimates the speech end time of A to be 5 seconds, which is the speech start time of 0 seconds plus the speech interval of (50) of 5 seconds. The estimation unit 23 estimates that the speech start time of B is 3 seconds, which is indicated by the label

[0030] . The estimation unit 23 estimates that the speech end time of B is 7 seconds, which is the sum of the speech start time of 3 seconds and the speech interval of 4 seconds, which is indicated by the label (40). The estimation unit 23 may estimate the speech start time, speech end time, and speech content in that order. As an example, when the speech content of a first speaker is A, the speech duration is 5 seconds (the time interval or length from 0 seconds, which is the speech start time, to 5 seconds, which is the speech end time, when the speech start time of the first speaker is set to 0 seconds), the speech content of a second speaker is B, the speech duration is 4 seconds (the time interval or length from 3 seconds, which is the speech start time, to 7 seconds, which is the speech end time, when the speech start time of the first speaker is set to 0 seconds), and the speech time is discretized every 0.1 seconds, the estimation unit 23 may estimate the speech start time of A, The speech content and speech time are sequentially estimated in the order of the labels trained on the machine learning model, that is, [0]

[0050] A[sep]

[0030]

[0070] B, which is the time label, [0], the speech end time label of A,

[0050] A label of the speech content of the first speaker, [sep] speaker change label,

[0030] B label of the speech start time,

[0070] B label of the speech end time, and

[0050] A label of the speech end time. This allows the estimation unit 23 to estimate the next information by referring to the information of the previously output label, and therefore the accuracy of the speech recognition itself can be improved to achieve even higher speech recognition performance. For example, the estimation unit 23 may estimate the end of an utterance using voice information as an input, based on a machine learning model in which model parameters have been trained so as to input voice information and output a label related to the end of an utterance that is the last in the order of labels as correct answer information. As an example, the estimation unit 23 may estimate the end of an utterance using voice information as an input, based on a machine learning model in which model parameters have been trained so as to input voice information and output a label related to the end of an utterance that is the last in the order of labels as correct answer information. In one example, the estimation unit 23 may estimate the end of an utterance using voice information as an input, based on a machine learning model in which model parameters have been trained so as to input voice information and output a label related to the end of an utterance that is the last in the order of labels as correct answer information. In one example, the estimation unit 23 may estimate the end of an utterance using voice information as an input, based on a machine learning model in which model parameters have been trained so as to input voice information and output a label related to the end of an utterance that is the last in the order of labels as correct answer information. In this case, the speech content and speech time are sequentially estimated in the order of [0], which is the speech start time label of A,

[0050] , which is the speech end time label of A, A, which is the speech content label of the first speaker, [sep], which is the speaker change label,

[0030] , which is the speech start time label of B,

[0070] , which is the speech end time label of B, B, which is the speech content of the second speaker, and [eos], which is the speech end label, that is, in the order of the labels trained by the machine learning model, that is, [0]

[0050] A [sep]

[0030]

[0070] B [eos]. As a result, the estimation unit 23 can further estimate the end of the speech, and therefore the accuracy of the speech recognition itself can be improved so that the speech recognition performance is particularly high. In the example shown in FIG. 5, the estimation unit 23 has an encoder unit 231 and a decoder unit 232. (Encoder section) The encoder unit 231 outputs a feature quantity for estimation based on a machine learning model with voice information as input. For example, the encoder unit 231 inputs an observed signal x including voice information to a machine learning model having a model parameter θ, and outputs a feature quantity h for estimation. The encoder unit 231 may be any one that has the above-mentioned functions and can use a machine learning model, and may be realized, for example, by a Transformer or the like, as in the conventional technology. (Decoder section) The decoder unit 232 estimates the speech content and speech time based on the feature amount output from the encoder unit 231, the machine learning model, and the speech content and speech time up to one step before. For example, the decoder unit 232 inputs the feature amount h output from the encoder unit 231, and the speech content and speech time Z1, ..., Z|t-1| up to the t-1th step before the tth step to a machine learning model with a model parameter of θ, and estimates the output information output from the output layer of the machine learning model as the speech content and speech time Z|t|. In the example shown in FIG. 6, the decoder unit 232 has a conversion unit 233 and an utterance content / utterance time output unit 234. (Conversion section) The conversion unit 233 converts the feature based on the feature output from the encoder unit 231, the machine learning model, and the speech content and speech time up to one step before. For example, the conversion unit 233 refers to the feature h output from the encoder unit 231, the speech content and speech time Z1, ..., Z|t-1| of its own estimation result up to one step before, and the model parameter θ of the machine learning model, and outputs a converted feature q obtained by converting the feature. The conversion unit 233 may be any one having the above-mentioned functions, and may be realized, for example, by a unit similar to the conventional technology. (Speech content and speech time output section) The speech content / speech time output unit 234 outputs the speech content and the speech time based on the feature converted by the conversion unit 233 and the machine learning model. For example, the speech content / speech time output unit 234 refers to a model parameter θ of the machine learning model, and outputs the speech content and the speech time Z|t| according to the converted feature q output from the conversion unit 233. The output layer of the speech content and speech time output unit 234 has an output unit that outputs the number of character tokens + the number of time tokens + the number of special symbol tokens. Here, the special symbol token refers to, for example, a speaker turn label, an end of speech label, etc. The number of tokens mentioned above may be any number, but for example, assuming that the maximum speech length is 60 seconds and the speech time is discretized every 0.1 seconds, the number of time tokens will be 600. In the conventional technology related to the multi-speaker speech recognition technology without time labels such as Non-Patent Document 1, this output layer has an output unit that outputs the number of character tokens + the number of special symbol tokens, etc., and in the conventional technology related to the single-speaker speech recognition technology with time labels such as Non-Patent Document 2, this output layer has an output unit that outputs the number of character tokens + the number of time tokens + the end of speech label, etc. (Output section) Returning to the description of Fig. 4, the output unit 24 outputs the estimation result estimated by the estimation unit 23. For example, the output unit 24 displays the speech content and speech time Z|t| output from the speech content / speech time output unit 234. The output unit 24 is realized by, for example, a display or the like. (Storage part) The storage unit 25 stores various information. For example, the storage unit 25 stores various information such as voice information, labels, model parameters, a machine learning model, an OS and various programs executed by the estimation device 20, etc. The storage unit 25 is realized by, for example, a storage device such as an HDD, SSD, or optical disk, or a semiconductor memory such as a RAM, flash memory, or NVSRAM that is capable of rewriting information. (An example of a processing flow) An example of the flow of processing executed by the learning device 10 and the estimation device 20 of the estimation system 1 will be described below with reference to Fig. 7 and Fig. 8. Fig. 7 is a flowchart showing an example of the flow of processing executed by the learning device. Fig. 8 is a flowchart showing an example of the flow of processing executed by the estimation device. First, an example of the flow of processing executed by the learning device 10 will be described with reference to Fig. 7. In step S11, the creation unit 131 creates learning data in which speech information in which the speeches of each of the multiple speakers are superimposed based on a predetermined setting is input, and the learning data is output with the speech content as correct answer information and the speech start time and speech end time determined based on the predetermined setting. For example, the creation unit 131 creates speech information X^1, ..., X^|D| in which the speeches of the multiple speakers are pseudo-mixed by supervising the single speaker speech X1, ..., X|D|, which is the speech of each of the multiple speakers, as input information when performing supervised machine learning of the learning data. In addition, the creation unit 131 further creates labels related to the speech content and the speech time as correct answer information (output information) when performing supervised machine learning of the learning data. As an example, the creation unit 131 determines the speech start time and speech end time of the single speaker speech X1, ..., X|D| used to create the speech information based on a predetermined setting. Next, the creation unit 131 creates labels Y^1, ..., Y^|D| for single speaker speech content Y1, ..., Y|D| and single speaker voices X1, ..., X|D| as correct answer information, for example, from single speaker utterance content Y1, ..., Y|D|, which is the speech content of each of the multiple speakers, and the speech start time and speech end time of the single speaker voices X1, ..., X|D| used to create the voice information, which are determined based on a predetermined setting. In step S12, the acquiring unit 11 acquires learning data. For example, the acquiring unit 11 acquires learning data D created by the creating unit 131, in which voice information with a single speaker voice superimposed is input, and labels related to the speech content and speech time, which are correct answer information, are output. In step S13, the learning unit 13 learns model parameters of the machine learning model so as to receive voice information as input and output the speech content and speech time of each of the multiple speakers, which is correct answer information. For example, the learning unit 13 receives voice information as input and learns model parameters θ of the machine learning model using learning data D, which outputs labels related to the speech content and speech time, which is correct answer information. Next, an example of a flow of processes executed by the estimation device 20 will be described with reference to Fig. 8. In step S21, the acquisition unit 21 acquires speech information including superimposed speech of multiple speakers. For example, the acquisition unit 21 acquires speech information including superimposed speech of multiple speakers that is generated by interrupting speech of one speaker with speech of another speaker. In step S22, the estimation unit 23 estimates the speech content and speech time included in the speech information based on a machine learning model that has learned model parameters so that speech information including the superimposed voices of multiple speakers is input and the speech content and speech time of each of the multiple speakers, which are correct answer information, are output. Hereinafter, an example will be described in which the speech content of the first speaker is A, the speech time is 5 seconds (the time interval or length from 0 seconds, which is the speech start time of the first speaker, to 5 seconds, which is the speech end time, when the speech start time of the first speaker is 0 seconds), the speech content of the second speaker is B, the speech time is 4 seconds (the time interval or length from 3 seconds, which is the speech start time of the first speaker, to 7 seconds, which is the speech end time, when the speech start time of the first speaker is 0 seconds), and the speech time is discretized every 0.1 seconds. In this case, the estimation unit 23 estimates the speech content and speech duration in the order of labels trained on the machine learning model, for example, [0] which is the speech start time label of A,

[0050] which is the speech end time label of A, A which is the speech content label of the first speaker, [sep] which is the speaker change label,

[0030] which is the speech start time label of B,

[0070] which is the speech end time label of B, and B which is the speech content of the second speaker, i.e., [0]

[0050] A [sep]

[0030]

[0070] B. (Modification) Each component of each part shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of the device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of each processing function performed by the device can be realized by a CPU and a program executed by the CPU, or can be realized as hardware by wired logic. For example, in the example shown in FIG. 1, the learning device 10 and the estimation device 20 are configured as independent devices, but in this embodiment, they are not limited to such a configuration, and they may be configured as an integrated estimation device. Furthermore, among the above-mentioned processes, all or part of the processes described as being performed automatically can be performed manually, and all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the process procedures, specific names, various information, parameters, etc. shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. (program) The estimation device 20 can be implemented by installing an estimation program as package software or online software on a computer. For example, the computer can function as the estimation device 20 by executing the estimation program. Similarly to the above example, the computer can function as the learning device 10 by executing a learning program. An example of the configuration of a computer that executes the estimation program will be described below with reference to FIG. 9. A description of an example of the configuration of a computer that executes the learning program will be omitted. Fig. 9 is a diagram showing an example of the configuration of a computer that executes an estimation program. In the example shown in Fig. 9, a computer 1000 has a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080. The memory 1010 includes a read only memory (ROM) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a basic input output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. For example, a removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130. The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process executed by the estimation device 20 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, the program module 1093 for executing a process similar to the functional configuration of the estimation device 20 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD. Data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, the memory 1010 or the hard disk drive 1090. The CPU 1020 reads out the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary and executes them. Note that the program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, and may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or wide area network (WAN)). The program module 1093 and the program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070. (Additional Note) The following supplementary notes are further disclosed regarding the above embodiment. (Additional note 1) Memory, at least one processor coupled to the memory; Including, The processor, Based on a machine learning model in which model parameters are trained so that speech information including superimposed speeches of a plurality of speakers is input and the speech content and speech time of each of the plurality of speakers, which are correct answer information, is output, speech information is input and speech information is used to estimate the speech content and speech time included in the speech information. An estimation device comprising: (Additional note 2) A non-transitory storage medium storing a program executable by a computer to execute an estimation process by the estimation device according to claim 1. (Additional note 3) Memory, at least one processor coupled to the memory; Including, The processor, A machine learning model is trained so that model parameters are trained to receive voice information including superimposed voices of a plurality of speakers as an input and to output the speech content and speech time of each of the plurality of speakers as correct answer information. A learning device comprising: 20 Estimation device 21 Acquisition Department 22 Control unit 23 Estimation section 24 Output section 25 Storage section

Claims

1. An estimation device having an estimation unit that receives audio information containing superimposed voices of multiple speakers as input and estimates the speech content and speech time contained in the audio information based on a machine learning model in which model parameters have been trained so as to output correct information, which is the speech content and speech time of each of the multiple speakers.

2. The estimation device according to claim 1, characterized in that the estimation unit further estimates speaker changes by adding the speech information as input to the speech content and the speech time based on the machine learning model.

3. The estimation device according to claim 1 or 2, characterized in that the speech time includes an utterance start time and an utterance end time.

4. The estimation device according to claim 3, wherein the estimation unit estimates the utterance start time, the utterance end time, and the utterance content in that order.

5. A learning device characterized by having a learning unit that learns model parameters of a machine learning model so that audio information containing superimposed voices of multiple speakers is input and the speech content and speech time of each of the multiple speakers, which are correct answer information, are output.

6. The learning device according to claim 5, further comprising a creation unit that creates learning data using the voice information obtained by superimposing the voices of each of the multiple speakers based on a predetermined setting as input and outputting the speech content, which is correct answer information, and the speech start time and speech end time determined based on the predetermined setting, and the learning unit learns the model parameters using the learning data.

7. An estimation method executed by an estimation device, comprising an estimation step of estimating the speech content and speech time contained in speech information using input speech information including superimposed voices of multiple speakers, based on a machine learning model in which model parameters have been trained so as to output correct answer information, that is, the speech content and speech time of each of the multiple speakers.

8. An estimation program that causes a computer to execute an estimation step of estimating the speech content and speech duration contained in the speech information based on a machine learning model in which model parameters are trained so that speech information containing superimposed speeches of multiple speakers is input and the speech content and speech duration of each of the multiple speakers, which are correct answer information, is output.

Images