Method for identifying language of input speech signal in multiple languages

Abstract

The language recognition device includes: a neural network for, in response to an input of a speech signal, outputting a score indicating that the speech represented by the speech signal is the speech of each of a plurality of languages; a selection device for selecting scores of a smaller number of languages ​​from the scores output by the neural network; a standardization device for standardizing the scores selected by the selection device; a determination device for determining whether the maximum value of the scores standardized by the standardization device is above a threshold value; and a language determination device for selectively performing a process of determining the language corresponding to the maximum value of the scores as the language of the speech represented by the speech signal in response to an affirmative determination by the determination device and a process of discarding the output of the score selection device.

Description

Technical Field

[0001] The present invention relates to a device for processing multilingual speech, and more particularly to a language recognition device for identifying the language of an input speech signal and a speech processing device using the language recognition device. This application claims priority based on Japanese application No. 2019-062346 filed on March 28, 2019, and all the contents recorded in the aforementioned Japanese application are cited here by reference. Background Art

[0002] Recently, devices that perform speech recognition, automatic translation, and speech synthesis on speech in multiple languages ​​to facilitate conversations between people who speak different languages ​​have become popular. It is expected that the number of languages ​​that can be used will also increase, and if such devices are used, people who speak various languages ​​can communicate with each other.

[0003] In order to perform such multi-language speech recognition with high accuracy, it is necessary to know which language the speech represented by the speech signal is in. Therefore, the language to be used is usually specified in advance to start the speech recognition.

[0004] However, when two people who use different languages ​​and do not know each other's language want to communicate using a multilingual speech processing device, there is a problem that it is difficult to specify the language to be used. If the owner of the device is basically set to the language to be used (for example, Japanese), there is no problem, but if the other party does not know Japanese at all, it is not easy to set the language. Since the display of the device is basically in Japanese, even if the other party sees it, they cannot understand the operation method. Even if the owner of the device knows how to operate the device, he does not know what setting to make because he does not know the other party's language.

[0005] As described above, when using a device that performs speech processing in multiple languages, there is a problem that it is difficult to start a conversation smoothly.

[0006] A solution to this problem is proposed in Patent Document 1 described later. In the technology described in Patent Document 1, speech recognition is performed on an input speech signal in multiple languages. As a result of the speech recognition, the reliability of the speech recognition is obtained. In the technology disclosed in Patent Document 1, the language with the highest reliability of the speech recognition is used as the language of the speech represented by the speech signal.

[0007] Prior Art Literature

[0008] Patent Literature

[0009] Patent Document 1: Japanese Patent Application Laid-Open No. 2019-23690 Summary of the invention

[0010] Problems to be solved by the invention

[0011] However, the technology described in Patent Document 1 has the following problems: since speech recognition is performed for all speech, the amount of calculation is large, and it takes time to perform them sequentially. If they are performed in parallel, since computing resources are consumed, if the requests overlap, for example, when a server processes multiple requests, the amount of processing in the server is too large, and there is a problem that speech recognition slows down.

[0012] Therefore, when performing speech recognition by a conventional multilingual speech processing device, there is a problem that the time required before starting actual speech processing becomes longer. In particular, when being asked something by a foreign tourist, it is desired to start speech processing for conversation in a short time and obtain the result as quickly as possible.

[0013] Therefore, an object of the present invention is to provide a speech recognition device and a speech processing device that can quickly obtain a speech processing result even when the language of the speech represented by a speech signal is unknown.

[0014] Means for solving problems

[0015] The language recognition device involved in the first embodiment of the present invention is a language recognition device that recognizes whether an input voice signal is a signal of a voice in any one of a first plurality of languages, and includes: a score output unit, which is used to respond to the input of the voice signal and output, for each of the first plurality of languages, a score indicating that the voice represented by the voice signal is the voice of the language; a selection unit, which is used to select, from the scores output by the score output unit, scores of a second plurality of languages ​​that are pre-specified from the first plurality of languages ​​and are less than the first plurality; a standardization unit, which is used to standardize each of the scores selected by the selection unit based on the total of the scores of the second plurality of languages; a determination unit, which determines whether the maximum value of the scores standardized by the standardization unit is above a threshold value; and a language determination unit, which, in response to whether the determination based on the determination unit is affirmative, selectively performs processing of determining the language corresponding to the maximum value of the score as the language of the voice represented by the voice signal and processing of discarding the output of the score selection unit.

[0016] Preferably, the score output unit includes: a neural network that is pre-trained to output a score of a language of speech represented by the speech signal in response to a time series of sound feature quantities calculated based on the speech signal.

[0017] More preferably, the scores output by the neural network are normalized relative to the sum of the scores.

[0018] A computer program according to a second aspect of the present invention causes a computer to function as each unit of any one of the above-mentioned devices.

[0019] The speech processing device involved in the third embodiment of the present invention includes: a language name storage unit, which stores the language name of a default language that is the object of speech processing; a speech processing unit, which is used to perform at least speech recognition for each of a plurality of languages, accept the designation of a language name, and process a provided speech signal as a speech of the designated language; a language recognition unit, which is used to identify which of a plurality of languages ​​the speech represented by the speech signal belongs to in response to the input of a speech signal; a first speech processing starting unit, which specifies the default language stored in the language name storage unit in response to the input of the speech signal and starts the processing of the speech signal by the speech processing unit; a consistency determination unit, which determines whether the language recognized by the language recognition unit is consistent with the default language; and a second speech processing starting unit, which, in response to a negative determination by the consistency determination unit, terminates the processing of the speech signal by the speech processing unit, specifies the language recognized by the language recognition unit from a given position of the speech signal, and starts the processing of the speech signal by the speech processing unit.

[0020] Preferably, the speech processing device also includes: a default language determination unit, which is used to determine, based on the judgment result performed by the consistency judgment unit and the default language stored in the language name storage unit, through a predetermined algorithm, to which of the multiple languages ​​the language name of the default language stored in the language name storage unit is set, and store it in the language name storage unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a schematic diagram showing an outline of language recognition in the first embodiment of the present invention.

[0022] Figure 2 This is a block diagram showing a schematic configuration of a speech processing device according to a first embodiment of the present invention.

[0023] Figure 3 This is a block diagram showing the structure of a core unit for speech recognition according to the first embodiment of the present invention.

[0024] Figure 4 This is a block diagram showing a schematic configuration of a multilingual speech translation device used in the multilingual speech processing device according to the first embodiment of the present invention.

[0025] Figure 5 It is shown Figure 4 A block diagram of a schematic structure of an automatic translation device is shown.

[0026] Figure 6This is a diagram showing the appearance of a computer system that realizes the multilingual speech processing device according to the first embodiment of the present invention.

[0027] Figure 7 It is shown Figure 6 A block diagram of the hardware structure of the computer shown.

[0028] Figure 8 It shows that Figure 6 The computer shown is a flowchart of a control structure of a computer program that enables the computer to function as the speech processing device according to the first embodiment.

[0029] Fig. 9 It is shown in Figure 8 A flow chart of a control structure showing a part of a program of the control structure is shown in FIG.

[0030] Fig.10 This is a diagram schematically showing line-by-line speech recognition in the speech processing device according to the first embodiment.

[0031] Fig.11 FIG. 1 is a schematic diagram showing the time course of speech processing by a conventional method.

[0032] Fig.12 This is a schematic diagram showing the time elapse of speech processing when a final speech processing result is obtained in a short time in the first embodiment of the present invention.

[0033] Fig.13 This is a schematic diagram showing the passage of time for speech processing in the case where the time required to obtain a final speech processing result is the longest in the first embodiment of the present invention.

[0034] Fig.14 This is a schematic diagram showing the overall configuration of a voice dialogue system according to a second embodiment of the present invention.

[0035] Fig.15 This is a block diagram showing a schematic hardware configuration of a smartphone used as a terminal in the second embodiment.

[0036] Fig.16 It is shown Fig.15 A block diagram showing the functional structure of a smartphone.

[0037] Fig.17 It is shown in Fig.15 The image shows the screen for setting the language on a smartphone.

[0038] Fig.18 yes Fig.15 The functional block diagram of the multi-language speech translation system on the server side is shown.

[0039] Fig.19 is a diagram showing a method for making a computer as Fig.18 A flowchart of a control structure of a computer program that functions in the multilingual speech translation system shown. DETAILED DESCRIPTION

[0040] In the following description and drawings, the same reference numerals are given to the same components, and therefore, their detailed description will not be repeated.

[0041] [First embodiment]

[0042] <Structure>

[0043] Figure 1 Schematic diagram showing the outline of language recognition in the first embodiment of the present invention. Figure 1 In the present embodiment, a neural network 50 is used to process the input speech signal, thereby calculating the probability 54 that the input speech is the input speech of the language for each of the eight predetermined languages ​​52. The probability 54 is standardized so that the total is 1. Furthermore, the probabilities of a plurality of (three in the following embodiment) selected languages ​​56 pre-specified by the user are extracted from the eight languages ​​52, and these probabilities are further standardized so that the total is 1. Among the standardized selected languages ​​56, the language 58 with the highest probability is estimated to be the language of the input speech. At this time, if the standardized probability is above a predetermined threshold, the language 58 is adopted as the estimated result, otherwise the estimated result is discarded. In the case where the estimated result is discarded, the language is estimated again using the continuation of the speech.

[0044] Figure 2 This is a block diagram showing a schematic structure of a multi-language speech processing device 100 according to a first embodiment of the present invention, which automatically recognizes the language represented by an input speech signal and performs speech processing on the speech signal based on the result. In this embodiment, the multi-language speech processing device 100 estimates which language the speech is in among eight languages ​​and performs speech recognition, performs automatic translation, synthesizes speech based on the translation result, and outputs it.

[0045] Reference Figure 2, a microphone (hereinafter referred to as "microphone") 102, a monitor 104, and a speaker 106 are connected to the multi-language speech processing device 100. The multi-language speech processing device 100 includes: a speech processing device 130, which receives a speech signal from the microphone 102, digitizes it, frames it, and transforms it into a time series of speech data vectors representing each speech; a language recognition neural network 132, which receives an input of a speech data vector sequence of a given length output by the speech processing device 130, and outputs the probability that the speech represented by the input speech signal is the aforementioned eight languages ​​for each of these languages; and a language determination unit 134, which determines the language of the input speech based on the eight probabilities output by the language recognition neural network 132.

[0046] In this embodiment, if Figure 1 As shown, the languages ​​to be determined are Japanese (abbreviated as "JA". Hereinafter, the abbreviations of the languages ​​are indicated in parentheses.), English (EN), Chinese (ZH), Korean (KO), Indonesian (ID), Vietnamese (VI), Thai (TH), and Burmese (MY).

[0047] The multi-language speech processing device 100 further includes: a basic information setting unit 120 for setting basic information including the language used in the interface with the user when using the multi-language speech processing device 100 through interactive processing; a counterpart language storage unit 122 for storing information related to the counterpart language of the speaker of the speech being processed; a language setting unit 124 for setting information related to eight languages ​​as processing targets of the speech recognition neural network 132, three languages ​​selected by the user, and a language pre-designated by the user as the language of the initial input speech through interactive processing; a selected language storage unit 128 for storing the information set in the language setting unit 124; and a default language storage unit 126 for storing the language estimated as the language of the next speech as a default value. The values ​​stored in the default language storage unit 126 and the counterpart language storage unit 122 are updated by a given algorithm as the multi-language speech processing device 100 operates and as the recognized language changes.

[0048] In addition, the language stored in the default language storage unit 126 is a language that is tentatively estimated when the language is unknown to the user, or a language that has been set by the user as the default language of the other party in the conversation. If the language of the other party in the conversation is clearly known, the language setting unit 124 can be used to set the language in the default language storage unit 126. However, in this case, the original determination of the language based on the language recognition neural network 132 becomes unnecessary. In this embodiment, the structure and description for such a case are not performed, and the case where the language of the other party in the conversation is unknown is always assumed.

[0049] The multilingual speech processing device 100 further includes: a language change determination unit 136, which determines whether the language determined by the language determination unit 134 for the input speech signal has changed from the default language stored in the default language storage unit 126, and outputs a determination signal indicating the result; and a language selection unit 138, which has two inputs respectively connected to the outputs of the default language storage unit 126 and the language determination unit 134, and is used to select and output any one of them according to the output of the language change determination unit 136.

[0050] The multi-language speech processing device 100 further includes: a FIFO buffer 140 for storing the amount of speech data vector sequence outputted by the speech processing device 130 corresponding to a certain time; a speech data selection unit 142 having two inputs connected to the output of the speech processing device 130 and the output of the buffer 140, for selecting and outputting one of them according to the output of the language change determination unit 136; and a multi-language speech translation device 144 having an input connected to the output of the speech data selection unit 142, for performing speech translation processing on the inputted speech data vector sequence assuming that the language of the speech is the speech specified by the output of the language selection unit 138. As described below, the buffer 140 is necessary in order to perform speech translation processing on the speech data from the beginning in the correct language when it is clear that the actual language is other than the default language after the speech translation processing on the speech data is started in the default language.

[0051] In this embodiment, the multilingual speech translation device 144 performs the following processing, namely, performs speech recognition processing on the input speech data vector column, automatically translates the result into the language stored in the other language storage unit 122, and then outputs the text as the result of the automatic translation and the parameters for speech synthesis.

[0052] The multilingual speech processing device 100 further includes: a speech signal synthesis device 146, which is used to synthesize a speech signal based on the parameters for speech synthesis output by the multilingual speech translation device 144 and the information stored in the other party language storage unit 122, and provide it to the speaker 106 to make it produce speech; and a display control device 148, which is used to control the monitor 104 based on the text information output by the multilingual speech translation device 144 and make it display the text.

[0053] Figure 3 1 is a block diagram showing the structure of a core unit for speech recognition according to the first embodiment of the present invention. Figure 3In this embodiment, the language recognition neural network 132 includes three language recognition models 180, 182, and 184. These are all neural networks. The language recognition model 180 is a model for taking 1.5 seconds of speech data as input to estimate the language of the speech. Similarly, the language recognition model 182 takes 2.0 seconds of speech data as input, and the language recognition model 184 takes 3.0 seconds of speech data as input.

[0054] For example, the language recognition model 180 includes: a convolution layer group 170, which includes a plurality of convolution blocks; and a fully connected layer 172, which receives the output of the convolution layer group 170 and outputs the probability 174 of eight languages. The fully connected layer 172 includes a plurality of layers, and the final layer is a SoftMax layer. Therefore, the total of the outputs corresponding to the eight languages ​​is 1, which can be processed as the probability 174. The language recognition model 182 and the language recognition model 184 have a structure that is substantially the same as the language recognition model 180, but there are differences in the structure that reflect the different sizes of the inputs (such as the number of nodes in the input layer).

[0055] The language recognition models 180, 182 and 184 are pre-trained respectively by combining a large number of speech signals of 8 languages ​​and correct data representing the language of the speech represented by the speech signals. The size of the speech signal uses different sizes according to each model. In the present embodiment, the correct data has eight elements, which are set as vectors in which the value of the element corresponding to the correct language is 1 and the value of the other elements is 0. The training itself can be performed by the usual error back propagation method for the neural network. The more training data there are, the higher the accuracy of language recognition is, so it is important to collect a sufficient amount of training data.

[0056] The language determination unit 134 includes: a model selection unit 190, which receives the input of probability 174 from the language recognition model 180, receives the input of probability 176 from the language recognition model 182, and receives the input of probability 178 from the language recognition model 184, and selects a group specified by an external signal based on the three groups of probabilities based on the three models; a language selection unit 200, which extracts the probabilities of the three selected languages ​​stored in the selected language storage unit 128 based on the eight probabilities included in the group of probabilities selected by the model selection unit 190; and a probability normalization unit 202, which is used to normalize the probabilities of the three languages ​​selected by the language selection unit 200 and output them as probabilities. The probability normalization unit 202 calculates the sum of the three probabilities, and divides the language of each probability by the sum as the probability of each language. The sum of the values ​​is 1.

[0057] The language determination unit 134 further includes: a maximum probability selection unit 204, which selects the maximum probability from the three probabilities normalized by the probability normalization unit 202 and outputs it; a threshold comparison unit 206, which determines whether the probability selected by the maximum probability selection unit 204 is above a given threshold and outputs the determination result; a language determination unit 208, which is used as follows, that is, accepts the normalized probability output by the probability normalization unit 202, the maximum probability selected by the maximum probability selection unit 204, and the comparison result from the threshold comparison unit 206, and attempts to recognize the language based on them, and if the language can be recognized, provides the result thereof to the language change determination unit 136, and provides a completion signal indicating that the determination is completed to the language change determination unit 136 ( Figure 2 ) and the other party language storage unit 122 ( Figure 2 ), when the language cannot be recognized, the standardized probability of each language at this time is stored in the probability storage unit 194, the value of the counter 196 is increased by 1, and an instruction is provided to the language selection unit 200 to select the output of the next language recognition model and perform the same process; and a threshold storage unit 192, which stores the threshold value compared by the threshold comparison unit 206. As the threshold value, it is determined in advance through experiments so that the desired accuracy and waiting time (latency) can be achieved. The desired threshold value is at least 0.5, more preferably 0.8, and even more preferably 0.85.

[0058] Figure 4 1 is a block diagram showing a schematic configuration of a multilingual speech translation device 144 used in the multilingual speech processing device according to the first embodiment of the present invention. Figure 4 The multilingual speech translation device 144 includes: speech recognition devices 220, ..., 234 for eight languages, which are prepared in advance corresponding to the eight languages; eight multilingual translation devices 240, ..., 254, which respectively receive the text of the speech recognition results output by these speech recognition devices 220, ..., 234, and can translate the text output by the speech recognition device from the language of each speech recognition device into the other seven languages, and translate it into the language specified by the other party language storage unit 122 and output it; a data distribution unit 258, which is used to receive the text from Figure 2 The speech data selection unit 142 provides a speech data vector column, and provides the speech data vector column to a device among the speech recognition devices 220, ..., 234 specified by the language selection signal from the language selection unit 138; and an output selection unit 256, which is connected in a manner of receiving outputs from each of the multilingual translation devices 240, ..., 254, and is used to select the speech processing result specified by the language selection signal from the language selection unit 138, and provide the parameters for speech synthesis to the speech signal synthesis device 146, and provide the text of the speech processing result to the display control device 148.

[0059] Reference Figure 5 The multi-language translation device 240 includes: seven automatic translation engines 262, ..., 274, which are used to translate the text of the language corresponding to the multi-language translation device 240 into the other seven languages, and are all connected in a manner of receiving a counterpart language selection signal from the counterpart language storage unit 122; a translation engine selection unit 260, which receives a language selection signal from the language selection unit 138 and a signal indicating the counterpart language from the counterpart language storage unit 122, and acts only when selected by the language selection unit 138, and performs processing of providing the text output by the speech recognition device 220 to the automatic translation engine 262, ..., 274 corresponding to the counterpart language selection signal from the counterpart language storage unit 122; and an output selection unit 278, which is connected in a manner of receiving a counterpart language selection signal from the counterpart language storage unit 122, and is used to select the output of the automatic translation engine 262, ..., 274 selected by the counterpart language selection signal, and provide it as an output from the multi-language translation device 240 Figure 4 The output selection unit 256 is shown.

[0060] exist Figure 6 The appearance of an example of a computer system 290 for implementing the multi-language speech processing device 100 is shown in FIG. Figure 7 A hardware block diagram showing an example of a computer constituting the computer system 290 .

[0061] Reference Figure 6 The computer system 290 includes a computer 300 having a DVD (Digital Versatile Disc) drive 310 and a keyboard 306, a mouse 308 and a monitor 104 all connected to the computer 300.

[0062] Reference Figure 7In addition to the DVD drive 310, the computer 300 also includes a CPU 316, a GPU (Graphic Processing Unit) 317, a bus 326 connected to the CPU 316, the GPU 317, and the DVD drive 310, a ROM 318 connected to the bus 326 and storing a startup program of the computer 300, a RAM 320 connected to the bus 326 and storing program instructions, system programs, and operation data, and a hard disk drive (HDD) 314 as a non-volatile memory connected to the bus 326. The hard disk 314 is used to store programs executed by the CPU 316 and the GPU 317, data used by the programs executed by the CPU 316 and the GPU 317, etc. The computer 300 also includes: a network I / F 304 that provides a connection to a network 328 that enables communication with other terminals; and a USB memory port 312 that can load and unload a USB memory 330 and provides communication between the USB memory 330 and various parts in the computer 300.

[0063] The computer 300 also includes: a voice I / F 324, which is connected to the microphone 102, the speaker 106 and the bus 326, and is used to read out the voice signal generated by the CPU 316 and stored in the RAM 320 or the HDD 314 according to the instructions of the CPU 316, perform analog conversion and amplification processing to drive the speaker 106, or digitize the analog voice signal from the microphone 102 and store it in any address specified by the CPU 316 in the RAM 320 or the HDD 314.

[0064] In the above embodiment, Figure 2 The basic information setting unit 120, the counterpart language storage unit 122, the language setting unit 124, and the default language storage unit 126 are all implemented by the HDD 314 or the RAM 320. Typically, they are written to the HDD 314 from the outside, loaded into the RAM 320 when the computer 300 is executed, and are periodically saved to the hard disk 314 as a backup.

[0065] The audio signal to be processed may be stored in the computer 300 via any means, but is usually transmitted to the computer 300 from another computer on the network 328 via the network I / F 304 and stored in the RAM 320 or the HDD 314 .

[0066] A computer program for causing the computer system to operate in a manner that realizes the functions of the multilingual speech processing device 100 and its components is stored in the DVD 322 installed in the DVD drive 310, and is transferred from the DVD drive 310 to the HDD 314. Alternatively, the program is stored in the USB memory 330, the USB memory 330 is installed in the USB memory port 312, and the program is transferred to the hard disk 314. Alternatively, the program can be sent to the computer 300 through the network 328 and stored in the HDD 314. The program is loaded into the RAM 320 when it is executed. In addition, the source program can be written using the keyboard 306, the mouse 308, and the monitor 104, and the target program output by the compiler can be saved in the HDD 314. In the case of a script language, a script as a program for realizing the above-mentioned processing can be input using the keyboard 306 and the like, and saved in the HDD 314.

[0067] CPU316 reads the program from RAM320 and interprets the instruction according to the address indicated by the register (not shown) inside it called the program counter, reads the data required for the execution of the instruction from RAM320, hard disk 314 or other devices according to the address specified by the instruction and performs the processing specified by the instruction. CPU316 stores the data of the execution result in the address specified by the program, such as the register in RAM320, hard disk 314, and CPU316. At this time, the value of the program counter is also updated by the program. The computer program can also be loaded directly into RAM320 from DVD322, USB memory 330, or via the network. In addition, in the program executed by CPU316, for a part of the tasks (mainly numerical calculations), according to the instructions in the program, or according to the analysis results when the instructions of CPU316 are executed, they are allocated to GPU317.

[0068] The program that implements the various functions of the computer 300 includes a plurality of instructions that enable the computer 300 to act as the above-mentioned devices. Some of the basic functions required for the computer 300 to perform the action are provided by the operating system (OS) or third-party programs that operate on the computer 300, or modules of various tool kits installed on the computer 300. Therefore, the program may not necessarily include all the functions required to implement the system and method of the present embodiment. The program only includes instructions in the instructions to perform the actions of the above-mentioned devices and their constituent elements by calling appropriate functions or "programming tool kits" in a method that is controlled to obtain the desired results. The operation method of the computer 300 is well known, so it is not repeated here. In addition, the GPU 317 is capable of parallel processing, and can simultaneously perform speech recognition processing, automatic translation processing, and speech synthesis processing related to a large amount of speech data in parallel or pipelined manner.

[0069] Figure 8 It shows that Figure 6 The computer shown is a flowchart of a control structure of a computer program that enables the computer to function as the speech processing device according to the first embodiment. The language recognition neural network 132 and the multilingual speech translation device 144 are executed by a process different from the program.

[0070] Reference Figure 8 The program includes: step 350, which performs initial settings when the program is started; step 352, which determines whether a voice is input and waits until the voice is input; step 354, which responds to the situation where a voice is input, respectively from Figure 2 The default language storage unit 126 reads the default language and the counterpart language is read from the counterpart language storage unit 122; step 356, which starts the multilingual speech translation device 144 with the combination of languages ​​read in step 354 (from the default language to the counterpart language); and step 358, which starts the language recognition processing based on the language recognition neural network 132.

[0071] according to Figure 2 It can be seen that the output of the speech processing device 130 is provided to the speech data selection unit 142 and the language recognition neural network 132 at the same time. When the program is started, the output of the language determination unit 134 selects the default language. Therefore, the same speech data is provided to the language recognition neural network 132 and the multilingual speech translation device 144 at the same time. The speech translation process is started in step 356, and the language recognition process is started in step 358, so the two will act in parallel thereafter.

[0072] The program also includes: step 360, which, after step 358, determines whether the recognized language is consistent with the language stored in Figure 2 The default language storage unit 126 of the default language storage unit 126 is consistent with the default language, and the control flow is branched according to the result; step 362, when the determination in step 360 is negative, the speech translation process started in step 356 by specifying the language of the translation target as the default language is terminated; and step 364, which specifies the language of the recognition result and the language stored in the default language storage unit 126 as the language of the recognition result. Figure 2 The combination of the counterpart language stored in the counterpart language storage unit 122 (speech translation from the language of the recognition result to the counterpart language) is used to start the speech translation processing of the multi-lingual speech translation device 144.

[0073] The program also includes step 366, which is executed after step 364 and when the determination of step 360 is affirmative, and stores the Figure 2 The partner language of the partner language storage unit 122 is stored in the default language storage unit 126 , the language of the recognition result output by the language selection unit 138 is stored in the partner language storage unit 122 , and the control returns to step 352 .

[0074] Fig. 9 It is shown in Figure 8 FIG. 3 is a flowchart of a part of the program of the control structure, that is, a part of the program routine executed in step 358. This program realizes the language recognition in a line-by-line manner described later. Fig. 9 , the program routine includes: step 380, which substitutes 0 into the variable i for controlling the repeated processing of the speech recognition processing based on the line-by-line method; step 382, ​​which substitutes the values ​​{1.75, 2.5, 0} corresponding to 1.75 seconds, 2.5 seconds and 0 seconds into the speech length threshold array prepared as the threshold of the speech length for determining whether to switch the speech recognition model in the speech recognition processing of the line-by-line method; and step 384, which determines whether the value of the variable i is greater than the value obtained by subtracting 1 from the number of models used in the speech recognition processing of the line-by-line method, and branches the control flow according to the determination result. In addition, in step 382, ​​an array of three elements that determine the recognition model (hereinafter referred to as the "model array") is also prepared to store the recognition information of each recognition model. Here, it is assumed that the input speech length to the three recognition models is used as the recognition information, and the value {1.5, 2.0, 3.0} representing the input voice data length of each recognition model is stored in the model array.

[0075] The program also includes: step 386, which, when the judgment of step 384 is negative, inputs the speech data of the input speech length of the model to model [i]; step 388, which, following step 386, selects the probabilities of given three languages ​​from the probabilities of each language output by model [i]; step 390, which standardizes the values ​​of the probabilities of the three languages ​​selected in step 388; step 392, which determines whether the maximum value of the standardized probability in step 390 is above a threshold, and branches the control process according to the judgment result; and step 394, which, when the judgment of step 392 is affirmative, determines the language corresponding to the maximum value as the recognition language and outputs the information, thereby terminating the processing.

[0076] The program further includes: step 398, which, when the judgment result of step 392 is negative, determines whether the speech length of the input speech is shorter than the speech length threshold [i], and branches the control process according to the judgment; step 400, which, when the judgment result of step 398 is negative, stores the probabilities of the three languages ​​standardized in step 390 in the probability storage unit 194 (refer to Figure 3 ); and step 402, which adds 1 to the value of variable i and returns control to step 384.

[0077] The program further includes a step 396, which stores the probability storage unit 194 (reference) when the determination of step 384 is positive (when the value of variable i is greater than the model number -1) and when the determination of step 398 is positive (when the speech length is shorter than the speech length threshold value [i]). Figure 3 ) is combined to determine the final recognized language and control is moved to step 394.

[0078] In step 396, for example, the average of each language is calculated for the probability of each language stored in the memory. In this case, in step 394, even if the maximum value of the average value is smaller than the threshold used in step 392, the language corresponding to the maximum value is determined as the recognition language.

[0079] exist Fig.10 The length of speech processed by the three speech recognition models is shown in Fig. 9 The relationship between the speech length threshold used in step 398 is Fig.10 , the language recognition model 180 will use 1.5 seconds of speech from the beginning of the speech as input to recognize the language. The language recognition model 182 will use 2.0 seconds of speech from the beginning of the speech as input to recognize the language. The language recognition model 184 will use 3.0 seconds of speech from the beginning of the speech as input to recognize the language. If the language recognition model 180 does not recognize the language, the language recognition model 182 is used to recognize the language. If the language recognition model 182 is also unable to recognize the language, the language recognition model 184 is used to recognize the language.

[0080] However, for example, even if the speech recognition model 180 cannot recognize the language, when the speech length is less than 2 seconds, there is a high possibility that the language cannot be recognized using the speech recognition model 182. Therefore, in the present embodiment, when the speech recognition model 180 cannot recognize the language, when the speech length is less than 2 seconds, the speech recognition model 182 is used to recognize the language. Fig.10 If the speech is shorter than 1.75 seconds as shown in FIG. 1 , the speech recognition model 182 does not recognize the speech, but recognizes the speech using the probability calculated by the speech recognition model 180. Similarly, even if the speech recognition model 182 cannot recognize the speech, in the case of a speech such as Fig.10 As shown, if the time is less than 2.5 seconds, the language recognition in the language recognition model 184 is not performed, and the language recognition is performed using the probability calculated for each language by the language recognition models 180 and 182. In this case, there is no problem as long as the maximum value of the probability calculated in this way is greater than the threshold value, but even if it is less than the threshold value, the language corresponding to the maximum value is used as the result of language recognition.

[0081] In this way, when the speech cannot be recognized with a short speech length, the speech recognition is performed by sequentially considering the subsequent speech data. In this specification, such a method is called a line-by-line method.

[0082] exist Fig. 9 The normalization process performed in step 390 is a process of recalculating the probability of each language by dividing the probability of each language by a value obtained by summing the probabilities of the three languages.

[0083] (Action>

[0084] The multi-language speech processing device 100 having the structure described above operates as follows. When the multi-language speech processing device 100 is started, the other party language storage unit 122 stores the language used by the user of the multi-language speech processing device 100 in the basic information stored in the basic information setting unit 120. The default language storage unit 126 stores the language temporarily selected as the language of speech input by the user from among the languages ​​stored in the language setting unit 124. Even if the user does not know the language of the other party in the conversation clearly, the language can be inferred and set in the default language storage unit 126.

[0085] In the following actions, it is assumed that the other party of the conversation starts talking instead of the user. In the actual conversation process, as a process for estimating the other party's language, it is necessary to let the other party of the conversation start talking first. Therefore, it is preferable to display a text such as "Please say something in your own language" on the screen in the language of a certain country. For example, if it is displayed in English, it is believed that the other party will not be at a loss.

[0086] Reference Figure 2 If the other party starts to speak, the microphone 102 converts the voice into a voice signal and provides it to Figure 2 The speech processing device 130 shown in the figure receives the speech signal, digitizes and frames it, and transforms it into a time series of speech data vectors representing each speech, and provides it to the language recognition neural network 132, the speech data selection unit 142 and the buffer 140. The buffer 140 buffers the speech data. At this time, refer to Figure 3 , respectively providing speech data for a period of 1.5 seconds from the beginning of the speech to the speech recognition model 180 within the speech recognition neural network 132, providing speech data for a period of 2.0 seconds from the beginning of the speech to the speech recognition model 182, and providing speech data for a period of 3.0 seconds from the beginning of the speech to the speech recognition model 184.

[0087] On the other hand, refer to Figure 2, the language change determination unit 136 sets the language selection unit 138 and the voice data selection unit 142 so that the voice data can be processed in the default language at startup. That is, the language selection unit 138 selects the output of the default language storage unit 126 and provides it to the multilingual voice translation device 144 and the voice signal synthesis device 146. Information related to the default language is also provided to the other party language storage unit 122, but the other party language storage unit 122 only takes in the output of the language selection unit 138 when the language is determined by the language determination unit 134. Therefore, the user's language is still stored in the other party language storage unit 122. This value is provided to the multilingual voice translation device 144 and the voice signal synthesis device 146 ( Figure 8 The speech data selection unit 142 selects the output of the speech processing device 130 instead of the output of the buffer 140 and provides it to the multi-language speech translation device 144.

[0088] As a result, the multilingual speech translation device 144 starts speech processing using the speech data provided from the speech processing device 130 via the speech data selection unit 142 as data representing speech in the language stored in the default language storage unit 126 ( Figure 8 Step 356).

[0089] Reference Figure 4 , the multilingual speech translation device 144 operates specifically as follows. The data distribution unit 258 distributes the speech data from the speech data selection unit 142 to the speech recognition devices 220, ..., 234 specified by the language selection signal from the language selection unit 138, and does not distribute it to other devices. In addition, the speech recognition devices 220, ..., 234 all receive the language selection signal from the language selection unit 138, and only the device (such as the speech recognition device 220) that processes the language specified by the language selection signal operates.

[0090] The speech recognition device designated by the language selection signal among the speech recognition devices 220 ..., 234 provides the recognized text to the corresponding multilingual translation device 240. In the following, as an example, the speech recognition device 220 is set as a device designated by the language selection signal, performs speech recognition on the speech data from the speech data selection unit 142, and provides the text of the result to the multilingual translation device 240, and the multilingual translation device 240 is set as a device for performing translation processing.

[0091] Reference Figure 5The translation engine selection unit 260 of the multi-language translation device 240 is provided with a language selection signal from the language selection unit 138 and a signal indicating the other party language from the other party language storage unit 122. The translation engine selection unit 260 provides the text output by the speech recognition device 220 to the automatic translation engine 262, ..., 274 corresponding to the other party language selection signal from the other party language storage unit 122 only when selected by the language selection unit 138.

[0092] The automatic translation engines 262, ..., 274 are all provided with the counterpart language selection signal from the counterpart language storage unit 122, and only the automatic translation engine selected by the counterpart language selection signal performs automatic translation. Figure 5 In the example shown, for example, if the automatic translation engine 262 is selected by the counterpart language selection signal, the automatic translation engine 262 is selected by the translation engine selection unit 260, and the output of the speech recognition device 220 is provided to the automatic translation engine 262. The automatic translation engine 262 automatically translates the input text into the counterpart language. The counterpart language is consistent with the counterpart language stored in the counterpart language storage unit 122.

[0093] The output selection unit 278 is also supplied with the other party language selection signal from the other party language storage unit 122. The output selection unit 278 selects the output of the automatic translation engine selected by the other party language selection signal from among the automatic translation engines 262, ..., 274 and supplies it to the multilingual translation device 240 as the output. Figure 4 The output selection unit 256 is shown.

[0094] The output selection unit 256 selects the output of the multilingual translation device (eg, multilingual translation device 240 ) corresponding to the language specified by the language selection signal from the language selection unit 138 , and outputs it to the speech signal synthesis device 146 and the display control device 148 .

[0095] That is, the multilingual speech translation device 144 performs the following operations, namely, performs speech recognition on input speech data, performs automatic translation, generates parameters for speech synthesis, and sequentially outputs them.

[0096] On the other hand, the language recognition models 180, 182, and 184 of the language recognition neural network 132 and the multi-language speech translation device 144 simultaneously start processing to recognize the language of the speech represented by the speech data ( Figure 8 The speech lengths required for each model are 1.5 seconds, 2.0 seconds, and 3.0 seconds, respectively, and they are Figure 2The speech processing device 130 shown processes and accumulates them as speech data vector sequences. At the time point when speech data corresponding to 1.5 seconds is accumulated, they are input to the speech recognition model 180. Similarly, at the time point when speech data corresponding to 2.0 seconds is accumulated, they are input to the speech recognition model 182, and at the time point when speech data corresponding to 3.0 seconds is accumulated, they are provided to the speech recognition model 184. In addition, when the speech is less than 1.75 seconds or 2.5 seconds, this information is saved by a flag not shown.

[0097] The recognition results are output from the language recognition models 180, 182 and 184 respectively. Figure 3 The probabilities 174, 176, and 178 shown in FIG. 17A ) are shown in FIG. 17B . At this time, the time until the language recognition model 180 starts the recognition process is earlier than that of the other two models, and its result is obtained first. The time until the language recognition model 182 starts the recognition process is longer than that of the language recognition model 180, but shorter than that of the language recognition model 184, so it is obtained after the recognition result based on the language recognition model 180 is obtained and before the recognition result based on the language recognition model 184 is obtained. The recognition result from the language recognition model 184 is obtained last.

[0098] Figure 3 The language determination unit 134 shown performs language recognition processing as follows using the probabilities output from the language recognition models 180, 182, and 184. In order to explain as simply as possible, the operation of the language determination unit 134 will be described below on the premise that the speech is longer than 2.5 seconds.

[0099] Reference Figure 3 If speech is detected, the language determination unit 208 first initializes the value of the counter 196 to 0 ( Fig. 9 Next, the language determination unit 208 provides the model (in this example, the language recognition model 180) determined by the value of the counter 196 (=0) with speech data corresponding to 1.5 seconds (step 386), and controls the model selection unit 190 to select its output (step 388). The model selection unit 190 selects the probability 174 output by the language recognition model 180 and provides it to the language selection unit 200. The language selection unit 200 selects the probabilities 174 related to the three languages ​​stored in the selected language storage unit 128 from the probabilities 174 of the eight languages ​​output by the language recognition model 180 ( Fig. 9 The probability normalization unit 202 calculates the sum of the probabilities of the three languages ​​and divides the values ​​of each probability by the sum, thereby normalizing the probabilities associated with the three languages ​​( Fig. 9The maximum probability selection unit 204 selects the maximum value, and the threshold comparison unit 206 compares the maximum value with the threshold value ( Fig. 9 Step 392 of ). If the maximum value is above the threshold value ("Yes" in step 392), the threshold comparison unit 206 provides a signal indicating that the probability of the language corresponding to the maximum value is above the threshold value to the language determination unit 208. The language determination unit 208 determines whether the language corresponding to the maximum probability is the language of the speech data based on the normalized probabilities of the three languages ​​output by the probability normalization unit 202, the maximum probability output by the maximum probability selection unit 204, and the comparison result of the threshold comparison unit 206. In this embodiment, basically, when the maximum probability is above the threshold value, the language corresponding to the probability is used as the recognition language, and when none of the probabilities is above the threshold value, the probabilities of each language at the past one or two judgments performed line by line and the current probability are merged (for example, the average value is calculated), and the language corresponding to the maximum value is used as the recognition language regardless of whether the value is above the threshold value. If it is determined to be the language of the speech data, the language determination unit 208 outputs the value to the language change determination unit 136. The language determination unit 208 further outputs a signal indicating that the recognition is completed to the language change determination unit 136 and the other party language storage unit 122. Thus, the language recognition is completed.

[0100] If the maximum value of the probability is less than the threshold, the recognition result is considered unreliable, the probabilities of the three languages ​​output by the probability normalization unit 202 are saved in the probability storage unit 194, the value of the variable i stored in the counter 196 is increased by 1, and the control model selection unit 190 selects the next language recognition model, namely, the language recognition model 182.

[0101] If the probability 176 as the recognition result based on the language recognition model 182 is output, the model selection unit 190 selects it and supplies it to the language selection unit 200. Hereinafter, the same processing as above is performed.

[0102] When the language cannot be determined by the recognition result based on the language recognition model 182, the language determination unit 208 further increases the counter 196 to 2, and controls the model selection unit 190 to select the probability 178 as the output of the language recognition model 184 next time. The model selection unit 190 selects the probability 178 from the language recognition model 184 and provides it to the language selection unit 200. Hereinafter, the same processing as above is repeated until the processing of the language determination unit 208.

[0103] In addition, Fig. 9If the result of the processing of step 392 is negative even when the speech recognition model 184 is used, the processing is performed as follows. In this case, the control proceeds to step 398. However, in this case, i=2, speech threshold length [2]=0, and therefore Fig. 9 The judgment of step 398 is definitely negative, and steps 400 and 402 are executed. If the judgment of step 384 at the beginning of the subsequent loop processing is negative, the language is recognized through steps 396 and 394.

[0104] In the language determination unit 208, when the maximum value of the probability is greater than the threshold value, the language corresponding to the probability is output as the result of language recognition to the language change determination unit 136, and a signal indicating that the language recognition is completed is output to the language change determination unit 136 and the counterpart language storage unit 122, as in other cases. This is the same as the case where i=0 and 1.

[0105] However, when the maximum value is less than the threshold value, the language determination unit 208 determines as follows. That is, in this case, the language determination unit 208 combines the probabilities of the languages ​​stored in the probability storage unit 194 so far. For example, the average value of the probabilities of the languages ​​is calculated and used as the combined probability of the language. Then, the language with the largest combined probability is determined as the language of the speech data. In this case, even if the probability is less than the threshold value, it is determined as the language. This is because, since the device is used for dialogue, it is necessary to output in a certain form within a given time.

[0106] In addition, when the speech length is less than 1.75 seconds, only the result of the language recognition model 180 is used to perform the above-mentioned recognition processing. In this case, even if the final maximum value of the probability is less than the threshold value, the result of language recognition is output in the same way as in the above-mentioned case of i=2. The same is true when the speech length is less than 2.5 seconds, and the recognition processing is performed only using the output of the language recognition model 180 and the output of the language recognition model 182. Even in this case, when the maximum value of the probability is less than the threshold value even when the result of the language recognition model 182 is used, the probability of each language stored so far is merged for each language (for example, the average value is calculated), and the language with the highest probability is used as the recognition result. It is not a necessary condition that this value is above the threshold value.

[0107] Refer again Figure 2In response to the signal from the language determination unit 134 indicating that the recognition is completed, the language change determination unit 136 compares the default language stored in the default language storage unit 126 with the language of the recognition result output by the language determination unit 134, and provides a determination signal indicating whether they are the same or changed to the language selection unit 138 and the voice data selection unit 142. The value of the determination signal in the case of the sameness is set to the first value, and the value of the determination signal in the case of the differentness is set to the second value. The operation of the multi-language voice processing device 100 is described below by dividing into the case where the language of the recognition result is consistent with the default language and the case where it is inconsistent.

[0108] -When the language of the recognition result is the same as the default language-

[0109] When the determination signal of the language change determination unit 136 is the first value, the language selection unit 138 selects the output of the default language storage unit 126 and provides it to the counterpart language storage unit 122, the multilingual speech translation device 144, and the speech signal synthesis device 146. In this case, the same information as the originally provided information is provided to the multilingual speech translation device 144 and the speech signal synthesis device 146.

[0110] In response to the supply of the determination signal having the first value from the language change determination unit 136 , the speech data selection unit 142 continues the process of supplying the speech signal from the speech processing device 130 to the multilingual speech translation device 144 .

[0111] Since the value of the language selection signal from the language selection unit 138 does not change, Figure 4 The data distribution unit 258 shown in FIG. 250 continues to provide voice data to the voice recognition device 220, ..., 234, which originally provided the voice data. Figure 4 The translation engine selection unit 260 ( Figure 5 ) continues to provide the speech recognition result to the same translation engine. In addition, since the value of the other party language selection signal from the other party language storage unit 122 does not change, in the multilingual translation device that operates according to the default language selection signal, the automatic translation engine selected by the other party language selection signal also continues to operate in the same manner. Other speech recognition devices and automatic translation devices do not operate. The same is true for the output selection unit 256, which selects the output of the speech recognition device and multilingual translation device in operation before determining the recognition result, and provides it to the speech signal synthesis device 146 and the display control device 148.

[0112] Reference Figure 2The speech signal synthesis device 146 and the display control device 148 also continue to process the speech translation result in the same language as before output from the multilingual speech translation device 144.

[0113] Then, the counterpart language stored in the counterpart language storage unit 122 is stored in the default language storage unit 126 , and information indicating the language of the new recognition result provided from the language selection unit 138 is stored in the counterpart language storage unit 122 .

[0114] If there is a new voice input, the voice processing is performed again by specifying the default language stored in the default language storage unit 126 and the counterpart language stored in the counterpart language storage unit 122 .

[0115] -When the language of the recognition result is inconsistent with the default language-

[0116] In this case, in response to the inconsistency between the two inputs, the language change determination unit 136 provides a determination signal of the second value to the language selection unit 138 and the voice data selection unit 142. In response to the determination signal, the language selection unit 138 selects the language of the recognition result output by the language determination unit 134 and provides it to the counterpart language storage unit 122, the multilingual voice translation device 144, and the voice signal synthesis device 146.

[0117] The speech data selection unit 142 selects the output of the buffer 140 in response to the determination signal from the language change determination unit 136 , and supplies the selected output to the multi-language speech translation device 144 .

[0118] Reference Figure 4 The data assigning unit 258 switches the speech recognition device and automatic translation device selected so far to a new speech recognition device and automatic translation device representing the language of the recognition result provided by the language selecting unit 138 and provides the output of the speech data selecting unit 142 .

[0119] In the speech recognition device 220, ..., 234, since the value of the selection signal from the language selection unit 138 has changed, the speech recognition device that has performed the speech translation process so far stops the speech recognition process. The multilingual translation device is also the same. On the other hand, in the speech recognition device 220, ..., 234, the speech recognition device specified by the new selection signal starts to process from the beginning of the voice data provided by the data distribution unit 258 and buffered. In addition, the multilingual translation device that receives the output of the speech recognition device also starts to move, and the automatic translation engine determined by the other party's language selection signal starts to translate. The translation result is output by the output selection unit 278 and the output selection unit 256.

[0120] Next, the output of the device that restarted the process is provided to Figure 2The speech signal synthesizer 146 and the display control device 148 are shown. The speech signal synthesizer 146 performs processing such as synthesizing the speech signal of the other language stored in the other language storage unit 122 based on the speech synthesis parameter output from the multilingual speech translation device 144 and providing it to the speaker 106.

[0121] If the processing of the input voice data is completed, the other party language stored in the other party language storage unit 122 is stored in the default language storage unit 126 and becomes the default language, and the language of the recognition result output by the language selection unit 138 is stored in the other party language storage unit 122 and becomes the target language for the voice translation processing of the next voice.

[0122] The operation of the multilingual speech processing device 100 described above can be summarized as follows. First, if the other party who is talking to the user speaks, the speech is assumed to be in the default language and speech recognition is started. The result of the speech recognition is output. If the recognition result of the language is consistent with the default language, the process is continued, and the speech recognition result is automatically translated into the language stored in the other party language storage unit 122 (the user's language) in sequence. The speech is synthesized based on the result of the automatic translation and output.

[0123] On the other hand, if the language recognition result does not match the default language, speech recognition is stopped midway, the language is switched to the language of the recognition result, speech recognition and automatic translation are started for the speech data accumulated in the buffer 140, and speech synthesis is started.

[0124] When the processing of the speech is completed, the language used as the other party's language in the previous processing is set as the default language, and the language of the recognition result is set as the other party's language. As a result, if the user of the multi-language speech processing device 100 then exchanges words with the other party, if the language recognition is good, the language of the recognition result is consistent with the default language, and the speech processing started by the multi-language speech translation device 144 is quickly output without interrupting it.

[0125] If the processing of the speech is completed, the other party's language and the default language are switched next time, and the speech of the other party's language is translated into the default language. Next time, since the other party's language is consistent with the recognition language, the speech translation processing started by the multilingual speech translation device 144 does not need to be stopped and continued, so that the speech translation result can be quickly obtained.

[0126] Reference Figures 11 to 13 , the effects of the multilingual speech processing device 100 involved in the first embodiment are described.

[0127] Fig.114 is a schematic diagram showing the time course of speech processing based on a conventional method. When speech is input, language recognition 440 is first started, and speech recognition 442 is started using the result. When the results of speech recognition 442 are sequentially output, automatic translation 444 is performed on speech recognition 442. Speech synthesis 446 is performed on the output of automatic translation 444.

[0128] like Fig.11 As shown, in comparison with the above-described embodiment, in the conventional method, the time from the start of speech recognition 440 to the start of speech synthesis 446 is referred to as Tc.

[0129] Fig.12 FIG. 1 is a schematic diagram showing the passage of time of speech processing when the final speech processing result is obtained in a short time in the first embodiment of the present invention. Fig.12 In the above embodiment, language recognition 440 and speech recognition 460 are started at the same time. Speech recognition 460 is speech recognition from the default language to the other party's language. If the result is output, automatic translation processing 462 is started. If the language based on the recognition result of language recognition 440 is consistent with the default language, speech recognition 460 and automatic translation processing 462 are continued without being stopped, and speech synthesis 464 is started for the result of automatic translation.

[0130] exist Fig.12 In the example shown, speech recognition 460 and language recognition 440 start simultaneously, so the time T1 required until speech synthesis 464 starts is significantly longer than Fig.11 The time Tc is short. Fig.11 and Fig.12 , it can be seen that time T1 is shorter than time Tc by an amount corresponding to the time required for language recognition 440.

[0131] Fig.13 This is a schematic diagram showing the passage of speech processing time when the time to obtain the final speech processing result is the longest in the first embodiment of the present invention. This example is a case where the language recognized by the language recognition 440 is different from the default language. In this case, the speech recognition 460 and the language recognition 440 are started at the same time, and the automatic translation process should be started after the speech recognition. Fig.12 The same as above, however, speech recognition 460 is stopped at the time when language recognition 440 is finished, speech recognition 480 in the recognized language is restarted, the result is automatically translated into the other language 482, and speech synthesis 484 for the result of automatic translation is restarted. Speech recognition 480 is performed from the beginning of the input speech, so in this case, the time T2 from speech input to the start of speech synthesis 484 is the same as that of speech recognition 460. Fig.11 The Tc shown is the same.

[0132] That is, according to the present embodiment, when the default language is consistent with the recognized language, the time required until the output of the result of the speech translation processing starts is shorter than the previous time. The time is shortened by at least the amount corresponding to the time required for the initial language recognition. Even if the default language is inconsistent with the recognized language, the time required until the output of the result of the speech translation processing starts is the same as the previous time.

[0133] As a result, it is possible to obtain a speech processing device that can quickly obtain a speech processing result even when the language of the speech represented by the speech signal is unknown.

[0134] In addition, instead of directly using the output of the language recognition neural network 132 to recognize the language, the probabilities of a plurality of languages ​​(three in the embodiment) specified in advance are selected from a plurality of languages ​​(eight in the embodiment), and the language with the highest probability is selected after they are standardized. The probability of the language is compared with the threshold to decide whether to accept or discard the recognition result. As a result, it is possible to perform highly accurate language recognition without complicating the determination process based on the language recognition neural network 132. In addition, since a part of the languages ​​can be further selected from the plurality of languages ​​to be designated as pre-processed objects, even when the language spoken by the other party is unknown, as long as the other party's place of origin, etc. is inferred, a combination of languages ​​such as the object of language recognition can be designated with a higher probability, and the possibility of starting a conversation smoothly is increased.

[0135] Furthermore, in the above-mentioned embodiment, after the language is recognized, the other party language of the speech processing is set as a new default language, and the recognized language is set as the next other party language. As a result, there is an effect that when two speakers are talking, the speech translation process from one language to another language can be smoothly performed. In addition, in this embodiment, the situation where two speakers speak alternately is assumed, but it can also be easily expanded to when there is only one speaker, when there are three speakers, etc. In addition, the process of setting them can be easily realized by a program that performs a normal conversational process.

[0136] In addition, in the above-mentioned embodiment (method 1), three models (speech length 1.5 seconds, 2.0 seconds, 3.0 seconds) are used as language recognition models. However, the present invention is not limited to such an embodiment. For example, four language recognition models can also be used, and the speech length as the object is set to 1.0 seconds, 1.5 seconds, 2.0 seconds, and 3.0 seconds (method 2). Alternatively, it is also possible to consider using the same three language recognition models, and setting the speech length to 1.0 seconds, 1.5 seconds, and 2.0 seconds (method 3). In such a structure, the accuracy is method 1>method 2>method 3, and the waiting time is method 3<method 2<method 1.

[0137] In the experiment, when the threshold value was 0.85, the accuracy of the above three methods was 95% or more. In addition, when using methods 2 and 3, the waiting time was a very short waiting time of 1.3 seconds on average.

[0138] In addition, various combinations of recognition models can be considered according to the purpose. Of course, the case of using only one recognition model is also an object.

[0139] In addition, in the above-mentioned method, the objects of the language recognition model are all voice data of a given length from the beginning of the speech. However, the present invention is not limited to such an embodiment. For example, it is also possible to make the voice data with the same speech length (for example, 1.0 seconds or 1.5 seconds) and each offset by 0.5 seconds at the beginning be used as the object for language recognition in sequence. In this case, the language recognition model can use the same language recognition model. In this case, the speech length as the object and the number of language recognition models used can also be selected according to the purpose.

[0140] [Second embodiment]

[0141] <Structure>

[0142] The multilingual speech processing device 100 according to the first embodiment is implemented by a single computer, but it is of course possible to perform processing such that each function of the multilingual speech processing device 100 is shared among a plurality of computers.

[0143] Fig.14 2 is a schematic diagram showing the overall structure of the voice dialogue system according to the second embodiment of the present invention. Fig.14As shown, the speech translation system 500 involved in the second embodiment prepares a multilingual speech translation processing server (hereinafter referred to as "server") 506 having most of the functions of the multilingual speech processing device 100 involved in the first embodiment, and performs speech language recognition, automatic translation and speech synthesis for a speech translation request provided from a device such as a smart phone 504 via the Internet 502, and returns a speech signal from the server 506 to the smart phone 504. In order to realize such a structure, in this embodiment, it is set to use the basic information (language used) of the user who uses the smart phone 504 stored in the smart phone 504, and further set to enable the user of the smart phone 504 to select three languages ​​from eight languages ​​of the other party as described in the first embodiment above. Such information is required in the language recognition processing in the server 506, so it is set to be sent to the server 506 together with the speech data.

[0144] Fig.15 2 is a block diagram showing a schematic hardware configuration of a smartphone used as a terminal in the second embodiment. Fig.15 The smartphone 504 includes a camera 520, a touch panel display 522, a speaker 524, and a microphone 526. The smartphone 504 also includes a processor 550 as a computer for performing various processes, a memory controller 556 connected to the processor 550, an SDRAM (Static Dynamic Access Memory) 558 and a flash memory 560 connected to the memory controller 556, a power supply and an audio LSI 570 connected to the processor 550, a modem 552 for wireless communication connected to the processor 550, the power supply and the audio LSI 570, and a wireless communication device 554 having a wireless transmission and reception antenna connected to the modem 552.

[0145] The smartphone 504 further includes a GPS (Global Positioning System) module 562 , a sensor group 564 including an acceleration sensor, a temperature sensor, a humidity sensor, etc., a Wi-Fi module 566 , and a short-range wireless communication device 568 as peripheral devices of the processor 550 .

[0146] The program executed by the processor 550 is stored in the flash memory 560 and loaded into the SDRAM 558 when executed. The processor 550 controls each module in the smartphone 504 by executing the program, and performs storage of setting values, acquisition of speech data of the speaker via the microphone 526, transmission of speech data to the server 506 via the modem 552 and the wireless communication device 554 or the GPS module 562, reception of speech translation results from the server 506, reproduction of speech using the speaker 524, and the like.

[0147] Fig.16 It is shown Fig.15 1 is a block diagram showing the functional structure of the smartphone. The diagram shows the functions implemented by the program executed by the processor 550 in the form of a block diagram.

[0148] Reference Fig.16 The smartphone 504 functionally includes: a basic information setting unit 600 for setting basic information of the smartphone 504 (including at least the language used by the user); a counterpart language storage unit 602 for storing the counterpart language of the speech translation; a language setting unit 604 for setting which three languages ​​are selected from eight languages ​​and which language is used as the default language, as in the embodiment; a default language storage unit 606 for storing the default language during speech translation processing; and a selected language storage unit 612 for storing information related to the language set by the language setting unit 604. When the speech translation processing is started, the counterpart language storage unit 602 stores the user's language in the basic information setting unit 600, and the default language storage unit 606 stores the default language set by the language setting unit 604.

[0149] The smart phone 504 further includes a preprocessing device 608 for performing a preprocessing operation on the voice signal obtained from the microphone 526. Figure 2 The speech processing device 130 shown in FIG. 1 is similar to the speech processing device 130 shown in FIG. 1 and outputs the speech data vector column and other preprocessing; a transmission data generating unit 610, which is used to generate transmission data for requesting speech translation to the server 506 based on the selected language stored in the selected language storage unit 612, the other language stored in the other language storage unit 602, the default language stored in the default language storage unit 606, and the speech data vector column output by the preprocessing device 608; and a communication I / F 614, which is used to transmit the speech translation to the server 506 via Fig.15 The modem 552 and the wireless communication device 554 or the Wi-Fi module 566 shown transmit the transmission data generated by the transmission data generation unit 610 to the server 506 and receive the processed data from the server 506 .

[0150] The smart phone 504 also includes: a language information extraction unit 616, which extracts information representing the other party's language from the data received by the communication I / F 614 from the server 506; a voice signal extraction unit 618, which is used to extract the voice signal after voice translation from the data received by the communication I / F 614; a voice signal amplification unit 620, which is used to output the voice after voice translation by analog conversion, amplification and driving the speaker 524 of the voice signal; a text extraction unit 622, which is used to extract the text after voice translation from the data received by the communication I / F 614 from the server 506; and a display control unit 624, which is used to display the text on the touch panel display 522.

[0151] Fig.17 It is shown in Fig.15 The following is a diagram of the language setting screen on a smartphone. Fig.17 In the language setting screen 650 displayed on the touch panel display 522, the names of eight languages ​​660 that can be selected are displayed, and a switch indicating whether to select is displayed in each of them. If the switch is moved to the right, the language is selected, and if it is moved to the left, the language is removed from the selection. Fig.17 In the example shown, Japanese, English, and Chinese are selected, and other languages ​​are not selected. In addition, the basic information setting screen and the default language setting screen can also be realized through the same interface.

[0152] Fig.18 yes Fig.15 A block diagram of the functions of the server 506 is shown. Fig.18 , server 506 has Figure 2 The structure of the multi-language speech processing device 100 shown in FIG. Figure 2 The structure shown similarly includes a selection language storage unit 128, a counterpart language storage unit 122, a language setting unit 124, a language recognition neural network 132, a language determination unit 134, a language selection unit 138, a buffer 140, a speech data selection unit 142, a multi-language speech translation device 144, and a speech signal synthesis device 146.

[0153] The multilingual speech translation processing server 506 also includes: a communication device 680, which is used to communicate with a terminal device such as a smart phone 504; a data separation device 682, which is used to separate information related to the selected language and information related to the default language from the data requesting speech translation received by the communication device 680, and store the former in the selected language storage unit 128 and the latter in the language setting unit 124 respectively; and a sending information generation device 684, which is used to generate sending information to be sent to a terminal device such as a smart phone 504 that sends a speech translation request based on the speech translated text output by the multilingual speech translation device 144, the speech translated synthesized speech signal output by the speech signal synthesis device 146, and the language selected The unit 138 outputs information related to the language recognized by the language recognition neural network 132 and the language determination unit 134, and provides it to the communication device 680.

[0154] This is not shown Fig.15 The smartphone 504 shown has a program structure that functions as a client of the speech translation system. In brief, the following processing is performed.

[0155] The basic information of the smartphone 504 is registered and stored in a storage device such as SDRAM 558 through a program for setting basic information. This process is not related to the speech translation system and is the basic process that the user who purchased the smartphone 504 performs initially. The basic information includes information related to the language used by the user.

[0156] The program for setting the language (a part of the client program of the system) receives the selection of three languages ​​among the eight languages ​​to be processed and stores them in a storage device such as SDRAM 558. At the same time, the input of designating any one of the three selected languages ​​as the default language is received and the default language is stored in the storage device.

[0157] When the speech translation program is started, the language information stored in the selected language storage unit 612, the default language stored in the default language storage unit 606, and the partner language stored in the partner language storage unit 602 are read out, and transmission data requesting speech translation is generated together with the input speech and transmitted to the server 506.

[0158] Wait for data to be sent from the server 506.

[0159] If data is sent from the server 506, the voice signal is extracted and played. In addition, the text after the voice translation is extracted and played. Furthermore, the information related to the recognized voice contained in the data is extracted and stored in the storage device as the other party's language.

[0160] Thereafter, the above-described processing is executed every time a voice is input.

[0161] Fig.19 is a diagram showing a method for making a computer as Fig.18 A flowchart of a control structure of a computer program that functions in the multilingual speech translation system shown.

[0162] Fig.19 The flowchart shown has Figure 8 The process shown Figure 10 The difference is that instead of Figure 8 The method further includes steps 720 of waiting until a speech translation request is received from a client such as the smartphone 504, and step 722 of extracting information related to the default language, the other party's language, and the selected language contained in the request and storing them in a given storage device in response to receiving the speech translation request, and step 723 of storing the information in a given storage device in response to receiving the speech translation request. Figure 8 After step 366 , the identified language is sent to the client and control returns to step 724 of step 720 .

[0163] (Action>

[0164] The operation of the smartphone 504 is obvious from the control structure of the above-mentioned program.

[0165] On the other hand, regarding the actions of server 506, if Figure 8 and Fig.19 It should be noted that in this embodiment, the situation where the language recognized by the server 506 needs to be sent to the smartphone 504 is different from the situation where the language recognized by the server 506 needs to be sent to the smartphone 504. Figure 8 In the example of using the output attached to the device to output the processing result, the server 506 needs to send it to the client such as the smartphone 504.

[0166] Regarding other points, the operation of the server 506 is the same as that of the multilingual speech processing device 100 in the first embodiment. Therefore, in this second embodiment, as in the first embodiment, if the default language and the language of the recognition result are consistent, the time until the result of the speech translation processing is obtained can be shorter than before, and even if they are different, the result can be obtained in the same time as before.

[0167] The mechanism of language recognition is also the same as that of the first embodiment, and the same effect can be obtained.

[0168] Furthermore, the language setting is also the same as that of the first embodiment, so there is an effect that the accuracy of language recognition can be improved even if the language recognition neural network 132 is simplified, as in the first embodiment.

[0169] In addition, in the above embodiment, there are eight languages ​​to be selected and three languages ​​to be selected from them, but the present invention is obviously not limited to such a number. If the number of languages ​​to be selected is greater than the number of languages ​​to be selected and there are multiple languages ​​to be selected, the same effect as in the above embodiment can be obtained.

[0170] Furthermore, in the above-described embodiment, there is only one default language. However, if the computing resources of the computer allow, the default language may be set to more than two. That is, speech is received, language recognition is started, and speech processing is started in two or more default languages ​​at the same time. At the time point when the result of language recognition is clear, only the processing of the language is continued, and processing based on other languages ​​is terminated. Compared with the case where speech recognition is performed in all languages ​​and any one of the speech recognition results is selected based on the final score as in Patent Document 1, the necessary computing resources can be reduced. In addition, the structure of the multilingual speech processing device 100, etc. at this time can be implemented even if not changed that much from the first embodiment. It is only necessary to store multiple default languages ​​and to make the speech recognition devices 220, ..., 234 ( Figure 4 ) etc., can specify a plurality of devices among them, and enable the data distribution unit 258 to supply voice data to these plurality of devices.

[0171] The embodiments disclosed this time are merely illustrative, and the present invention is not limited to the above-described embodiments. The scope of the present invention is indicated by each claim of the claims based on the description of the detailed description of the invention, and includes all changes within the meaning and scope equivalent to the words described herein.

[0172] Explanation of symbols

[0173] 100: Multi-language speech processing device;

[0174] 120, 600: basic information setting unit;

[0175] 122, 602: other party language storage unit;

[0176] 124, 604: Language setting unit;

[0177] 126, 606: default language storage unit;

[0178] 128, 612: select language storage unit;

[0179] 130: speech processing device;

[0180] 132: Language recognition neural network;

[0181] 134: Language determination unit;

[0182] 142: voice data selection unit;

[0183] 144: Multi-language voice translation device;

[0184] 146: speech signal synthesis device;

[0185] 170: convolutional layer group;

[0186] 172: fully connected layer;

[0187] 202: Probability Standardization Department;

[0188] 204: Maximum probability selection unit;

[0189] 206: threshold comparison unit;

[0190] 220, 222, 224, 226, 228, 230, 232, 234: speech recognition device;

[0191] 240, 242, 244, 246, 248, 250, 252, 254: multi-language translation device;

[0192] 262, 264, 266, 268, 270, 272, 274: automatic translation engine;

[0193] 440: Language identification;

[0194] 442, 460, 480: speech recognition;

[0195] 444, 462, 482: automatic translation;

[0196] 446, 464, 484: speech synthesis;

[0197] 500: Voice translation system;

[0198] 506: Multi-language speech translation processing server.

Claims

1. A method for identifying a language of an input speech signal in a plurality of languages, comprising: A first step of attempting to recognize a language of the speech signal from among the plurality of languages ​​based on the speech signal until a first time has passed since the beginning of the speech signal; In response to the language of the speech signal being identified in the first step, outputting information indicating the language; A second step of attempting to identify a language of the speech signal from among the plurality of languages ​​based on the speech signal from the beginning of the speech signal to a second time that is longer than the first time, in response to the language of the speech signal not being identified in the first step; In response to the language of the speech signal being identified in the second step, outputting information indicating the language; and In response to the language of the speech signal not being recognized in the second step, an additional recognition step of performing a given additional process for recognizing the language of the speech signal, The first step comprises: a score calculation step of repeatedly calculating scores corresponding to probabilities that the language of the speech signal is the language for each of the plurality of languages ​​based on the speech signal from the beginning of the speech signal until the first time has passed from the beginning of the speech signal; as well as a comparing step, in response to the scores being calculated for the plurality of languages ​​in the score calculating step, selectively performing a process of identifying the language of the speech signal and a process of discarding the scores based on a comparison result of each score with a given threshold value, The threshold value is predetermined so as to achieve a given accuracy or latency for speech recognition based on the method.

2. The method according to claim 1, wherein: The additional recognition step includes: a third step, in response to the language of the voice signal not being recognized in the second step, attempting to recognize the language of the voice signal from among the plurality of languages ​​based on the voice signal from the beginning of the voice signal to a third time that is longer than the second time.

Images