Speech processing system, audio processing method

Abstract

To achieve a suitable process for speech-to-text conversion with translation.SOLUTION: Input speech data is converted into text, the text data is sent to respective speech input / output devices, and output in the respective speech input / output devices based on a specified language and at least one of the text data and a conversion result of the text data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a speech processing system and a speech processing method that realize suitable processing of speech-to-text conversion involving translation. [Background technology]

[0002] In data processing for dialogue in different languages, it is generally known that first, input speech in a first language is acquired, the input speech is converted into first text data in the first language, the first text data in the first language is translated into first text data in a second language, and the first text data in the second language is output. It is also known that dialogue in different languages ​​can be realized by converting the first text data in the second language into output speech in the second language and outputting it.

[0003] The two-way speech translation system described in Patent Document 1 discloses that speech translation can be performed by combining an appropriate speech recognition engine, translation engine, and speech synthesis engine according to the received speech or the language of the speech.

[0004] In this way, spoken dialogue between different languages ​​is understood to be a complex configuration, involving a large amount of data and processing, such as controlling various engines to perform translation processing between a first language and a second language, and generating speech data and first text data in the first language, as well as speech data and text data in the second language that are the translation results. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2019 / 111346 Summary of the Invention [Problem to be solved by the invention]

[0006] Processing voice data with translation requires a lot of labor and data, which means high costs in terms of processing speed and processing burden.

[0007] An object of the present invention is to provide a technique for achieving suitable processing of speech-to-text involving translation. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, the present invention provides a voice processing system comprising a voice processing device and a plurality of voice input / output devices, wherein the voice processing device comprises a text processing unit that converts input voice data into text and transmits the text data to each voice input / output device, and each of the voice input / output devices comprises an output unit that outputs based on a specified language and at least one of the text data and the conversion result of the text data.

[0009] The present invention also provides a voice processing method using a voice processing device and multiple voice input / output devices, which converts input voice data into text, transmits the text data to each voice input / output device, and outputs it on each voice input / output device based on a specified language and at least one of the text data and the conversion result of the text data.

[0010] With this configuration, it is possible to determine whether text data corresponding to voice data should be output as a conversion result depending on the designated language of each voice input / output device, thereby improving the real-time nature of the output.

[0011] In a preferred embodiment of the present invention, the voice processing device includes a voice acquisition unit that acquires voice data from the voice input / output device, and a control instruction acquisition unit that acquires control instructions for controlling a text conversion area of ​​the voice data from the voice input / output device. With this configuration, the text region can be controlled, thereby improving the accuracy and speed of output.

[0012] In a preferred form of the present invention, the output unit displays the text data or the conversion result of the text data as a chat screen on a display unit of the voice input / output device, the chat screen has a control instruction button, and the control instruction acquisition unit acquires control instructions regarding the text conversion area generated in response to an on / off operation of the control instruction button. In a preferred embodiment of the present invention, the voice input / output device includes a control instruction button, and the control instruction acquisition unit acquires a control instruction related to the text conversion area generated in response to an on / off operation of the control instruction button. With this configuration, it is possible to realize a suitable interface that allows intuitive control instructions regarding the text conversion area in response to voice input.

[0013] In a preferred embodiment of the present invention, the output unit outputs at least one of voice data transmitted from another voice input / output device and synthesized voice data corresponding to the conversion result. With this configuration, it is possible to control the desired conversion of voice data, text data, and synthetic voice data. Effect of the Invention

[0014] According to the present invention, it is possible to provide a technique for achieving suitable processing of speech-to-text conversion involving translation. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram of a system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing the hardware configuration of the present embodiment. [Diagram 3] FIG. 4 is a process sequence diagram of the present embodiment. [Figure 4] 4 is a process flowchart of a speech input / output device on the speaking side according to the embodiment. [Diagram 5] 5 is a process flowchart of a receiving side voice input / output device according to the embodiment. [Figure 6] 4 shows an example of a chat screen display according to the present embodiment. [Figure 7] 1 is a block diagram of a system according to different embodiments; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, a voice processing system according to an embodiment of the present invention will be described with reference to the drawings. Note that the embodiment described below is an example of the present invention, and the present invention is not limited to the embodiment described below, and various configurations can be adopted.

[0017] In this embodiment, the configuration, operation, etc. of a voice processing system, a voice processing device, and a voice processing program are described, but a method with a similar configuration, a computer program, and a program recording medium on which the program is recorded also have the same effects. By using a program recording medium, for example, the program can be installed in a computer. A series of processes according to this embodiment described below is provided as a computer-executable program, and can be provided via a non-transitory computer-readable recording medium such as a CD-ROM or a flexible disk, or even via a communication line.

[0018] The voice processing system is composed of a computer device. The computer device has an arithmetic device such as a CPU (Central Processing Unit) and a storage device. The computer device can function as a voice processing device by executing a voice processing program stored in the storage device with the arithmetic device. The voice processing method is realized by processing of the computer device including the voice processing device.

[0019] In this embodiment, the voice processing system controls two-way communication of voice data and text data via a plurality of voice input / output devices. For example, a first user and a second user each use a voice input / output device. When the first user and the second user each speak different languages, the language of one user is translated by a voice-to-text process via the voice processing system and output to the other user. The number of users is not limited. Furthermore, the types of languages ​​of each user and the number of translation languages ​​are not limited.

[0020] <Embodiment 1> 1 shows a block diagram of a speech processing system 1. The speech processing system 1 includes a speech processing device 2, a speech input / output device 3, and a translation device 4, and these components are communicatively connected via a communication network NW.

[0021] The voice processing device 2 has, as functional components, a voice acquisition unit 21 that acquires voice data from the voice input / output device 3, a control instruction acquisition unit 22 that acquires control instructions regarding the text conversion area of ​​the voice data from the voice input / output device 3, and a text processing unit 23 that generates text data based on the voice data and the control instructions and transmits it to each voice input / output device 3.

[0022] The voice processing device 2 receives voice data from the voice input / output device 3. The voice processing device 2 transmits text data corresponding to the received voice data to each voice input / output device 3. The voice processing device 2 may transmit the voice data to an external voice recognition device and obtain the text data from the voice recognition device.

[0023] The voice input / output device 3 includes an input unit 31 that receives voice input, and an output unit 32 that performs output based on the processing result of the voice processing device 2.

[0024] The voice input / output device 3 transmits voice data to the voice processing device 2. The voice input / output device 3 can also transmit control instructions related to the text conversion area of ​​the voice data to the voice input / output device 3. The voice input / output device 3 acquires text data from the voice processing device 2. The voice input / output device 3 transmits a translation instruction including a specified language and text data to the translation device 4, and acquires the conversion result of the text data. The voice input / output device 3 outputs based on the specified language and at least one of the text data and the conversion result of the text data.

[0025] The translation device 4 executes a process of converting the text data by translation in accordance with a translation instruction from the voice input / output device 3, and transmits the conversion result to at least the source of the translation instruction. The conversion result includes at least the text data converted from the language, and may further include synthetic voice data corresponding to the converted text data. In this embodiment, the source of the translation instruction is the voice input / output device 3, but is not limited to this, and a mode in which the source is the voice processing device 2 may also be adopted.

[0026] Fig. 2(a) shows a hardware configuration diagram of the voice processing device 2. The voice processing device 2 includes, as its hardware configuration, a control device 201, a storage device 202, and a communication device 203. In this embodiment, the voice processing device 2 may use a computer device such as a server device or a personal computer. Note that the voice processing device 2 may be configured by a plurality of computer devices, and is not limited to the configuration shown in Fig. 2(a) as long as it can realize the above-mentioned functional components (21-24) as a whole.

[0027] The control device 201 is configured with one or more processors such as a CPU, and executes a voice processing program, an OS (Operating System), and other applications to control the overall processing in the voice processing device 2. The storage device 202 is a hard disk drive (HDD), a solid state drive (SSD), a flash memory, a random access memory (RAM), etc., and stores the voice processing program and various data. The communication device 203 controls communication with the communication network NW, and realizes data communication with the voice input / output device 3, etc.

[0028] The translation device 4 may have the same hardware configuration as the speech processing device 2. In addition, the translation device 4 may be implemented in the same computer device as the speech processing device 2.

[0029] 2(b) shows a hardware configuration diagram of the audio input / output device 3. The audio input / output device 3 includes, as its hardware configuration, a control device 301, a storage device 302, a communication device 303, an input device 304, and an output device 305. In this embodiment, the audio input / output device 3 can be a smartphone, a personal computer, a tablet terminal, an earphone device, or the like.

[0030] The control device 301 is configured with one or more processors such as a CPU, and controls the overall processing in the voice input / output device 3 by executing an OS and other applications. The storage device 302 is a HDD, SSD, flash memory, RAM, etc., and stores a browser application and various data. The communication device 303 controls communication with the communication network NW, and realizes data communication with at least the voice processing device 2. The input device 304 is an input interface that accepts input operations by a user, and is configured with a touch panel, a mouse, a keyboard, etc. The output device 305 is configured with a display that displays the processing results by the control device 301, etc.

[0031] In one embodiment, the audio input / output device 3 is configured to be able to access the audio processing system 1 via a web browser. In one embodiment, the audio input / output device 3 is configured to store an audio input / output program in the storage device 302, and execute the program by the control device 301 to start an application and access the audio processing system 1. The audio input / output device 3 may also be configured to realize some of the functional components (21-23) of the audio processing device 2.

[0032] Fig. 3 illustrates a processing sequence in the voice processing system 1. In the example of Fig. 3, an example will be described in which the voice processing system 1 processes dialogue voice between a voice input / output device 3A operated by a user who uses a first language and a voice input / output device 3B operated by a user who uses a second language.

[0033] First, the voice input / output device 3A transmits voice data to the voice processing device 2 (S101). Here, the voice data is data based on an utterance in a first language. In addition, the voice input / output device 3A is capable of transmitting a control instruction for controlling a text conversion area of ​​the voice data transmitted in S101 to the voice processing device 2 (S102).

[0034] The voice processing device 2 converts the input voice data into text (S103). When a control instruction is input, the voice processing device 2 converts the voice data into text based on the text area included in the control instruction. Here, the text data converted into text is in the first language. The voice processing device 2 transmits the converted text data to each of the voice input / output devices 3A and 3B (S104A, S104B).

[0035] In S103, the voice processing device 2 may transmit an instruction to convert the input voice data into text to an external voice recognition device. The voice recognition device recognizes the voice data, converts it into text, and transmits the text data to the voice processing device 2. As a result, the voice processing device 2 obtains text data corresponding to the input voice data.

[0036] The voice input / output device 3B transmits a translation instruction including a specified language and text data to the translation device 4 (S105). Here, for example, the specified language is a second language different from the first language. The translation device 4 transmits a conversion result based on the translation instruction to the voice input / output device 3B (S106). Specifically, the translation device 4 generates text data in the second language based on the text data in the first language and the specified language in the second language included in the translation instruction. The translation device 4 transmits the text data in the second language to the voice input / output device 3B as a conversion result. Note that the translation device 4 may include synthetic voice data corresponding to the text data in the second language as a conversion result.

[0037] The voice processing device 2 transmits the voice data transmitted from the voice input / output device 3A in S101 to the voice input / output device 3B. The voice processing device 2 may receive the voice data and transmit the voice data to another voice input / output device 3, or may transmit the voice data to the voice input / output device 3 together with the text data in S104.

[0038] 3, when the speaker is the voice input / output device B, the voice input / output device 3A that acquired the text data in S104A executes the same steps as S105 and S106. Also, although only two voice input / output devices 3 are shown, even if more voice input / output devices 3 exist, the other voice input / output devices 3 execute the same steps as S105 and S106.

[0039] The voice acquisition unit 21 of the voice processing device 2 acquires voice data from the voice input / output device 3. The control instruction acquisition unit 22 acquires a control instruction for controlling a text area from the voice input / output device 3. The control instruction acquisition unit 22 generates a text area based on the acquired voice data and the control instruction. Specifically, the control instruction acquisition unit 22 sets the voice data acquired when the control instruction is in the ON state as a text area, and sets the voice data acquired when the control instruction is in the OFF state as a non-text area. The text processing unit 23 converts the acquired voice data into text and transmits the text data to each voice input / output device 3. The text processing unit 23 converts the voice data into text based on the text area included in the control instruction, and transmits the text data to each voice input / output device 3.

[0040] FIG. 4 shows an example of a process flowchart in the voice input / output device 3A on the speaking side that transmits voice data.

[0041] In this embodiment, the voice input / output device 3A of the first language user and the voice input / output device 3B of the second language user each participate in a chat room to realize the exchange of voice data and text data. Each user who participates in a chat room sets a designated language. The designated language may be linked to the user ID and automatically set.

[0042] The voice processing device 2 stores chat room information including a chat room ID, the user IDs of the participants, and chat data. The chat data holds text data corresponding to the voice data of each voice input / output device 3 as a dialogue history.

[0043] The input unit 31 of the voice input / output device 3A accepts an on / off operation of a control instruction button for controlling a text conversion area when voice is input (S201). When the control instruction button is in an on state, it is used to control the voice processing device 2 not to perform text conversion processing on the voice data transmitted, and when the control instruction button is in an off state, it is used to control the voice processing device 2 to perform text conversion processing on the voice data transmitted. The control instruction button can be configured to be switchable between on / off states by pressing, or can be configured to be in an on state (or off state) while pressed.

[0044] The input unit 31 of the voice input / output device 3A accepts voice input in the first language via the input device 304 such as a microphone (S202). The voice input / output device 3A transmits the input voice as voice data to the voice processing device 2 (S203). The voice input / output device 3A may transmit the designated language together with the voice data. The voice input / output device 3A acquires text data in the first language corresponding to the voice data in the first language transmitted in S203 from the voice processing device 2 (S204).

[0045] The output unit 32 of the voice input / output device 3A outputs via the output device 305 based on the text data in the first language (S205). Here, the output is output as voice and / or screen display. The voice output is converted into voice data in the first language by a voice synthesis process based on the text data in the first language, and is output as voice. The screen display output is output as a chat-style display based on the text data in the first language.

[0046] 5 shows an example of a processing flowchart in the voice input / output device 3B on the receiver side that receives voice data. Here, the voice input / output device 3A is operated by a user of a first language, and the voice input / output device 3B is operated by a user of a second language.

[0047] The voice input / output device 3B acquires voice data in a first language input via another voice input / output device 3A from the voice processing device 2 (S301). The voice input / output device 3B acquires text data in the first language corresponding to S301 (S302).

[0048] The voice input / output device 3B transmits a conversion instruction including the specified language set as the second language and the acquired text data of the first language to the translation device 4 (S303). The voice input / output device 3B acquires the text data of the second language as the conversion result from the translation device 4 (S304). The voice input / output device 3B may acquire synthesized voice data of the second language as the conversion result.

[0049] The output unit 32 of the voice input / output device 3B outputs via the output device 305 based on at least one of the text data in the first language and the text data in the second language which is the result of conversion of the text data in the first language (S305). Here, the output is output as a voice and / or a screen display. The voice output is converted into voice data in the second language by a voice synthesis process based on the text data in the second language, and output as a voice. The voice synthesis process can be executed by the translation device 4 or the voice input / output device 3B, and refers to a process of generating synthetic voice data. The screen display output is output as a chat-style display based on at least the text data in the second language. The screen display output may also output the text data in the first language as a chat-style display.

[0050] Fig. 6 shows examples of chat screens W10, W20, W30, and W40. Fig. 6(a) and (b) respectively show examples of chat screens W10 and W20 displayed on a voice input / output device 3A of a user of a first language (Japanese). Fig. 6(c) and (d) respectively show examples of chat screens W30 and W40 displayed on a voice input / output device 3B of a user of a second language (English). The chat screens W10, W20, W30, and W40 sequentially show screen transitions when a user of the first language speaks first, followed by a user of the second language.

[0051] In Fig. 6(a), the chat screen W10 includes a real-time display section W11 and a control instruction button W12. The real-time display section W11 displays text data that is obtained by converting voice data of a first language into text in real time. The control instruction button W12 receives an on / off operation by pressing it, and outputs a control instruction regarding the text area of ​​the voice data. In Fig. 6(a), for example, even if the actual voice data of the first user is "Um... Hello," by operating the control instruction button W12 to turn off the text area when "Hello" is spoken, only the voice data of "Hello" is converted into text.

[0052] 6(b), chat W21 is output as a chat-format display based on text data in a first language uttered by a first user. Since the first language is set as a designated language in the voice input / output device 3A, only the text data in the first language is displayed as chat W21. Note that a language designated by a participant in the chat room (e.g., a second language) may be displayed in the same manner as chat W21.

[0053] In Fig. 6(c), chat W31 corresponds to chat W21. Chat W31 displays, at least in chat format, text data in the second language, which is the result of conversion of text data in the first language, based on the second language, which is the specified language. In the illustrated example, chat W31 displays the text data in the first language, which is the source of conversion, in parallel. A real-time display unit W32 displays text data obtained by converting voice data in the second language into text in real time.

[0054] 6(d), the chat W42 is output as a chat-format display based on text data in the second language uttered by the second user. For example, the first language and the second language are set as designated languages ​​in the voice input / output device 3B, and therefore the text data in the first language and the text data in the second language are displayed as the chat W42. Note that the text data in the designated language set in the voice input / output device 3B may not be converted.

[0055] In this embodiment, each chat displayed on the chat screen corresponds to chat data including text data. The voice processing device 2 is configured to be able to output a dialogue history based on the chat data when it acquires a dialogue history output request from the voice input / output device 3. The dialogue history shows chat data including at least the speaker and the content of the conversation (text data) in chronological order, and can be used as minutes of a meeting, etc.

[0056] <Embodiment 2> Fig. 7 shows a block diagram of a speech processing system 100 according to the second embodiment. The speech processing system 100 includes a speech processing device 2, a plurality of user devices 300, and a translation device 4, and these components are communicatively connected via a communication network NW. In Fig. 7, two user devices 300 are shown as an example, but the number is not limited. In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals and will not be described.

[0057] The user device 300 includes a voice input / output device 310 and a user terminal 320, which are communicatively connected. The communicative connection may be wireless or wired. In this embodiment, the communicative connection is short-range wireless. The communication standard is not limited.

[0058] In this embodiment, the audio input / output device 310 is configured as a microphone-equipped earphone device. The audio input / output device 310 includes, as hardware components, a control device such as a CPU, a storage device such as a HDD, SSD, or RAM, a communication device, a microphone, a speaker, an input device, and the like.

[0059] The voice input / output device 310 accepts voice input from the user via a microphone. A control device of the voice input / output device 310 processes the voice input into voice data and transmits it to the user terminal 320 via a communication device. The voice input / output device 310 accepts voice data and / or synthetic voice data from the user terminal 320. A control device of the voice input / output device 310 processes the voice data and / or synthetic voice data into output voice and outputs the voice via a speaker.

[0060] In this embodiment, the audio input / output device 310 includes an input device such as one or more physical switches. The audio input / output device 310 can issue control commands to the control device, such as power on / off, a command to establish a communication connection with the user terminal 320, audio volume adjustment, mute setting, play / stop, etc., by operating the physical switch. The physical switch can be configured to issue different control commands depending on the method of operating the input, such as pressing, pressing and holding, or the number of times pressed.

[0061] The physical switch preferably includes a control instruction button for controlling the text conversion area when voice is input. The control instruction button can output a control instruction regarding the text conversion area of ​​the voice data to the user terminal via the control device by accepting an on / off operation.

[0062] In this embodiment, the user terminal 320 is configured by a computer device selected from a smartphone, a tablet terminal, a personal computer, etc. The user terminal 320 includes, as hardware components, a control device such as a CPU, a storage device such as a HDD, SSD, RAM, etc., a communication device, an input interface such as a touch panel, keyboard, mouse, etc., a display unit such as a display, etc.

[0063] The user terminal 320 transmits the voice data acquired from the voice input / output device 310 to the voice processing device 2. In addition, the user terminal 320 generates a control instruction related to the text conversion area in response to an on / off operation of a control instruction button of the voice input / output device 310, and transmits the control instruction to the voice processing device 2.

[0064] The user terminal 320 acquires, via the voice processing device 2, voice data input via another user device 300 and text data obtained by converting the voice data into text. The user terminal 320 transmits a conversion instruction to the translation device 4 based on the designated language and the text data. For example, when the designated language is a first language and the text data is a second language, the user terminal 320 transmits a conversion instruction to convert the text data in the second language into the first language. The user terminal 320 may be configured not to transmit a conversion instruction when the text data acquired from the voice processing device 2 and the designated language are the same. The user terminal 320 acquires a conversion result from the translation device 4. The conversion result includes at least one of the converted text data and synthetic voice data corresponding to the text data.

[0065] The user terminal 320 outputs based on the specified language and at least one of the text data and the conversion result of the text data. Here, the output includes a display output on a display unit of the user terminal 320. The output may also be an audio output including at least one of the voice data transmitted from another user device 300 and the synthetic voice data generated based on the conversion result. The audio output may also be an output instruction to the voice input / output device 310.

[0066] As described above, according to the present invention, real-time output is realized by processing voice data and text data that is the result of processing the voice data. The present invention realizes optimization of the conversion process by allowing the user's device to partially control whether or not conversion such as converting text data to text or translating the text data is required. [Explanation of symbols]

[0067] 1. Audio processing system 2. Audio processing device 21 Voice Acquisition Unit 22 Control instruction acquisition unit 23 Text Processing Section 3 Audio input / output device 31 Input section 32 Output section 4. Translation Device 100 Audio Processing System 300 User equipment 310 Audio input / output device 320 User terminal

Claims

1. 1. A voice processing system comprising: A voice processing device and a plurality of voice input / output devices, The voice processing device includes a text processing unit that converts input voice data into text and transmits the text data to each voice input / output device; A voice processing system, wherein each of the voice input / output devices has an output unit that outputs based on a specified language and at least one of the text data and a conversion result of the text data.

2. The voice processing device includes a voice acquisition unit that acquires voice data from the voice input / output device; 2. The voice processing system according to claim 1, further comprising: a control instruction acquisition unit that acquires, from the voice input / output device, a control instruction for controlling a text conversion area of ​​the voice data.

3. the output unit displays the text data or a result of the conversion of the text data as a chat screen on a display unit of the voice input / output device; the chat screen includes a control instruction button; The voice processing system according to claim 2 , wherein the control instruction acquisition unit acquires a control instruction related to the text conversion area generated in response to an on / off operation of the control instruction button.

4. The audio input / output device includes a control instruction button, The voice processing system according to claim 2 , wherein the control instruction acquisition unit acquires a control instruction related to the text conversion area generated in response to an on / off operation of the control instruction button.

5. The voice processing system according to any one of claims 1 to 4, wherein the output unit outputs at least one of voice data transmitted from another voice input / output device and synthesized voice data corresponding to the conversion result.

6. A voice processing method using a voice processing device and a plurality of voice input / output devices, Converting the input voice data into text and transmitting the text data to each voice input / output device; a voice processing method for outputting, in each of the voice input / output devices, a specified language and at least one of the text data and a conversion result of the text data.

Images