System

Abstract

To provide an educational and child-raising support system capable of making a child continue to be familiar and interested.SOLUTION: A voice input unit that allows a user to input a voice, a transmission unit that transmits voice data acquired by the voice input unit to a server, a voice recognition unit that converts the voice data received by the server by the transmission unit into text data, a natural language processing unit that analyzes the text data converted by the voice recognition unit to understand an intention of the user and generate a response, a voice synthesis unit that converts the response generated by the natural language processing unit into voice data of a selected character, and a voice synthesis unit that transmits the voice data generated by the voice synthesis unit to the terminal; Playback means for providing a response to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional education and childcare settings require direct instruction and dialogue from teachers, parents, or other people, but this method is limited by time and location, making it particularly difficult to respond to individual children. While there are dialogue systems that use characters that children find appealing and interesting, they face the problem of difficulty in generating real-time responses and responses that adapt to the situation. Furthermore, there are technical challenges in implementing a feature that responds in the voice of a selected character to keep children engaged. [Means for solving the problem]

[0005] To solve these problems, the present invention provides a system including a voice input means that enables a user to input voice, a transmission means that transmits voice data acquired by the voice input means to a server, a voice recognition means that converts the voice data received by the server into text data, a natural language processing means that analyzes the converted text data, understands the user's intention, and generates a response, a voice synthesis means that converts the generated response into voice data in the voice of a selected character, and a playback means that provides the generated voice data to the user.Furthermore, by including in the system a means that takes into account a character setting selected by the user in the analysis results and response generation by the natural language processing means, and a means that enables the voices to be provided in the voices of multiple different characters, it is possible to realize an education and childcare support system that is easy for children to get along with and that keeps them interested.

[0006] The "voice input means" is a device or function that receives voice uttered by the user and records it as digital data.

[0007] The "transmitting means" is a device or function for transferring the recorded voice data to a designated server.

[0008] "Speech recognition means" refers to a device or function that analyzes transmitted voice data and converts it into corresponding text data.

[0009] The "natural language processing means" is a device or function that analyzes the user's intention based on the text data generated by the speech recognition means and generates an appropriate response.

[0010] The "voice synthesis means" is a device or function that converts the response text generated by the natural language processing means into voice data in the voice of a designated character.

[0011] The "playback means" is a device or function that plays back the generated audio data to provide it to the user.

[0012] "Character settings" are information for specifying the voice and behavior of a specific character selected by the user.

[0013] A "server" is a computer system that performs processes such as analyzing voice data, converting it to text data, generating responses, and synthesizing voice data.

[0014] A "terminal" is a device through which a user inputs voice and receives and plays back responses from a server.

[0015] "User" refers to a person who utilizes the system to provide voice input and receive responses. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] System Overview

[0038] The present invention provides a system that allows users to speak, converts the speech into text, generates appropriate responses based on information obtained from the text, and plays back the responses in the voice of a character. It is designed to be used in education and childcare settings, particularly to keep children engaged.

[0039] Program processing overview

[0040] 1. Accepting user input

[0041] The device is always on standby for the user's (especially children's) voice. When the user speaks, the device records the voice and transmits it to the server in real time.

[0042] 2. Speech Recognition Processing

[0043] The server receives the voice data sent from the terminal. The received voice data is passed to a voice recognition engine and converted into text data. This text data is used in the next processing step.

[0044] 3. Context Understanding and Response Generation

[0045] The server passes the text data obtained from the speech recognition engine to a natural language processing (NLP) engine. The NLP engine analyzes the text and understands the user's intent. For example, if a user says, "Tell me about today's homework," the NLP engine generates an appropriate response to that question.

[0046] 4. Speech synthesis processing

[0047] The generated response text is passed to a speech synthesis engine, which uses the voice of the character selected by the user to convert the response text into voice data that is used to respond to the user in the character's voice.

[0048] 5. Responding to the User

[0049] The generated voice data is sent from the server to the device, which then plays the received voice data and responds to the user (child) in the voice of the selected character.

[0050] Specific examples

[0051] Conversation example 1: Helping with homework

[0052] 1. Accepting user input

[0053] The terminal waits for the user's voice to say, "Doraemon, tell me today's homework."

[0054] When the user speaks, the device records the voice and immediately sends it to the server.

[0055] 2. Speech Recognition Processing

[0056] The server receives the voice data "Doraemon, tell me today's homework," passes it to a voice recognition engine, and converts it into text data.

[0057] 3. Context Understanding and Response Generation

[0058] The server passes the text data to an NLP engine to understand the user's intent (i.e., please help me with my homework).

[0059] The NLP engine generates the response, "Today's homework is to do pages 10 to 15 of math."

[0060] 4. Speech synthesis processing

[0061] The server passes the response text to a speech synthesis engine and converts it into audio data in Doraemon's voice.

[0062] 5. Responding to the User

[0063] The server transmits the generated voice data to the terminal.

[0064] The device plays the audio data and responds to the user in Doraemon's voice, saying, "Today's homework is to do pages 10 to 15 of math."

[0065] In this way, the system of the present invention can receive a user's voice input, generate an appropriate response, and play it back in the character's voice, enabling engaging and sustainable learning and parenting support, especially for children.

[0066] The processing flow will be explained below.

[0067] Step 1:

[0068] The device is always on standby for user voice input. When the user says, "Character (e.g., Doraemon), tell me what my homework is today," the device records the voice.

[0069] Step 2:

[0070] The device immediately sends the recorded audio data to a server, and the data is often encrypted before being sent.

[0071] Step 3:

[0072] The server receives the voice data sent from the device and then passes it to a voice recognition engine for analysis.

[0073] Step 4:

[0074] The speech recognition engine converts the received voice data into text data, generating text such as "Character, please tell me today's homework."

[0075] Step 5:

[0076] The server receives the text data from the speech recognition engine and passes it to a natural language processing (NLP) engine, which analyzes the context and understands the user's intent.

[0077] Step 6:

[0078] The NLP engine generates an appropriate response based on the user's intent, in this case, a response text such as "Today's homework is to do pages 10 to 15 of math."

[0079] Step 7:

[0080] The server passes the generated response text to the speech synthesis engine, instructing it to generate voice data in the voice of the character selected by the user (e.g., Doraemon).

[0081] Step 8:

[0082] The speech synthesis engine converts the response text into speech data in the voice of the designated character, utilizing the character's distinctive speech patterns.

[0083] Step 9:

[0084] The server sends the generated voice data to the terminal, allowing the response content to be delivered to the user in real time.

[0085] Step 10:

[0086] The device plays back the received voice data and responds to the user in the character's voice, saying, "Today's homework is to do pages 10 to 15 of math." This allows the user to enjoy the character's response.

[0087] Example 1

[0088] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] In recent years, there has been a demand in education and childcare for interactive learning support systems that can sustain children's interest. However, conventional systems lack the technology to properly analyze a user's voice input, understand their intent, generate a response, and then play that response in a character's voice, limiting their ability to sustain children's interest. The present invention aims to solve this problem and provide engaging and sustained educational support, especially for children.

[0090] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0091] In this invention, the server includes voice input means that allows a user to input voice, transmission means that transmits voice data acquired by the voice input means to a processing device, voice recognition means that converts the voice data received by the processing device by the transmission means into text data, natural language processing means that analyzes the text data converted by the voice recognition means, understands the user's intention, and generates a response, voice synthesis means that converts the response generated by the natural language processing means into voice data in the voice of a selected character, and playback means that transmits the voice data generated by the voice synthesis means to a receiving device and provides the response to the user. This makes it possible to analyze the user's voice instructions and provide an appropriate response in real time in the voice of a character.

[0092] "Voice input means" refers to a device or system that allows a user to input voice, and includes a microphone and the like.

[0093] "Transmission means" refers to a function for transferring acquired voice data to a processing device or a server.

[0094] "Speech recognition means" refers to the technology or software used to analyze received voice data and convert it into text data.

[0095] "Natural language processing means" refers to technology or software that analyzes text data converted by speech recognition means, understands the user's intent, and generates an appropriate response.

[0096] "Speech synthesis means" refers to technology or software for converting the generated response into audio data in the voice of a selected character.

[0097] The "playback means" refers to a device or system that outputs sound to provide the generated sound data to the user.

[0098] "Processing device" refers to a server or computer that receives, analyzes, and converts voice data sent from a terminal.

[0099] "Receiving device" refers to a terminal or device that receives audio data transmitted from a server.

[0100] The present invention is a system for use in education and childcare that converts user-input speech into text, generates a response based on information obtained from the text, and plays back the response in the voice of a character. The system includes a speech input unit, a transmission unit, a speech recognition unit, a natural language processing unit, a speech synthesis unit, and a playback unit.

[0101] The device is always on standby for the user to speak and records the voice through a microphone. For example, when a user says, "Doraemon, please tell me today's homework," the device transmits this voice data to a processing unit in real time.

[0102] The server receives the voice data sent from the device and converts it into text data using the IBM Watson® speech recognition engine. The converted text data is analyzed using the Google® Cloud Natural Language API to understand the user's intent. As a result of this analysis, the intent, for example, "Please tell me about my homework," is recognized. Based on this information, the NLP engine generates an appropriate response text, such as "Today's homework is to do pages 10 to 15 of the math book."

[0103] The server then passes the generated response text to the Amazon Polly speech synthesis engine, which converts it into audio data in the voice of the specified character. This audio data is then sent back to the device, where it is played back through the device's speaker, allowing the user to hear the response in Doraemon's voice.

[0104] As a concrete example, consider the case where a user talks about homework. When the user says, "Doraemon, tell me today's homework," the device records this voice and sends it to the server. The server receives the voice data, converts it into text data using a speech recognition engine, then analyzes it using a natural language processing engine to generate an appropriate response. The response is then converted into Doraemon's voice using a speech synthesis engine and sent to the device, which then plays it back. As a result, the user can hear Doraemon's voice saying, "Today's homework is to do pages 10 to 15 of math."

[0105] As an example of a prompt sentence, if you say "Doraemon, tell me the weather," the following will happen:

[0106] 1. User: "Doraemon, tell me the weather."

[0107] 2. Device: Records audio using a microphone and sends it to the server

[0108] 3. Server: Receives voice data and converts it into text data using a speech recognition engine

[0109] 4. Server: Passes text data to the NLP engine, understands user intent, and generates a response

[0110] 5. Server: Passes the response text to the speech synthesis engine and converts it into audio data in Doraemon's voice

[0111] 6. Server: Sends the generated voice data to the device

[0112] 7. Device: Plays audio data through the speaker and responds to the user in Doraemon's voice, saying "Today's weather is sunny."

[0113] In this way, the system of the present invention can analyze the user's voice instructions and provide appropriate responses in real time using the character's voice, thereby enabling attractive and sustainable learning and childcare support, especially for children.

[0114] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0115] Step 1: Input reception

[0116] The user stands in front of the device and says, "Doraemon, tell me what my homework is today." The device uses a microphone to wait for the user's voice input. When voice input is made, the device records this voice data. After recording is complete, the voice data is sent to the server.

[0117] input:

[0118] User voice data

[0119] output:

[0120] Audio data sent to the server

[0121] Specific behavior:

[0122] The device records audio through a microphone and transmits the recorded data to a server in real time.

[0123] Step 2: Speech recognition processing

[0124] The server receives the voice data sent from the device. The received voice data is passed to the IBM Watson speech recognition engine. The speech recognition engine analyzes the voice waveform and converts the voice data into text data.

[0125] input:

[0126] Audio data sent from the device

[0127] output:

[0128] Converting audio data into text data

[0129] Specific behavior:

[0130] The server passes the voice data to a voice recognition engine, which generates text data through waveform analysis.

[0131] Step 3: Context understanding and response generation

[0132] The server passes the text data obtained from the speech recognition engine to the Google Cloud Natural Language API. The NLP engine breaks the text down into tokens, performs grammatical analysis, and understands the user's intent. Specifically, it recognizes the intent as "Please tell me what my homework is." Based on this information, the NLP engine generates a response text such as "Today's homework is to do pages 10 to 15 of math."

[0133] input:

[0134] Text data obtained from the speech recognition engine

[0135] output:

[0136] Generated response text

[0137] Specific behavior:

[0138] The server passes the text data to an NLP engine, which analyzes the intent and generates a response text.

[0139] Step 4: Speech synthesis processing

[0140] The server passes the generated response text to the Amazon Polly speech synthesis engine, which converts the response text into audio data in the voice of the specified character.

[0141] input:

[0142] Generated response text

[0143] output:

[0144] Audio data converted into a character's voice

[0145] Specific behavior:

[0146] The server passes the response text to a speech synthesis engine, which generates voice data in the voice of the specified character.

[0147] Step 5: Respond to the user

[0148] The server sends the generated voice data to the device, which then plays it back through the speaker, allowing the user to hear Doraemon's voice reply, "Today's homework is to do pages 10 to 15 of math."

[0149] input:

[0150] Audio data sent from the server

[0151] output:

[0152] Audio that users can hear

[0153] Specific behavior:

[0154] The device receives the voice data and plays it over the speaker, and the user hears the response in Doraemon's voice.

[0155] Through the above steps, the system of the present invention analyzes the user's voice instructions and provides appropriate responses in real time using the character's voice.

[0156] (Application example 1)

[0157] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0158] In conventional factories, workers had to manually check instructions or ask other workers to understand work content and maintenance instructions, which resulted in a decrease in efficiency. Furthermore, systems that use voice input could only provide simple voice responses, making it difficult to provide the specific and immediate instructions workers needed. This significantly reduced work efficiency within the factory, and ultimately became a factor hindering productivity improvements.

[0159] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0160] In this invention, the server includes a voice input means that enables a user to input voice data, a transmission means that transmits voice data acquired by the voice input means to the server, a voice recognition means that converts the voice data received by the server by the transmission means into text data, a natural language processing means that analyzes the text data converted by the voice recognition means, understands the user's intention, and generates a response, a voice synthesis means that converts the response generated by the natural language processing means into voice data in the voice of a selected character, a playback means that transmits the voice data generated by the voice synthesis means to a terminal and provides the response to the user, and a means that recognizes voice input from a field worker and provides the character's response to the work instructions or maintenance questions. This enables work instructions and maintenance instructions to be given quickly and efficiently within a factory.

[0161] The "voice input means" is a device or system that allows a user to input voice.

[0162] The "transmitting means" is a device or system for transmitting the acquired voice data to the server.

[0163] A "voice recognition means" is a device or system for converting received voice data into text data.

[0164] A "natural language processing means" is a device or system that analyzes text data, understands the user's intent, and generates an appropriate response.

[0165] A "voice synthesis means" is a device or system that converts the generated response into voice data in the voice of a selected character.

[0166] A "playback means" is a device or system for providing the generated audio data to a user.

[0167] "Field worker" means a worker who performs work in a factory or work site.

[0168] "Work instructions" are instructions or guidance for carrying out a specific task or process.

[0169] "Maintenance questions" are questions about the maintenance and repair of machines and equipment.

[0170] A "character response" is a verbal response provided by the selected character.

[0171] This invention is a system for improving the efficiency of work instructions and maintenance in a factory. A user makes a voice inquiry, and an appropriate response is provided in the character's voice.

[0172] The main components of the system are:

[0173] 1. Voice input method

[0174] The microphones installed at each work station in the factory are used as devices for users to input voice, allowing workers to input voice in a hands-free manner.

[0175] 2. Transmission Method

[0176] The voice data acquired by the voice input means is transmitted to the server in real time via Wi-Fi or a wired network.

[0177] 3. Voice Recognition Methods

[0178] The server converts the received audio data into text data using the Google Cloud Speech-to-Text API, which prepares the audio data for analysis as text data.

[0179] 4. Natural Language Processing Methods

[0180] The text data is analyzed using the Google Cloud Natural Language API to understand the user's intent and generate an appropriate response, with the use of generative AI models to enable more accurate responses.

[0181] 5. Speech synthesis means

[0182] The generated response text is converted into audio data in a character's voice using the Google Cloud Text-to-Speech API. This character's voice is pre-set and can be selected from multiple options.

[0183] 6. Regeneration means

[0184] The resulting audio data is played back through the factory's speaker system to provide a response to the user, allowing workers to instantly obtain the information they need.

[0185] Specific examples

[0186] 1. User Input

[0187] The worker speaks into the microphone, asking, "What's the next step?" This voice is input into the system via the microphone.

[0188] 2. Voice Recognition

[0189] The input voice data is sent to the server and converted into text data such as "What's the next step?" using the Google Cloud Speech-to-Text API.

[0190] 3. Natural Language Processing

[0191] The text data is parsed by the Google Cloud Natural Language API, which understands that the request is for instructions on the next step, and the generative AI model generates a response such as, "The next step is to assemble the parts."

[0192] 4. Speech Synthesis

[0193] The generated response text is converted into audio data using the Google Cloud Text-to-Speech API, and a character's voice is generated saying, "The next step is assembling the parts."

[0194] 5. Responding to the User

[0195] The final audio data is played back through factory speakers and provided to workers.

[0196] Prompt Sentence Examples

[0197] User input: "What's next?"

[0198] This system will enable work instructions and maintenance instructions within the factory to be given quickly and efficiently, which is expected to improve work efficiency.

[0199] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0200] Step 1:

[0201] The user inputs voice using the voice input means. Specifically, a factory worker speaks into the microphone, saying, "What's the next process?" This voice data is acquired and sent to the server via the transmission means.

[0202] Input: Voice data (e.g., "What's the next step?")

[0203] Output: Audio data sent to the server

[0204] Step 2:

[0205] The server passes the received voice data to the speech recognition means and converts it into text data using the Google Cloud Speech-to-Text API, which converts the voice data into a string of characters.

[0206] Input: Audio data

[0207] Output: Text data (e.g. "What's the next step?")

[0208] How it works: Converts audio into text using the Google Cloud Speech-to-Text API.

[0209] Step 3:

[0210] The server passes the text data to a natural language processor, which uses the Google Cloud Natural Language API to analyze the text and understand the user's intent. A generative AI model is then used to generate an appropriate response to the user's question.

[0211] Input: Text data (e.g., "What's the next step?")

[0212] Output: Response text data (e.g. "The next step is assembling the parts.")

[0213] How it works: Uses the Google Cloud Natural Language API for text analysis and response generation.

[0214] Step 4:

[0215] The server passes the generated response text to the speech synthesis means, which converts it into audio data in the character's voice using the Google Cloud Text-to-Speech API, which converts the text into audio.

[0216] Input: Response text data (e.g., "The next step is assembling the parts.")

[0217] Output: Character voice data

[0218] How it works: Speech synthesis is performed using the Google Cloud Text-to-Speech API.

[0219] Step 5:

[0220] The server transmits the generated voice data to the terminal and uses a playback means to play back the response from the factory speaker system, allowing the worker to receive voice instructions regarding the next process.

[0221] Input: Character voice data

[0222] Output: Audio played through the speaker (e.g. "The next step is to assemble the parts.")

[0223] Action: Plays audio data on the device.

[0224] As described above, by executing each step consecutively, work instructions within the factory can be issued efficiently.

[0225] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0226] System Overview

[0227] This invention provides a system that allows users to input speech, converts the speech into text, generates appropriate responses based on information obtained from the text, and plays back the responses in a character's voice. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and generates responses based on those emotions, this system achieves more friendly and effective dialogue. It is designed to be used in education and childcare settings, particularly to keep children engaged.

[0228] Program processing overview

[0229] 1. Accepting user input

[0230] The device is always on standby for the user's (especially children's) voice. When the user speaks, the device records the voice and transmits it to the server in real time.

[0231] 2. Speech Recognition Processing

[0232] The server receives the voice data sent from the terminal. The received voice data is passed to a voice recognition engine and converted into text data. This text data is used in the next processing step.

[0233] 3. Emotion Recognition Processing

[0234] The server passes the text data obtained from the speech recognition engine to the emotion engine, which analyzes the text data and voice data and recognizes the user's emotional state (e.g., joy, sadness, anger, etc.).

[0235] 4. Context Understanding and Response Generation

[0236] The server passes the text data from the speech recognition engine to a natural language processing (NLP) engine, taking into account the user's emotional state obtained from the emotion engine. The NLP engine analyzes the text and understands the user's intent. For example, if a user says, "What is today's homework?", the NLP engine generates an appropriate response to the question: "Today's homework is to do pages 10 to 15 of math."

[0237] 5. Speech synthesis processing

[0238] The generated response text is passed to a speech synthesis engine, which converts the response text into audio data using the voice of a character selected by the user, while simultaneously adjusting the tone and expression of the voice to take into account the user's emotional state.

[0239] 6. Responding to the User

[0240] The generated voice data is sent from the server to the device. The device plays the received voice data and responds to the user (child) in the voice of the selected character. For example, the device could respond in Doraemon's voice, saying, "Today's homework is to do pages 10 to 15 of math," and if the user is feeling a little down, it could play an encouraging tone.

[0241] Specific examples

[0242] Conversation example 1: Helping with homework

[0243] 1. Accepting user input

[0244] The device waits for the user to say, "Doraemon, tell me what my homework is today." When the user speaks, the device records the voice and immediately sends it to the server.

[0245] 2. Speech Recognition Processing

[0246] The server receives the voice data "Doraemon, tell me today's homework," passes it to a voice recognition engine, and converts it into text data.

[0247] 3. Emotion Recognition Processing

[0248] The server passes the text data and voice data obtained from the voice recognition engine to the emotion engine, and recognizes the user's emotion (e.g., depressed).

[0249] 4. Context Understanding and Response Generation

[0250] The server passes the text data to the NLP engine based on the emotional information obtained from the emotion engine, understands the user's intention (please help me with my homework), and generates a response: "Today's homework is to do pages 10 to 15 of math."

[0251] 5. Speech synthesis processing

[0252] The server passes the response text to a speech synthesis engine, which converts it into audio data in Doraemon's voice, generating a cheerful tone to encourage the depressed user.

[0253] 6. Responding to the User

[0254] The server then sends the generated voice data to the device, which then plays it back, responding in Doraemon's cheerful voice, "Today's homework is to do pages 10 to 15 of math."

[0255] As described above, the present invention realizes a dialogue response system that takes emotions into consideration, thereby making it possible to provide users with friendly and effective educational and child-rearing support.

[0256] The processing flow will be explained below.

[0257] Step 1:

[0258] The device is always on standby for user voice input. When the user says, "Character (e.g., Doraemon), tell me what my homework is today," the device records the voice.

[0259] Step 2:

[0260] The device immediately sends the recorded audio data to a server, and the data is often encrypted before being sent.

[0261] Step 3:

[0262] The server receives the voice data sent from the device and then passes it to a voice recognition engine for analysis.

[0263] Step 4:

[0264] The speech recognition engine converts the received voice data into text data, generating text such as "Character, please tell me today's homework."

[0265] Step 5:

[0266] The server passes the text and voice data obtained from the speech recognition engine to the emotion engine, which analyzes the text and voice characteristics and recognizes the user's emotional state (e.g., joy, sadness, anger, etc.).

[0267] Step 6:

[0268] The server passes the text data from the speech recognition engine to the natural language processing (NLP) engine based on the user's emotional information obtained from the emotion engine. The NLP engine analyzes the text and understands the user's intent.

[0269] Step 7:

[0270] The NLP engine generates an appropriate response based on the user's intent and emotional information, in this case, a response text such as "Today's homework is to do pages 10 to 15 of math."

[0271] Step 8:

[0272] The server then passes the generated response text to the speech synthesis engine, which instructs it to generate voice data in the voice of a character selected by the user (e.g., Doraemon).The tone and expression of the voice are also adjusted according to the user's emotional state.

[0273] Step 9:

[0274] The speech synthesis engine converts the response text into voice data in the voice of the specified character, taking into account the user's emotions and using encouraging or gentle tones.

[0275] Step 10:

[0276] The server sends the generated voice data to the terminal, allowing the response content to be delivered to the user in real time.

[0277] Step 11:

[0278] The device plays back the received voice data and responds to the user in the character's voice, saying, "Today's homework is to do pages 10 to 15 of math." By playing back the voice in a tone that corresponds to the user's emotions, a more friendly dialogue is achieved.

[0279] Example 2

[0280] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0281] Conventional voice response systems generate responses without considering the user's emotions, which makes it difficult to realize more friendly and effective dialogue. Particularly in education and childcare settings, responses that respond to the user's emotions are required, but no system has been able to meet this demand. Furthermore, systems have been limited in their ability to provide responses in the voices of multiple characters, lacking the flexibility to allow users to choose from a variety of options.

[0282] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice input means, a transmission means, a voice recognition means, an emotion recognition means, a natural language processing means, a voice synthesis means, and a playback means. This makes it possible to recognize the user's emotional state and generate a response according to the emotion. Furthermore, by dynamically providing responses using the voices of multiple characters, it is possible to provide the user with a more familiar and effective dialogue experience.

[0283] "Voice input means" refers to a device or function that allows a user to input voice.

[0284] "Transmitting means" refers to a device or function for transmitting acquired voice data to a server.

[0285] "Speech recognition means" refers to a device or function for converting received voice data into text data.

[0286] "Emotion recognition means" refers to a device or function for analyzing the text data and voice data converted by the voice recognition means and recognizing the emotional state of the user.

[0287] "Natural language processing means" refers to a device or function that analyzes text data taking into account the emotional state recognized by the emotion recognition means, understands the user's intention, and generates a response.

[0288] "Speech synthesis means" refers to a device or function for converting a response generated by a natural language processing means into voice data in the voice of a selected character.

[0289] "Playback means" refers to a device or function for transmitting generated voice data to a terminal and providing a response to a user.

[0290] This invention provides a system that accepts user voice input, converts the speech into text, generates appropriate responses based on information obtained from the text, and plays the responses in a character's voice. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and generates responses based on those emotions, it achieves more friendly and effective dialogue. This system is specifically designed for use in education and childcare settings.

[0291] System Configuration

[0292] The system includes the following means:

[0293] 1. Voice input means: A device or function that allows a user to input voice. For example, when a user wants to ask a question, they can speak into the terminal.

[0294] 2. Transmission means: A device or function for transmitting acquired voice data to a server. The voice data acquired by the voice input means is transmitted to the server in real time.

[0295] 3. Speech recognition means: A device or function for converting received voice data into text data. Specifically, the voice data is converted into text data using the Google Cloud Speech-to-Text API.

[0296] 4. Emotion Recognition Means: A device or function for analyzing the text data and voice data converted by the voice recognition means and recognizing the user's emotional state. The Microsoft® Azure® Emotion API is used to analyze the user's emotions.

[0297] 5. Natural language processing means: A device or function that analyzes text data taking into account the emotional state recognized by the emotion recognition means, understands the user's intention, and generates a response. OpenAI's (registered trademark) GPT-4 (registered trademark) is used to generate appropriate responses to user questions.

[0298] 6. Speech synthesis means: A device or function for converting the response generated by the natural language processing means into voice data in the voice of the selected character. Amazon Polly is used to convert into voice data in the voice of the selected character.

[0299] 7. Playback means: A device or function for transmitting the generated voice data to the terminal and providing a response to the user. The generated voice is played back through the terminal's speaker.

[0300] Specific processing operations

[0301] The specific processing operations of this system are shown below.

[0302] 1. Accepting user input

[0303] The device is always on standby for user voice input. For example, if a user says to the device, "Doraemon, tell me what my homework is today," the device records this voice and sends it to the server in real time.

[0304] 2. Speech Recognition Processing

[0305] The server receives the voice data sent from the device, passes it to the Google Cloud Speech-to-Text API, and converts it into text data, which is used for the next processing step.

[0306] 3. Emotion Recognition Processing

[0307] The server passes the text and voice data acquired by the speech recognition tool to the Microsoft Azure Emotion API, which analyzes the data and recognizes the user's emotional state. For example, it may recognize that the user is feeling a little depressed.

[0308] 4. Context Understanding and Response Generation

[0309] The server passes the text data from the speech recognition engine to OpenAI's GPT-4, taking into account the user's emotional state obtained from the emotion recognition engine. The NLP engine analyzes the text and understands the user's intent. For example, in response to the question "What is today's homework?", it generates a response such as "Today's homework is pages 10 to 15 of math."

[0310] 5. Speech synthesis processing

[0311] The generated response text is passed to Amazon Polly, which converts it into voice data in the voice of a character selected by the user, such as "Doraemon's voice."

[0312] 6. Responding to the User

[0313] The server sends the generated voice data to the device, which then plays the received voice data from its speaker and responds to the user in Doraemon's voice, saying, "Today's homework is to do pages 10 to 15 of math."

[0314] Specific examples

[0315] For example, if a user says, "Doraemon, tell me my homework today," the above process will result in a response in Doraemon's voice saying, "Today's homework is to do pages 10 to 15 of math." If the user is feeling down, the system can respond in a positive, encouraging tone.

[0316] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0317] Step 1: Accepting user input

[0318] The device is always on standby for user voice input. When the user says, "Doraemon, tell me what my homework is today," the device records the voice.

[0319] Input: User voice input

[0320] How it works: The device's built-in microphone captures and records audio. The voice recognition chip detects what the user is saying and starts the recording process.

[0321] Output: Recorded audio data

[0322] Step 2: Sending audio data

[0323] The recorded audio data is sent to the server in real time.

[0324] Input: Audio data recorded on the device

[0325] What it does: Uses the device's communications module to send audio data to a server over the internet.

[0326] Output: Audio data sent to the server

[0327] Step 3: Speech recognition processing

[0328] The server passes the received audio data to the Google Cloud Speech-to-Text API, which converts the audio data into text data, which is used for the next processing step.

[0329] Input: Audio data sent to the server

[0330] How it works: The server sends the audio data to the Google Cloud Speech-to-Text API, which uses a voice filtering function to remove background noise, then analyzes the audio data and converts it into text.

[0331] Output: Converted text data (e.g. "Doraemon, tell me today's homework.")

[0332] Step 4: Emotion recognition processing

[0333] The server passes the text and voice data acquired by the voice recognition means to the Microsoft Azure Emotion API, which analyzes the user's emotional state.

[0334] Input: Text data and audio data converted by speech recognition

[0335] How it works: The server sends the data to the Microsoft Azure Emotion API, which analyzes the linguistic features of the audio waveform data and text data to recognize the user's emotional state (e.g., depression).

[0336] Output: User's emotional state (e.g., "depressed")

[0337] Step 5: Context understanding and response generation

[0338] The server then passes the text data based on the emotion and speech recognition results to OpenAI's GPT-4, which analyzes the text, understands the user's intent, and generates an appropriate response.

[0339] Input: User's emotional state and text data obtained by speech recognition means

[0340] How it works: The server inputs emotional state and text data as prompts into GPT-4, which generates an appropriate response to the user's question (e.g., "Today's homework is to do pages 10 to 15 of math").

[0341] Output: The generated response text

[0342] Step 6: Speech synthesis processing

[0343] The generated response text is passed to Amazon Polly, which converts it into voice data in the voice of the character selected by the user.

[0344] Input: GPT-4 generated response text

[0345] How it works: The server sends the response text to Amazon Polly, which converts it into speech data in the voice of a selected character (e.g., Doraemon), while also adjusting the tone of the voice to take into account emotional information.

[0346] Output: Generated audio data

[0347] Step 7: Respond to the user

[0348] The generated voice data is transmitted from the server to the terminal, which then plays the received voice data and provides a response to the user.

[0349] Input: Audio data sent from the server

[0350] How it works: The device's speaker plays audio data, and responds in a cheerful, encouraging tone, for example, in Doraemon's voice, saying, "Today's homework is to do pages 10 to 15 of math."

[0351] Output: A spoken response to the user

[0352] (Application example 2)

[0353] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0354] Modern work environments, especially logistics centers, require efficient work performance and reduced employee fatigue. Conversational support systems are useful for enabling workers to work efficiently while taking appropriate breaks. However, conventional dialogue systems have limitations in recognizing users' emotions and are sometimes unable to provide appropriate responses. Furthermore, there is a lack of systems that can properly analyze users' voice input and provide appropriate instructions and support information in real time.

[0355] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a voice input means that enables a user to input voice; a transmission means that transmits voice data acquired by the voice input means to the server; a voice recognition means that converts the voice data received by the server by the transmission means into text data; a natural language processing means that analyzes the text data converted by the voice recognition means, understands the user's intention, and generates a response; a voice synthesis means that converts the response generated by the natural language processing means into voice data in the voice of a selected character; a playback means that transmits the voice data generated by the voice synthesis means to a terminal and provides the response to the user; and an emotion recognition means that analyzes the user's emotional state and adjusts the response based on the analysis results. This allows appropriate instructions and support to be provided in real time according to the user's emotional state, improving work efficiency and reducing worker fatigue.

[0356] The "voice input means" is a device that allows the user to input voice (for example, a microphone).

[0357] The "transmission means" is a means having a function of transmitting the acquired voice data to the server.

[0358] The "voice recognition means" is a system that converts received voice data into text data.

[0359] "Natural language processing means" is a function that analyzes text data, understands the user's intent, and generates an appropriate response.

[0360] The "voice synthesis means" is a system that converts the generated response text into voice data in the voice of a selected character.

[0361] The "playback means" is a function that transmits the generated voice data to the terminal and provides a response to the user.

[0362] The "emotion recognition means" is a system that analyzes the user's voice data and text data to analyze the user's emotional state.

[0363] This invention provides a system that allows a user to receive work instructions using voice input. The system picks up the user's voice using voice input means and transmits the voice data to a server via transmission means. The server converts the received voice data into text data using voice recognition means and analyzes it using natural language processing means to understand the user's instructions. It also analyzes the user's emotional state using emotion recognition means and generates a response in an appropriate tone. The response is converted into voice data in the voice of a specified character by voice synthesis means and transmitted to a terminal via playback means.

[0364] Hardware and software used

[0365] Audio input method: Microphone

[0366] Transmission method: Network module or communication adapter

[0367] Speech recognition method: Google Speech-to-Text API

[0368] Natural language processing tools: Various NLP libraries (e.g. spaCy, NLTK, etc.)

[0369] Emotion recognition: sentiment_analysis_spanish library

[0370] Speech synthesis method: Google Text-to-Speech API

[0371] Playback via: Speakers, smart glasses, and smartphones

[0372] System processing flow

[0373] The server acquires the user's speech input, performs speech recognition, and analyzes the resulting text data. The analyzed text data is passed to a natural language processing means, which understands the user's intention and generates an appropriate response. The emotion recognition means analyzes the user's emotional state and adjusts the content and tone of the response. The generated response text is converted into voice data by a voice synthesis means and transmitted to the terminal via a playback means.

[0374] Specific examples

[0375] When a worker puts on the smart glasses and says, "What are the next work instructions?", the voice is recognized by the system and sent to the server. The server converts the voice into text data, performs emotion recognition, and generates the next work instruction using natural language processing. The generated response is synthesized in the character's voice and provided to the worker through the smart glasses. For example, if the worker is tired, instructions such as "Next, please pick up container B. Please take a break while working" are provided in an encouraging tone.

[0376] Prompt Sentence Examples

[0377] "Please tell me the next work instructions."

[0378] Check the current progress

[0379] "Please tell me when to take a break."

[0380] In this way, appropriate instructions and support can be provided in real time according to the user's emotional state, improving work efficiency and reducing worker fatigue.

[0381] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0382] Step 1:

[0383] The user inputs the voice. The user inputs the voice through a microphone. For example, the user might say, "Please tell me the next work instruction." This voice data is generated and sent to the next step.

[0384] Step 2:

[0385] The terminal transmits the user's voice data to the server via a transmission means. The voice data is input and transferred to the server as output. During this process, data compression and encryption may be performed.

[0386] Step 3:

[0387] The server receives the voice data. It converts the received voice data into text data using a speech recognition tool. The input is voice data, and the output is text data. This conversion is specifically performed using the Google Speech-to-Text API.

[0388] Step 4:

[0389] The server passes the converted text data to a natural language processing means for analysis. The input is text data, and the output is response data that understands the user's intent. In natural language processing, a specific NLP library (e.g., spaCy or NLTK) is used to analyze the user's intent.

[0390] Step 5:

[0391] The server passes the text data and voice data to the emotion recognition means to recognize the user's emotional state. The input is text data and voice data, and the output is emotional state information. The sentiment_analysis_spanish library is used for emotion recognition.

[0392] Step 6:

[0393] The server generates an appropriate response taking into account the emotional state and passes the response text to a speech synthesis means. The input is the response text and emotional state information, and the output is audio data. The response text is converted into audio using the Google Text-to-Speech API.

[0394] Step 7:

[0395] The server transmits the generated voice data to the terminal. The terminal uses a playback means to play the voice data to the user. Specifically, the voice is played using smart glasses or a smartphone. The input is the voice data, and the output is the voice heard by the user.

[0396] Step 8:

[0397] The user receives the voice instructions and performs the next task. For example, if the instruction is "Next, pick up container B. Please take a break while working," the user will continue working in accordance with the instructions.

[0398] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0399] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0400] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0401] [Second embodiment]

[0402] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0403] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0404] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0405] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0406] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0407] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0408] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0409] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0410] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0411] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0412] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0413] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0414] System Overview

[0415] The present invention provides a system that allows users to speak, converts the speech into text, generates appropriate responses based on information obtained from the text, and plays back the responses in the voice of a character. It is designed to be used in education and childcare settings, particularly to keep children engaged.

[0416] Program processing overview

[0417] 1. Accepting user input

[0418] The device is always on standby for the user's (especially children's) voice. When the user speaks, the device records the voice and transmits it to the server in real time.

[0419] 2. Speech Recognition Processing

[0420] The server receives the voice data sent from the terminal. The received voice data is passed to a voice recognition engine and converted into text data. This text data is used in the next processing step.

[0421] 3. Context Understanding and Response Generation

[0422] The server passes the text data obtained from the speech recognition engine to a natural language processing (NLP) engine. The NLP engine analyzes the text and understands the user's intent. For example, if a user says, "Tell me about today's homework," the NLP engine generates an appropriate response to that question.

[0423] 4. Speech synthesis processing

[0424] The generated response text is passed to a speech synthesis engine, which uses the voice of the character selected by the user to convert the response text into voice data that is used to respond to the user in the character's voice.

[0425] 5. Responding to the User

[0426] The generated voice data is sent from the server to the device, which then plays the received voice data and responds to the user (child) in the voice of the selected character.

[0427] Specific examples

[0428] Conversation example 1: Helping with homework

[0429] 1. Accepting user input

[0430] The terminal waits for the user's voice to say, "Doraemon, tell me today's homework."

[0431] When the user speaks, the device records the voice and immediately sends it to the server.

[0432] 2. Speech Recognition Processing

[0433] The server receives the voice data "Doraemon, tell me today's homework," passes it to a voice recognition engine, and converts it into text data.

[0434] 3. Context Understanding and Response Generation

[0435] The server passes the text data to an NLP engine to understand the user's intent (i.e., please help me with my homework).

[0436] The NLP engine generates the response, "Today's homework is to do pages 10 to 15 of math."

[0437] 4. Speech synthesis processing

[0438] The server passes the response text to a speech synthesis engine and converts it into audio data in Doraemon's voice.

[0439] 5. Responding to the User

[0440] The server transmits the generated voice data to the terminal.

[0441] The device plays the audio data and responds to the user in Doraemon's voice, saying, "Today's homework is to do pages 10 to 15 of math."

[0442] In this way, the system of the present invention can receive a user's voice input, generate an appropriate response, and play it back in the character's voice, enabling engaging and sustainable learning and parenting support, especially for children.

[0443] The processing flow will be explained below.

[0444] Step 1:

[0445] The device is always on standby for user voice input. When the user says, "Character (e.g., Doraemon), tell me what my homework is today," the device records the voice.

[0446] Step 2:

[0447] The device immediately sends the recorded audio data to a server, and the data is often encrypted before being sent.

[0448] Step 3:

[0449] The server receives the voice data sent from the device and then passes it to a voice recognition engine for analysis.

[0450] Step 4:

[0451] The speech recognition engine converts the received voice data into text data, generating text such as "Character, please tell me today's homework."

[0452] Step 5:

[0453] The server receives the text data from the speech recognition engine and passes it to a natural language processing (NLP) engine, which analyzes the context and understands the user's intent.

[0454] Step 6:

[0455] The NLP engine generates an appropriate response based on the user's intent, in this case, a response text such as "Today's homework is to do pages 10 to 15 of math."

[0456] Step 7:

[0457] The server passes the generated response text to the speech synthesis engine, instructing it to generate voice data in the voice of the character selected by the user (e.g., Doraemon).

[0458] Step 8:

[0459] The speech synthesis engine converts the response text into speech data in the voice of the designated character, utilizing the character's distinctive speech patterns.

[0460] Step 9:

[0461] The server sends the generated voice data to the terminal, allowing the response content to be delivered to the user in real time.

[0462] Step 10:

[0463] The device plays back the received voice data and responds to the user in the character's voice, saying, "Today's homework is to do pages 10 to 15 of math." This allows the user to enjoy the character's response.

[0464] Example 1

[0465] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0466] In recent years, there has been a demand in education and childcare for interactive learning support systems that can sustain children's interest. However, conventional systems lack the technology to properly analyze a user's voice input, understand their intent, generate a response, and then play that response in a character's voice, limiting their ability to sustain children's interest. The present invention aims to solve this problem and provide engaging and sustained educational support, especially for children.

[0467] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0468] In this invention, the server includes voice input means that allows a user to input voice, transmission means that transmits voice data acquired by the voice input means to a processing device, voice recognition means that converts the voice data received by the processing device by the transmission means into text data, natural language processing means that analyzes the text data converted by the voice recognition means, understands the user's intention, and generates a response, voice synthesis means that converts the response generated by the natural language processing means into voice data in the voice of a selected character, and playback means that transmits the voice data generated by the voice synthesis means to a receiving device and provides the response to the user. This makes it possible to analyze the user's voice instructions and provide an appropriate response in real time in the voice of a character.

[0469] "Voice input means" refers to a device or system that allows a user to input voice, and includes a microphone and the like.

[0470] "Transmission means" refers to a function for transferring acquired voice data to a processing device or a server.

[0471] "Speech recognition means" refers to the technology or software used to analyze received voice data and convert it into text data.

[0472] "Natural language processing means" refers to technology or software that analyzes text data converted by speech recognition means, understands the user's intent, and generates an appropriate response.

[0473] "Speech synthesis means" refers to technology or software for converting the generated response into audio data in the voice of a selected character.

[0474] The "playback means" refers to a device or system that outputs sound to provide the generated sound data to the user.

[0475] "Processing device" refers to a server or computer that receives, analyzes, and converts voice data sent from a terminal.

[0476] "Receiving device" refers to a terminal or device that receives audio data transmitted from a server.

[0477] The present invention is a system for use in education and childcare that converts user-input speech into text, generates a response based on information obtained from the text, and plays back the response in the voice of a character. The system includes a speech input unit, a transmission unit, a speech recognition unit, a natural language processing unit, a speech synthesis unit, and a playback unit.

[0478] The device is always on standby for the user to speak and records the voice through a microphone. For example, when a user says, "Doraemon, please tell me today's homework," the device transmits this voice data to a processing unit in real time.

[0479] The server receives the voice data sent from the device and converts it into text using the IBM Watson speech recognition engine. The converted text data is then analyzed using the Google Cloud Natural Language API to understand the user's intent. As a result of this analysis, the intent may be recognized as, for example, "Please tell me about my homework." Based on this information, the NLP engine generates an appropriate response text, such as, "Today's homework is to do pages 10 to 15 of math."

[0480] The server then passes the generated response text to the Amazon Polly speech synthesis engine, which converts it into audio data in the voice of the specified character. This audio data is then sent back to the device, where it is played back through the device's speaker, allowing the user to hear the response in Doraemon's voice.

[0481] As a concrete example, consider the case where a user talks about homework. When the user says, "Doraemon, tell me today's homework," the device records this voice and sends it to the server. The server receives the voice data, converts it into text data using a speech recognition engine, then analyzes it using a natural language processing engine to generate an appropriate response. The response is then converted into Doraemon's voice using a speech synthesis engine and sent to the device, which then plays it back. As a result, the user can hear Doraemon's voice saying, "Today's homework is to do pages 10 to 15 of math."

[0482] As an example of a prompt sentence, if you say "Doraemon, tell me the weather," the following will happen:

[0483] 1. User: "Doraemon, tell me the weather."

[0484] 2. Device: Records audio using a microphone and sends it to the server

[0485] 3. Server: Receives voice data and converts it into text data using a speech recognition engine

[0486] 4. Server: Passes text data to the NLP engine, understands user intent, and generates a response

[0487] 5. Server: Passes the response text to the speech synthesis engine and converts it into audio data in Doraemon's voice

[0488] 6. Server: Sends the generated voice data to the device

[0489] 7. Device: Plays audio data through the speaker and responds to the user in Doraemon's voice, saying "Today's weather is sunny."

[0490] In this way, the system of the present invention can analyze the user's voice instructions and provide appropriate responses in real time using the character's voice, thereby enabling attractive and sustainable learning and childcare support, especially for children.

[0491] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0492] Step 1: Input reception

[0493] The user stands in front of the device and says, "Doraemon, tell me what my homework is today." The device uses a microphone to wait for the user's voice input. When voice input is made, the device records this voice data. After recording is complete, the voice data is sent to the server.

[0494] input:

[0495] User voice data

[0496] output:

[0497] Audio data sent to the server

[0498] Specific behavior:

[0499] The device records audio through a microphone and transmits the recorded data to a server in real time.

[0500] Step 2: Speech recognition processing

[0501] The server receives the voice data sent from the device. The received voice data is passed to the IBM Watson speech recognition engine. The speech recognition engine analyzes the voice waveform and converts the voice data into text data.

[0502] input:

[0503] Audio data sent from the device

[0504] output:

[0505] Converting audio data into text data

[0506] Specific behavior:

[0507] The server passes the voice data to a voice recognition engine, which generates text data through waveform analysis.

[0508] Step 3: Context understanding and response generation

[0509] The server passes the text data obtained from the speech recognition engine to the Google Cloud Natural Language API. The NLP engine breaks the text down into tokens, performs grammatical analysis, and understands the user's intent. Specifically, it recognizes the intent as "Please tell me what my homework is." Based on this information, the NLP engine generates a response text such as "Today's homework is to do pages 10 to 15 of math."

[0510] input:

[0511] Text data obtained from the speech recognition engine

[0512] output:

[0513] Generated response text

[0514] Specific behavior:

[0515] The server passes the text data to an NLP engine, which analyzes the intent and generates a response text.

[0516] Step 4: Speech synthesis processing

[0517] The server passes the generated response text to the Amazon Polly speech synthesis engine, which converts the response text into audio data in the voice of the specified character.

[0518] input:

[0519] Generated response text

[0520] output:

[0521] Audio data converted into a character's voice

[0522] Specific behavior:

[0523] The server passes the response text to a speech synthesis engine, which generates voice data in the voice of the specified character.

[0524] Step 5: Respond to the user

[0525] The server sends the generated voice data to the device, which then plays it back through the speaker, allowing the user to hear Doraemon's voice reply, "Today's homework is to do pages 10 to 15 of math."

[0526] input:

[0527] Audio data sent from the server

[0528] output:

[0529] Audio that users can hear

[0530] Specific behavior:

[0531] The device receives the voice data and plays it over the speaker, and the user hears the response in Doraemon's voice.

[0532] Through the above steps, the system of the present invention analyzes the user's voice instructions and provides appropriate responses in real time using the character's voice.

[0533] (Application example 1)

[0534] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0535] In conventional factories, workers had to manually check instructions or ask other workers to understand work content and maintenance instructions, which resulted in a decrease in efficiency. Furthermore, systems that use voice input could only provide simple voice responses, making it difficult to provide the specific and immediate instructions workers needed. This significantly reduced work efficiency within the factory, and ultimately became a factor hindering productivity improvements.

[0536] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0537] In this invention, the server includes a voice input means that enables a user to input voice data, a transmission means that transmits voice data acquired by the voice input means to the server, a voice recognition means that converts the voice data received by the server by the transmission means into text data, a natural language processing means that analyzes the text data converted by the voice recognition means, understands the user's intention, and generates a response, a voice synthesis means that converts the response generated by the natural language processing means into voice data in the voice of a selected character, a playback means that transmits the voice data generated by the voice synthesis means to a terminal and provides the response to the user, and a means that recognizes voice input from a field worker and provides the character's response to the work instructions or maintenance questions. This enables work instructions and maintenance instructions to be given quickly and efficiently within a factory.

[0538] The "voice input means" is a device or system that allows a user to input voice.

[0539] The "transmitting means" is a device or system for transmitting the acquired voice data to the server.

[0540] A "voice recognition means" is a device or system for converting received voice data into text data.

[0541] A "natural language processing means" is a device or system that analyzes text data, understands the user's intent, and generates an appropriate response.

[0542] A "voice synthesis means" is a device or system that converts the generated response into voice data in the voice of a selected character.

[0543] A "playback means" is a device or system for providing the generated audio data to a user.

[0544] "Field worker" means a worker who performs work in a factory or work site.

[0545] "Work instructions" are instructions or guidance for carrying out a specific task or process.

[0546] "Maintenance questions" are questions about the maintenance and repair of machines and equipment.

[0547] A "character response" is a verbal response provided by the selected character.

[0548] This invention is a system for improving the efficiency of work instructions and maintenance in a factory. A user makes a voice inquiry, and an appropriate response is provided in the character's voice.

[0549] The main components of the system are:

[0550] 1. Voice input method

[0551] The microphones installed at each work station in the factory are used as devices for users to input voice, allowing workers to input voice in a hands-free manner.

[0552] 2. Transmission Method

[0553] The voice data acquired by the voice input means is transmitted to the server in real time via Wi-Fi or a wired network.

[0554] 3. Voice Recognition Methods

[0555] The server converts the received audio data into text data using the Google Cloud Speech-to-Text API, which prepares the audio data for analysis as text data.

[0556] 4. Natural Language Processing Methods

[0557] The text data is analyzed using the Google Cloud Natural Language API to understand the user's intent and generate an appropriate response, with the use of generative AI models to enable more accurate responses.

[0558] 5. Speech synthesis means

[0559] The generated response text is converted into audio data in a character's voice using the Google Cloud Text-to-Speech API. This character's voice is pre-set and can be selected from multiple options.

[0560] 6. Regeneration means

[0561] The resulting audio data is played back through the factory's speaker system to provide a response to the user, allowing workers to instantly obtain the information they need.

[0562] Specific examples

[0563] 1. User Input

[0564] The worker speaks into the microphone, asking, "What's the next step?" This voice is input into the system via the microphone.

[0565] 2. Voice Recognition

[0566] The input voice data is sent to the server and converted into text data such as "What's the next step?" using the Google Cloud Speech-to-Text API.

[0567] 3. Natural Language Processing

[0568] The text data is parsed by the Google Cloud Natural Language API, which understands that the request is for instructions on the next step, and the generative AI model generates a response such as, "The next step is to assemble the parts."

[0569] 4. Speech Synthesis

[0570] The generated response text is converted into audio data using the Google Cloud Text-to-Speech API, and a character's voice is generated saying, "The next step is assembling the parts."

[0571] 5. Responding to the User

[0572] The final audio data is played back through factory speakers and provided to workers.

[0573] Prompt Sentence Examples

[0574] User input: "What's next?"

[0575] This system will enable work instructions and maintenance instructions within the factory to be given quickly and efficiently, which is expected to improve work efficiency.

[0576] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0577] Step 1:

[0578] The user inputs voice using the voice input means. Specifically, a factory worker speaks into the microphone, saying, "What's the next process?" This voice data is acquired and sent to the server via the transmission means.

[0579] Input: Voice data (e.g., "What's the next step?")

[0580] Output: Audio data sent to the server

[0581] Step 2:

[0582] The server passes the received voice data to the speech recognition means and converts it into text data using the Google Cloud Speech-to-Text API, which converts the voice data into a string of characters.

[0583] Input: Audio data

[0584] Output: Text data (e.g. "What's the next step?")

[0585] How it works: Converts audio into text using the Google Cloud Speech-to-Text API.

[0586] Step 3:

[0587] The server passes the text data to a natural language processor, which uses the Google Cloud Natural Language API to analyze the text and understand the user's intent. A generative AI model is then used to generate an appropriate response to the user's question.

[0588] Input: Text data (e.g., "What's the next step?")

[0589] Output: Response text data (e.g. "The next step is assembling the parts.")

[0590] How it works: Uses the Google Cloud Natural Language API for text analysis and response generation.

[0591] Step 4:

[0592] The server passes the generated response text to the speech synthesis means, which converts it into audio data in the character's voice using the Google Cloud Text-to-Speech API, which converts the text into audio.

[0593] Input: Response text data (e.g., "The next step is assembling the parts.")

[0594] Output: Character voice data

[0595] How it works: Speech synthesis is performed using the Google Cloud Text-to-Speech API.

[0596] Step 5:

[0597] The server transmits the generated voice data to the terminal and uses a playback means to play back the response from the factory speaker system, allowing the worker to receive voice instructions regarding the next process.

[0598] Input: Character voice data

[0599] Output: Audio played through the speaker (e.g. "The next step is to assemble the parts.")

[0600] Action: Plays audio data on the device.

[0601] As described above, by executing each step consecutively, work instructions within the factory can be issued efficiently.

[0602] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0603] System Overview

[0604] This invention provides a system that allows users to input speech, converts the speech into text, generates appropriate responses based on information obtained from the text, and plays back the responses in a character's voice. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and generates responses based on those emotions, this system achieves more friendly and effective dialogue. It is designed to be used in education and childcare settings, particularly to keep children engaged.

[0605] Program processing overview

[0606] 1. Accepting user input

[0607] The device is always on standby for the user's (especially children's) voice. When the user speaks, the device records the voice and transmits it to the server in real time.

[0608] 2. Speech Recognition Processing

[0609] The server receives the voice data sent from the terminal. The received voice data is passed to a voice recognition engine and converted into text data. This text data is used in the next processing step.

[0610] 3. Emotion Recognition Processing

[0611] The server passes the text data obtained from the speech recognition engine to the emotion engine, which analyzes the text data and voice data and recognizes the user's emotional state (e.g., joy, sadness, anger, etc.).

[0612] 4. Context Understanding and Response Generation

[0613] The server passes the text data from the speech recognition engine to a natural language processing (NLP) engine, taking into account the user's emotional state obtained from the emotion engine. The NLP engine analyzes the text and understands the user's intent. For example, if a user says, "What is today's homework?", the NLP engine generates an appropriate response to the question: "Today's homework is to do pages 10 to 15 of math."

[0614] 5. Speech synthesis processing

[0615] The generated response text is passed to a speech synthesis engine, which converts the response text into audio data using the voice of a character selected by the user, while simultaneously adjusting the tone and expression of the voice to take into account the user's emotional state.

[0616] 6. Responding to the User

[0617] The generated voice data is sent from the server to the device. The device plays the received voice data and responds to the user (child) in the voice of the selected character. For example, the device could respond in Doraemon's voice, saying, "Today's homework is to do pages 10 to 15 of math," and if the user is feeling a little down, it could play an encouraging tone.

[0618] Specific examples

[0619] Conversation example 1: Helping with homework

[0620] 1. Accepting user input

[0621] The device waits for the user to say, "Doraemon, tell me what my homework is today." When the user speaks, the device records the voice and immediately sends it to the server.

[0622] 2. Speech Recognition Processing

[0623] The server receives the voice data "Doraemon, tell me today's homework," passes it to a voice recognition engine, and converts it into text data.

[0624] 3. Emotion Recognition Processing

[0625] The server passes the text data and voice data obtained from the voice recognition engine to the emotion engine, and recognizes the user's emotion (e.g., depressed).

[0626] 4. Context Understanding and Response Generation

[0627] The server passes the text data to the NLP engine based on the emotional information obtained from the emotion engine, understands the user's intention (please help me with my homework), and generates a response: "Today's homework is to do pages 10 to 15 of math."

[0628] 5. Speech synthesis processing

[0629] The server passes the response text to a speech synthesis engine, which converts it into audio data in Doraemon's voice, generating a cheerful tone to encourage the depressed user.

[0630] 6. Responding to the User

[0631] The server then sends the generated voice data to the device, which then plays it back, responding in Doraemon's cheerful voice, "Today's homework is to do pages 10 to 15 of math."

[0632] As described above, the present invention realizes a dialogue response system that takes emotions into consideration, thereby making it possible to provide users with friendly and effective educational and child-rearing support.

[0633] The processing flow will be explained below.

[0634] Step 1:

[0635] The device is always on standby for user voice input. When the user says, "Character (e.g., Doraemon), tell me what my homework is today," the device records the voice.

[0636] Step 2:

[0637] The device immediately sends the recorded audio data to a server, and the data is often encrypted before being sent.

[0638] Step 3:

[0639] The server receives the voice data sent from the device and then passes it to a voice recognition engine for analysis.

[0640] Step 4:

[0641] The speech recognition engine converts the received voice data into text data, generating text such as "Character, please tell me today's homework."

[0642] Step 5:

[0643] The server passes the text and voice data obtained from the speech recognition engine to the emotion engine, which analyzes the text and voice characteristics and recognizes the user's emotional state (e.g., joy, sadness, anger, etc.).

[0644] Step 6:

[0645] The server passes the text data from the speech recognition engine to the natural language processing (NLP) engine based on the user's emotional information obtained from the emotion engine. The NLP engine analyzes the text and understands the user's intent.

[0646] Step 7:

[0647] The NLP engine generates an appropriate response based on the user's intent and emotional information, in this case, a response text such as "Today's homework is to do pages 10 to 15 of math."

[0648] Step 8:

[0649] The server then passes the generated response text to the speech synthesis engine, which instructs it to generate voice data in the voice of a character selected by the user (e.g., Doraemon).The tone and expression of the voice are also adjusted according to the user's emotional state.

[0650] Step 9:

[0651] The speech synthesis engine converts the response text into voice data in the voice of the specified character, taking into account the user's emotions and using encouraging or gentle tones.

[0652] Step 10:

[0653] The server sends the generated voice data to the terminal, allowing the response content to be delivered to the user in real time.

[0654] Step 11:

[0655] The device plays back the received voice data and responds to the user in the character's voice, saying, "Today's homework is to do pages 10 to 15 of math." By playing back the voice in a tone that corresponds to the user's emotions, a more friendly dialogue is achieved.

[0656] Example 2

[0657] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0658] Conventional voice response systems generate responses without considering the user's emotions, which makes it difficult to realize more friendly and effective dialogue. Particularly in education and childcare settings, responses that respond to the user's emotions are required, but no system has been able to meet this demand. Furthermore, systems have been limited in their ability to provide responses in the voices of multiple characters, lacking the flexibility to allow users to choose from a variety of options.

[0659] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice input means, a transmission means, a voice recognition means, an emotion recognition means, a natural language processing means, a voice synthesis means, and a playback means. This makes it possible to recognize the user's emotional state and generate a response according to the emotion. Furthermore, by dynamically providing responses using the voices of multiple characters, it is possible to provide the user with a more familiar and effective dialogue experience.

[0660] "Voice input means" refers to a device or function that allows a user to input voice.

[0661] "Transmitting means" refers to a device or function for transmitting acquired voice data to a server.

[0662] "Speech recognition means" refers to a device or function for converting received voice data into text data.

[0663] "Emotion recognition means" refers to a device or function for analyzing the text data and voice data converted by the voice recognition means and recognizing the emotional state of the user.

[0664] "Natural language processing means" refers to a device or function that analyzes text data taking into account the emotional state recognized by the emotion recognition means, understands the user's intention, and generates a response.

[0665] "Speech synthesis means" refers to a device or function for converting a response generated by a natural language processing means into voice data in the voice of a selected character.

[0666] "Playback means" refers to a device or function for transmitting generated voice data to a terminal and providing a response to a user.

[0667] This invention provides a system that accepts user voice input, converts the speech into text, generates appropriate responses based on information obtained from the text, and plays the responses in a character's voice. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and generates responses based on those emotions, it achieves more friendly and effective dialogue. This system is specifically designed for use in education and childcare settings.

[0668] System Configuration

[0669] The system includes the following means:

[0670] 1. Voice input means: A device or function that allows a user to input voice. For example, when a user wants to ask a question, they can speak into the terminal.

[0671] 2. Transmission means: A device or function for transmitting acquired voice data to a server. The voice data acquired by the voice input means is transmitted to the server in real time.

[0672] 3. Speech recognition means: A device or function for converting received voice data into text data. Specifically, the voice data is converted into text data using the Google Cloud Speech-to-Text API.

[0673] 4. Emotion Recognition Means: A device or function for analyzing the text data and voice data converted by the Voice Recognition Means and recognizing the user's emotional state. The Microsoft Azure Emotion API is used to analyze the user's emotions.

[0674] 5. Natural language processing means: A device or function that analyzes text data taking into account the emotional state recognized by the emotion recognition means, understands the user's intention, and generates a response. OpenAI's GPT-4 is used to generate appropriate responses to user questions.

[0675] 6. Speech synthesis means: A device or function for converting the response generated by the natural language processing means into voice data in the voice of the selected character. Amazon Polly is used to convert into voice data in the voice of the selected character.

[0676] 7. Playback means: A device or function for transmitting the generated voice data to the terminal and providing a response to the user. The generated voice is played back through the terminal's speaker.

[0677] Specific processing operations

[0678] The specific processing operations of this system are shown below.

[0679] 1. Accepting user input

[0680] The device is always on standby for user voice input. For example, if a user says to the device, "Doraemon, tell me what my homework is today," the device records this voice and sends it to the server in real time.

[0681] 2. Speech Recognition Processing

[0682] The server receives the voice data sent from the device, passes it to the Google Cloud Speech-to-Text API, and converts it into text data, which is used for the next processing step.

[0683] 3. Emotion Recognition Processing

[0684] The server passes the text and voice data acquired by the speech recognition tool to the Microsoft Azure Emotion API, which analyzes the data and recognizes the user's emotional state. For example, it may recognize that the user is feeling a little depressed.

[0685] 4. Context Understanding and Response Generation

[0686] The server passes the text data from the speech recognition engine to OpenAI's GPT-4, taking into account the user's emotional state obtained from the emotion recognition engine. The NLP engine analyzes the text and understands the user's intent. For example, in response to the question "What is today's homework?", it generates a response such as "Today's homework is pages 10 to 15 of math."

[0687] 5. Speech synthesis processing

[0688] The generated response text is passed to Amazon Polly, which converts it into voice data in the voice of a character selected by the user, such as "Doraemon's voice."

[0689] 6. Responding to the User

[0690] The server sends the generated voice data to the device, which then plays the received voice data from its speaker and responds to the user in Doraemon's voice, saying, "Today's homework is to do pages 10 to 15 of math."

[0691] Specific examples

[0692] For example, if a user says, "Doraemon, tell me my homework today," the above process will result in a response in Doraemon's voice saying, "Today's homework is to do pages 10 to 15 of math." If the user is feeling down, the system can respond in a positive, encouraging tone.

[0693] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0694] Step 1: Accepting user input

[0695] The device is always on standby for user voice input. When the user says, "Doraemon, tell me what my homework is today," the device records the voice.

[0696] Input: User voice input

[0697] How it works: The device's built-in microphone captures and records audio. The voice recognition chip detects what the user is saying and starts the recording process.

[0698] Output: Recorded audio data

[0699] Step 2: Sending audio data

[0700] The recorded audio data is sent to the server in real time.

[0701] Input: Audio data recorded on the device

[0702] What it does: Uses the device's communications module to send audio data to a server over the internet.

[0703] Output: Audio data sent to the server

[0704] Step 3: Speech recognition processing

[0705] The server passes the received audio data to the Google Cloud Speech-to-Text API, which converts the audio data into text data, which is used for the next processing step.

[0706] Input: Audio data sent to the server

[0707] How it works: The server sends the audio data to the Google Cloud Speech-to-Text API, which uses a voice filtering function to remove background noise, then analyzes the audio data and converts it into text.

[0708] Output: Converted text data (e.g. "Doraemon, tell me today's homework.")

[0709] Step 4: Emotion recognition processing

[0710] The server passes the text and voice data acquired by the voice recognition means to the Microsoft Azure Emotion API, which analyzes the user's emotional state.

[0711] Input: Text data and audio data converted by speech recognition

[0712] How it works: The server sends the data to the Microsoft Azure Emotion API, which analyzes the linguistic features of the audio waveform data and text data to recognize the user's emotional state (e.g., depression).

[0713] Output: User's emotional state (e.g., "depressed")

[0714] Step 5: Context understanding and response generation

[0715] The server then passes the text data based on the emotion and speech recognition results to OpenAI's GPT-4, which analyzes the text, understands the user's intent, and generates an appropriate response.

[0716] Input: User's emotional state and text data obtained by speech recognition means

[0717] How it works: The server inputs emotional state and text data as prompts into GPT-4, which generates an appropriate response to the user's question (e.g., "Today's homework is to do pages 10 to 15 of math").

[0718] Output: The generated response text

[0719] Step 6: Speech synthesis processing

[0720] The generated response text is passed to Amazon Polly, which converts it into voice data in the voice of the character selected by the user.

[0721] Input: GPT-4 generated response text

[0722] How it works: The server sends the response text to Amazon Polly, which converts it into speech data in the voice of a selected character (e.g., Doraemon), while also adjusting the tone of the voice to take into account emotional information.

[0723] Output: Generated audio data

[0724] Step 7: Respond to the user

[0725] The generated voice data is transmitted from the server to the terminal, which then plays the received voice data and provides a response to the user.

[0726] Input: Audio data sent from the server

[0727] How it works: The device's speaker plays audio data, and responds in a cheerful, encouraging tone, for example, in Doraemon's voice, saying, "Today's homework is to do pages 10 to 15 of math."

[0728] Output: A spoken response to the user

[0729] (Application example 2)

[0730] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0731] Modern work environments, especially logistics centers, require efficient work performance and reduced employee fatigue. Conversational support systems are useful for enabling workers to work efficiently while taking appropriate breaks. However, conventional dialogue systems have limitations in recognizing users' emotions and are sometimes unable to provide appropriate responses. Furthermore, there is a lack of systems that can properly analyze users' voice input and provide appropriate instructions and support information in real time.

[0732] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a voice input means that enables a user to input voice; a transmission means that transmits voice data acquired by the voice input means to the server; a voice recognition means that converts the voice data received by the server by the transmission means into text data; a natural language processing means that analyzes the text data converted by the voice recognition means, understands the user's intention, and generates a response; a voice synthesis means that converts the response generated by the natural language processing means into voice data in the voice of a selected character; a playback means that transmits the voice data generated by the voice synthesis means to a terminal and provides the response to the user; and an emotion recognition means that analyzes the user's emotional state and adjusts the response based on the analysis results. This allows appropriate instructions and support to be provided in real time according to the user's emotional state, improving work efficiency and reducing worker fatigue.

[0733] The "voice input means" is a device that allows the user to input voice (for example, a microphone).

[0734] The "transmission means" is a means having a function of transmitting the acquired voice data to the server.

[0735] The "voice recognition means" is a system that converts received voice data into text data.

[0736] "Natural language processing means" is a function that analyzes text data, understands the user's intent, and generates an appropriate response.

[0737] The "voice synthesis means" is a system that converts the generated response text into voice data in the voice of a selected character.

[0738] The "playback means" is a function that transmits the generated voice data to the terminal and provides a response to the user.

[0739] The "emotion recognition means" is a system that analyzes the user's voice data and text data to analyze the user's emotional state.

[0740] This invention provides a system that allows a user to receive work instructions using voice input. The system picks up the user's voice using voice input means and transmits the voice data to a server via transmission means. The server converts the received voice data into text data using voice recognition means and analyzes it using natural language processing means to understand the user's instructions. It also analyzes the user's emotional state using emotion recognition means and generates a response in an appropriate tone. The response is converted into voice data in the voice of a specified character by voice synthesis means and transmitted to a terminal via playback means.

[0741] Hardware and software used

[0742] Audio input method: Microphone

[0743] Transmission method: Network module or communication adapter

[0744] Speech recognition method: Google Speech-to-Text API

[0745] Natural language processing tools: Various NLP libraries (e.g. spaCy, NLTK, etc.)

[0746] Emotion recognition: sentiment_analysis_spanish library

[0747] Speech synthesis method: Google Text-to-Speech API

[0748] Playback via: Speakers, smart glasses, and smartphones

[0749] System processing flow

[0750] The server acquires the user's speech input, performs speech recognition, and analyzes the resulting text data. The analyzed text data is passed to a natural language processing means, which understands the user's intention and generates an appropriate response. The emotion recognition means analyzes the user's emotional state and adjusts the content and tone of the response. The generated response text is converted into voice data by a voice synthesis means and transmitted to the terminal via a playback means.

[0751] Specific examples

[0752] When a worker puts on the smart glasses and says, "What are the next work instructions?", the voice is recognized by the system and sent to the server. The server converts the voice into text data, performs emotion recognition, and generates the next work instruction using natural language processing. The generated response is synthesized in the character's voice and provided to the worker through the smart glasses. For example, if the worker is tired, instructions such as "Next, please pick up container B. Please take a break while working" are provided in an encouraging tone.

[0753] Prompt Sentence Examples

[0754] "Please tell me the next work instructions."

[0755] Check the current progress

[0756] "Please tell me when to take a break."

[0757] In this way, appropriate instructions and support can be provided in real time according to the user's emotional state, improving work efficiency and reducing worker fatigue.

[0758] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0759] Step 1:

[0760] The user inputs the voice. The user inputs the voice through a microphone. For example, the user might say, "Please tell me the next work instruction." This voice data is generated and sent to the next step.

[0761] Step 2:

[0762] The terminal transmits the user's voice data to the server via a transmission means. The voice data is input and transferred to the server as output. During this process, data compression and encryption may be performed.

[0763] Step 3:

[0764] The server receives the voice data. It converts the received voice data into text data using a speech recognition tool. The input is voice data, and the output is text data. This conversion is specifically performed using the Google Speech-to-Text API.

[0765] Step 4:

[0766] The server passes the converted text data to a natural language processing means for analysis. The input is text data, and the output is response data that understands the user's intent. In natural language processing, a specific NLP library (e.g., spaCy or NLTK) is used to analyze the user's intent.

[0767] Step 5:

[0768] The server passes the text data and voice data to the emotion recognition means to recognize the user's emotional state. The input is text data and voice data, and the output is emotional state information. The sentiment_analysis_spanish library is used for emotion recognition.

[0769] Step 6:

[0770] The server generates an appropriate response taking into account the emotional state and passes the response text to a speech synthesis means. The input is the response text and emotional state information, and the output is audio data. The response text is converted into audio using the Google Text-to-Speech API.

[0771] Step 7:

[0772] The server transmits the generated voice data to the terminal. The terminal uses a playback means to play the voice data to the user. Specifically, the voice is played using smart glasses or a smartphone. The input is the voice data, and the output is the voice heard by the user.

[0773] Step 8:

[0774] The user receives the voice instructions and performs the next task. For example, if the instruction is "Next, pick up container B. Please take a break while working," the user will continue working in accordance with the instructions.

[0775] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0776] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0777] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0778] [Third embodiment]

[0779] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0780] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0781] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0782] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0783] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0784] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0785] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0786] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0787] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0788] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0789] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0790] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0791] System Overview

[0792] The present invention provides a system that allows users to speak, converts the speech into text, generates appropriate responses based on information obtained from the text, and plays back the responses in the voice of a character. It is designed to be used in education and childcare settings, particularly to keep children engaged.

[0793] Program processing overview

[0794] 1. Accepting user input

[0795] The device is always on standby for the user's (especially children's) voice. When the user speaks, the device records the voice and transmits it to the server in real time.

[0796] 2. Speech Recognition Processing

[0797] The server receives the voice data sent from the terminal. The received voice data is passed to a voice recognition engine and converted into text data. This text data is used in the next processing step.

[0798] 3. Context Understanding and Response Generation

[0799] The server passes the text data obtained from the speech recognition engine to a natural language processing (NLP) engine. The NLP engine analyzes the text and understands the user's intent. For example, if a user says, "Tell me about today's homework," the NLP engine generates an appropriate response to that question.

[0800] 4. Speech synthesis processing

[0801] The generated response text is passed to a speech synthesis engine, which uses the voice of the character selected by the user to convert the response text into voice data that is used to respond to the user in the character's voice.

[0802] 5. Responding to the User

[0803] The generated voice data is sent from the server to the device, which then plays the received voice data and responds to the user (child) in the voice of the selected character.

[0804] Specific examples

[0805] Conversation example 1: Helping with homework

[0806] 1. Accepting user input

[0807] The terminal waits for the user's voice to say, "Doraemon, tell me today's homework."

[0808] When the user speaks, the device records the voice and immediately sends it to the server.

[0809] 2. Speech Recognition Processing

[0810] The server receives the voice data "Doraemon, tell me today's homework," passes it to a voice recognition engine, and converts it into text data.

[0811] 3. Context Understanding and Response Generation

[0812] The server passes the text data to an NLP engine to understand the user's intent (i.e., please help me with my homework).

[0813] The NLP engine generates the response, "Today's homework is to do pages 10 to 15 of math."

[0814] 4. Speech synthesis processing

[0815] The server passes the response text to a speech synthesis engine and converts it into audio data in Doraemon's voice.

[0816] 5. Responding to the User

[0817] The server transmits the generated voice data to the terminal.

[0818] The device plays the audio data and responds to the user in Doraemon's voice, saying, "Today's homework is to do pages 10 to 15 of math."

[0819] In this way, the system of the present invention can receive a user's voice input, generate an appropriate response, and play it back in the character's voice, enabling engaging and sustainable learning and parenting support, especially for children.

[0820] The processing flow will be explained below.

[0821] Step 1:

[0822] The device is always on standby for user voice input. When the user says, "Character (e.g., Doraemon), tell me what my homework is today," the device records the voice.

[0823] Step 2:

[0824] The device immediately sends the recorded audio data to a server, and the data is often encrypted before being sent.

[0825] Step 3:

[0826] The server receives the voice data sent from the device and then passes it to a voice recognition engine for analysis.

[0827] Step 4:

[0828] The speech recognition engine converts the received voice data into text data, generating text such as "Character, please tell me today's homework."

[0829] Step 5:

[0830] The server receives the text data from the speech recognition engine and passes it to a natural language processing (NLP) engine, which analyzes the context and understands the user's intent.

[0831] Step 6:

[0832] The NLP engine generates an appropriate response based on the user's intent, in this case, a response text such as "Today's homework is to do pages 10 to 15 of math."

[0833] Step 7:

[0834] The server passes the generated response text to the speech synthesis engine, instructing it to generate voice data in the voice of the character selected by the user (e.g., Doraemon).

[0835] Step 8:

[0836] The speech synthesis engine converts the response text into speech data in the voice of the designated character, utilizing the character's distinctive speech patterns.

[0837] Step 9:

[0838] The server sends the generated voice data to the terminal, allowing the response content to be delivered to the user in real time.

[0839] Step 10:

[0840] The device plays back the received voice data and responds to the user in the character's voice, saying, "Today's homework is to do pages 10 to 15 of math." This allows the user to enjoy the character's response.

[0841] Example 1

[0842] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0843] In recent years, there has been a demand in education and childcare for interactive learning support systems that can sustain children's interest. However, conventional systems lack the technology to properly analyze a user's voice input, understand their intent, generate a response, and then play that response in a character's voice, limiting their ability to sustain children's interest. The present invention aims to solve this problem and provide engaging and sustained educational support, especially for children.

[0844] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0845] In this invention, the server includes voice input means that allows a user to input voice, transmission means that transmits voice data acquired by the voice input means to a processing device, voice recognition means that converts the voice data received by the processing device by the transmission means into text data, natural language processing means that analyzes the text data converted by the voice recognition means, understands the user's intention, and generates a response, voice synthesis means that converts the response generated by the natural language processing means into voice data in the voice of a selected character, and playback means that transmits the voice data generated by the voice synthesis means to a receiving device and provides the response to the user. This makes it possible to analyze the user's voice instructions and provide an appropriate response in real time in the voice of a character.

[0846] "Voice input means" refers to a device or system that allows a user to input voice, and includes a microphone and the like.

[0847] "Transmission means" refers to a function for transferring acquired voice data to a processing device or a server.

[0848] "Speech recognition means" refers to the technology or software used to analyze received voice data and convert it into text data.

[0849] "Natural language processing means" refers to technology or software that analyzes text data converted by speech recognition means, understands the user's intent, and generates an appropriate response.

[0850] "Speech synthesis means" refers to technology or software for converting the generated response into audio data in the voice of a selected character.

[0851] The "playback means" refers to a device or system that outputs sound to provide the generated sound data to the user.

[0852] "Processing device" refers to a server or computer that receives, analyzes, and converts voice data sent from a terminal.

[0853] "Receiving device" refers to a terminal or device that receives audio data transmitted from a server.

[0854] The present invention is a system for use in education and childcare that converts user-input speech into text, generates a response based on information obtained from the text, and plays back the response in the voice of a character. The system includes a speech input unit, a transmission unit, a speech recognition unit, a natural language processing unit, a speech synthesis unit, and a playback unit.

[0855] The device is always on standby for the user to speak and records the voice through a microphone. For example, when a user says, "Doraemon, please tell me today's homework," the device transmits this voice data to a processing unit in real time.

[0856] The server receives the voice data sent from the device and converts it into text using the IBM Watson speech recognition engine. The converted text data is then analyzed using the Google Cloud Natural Language API to understand the user's intent. As a result of this analysis, the intent may be recognized as, for example, "Please tell me about my homework." Based on this information, the NLP engine generates an appropriate response text, such as, "Today's homework is to do pages 10 to 15 of math."

[0857] The server then passes the generated response text to the Amazon Polly speech synthesis engine, which converts it into audio data in the voice of the specified character. This audio data is then sent back to the device, where it is played back through the device's speaker, allowing the user to hear the response in Doraemon's voice.

[0858] As a concrete example, consider the case where a user talks about homework. When the user says, "Doraemon, tell me today's homework," the device records this voice and sends it to the server. The server receives the voice data, converts it into text data using a speech recognition engine, then analyzes it using a natural language processing engine to generate an appropriate response. The response is then converted into Doraemon's voice using a speech synthesis engine and sent to the device, which then plays it back. As a result, the user can hear Doraemon's voice saying, "Today's homework is to do pages 10 to 15 of math."

[0859] As an example of a prompt sentence, if you say "Doraemon, tell me the weather," the following will happen:

[0860] 1. User: "Doraemon, tell me the weather."

[0861] 2. Device: Records audio using a microphone and sends it to the server

[0862] 3. Server: Receives voice data and converts it into text data using a speech recognition engine

[0863] 4. Server: Passes text data to the NLP engine, understands user intent, and generates a response

[0864] 5. Server: Passes the response text to the speech synthesis engine and converts it into audio data in Doraemon's voice

[0865] 6. Server: Sends the generated voice data to the device

[0866] 7. Device: Plays audio data through the speaker and responds to the user in Doraemon's voice, saying "Today's weather is sunny."

[0867] In this way, the system of the present invention can analyze the user's voice instructions and provide appropriate responses in real time using the character's voice, thereby enabling attractive and sustainable learning and childcare support, especially for children.

[0868] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0869] Step 1: Input reception

[0870] The user stands in front of the device and says, "Doraemon, tell me what my homework is today." The device uses a microphone to wait for the user's voice input. When voice input is made, the device records this voice data. After recording is complete, the voice data is sent to the server.

[0871] input:

[0872] User voice data

[0873] output:

[0874] Audio data sent to the server

[0875] Specific behavior:

[0876] The device records audio through a microphone and transmits the recorded data to a server in real time.

[0877] Step 2: Speech recognition processing

[0878] The server receives the voice data sent from the device. The received voice data is passed to the IBM Watson speech recognition engine. The speech recognition engine analyzes the voice waveform and converts the voice data into text data.

[0879] input:

[0880] Audio data sent from the device

[0881] output:

[0882] Converting audio data into text data

[0883] Specific behavior:

[0884] The server passes the voice data to a voice recognition engine, which generates text data through waveform analysis.

[0885] Step 3: Context understanding and response generation

[0886] The server passes the text data obtained from the speech recognition engine to the Google Cloud Natural Language API. The NLP engine breaks the text down into tokens, performs grammatical analysis, and understands the user's intent. Specifically, it recognizes the intent as "Please tell me what my homework is." Based on this information, the NLP engine generates a response text such as "Today's homework is to do pages 10 to 15 of math."

[0887] input:

[0888] Text data obtained from the speech recognition engine

[0889] output:

[0890] Generated response text

[0891] Specific behavior:

[0892] The server passes the text data to an NLP engine, which analyzes the intent and generates a response text.

[0893] Step 4: Speech synthesis processing

[0894] The server passes the generated response text to the Amazon Polly speech synthesis engine, which converts the response text into audio data in the voice of the specified character.

[0895] input:

[0896] Generated response text

[0897] output:

[0898] Audio data converted into a character's voice

[0899] Specific behavior:

[0900] The server passes the response text to a speech synthesis engine, which generates voice data in the voice of the specified character.

[0901] Step 5: Respond to the user

[0902] The server sends the generated voice data to the device, which then plays it back through the speaker, allowing the user to hear Doraemon's voice reply, "Today's homework is to do pages 10 to 15 of math."

[0903] input:

[0904] Audio data sent from the server

[0905] output:

[0906] Audio that users can hear

[0907] Specific behavior:

[0908] The device receives the voice data and plays it over the speaker, and the user hears the response in Doraemon's voice.

[0909] Through the above steps, the system of the present invention analyzes the user's voice instructions and provides appropriate responses in real time using the character's voice.

[0910] (Application example 1)

[0911] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0912] In conventional factories, workers had to manually check instructions or ask other workers to understand work content and maintenance instructions, which resulted in a decrease in efficiency. Furthermore, systems that use voice input could only provide simple voice responses, making it difficult to provide the specific and immediate instructions workers needed. This significantly reduced work efficiency within the factory, and ultimately became a factor hindering productivity improvements.

[0913] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0914] In this invention, the server includes a voice input means that enables a user to input voice data, a transmission means that transmits voice data acquired by the voice input means to the server, a voice recognition means that converts the voice data received by the server by the transmission means into text data, a natural language processing means that analyzes the text data converted by the voice recognition means, understands the user's intention, and generates a response, a voice synthesis means that converts the response generated by the natural language processing means into voice data in the voice of a selected character, a playback means that transmits the voice data generated by the voice synthesis means to a terminal and provides the response to the user, and a means that recognizes voice input from a field worker and provides the character's response to the work instructions or maintenance questions. This enables work instructions and maintenance instructions to be given quickly and efficiently within a factory.

[0915] The "voice input means" is a device or system that allows a user to input voice.

[0916] The "transmitting means" is a device or system for transmitting the acquired voice data to the server.

[0917] A "voice recognition means" is a device or system for converting received voice data into text data.

[0918] A "natural language processing means" is a device or system that analyzes text data, understands the user's intent, and generates an appropriate response.

[0919] A "voice synthesis means" is a device or system that converts the generated response into voice data in the voice of a selected character.

[0920] A "playback means" is a device or system for providing the generated audio data to a user.

[0921] "Field worker" means a worker who performs work in a factory or work site.

[0922] "Work instructions" are instructions or guidance for carrying out a specific task or process.

[0923] "Maintenance questions" are questions about the maintenance and repair of machines and equipment.

[0924] A "character response" is a verbal response provided by the selected character.

[0925] This invention is a system for improving the efficiency of work instructions and maintenance in a factory. A user makes a voice inquiry, and an appropriate response is provided in the character's voice.

[0926] The main components of the system are:

[0927] 1. Voice input method

[0928] The microphones installed at each work station in the factory are used as devices for users to input voice, allowing workers to input voice in a hands-free manner.

[0929] 2. Transmission Method

[0930] The voice data acquired by the voice input means is transmitted to the server in real time via Wi-Fi or a wired network.

[0931] 3. Voice Recognition Methods

[0932] The server converts the received audio data into text data using the Google Cloud Speech-to-Text API, which prepares the audio data for analysis as text data.

[0933] 4. Natural Language Processing Methods

[0934] The text data is analyzed using the Google Cloud Natural Language API to understand the user's intent and generate an appropriate response, with the use of generative AI models to enable more accurate responses.

[0935] 5. Speech synthesis means

[0936] The generated response text is converted into audio data in a character's voice using the Google Cloud Text-to-Speech API. This character's voice is pre-set and can be selected from multiple options.

[0937] 6. Regeneration means

[0938] The resulting audio data is played back through the factory's speaker system to provide a response to the user, allowing workers to instantly obtain the information they need.

[0939] Specific examples

[0940] 1. User Input

[0941] The worker speaks into the microphone, asking, "What's the next step?" This voice is input into the system via the microphone.

[0942] 2. Voice Recognition

[0943] The input voice data is sent to the server and converted into text data such as "What's the next step?" using the Google Cloud Speech-to-Text API.

[0944] 3. Natural Language Processing

[0945] The text data is parsed by the Google Cloud Natural Language API, which understands that the request is for instructions on the next step, and the generative AI model generates a response such as, "The next step is to assemble the parts."

[0946] 4. Speech Synthesis

[0947] The generated response text is converted into audio data using the Google Cloud Text-to-Speech API, and a character's voice is generated saying, "The next step is assembling the parts."

[0948] 5. Responding to the User

[0949] The final audio data is played back through factory speakers and provided to workers.

[0950] Prompt Sentence Examples

[0951] User input: "What's next?"

[0952] This system will enable work instructions and maintenance instructions within the factory to be given quickly and efficiently, which is expected to improve work efficiency.

[0953] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0954] Step 1:

[0955] The user inputs voice using the voice input means. Specifically, a factory worker speaks into the microphone, saying, "What's the next process?" This voice data is acquired and sent to the server via the transmission means.

[0956] Input: Voice data (e.g., "What's the next step?")

[0957] Output: Audio data sent to the server

[0958] Step 2:

[0959] The server passes the received voice data to the speech recognition means and converts it into text data using the Google Cloud Speech-to-Text API, which converts the voice data into a string of characters.

[0960] Input: Audio data

[0961] Output: Text data (e.g. "What's the next step?")

[0962] How it works: Converts audio into text using the Google Cloud Speech-to-Text API.

[0963] Step 3:

[0964] The server passes the text data to a natural language processor, which uses the Google Cloud Natural Language API to analyze the text and understand the user's intent. A generative AI model is then used to generate an appropriate response to the user's question.

[0965] Input: Text data (e.g., "What's the next step?")

[0966] Output: Response text data (e.g. "The next step is assembling the parts.")

[0967] How it works: Uses the Google Cloud Natural Language API for text analysis and response generation.

[0968] Step 4:

[0969] The server passes the generated response text to the speech synthesis means, which converts it into audio data in the character's voice using the Google Cloud Text-to-Speech API, which converts the text into audio.

[0970] Input: Response text data (e.g., "The next step is assembling the parts.")

[0971] Output: Character voice data

[0972] How it works: Speech synthesis is performed using the Google Cloud Text-to-Speech API.

[0973] Step 5:

[0974] The server transmits the generated voice data to the terminal and uses a playback means to play back the response from the factory speaker system, allowing the worker to receive voice instructions regarding the next process.

[0975] Input: Character voice data

[0976] Output: Audio played through the speaker (e.g. "The next step is to assemble the parts.")

[0977] Action: Plays audio data on the device.

[0978] As described above, by executing each step consecutively, work instructions within the factory can be issued efficiently.

[0979] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0980] System Overview

[0981] This invention provides a system that allows users to input speech, converts the speech into text, generates appropriate responses based on information obtained from the text, and plays back the responses in a character's voice. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and generates responses based on those emotions, this system achieves more friendly and effective dialogue. It is designed to be used in education and childcare settings, particularly to keep children engaged.

[0982] Program processing overview

[0983] 1. Accepting user input

[0984] The device is always on standby for the user's (especially children's) voice. When the user speaks, the device records the voice and transmits it to the server in real time.

[0985] 2. Speech Recognition Processing

[0986] The server receives the voice data sent from the terminal. The received voice data is passed to a voice recognition engine and converted into text data. This text data is used in the next processing step.

[0987] 3. Emotion Recognition Processing

[0988] The server passes the text data obtained from the speech recognition engine to the emotion engine, which analyzes the text data and voice data and recognizes the user's emotional state (e.g., joy, sadness, anger, etc.).

[0989] 4. Context Understanding and Response Generation

[0990] The server passes the text data from the speech recognition engine to a natural language processing (NLP) engine, taking into account the user's emotional state obtained from the emotion engine. The NLP engine analyzes the text and understands the user's intent. For example, if a user says, "What is today's homework?", the NLP engine generates an appropriate response to the question: "Today's homework is to do pages 10 to 15 of math."

[0991] 5. Speech synthesis processing

[0992] The generated response text is passed to a speech synthesis engine, which converts the response text into audio data using the voice of a character selected by the user, while simultaneously adjusting the tone and expression of the voice to take into account the user's emotional state.

[0993] 6. Responding to the User

[0994] The generated voice data is sent from the server to the device. The device plays the received voice data and responds to the user (child) in the voice of the selected character. For example, the device could respond in Doraemon's voice, saying, "Today's homework is to do pages 10 to 15 of math," and if the user is feeling a little down, it could play an encouraging tone.

[0995] Specific examples

[0996] Conversation example 1: Helping with homework

[0997] 1. Accepting user input

[0998] The device waits for the user to say, "Doraemon, tell me what my homework is today." When the user speaks, the device records the voice and immediately sends it to the server.

[0999] 2. Speech Recognition Processing

[1000] The server receives the voice data "Doraemon, tell me today's homework," passes it to a voice recognition engine, and converts it into text data.

[1001] 3. Emotion Recognition Processing

[1002] The server passes the text data and voice data obtained from the voice recognition engine to the emotion engine, and recognizes the user's emotion (e.g., depressed).

[1003] 4. Context Understanding and Response Generation

[1004] The server passes the text data to the NLP engine based on the emotional information obtained from the emotion engine, understands the user's intention (please help me with my homework), and generates a response: "Today's homework is to do pages 10 to 15 of math."

[1005] 5. Speech synthesis processing

[1006] The server passes the response text to a speech synthesis engine, which converts it into audio data in Doraemon's voice, generating a cheerful tone to encourage the depressed user.

[1007] 6. Responding to the User

[1008] The server then sends the generated voice data to the device, which then plays it back, responding in Doraemon's cheerful voice, "Today's homework is to do pages 10 to 15 of math."

[1009] As described above, the present invention realizes a dialogue response system that takes emotions into consideration, thereby making it possible to provide users with friendly and effective educational and child-rearing support.

[1010] The processing flow will be explained below.

[1011] Step 1:

[1012] The device is always on standby for user voice input. When the user says, "Character (e.g., Doraemon), tell me what my homework is today," the device records the voice.

[1013] Step 2:

[1014] The device immediately sends the recorded audio data to a server, and the data is often encrypted before being sent.

[1015] Step 3:

[1016] The server receives the voice data sent from the device and then passes it to a voice recognition engine for analysis.

[1017] Step 4:

[1018] The speech recognition engine converts the received voice data into text data, generating text such as "Character, please tell me today's homework."

[1019] Step 5:

[1020] The server passes the text and voice data obtained from the speech recognition engine to the emotion engine, which analyzes the text and voice characteristics and recognizes the user's emotional state (e.g., joy, sadness, anger, etc.).

[1021] Step 6:

[1022] The server passes the text data from the speech recognition engine to the natural language processing (NLP) engine based on the user's emotional information obtained from the emotion engine. The NLP engine analyzes the text and understands the user's intent.

[1023] Step 7:

[1024] The NLP engine generates an appropriate response based on the user's intent and emotional information, in this case, a response text such as "Today's homework is to do pages 10 to 15 of math."

[1025] Step 8:

[1026] The server then passes the generated response text to the speech synthesis engine, which instructs it to generate voice data in the voice of a character selected by the user (e.g., Doraemon).The tone and expression of the voice are also adjusted according to the user's emotional state.

[1027] Step 9:

[1028] The speech synthesis engine converts the response text into voice data in the voice of the specified character, taking into account the user's emotions and using encouraging or gentle tones.

[1029] Step 10:

[1030] The server sends the generated voice data to the terminal, allowing the response content to be delivered to the user in real time.

[1031] Step 11:

[1032] The device plays back the received voice data and responds to the user in the character's voice, saying, "Today's homework is to do pages 10 to 15 of math." By playing back the voice in a tone that corresponds to the user's emotions, a more friendly dialogue is achieved.

[1033] Example 2

[1034] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1035] Conventional voice response systems generate responses without considering the user's emotions, which makes it difficult to realize more friendly and effective dialogue. Particularly in education and childcare settings, responses that respond to the user's emotions are required, but no system has been able to meet this demand. Furthermore, systems have been limited in their ability to provide responses in the voices of multiple characters, lacking the flexibility to allow users to choose from a variety of options.

[1036] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice input means, a transmission means, a voice recognition means, an emotion recognition means, a natural language processing means, a voice synthesis means, and a playback means. This makes it possible to recognize the user's emotional state and generate a response according to the emotion. Furthermore, by dynamically providing responses using the voices of multiple characters, it is possible to provide the user with a more familiar and effective dialogue experience.

[1037] "Voice input means" refers to a device or function that allows a user to input voice.

[1038] "Transmitting means" refers to a device or function for transmitting acquired voice data to a server.

[1039] "Speech recognition means" refers to a device or function for converting received voice data into text data.

[1040] "Emotion recognition means" refers to a device or function for analyzing the text data and voice data converted by the voice recognition means and recognizing the emotional state of the user.

[1041] "Natural language processing means" refers to a device or function that analyzes text data taking into account the emotional state recognized by the emotion recognition means, understands the user's intention, and generates a response.

[1042] "Speech synthesis means" refers to a device or function for converting a response generated by a natural language processing means into voice data in the voice of a selected character.

[1043] "Playback means" refers to a device or function for transmitting generated voice data to a terminal and providing a response to a user.

[1044] This invention provides a system that accepts user voice input, converts the speech into text, generates appropriate responses based on information obtained from the text, and plays the responses in a character's voice. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and generates responses based on those emotions, it achieves more friendly and effective dialogue. This system is specifically designed for use in education and childcare settings.

[1045] System Configuration

[1046] The system includes the following means:

[1047] 1. Voice input means: A device or function that allows a user to input voice. For example, when a user wants to ask a question, they can speak into the terminal.

[1048] 2. Transmission means: A device or function for transmitting acquired voice data to a server. The voice data acquired by the voice input means is transmitted to the server in real time.

[1049] 3. Speech recognition means: A device or function for converting received voice data into text data. Specifically, the voice data is converted into text data using the Google Cloud Speech-to-Text API.

[1050] 4. Emotion Recognition Means: A device or function for analyzing the text data and voice data converted by the Voice Recognition Means and recognizing the user's emotional state. The Microsoft Azure Emotion API is used to analyze the user's emotions.

[1051] 5. Natural language processing means: A device or function that analyzes text data taking into account the emotional state recognized by the emotion recognition means, understands the user's intention, and generates a response. OpenAI's GPT-4 is used to generate appropriate responses to user questions.

[1052] 6. Speech synthesis means: A device or function for converting the response generated by the natural language processing means into voice data in the voice of the selected character. Amazon Polly is used to convert into voice data in the voice of the selected character.

[1053] 7. Playback means: A device or function for transmitting the generated voice data to the terminal and providing a response to the user. The generated voice is played back through the terminal's speaker.

[1054] Specific processing operations

[1055] The specific processing operations of this system are shown below.

[1056] 1. Accepting user input

[1057] The device is always on standby for user voice input. For example, if a user says to the device, "Doraemon, tell me what my homework is today," the device records this voice and sends it to the server in real time.

[1058] 2. Speech Recognition Processing

[1059] The server receives the voice data sent from the device, passes it to the Google Cloud Speech-to-Text API, and converts it into text data, which is used for the next processing step.

[1060] 3. Emotion Recognition Processing

[1061] The server passes the text and voice data acquired by the speech recognition tool to the Microsoft Azure Emotion API, which analyzes the data and recognizes the user's emotional state. For example, it may recognize that the user is feeling a little depressed.

[1062] 4. Context Understanding and Response Generation

[1063] The server passes the text data from the speech recognition engine to OpenAI's GPT-4, taking into account the user's emotional state obtained from the emotion recognition engine. The NLP engine analyzes the text and understands the user's intent. For example, in response to the question "What is today's homework?", it generates a response such as "Today's homework is pages 10 to 15 of math."

[1064] 5. Speech synthesis processing

[1065] The generated response text is passed to Amazon Polly, which converts it into voice data in the voice of a character selected by the user, such as "Doraemon's voice."

[1066] 6. Responding to the User

[1067] The server sends the generated voice data to the device, which then plays the received voice data from its speaker and responds to the user in Doraemon's voice, saying, "Today's homework is to do pages 10 to 15 of math."

[1068] Specific examples

[1069] For example, if a user says, "Doraemon, tell me my homework today," the above process will result in a response in Doraemon's voice saying, "Today's homework is to do pages 10 to 15 of math." If the user is feeling down, the system can respond in a positive, encouraging tone.

[1070] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1071] Step 1: Accepting user input

[1072] The device is always on standby for user voice input. When the user says, "Doraemon, tell me what my homework is today," the device records the voice.

[1073] Input: User voice input

[1074] How it works: The device's built-in microphone captures and records audio. The voice recognition chip detects what the user is saying and starts the recording process.

[1075] Output: Recorded audio data

[1076] Step 2: Sending audio data

[1077] The recorded audio data is sent to the server in real time.

[1078] Input: Audio data recorded on the device

[1079] What it does: Uses the device's communications module to send audio data to a server over the internet.

[1080] Output: Audio data sent to the server

[1081] Step 3: Speech recognition processing

[1082] The server passes the received audio data to the Google Cloud Speech-to-Text API, which converts the audio data into text data, which is used for the next processing step.

[1083] Input: Audio data sent to the server

[1084] How it works: The server sends the audio data to the Google Cloud Speech-to-Text API, which uses a voice filtering function to remove background noise, then analyzes the audio data and converts it into text.

[1085] Output: Converted text data (e.g. "Doraemon, tell me today's homework.")

[1086] Step 4: Emotion recognition processing

[1087] The server passes the text and voice data acquired by the voice recognition means to the Microsoft Azure Emotion API, which analyzes the user's emotional state.

[1088] Input: Text data and audio data converted by speech recognition

[1089] How it works: The server sends the data to the Microsoft Azure Emotion API, which analyzes the linguistic features of the audio waveform data and text data to recognize the user's emotional state (e.g., depression).

[1090] Output: User's emotional state (e.g., "depressed")

[1091] Step 5: Context understanding and response generation

[1092] The server then passes the text data based on the emotion and speech recognition results to OpenAI's GPT-4, which analyzes the text, understands the user's intent, and generates an appropriate response.

[1093] Input: User's emotional state and text data obtained by speech recognition means

[1094] How it works: The server inputs emotional state and text data as prompts into GPT-4, which generates an appropriate response to the user's question (e.g., "Today's homework is to do pages 10 to 15 of math").

[1095] Output: The generated response text

[1096] Step 6: Speech synthesis processing

[1097] The generated response text is passed to Amazon Polly, which converts it into voice data in the voice of the character selected by the user.

[1098] Input: GPT-4 generated response text

[1099] How it works: The server sends the response text to Amazon Polly, which converts it into speech data in the voice of a selected character (e.g., Doraemon), while also adjusting the tone of the voice to take into account emotional information.

[1100] Output: Generated audio data

[1101] Step 7: Respond to the user

[1102] The generated voice data is transmitted from the server to the terminal, which then plays the received voice data and provides a response to the user.

[1103] Input: Audio data sent from the server

[1104] How it works: The device's speaker plays audio data, and responds in a cheerful, encouraging tone, for example, in Doraemon's voice, saying, "Today's homework is to do pages 10 to 15 of math."

[1105] Output: A spoken response to the user

[1106] (Application example 2)

[1107] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1108] Modern work environments, especially logistics centers, require efficient work performance and reduced employee fatigue. Conversational support systems are useful for enabling workers to work efficiently while taking appropriate breaks. However, conventional dialogue systems have limitations in recognizing users' emotions and are sometimes unable to provide appropriate responses. Furthermore, there is a lack of systems that can properly analyze users' voice input and provide appropriate instructions and support information in real time.

[1109] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a voice input means that enables a user to input voice; a transmission means that transmits voice data acquired by the voice input means to the server; a voice recognition means that converts the voice data received by the server by the transmission means into text data; a natural language processing means that analyzes the text data converted by the voice recognition means, understands the user's intention, and generates a response; a voice synthesis means that converts the response generated by the natural language processing means into voice data in the voice of a selected character; a playback means that transmits the voice data generated by the voice synthesis means to a terminal and provides the response to the user; and an emotion recognition means that analyzes the user's emotional state and adjusts the response based on the analysis results. This allows appropriate instructions and support to be provided in real time according to the user's emotional state, improving work efficiency and reducing worker fatigue.

[1110] The "voice input means" is a device that allows the user to input voice (for example, a microphone).

[1111] The "transmission means" is a means having a function of transmitting the acquired voice data to the server.

[1112] The "voice recognition means" is a system that converts received voice data into text data.

[1113] "Natural language processing means" is a function that analyzes text data, understands the user's intent, and generates an appropriate response.

[1114] The "voice synthesis means" is a system that converts the generated response text into voice data in the voice of a selected character.

[1115] The "playback means" is a function that transmits the generated voice data to the terminal and provides a response to the user.

[1116] The "emotion recognition means" is a system that analyzes the user's voice data and text data to analyze the user's emotional state.

[1117] This invention provides a system that allows a user to receive work instructions using voice input. The system picks up the user's voice using voice input means and transmits the voice data to a server via transmission means. The server converts the received voice data into text data using voice recognition means and analyzes it using natural language processing means to understand the user's instructions. It also analyzes the user's emotional state using emotion recognition means and generates a response in an appropriate tone. The response is converted into voice data in the voice of a specified character by voice synthesis means and transmitted to a terminal via playback means.

[1118] Hardware and software used

[1119] Audio input method: Microphone

[1120] Transmission method: Network module or communication adapter

[1121] Speech recognition method: Google Speech-to-Text API

[1122] Natural language processing tools: Various NLP libraries (e.g. spaCy, NLTK, etc.)

[1123] Emotion recognition: sentiment_analysis_spanish library

[1124] Speech synthesis method: Google Text-to-Speech API

[1125] Playback via: Speakers, smart glasses, and smartphones

[1126] System processing flow

[1127] The server acquires the user's speech input, performs speech recognition, and analyzes the resulting text data. The analyzed text data is passed to a natural language processing means, which understands the user's intention and generates an appropriate response. The emotion recognition means analyzes the user's emotional state and adjusts the content and tone of the response. The generated response text is converted into voice data by a voice synthesis means and transmitted to the terminal via a playback means.

[1128] Specific examples

[1129] When a worker puts on the smart glasses and says, "What are the next work instructions?", the voice is recognized by the system and sent to the server. The server converts the voice into text data, performs emotion recognition, and generates the next work instruction using natural language processing. The generated response is synthesized in the character's voice and provided to the worker through the smart glasses. For example, if the worker is tired, instructions such as "Next, please pick up container B. Please take a break while working" are provided in an encouraging tone.

[1130] Prompt Sentence Examples

[1131] "Please tell me the next work instructions."

[1132] Check the current progress

[1133] "Please tell me when to take a break."

[1134] In this way, appropriate instructions and support can be provided in real time according to the user's emotional state, improving work efficiency and reducing worker fatigue.

[1135] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1136] Step 1:

[1137] The user inputs the voice. The user inputs the voice through a microphone. For example, the user might say, "Please tell me the next work instruction." This voice data is generated and sent to the next step.

[1138] Step 2:

[1139] The terminal transmits the user's voice data to the server via a transmission means. The voice data is input and transferred to the server as output. During this process, data compression and encryption may be performed.

[1140] Step 3:

[1141] The server receives the voice data. It converts the received voice data into text data using a speech recognition tool. The input is voice data, and the output is text data. This conversion is specifically performed using the Google Speech-to-Text API.

[1142] Step 4:

[1143] The server passes the converted text data to a natural language processing means for analysis. The input is text data, and the output is response data that understands the user's intent. In natural language processing, a specific NLP library (e.g., spaCy or NLTK) is used to analyze the user's intent.

[1144] Step 5:

[1145] The server passes the text data and voice data to the emotion recognition means to recognize the user's emotional state. The input is text data and voice data, and the output is emotional state information. The sentiment_analysis_spanish library is used for emotion recognition.

[1146] Step 6:

[1147] The server generates an appropriate response taking into account the emotional state and passes the response text to a speech synthesis means. The input is the response text and emotional state information, and the output is audio data. The response text is converted into audio using the Google Text-to-Speech API.

[1148] Step 7:

[1149] The server transmits the generated voice data to the terminal. The terminal uses a playback means to play the voice data to the user. Specifically, the voice is played using smart glasses or a smartphone. The input is the voice data, and the output is the voice heard by the user.

[1150] Step 8:

[1151] The user receives the voice instructions and performs the next task. For example, if the instruction is "Next, pick up container B. Please take a break while working," the user will continue working in accordance with the instructions.

[1152] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1153] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1154] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1155] [Fourth embodiment]

[1156] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1157] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1158] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1159] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1160] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1161] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1162] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1163] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1164] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1165] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1166] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1167] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1168] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1169] System Overview

[1170] The present invention provides a system that allows users to speak, converts the speech into text, generates appropriate responses based on information obtained from the text, and plays back the responses in the voice of a character. It is designed to be used in education and childcare settings, particularly to keep children engaged.

[1171] Program processing overview

[1172] 1. Accepting user input

[1173] The device is always on standby for the user's (especially children's) voice. When the user speaks, the device records the voice and transmits it to the server in real time.

[1174] 2. Speech Recognition Processing

[1175] The server receives the voice data sent from the terminal. The received voice data is passed to a voice recognition engine and converted into text data. This text data is used in the next processing step.

[1176] 3. Context Understanding and Response Generation

[1177] The server passes the text data obtained from the speech recognition engine to a natural language processing (NLP) engine. The NLP engine analyzes the text and understands the user's intent. For example, if a user says, "Tell me about today's homework," the NLP engine generates an appropriate response to that question.

[1178] 4. Speech synthesis processing

[1179] The generated response text is passed to a speech synthesis engine, which uses the voice of the character selected by the user to convert the response text into voice data that is used to respond to the user in the character's voice.

[1180] 5. Responding to the User

[1181] The generated voice data is sent from the server to the device, which then plays the received voice data and responds to the user (child) in the voice of the selected character.

[1182] Specific examples

[1183] Conversation example 1: Helping with homework

[1184] 1. Accepting user input

[1185] The terminal waits for the user's voice to say, "Doraemon, tell me today's homework."

[1186] When the user speaks, the device records the voice and immediately sends it to the server.

[1187] 2. Speech Recognition Processing

[1188] The server receives the voice data "Doraemon, tell me today's homework," passes it to a voice recognition engine, and converts it into text data.

[1189] 3. Context Understanding and Response Generation

[1190] The server passes the text data to an NLP engine to understand the user's intent (i.e., please help me with my homework).

[1191] The NLP engine generates the response, "Today's homework is to do pages 10 to 15 of math."

[1192] 4. Speech synthesis processing

[1193] The server passes the response text to a speech synthesis engine and converts it into audio data in Doraemon's voice.

[1194] 5. Responding to the User

[1195] The server transmits the generated voice data to the terminal.

[1196] The device plays the audio data and responds to the user in Doraemon's voice, saying, "Today's homework is to do pages 10 to 15 of math."

[1197] In this way, the system of the present invention can receive a user's voice input, generate an appropriate response, and play it back in the character's voice, enabling engaging and sustainable learning and parenting support, especially for children.

[1198] The processing flow will be explained below.

[1199] Step 1:

[1200] The device is always on standby for user voice input. When the user says, "Character (e.g., Doraemon), tell me what my homework is today," the device records the voice.

[1201] Step 2:

[1202] The device immediately sends the recorded audio data to a server, and the data is often encrypted before being sent.

[1203] Step 3:

[1204] The server receives the voice data sent from the device and then passes it to a voice recognition engine for analysis.

[1205] Step 4:

[1206] The speech recognition engine converts the received voice data into text data, generating text such as "Character, please tell me today's homework."

[1207] Step 5:

[1208] The server receives the text data from the speech recognition engine and passes it to a natural language processing (NLP) engine, which analyzes the context and understands the user's intent.

[1209] Step 6:

[1210] The NLP engine generates an appropriate response based on the user's intent, in this case, a response text such as "Today's homework is to do pages 10 to 15 of math."

[1211] Step 7:

[1212] The server passes the generated response text to the speech synthesis engine, instructing it to generate voice data in the voice of the character selected by the user (e.g., Doraemon).

[1213] Step 8:

[1214] The speech synthesis engine converts the response text into speech data in the voice of the designated character, utilizing the character's distinctive speech patterns.

[1215] Step 9:

[1216] The server sends the generated voice data to the terminal, allowing the response content to be delivered to the user in real time.

[1217] Step 10:

[1218] The device plays back the received voice data and responds to the user in the character's voice, saying, "Today's homework is to do pages 10 to 15 of math." This allows the user to enjoy the character's response.

[1219] Example 1

[1220] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1221] In recent years, there has been a demand in education and childcare for interactive learning support systems that can sustain children's interest. However, conventional systems lack the technology to properly analyze a user's voice input, understand their intent, generate a response, and then play that response in a character's voice, limiting their ability to sustain children's interest. The present invention aims to solve this problem and provide engaging and sustained educational support, especially for children.

[1222] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1223] In this invention, the server includes voice input means that allows a user to input voice, transmission means that transmits voice data acquired by the voice input means to a processing device, voice recognition means that converts the voice data received by the processing device by the transmission means into text data, natural language processing means that analyzes the text data converted by the voice recognition means, understands the user's intention, and generates a response, voice synthesis means that converts the response generated by the natural language processing means into voice data in the voice of a selected character, and playback means that transmits the voice data generated by the voice synthesis means to a receiving device and provides the response to the user. This makes it possible to analyze the user's voice instructions and provide an appropriate response in real time in the voice of a character.

[1224] "Voice input means" refers to a device or system that allows a user to input voice, and includes a microphone and the like.

[1225] "Transmission means" refers to a function for transferring acquired voice data to a processing device or a server.

[1226] "Speech recognition means" refers to the technology or software used to analyze received voice data and convert it into text data.

[1227] "Natural language processing means" refers to technology or software that analyzes text data converted by speech recognition means, understands the user's intent, and generates an appropriate response.

[1228] "Speech synthesis means" refers to technology or software for converting the generated response into audio data in the voice of a selected character.

[1229] The "playback means" refers to a device or system that outputs sound to provide the generated sound data to the user.

[1230] "Processing device" refers to a server or computer that receives, analyzes, and converts voice data sent from a terminal.

[1231] "Receiving device" refers to a terminal or device that receives audio data transmitted from a server.

[1232] The present invention is a system for use in education and childcare that converts user-input speech into text, generates a response based on information obtained from the text, and plays back the response in the voice of a character. The system includes a speech input unit, a transmission unit, a speech recognition unit, a natural language processing unit, a speech synthesis unit, and a playback unit.

[1233] The device is always on standby for the user to speak and records the voice through a microphone. For example, when a user says, "Doraemon, please tell me today's homework," the device transmits this voice data to a processing unit in real time.

[1234] The server receives the voice data sent from the device and converts it into text using the IBM Watson speech recognition engine. The converted text data is then analyzed using the Google Cloud Natural Language API to understand the user's intent. As a result of this analysis, the intent may be recognized as, for example, "Please tell me about my homework." Based on this information, the NLP engine generates an appropriate response text, such as, "Today's homework is to do pages 10 to 15 of math."

[1235] The server then passes the generated response text to the Amazon Polly speech synthesis engine, which converts it into audio data in the voice of the specified character. This audio data is then sent back to the device, where it is played back through the device's speaker, allowing the user to hear the response in Doraemon's voice.

[1236] As a concrete example, consider the case where a user talks about homework. When the user says, "Doraemon, tell me today's homework," the device records this voice and sends it to the server. The server receives the voice data, converts it into text data using a speech recognition engine, then analyzes it using a natural language processing engine to generate an appropriate response. The response is then converted into Doraemon's voice using a speech synthesis engine and sent to the device, which then plays it back. As a result, the user can hear Doraemon's voice saying, "Today's homework is to do pages 10 to 15 of math."

[1237] As an example of a prompt sentence, if you say "Doraemon, tell me the weather," the following will happen:

[1238] 1. User: "Doraemon, tell me the weather."

[1239] 2. Device: Records audio using a microphone and sends it to the server

[1240] 3. Server: Receives voice data and converts it into text data using a speech recognition engine

[1241] 4. Server: Passes text data to the NLP engine, understands user intent, and generates a response

[1242] 5. Server: Passes the response text to the speech synthesis engine and converts it into audio data in Doraemon's voice

[1243] 6. Server: Sends the generated voice data to the device

[1244] 7. Device: Plays audio data through the speaker and responds to the user in Doraemon's voice, saying "Today's weather is sunny."

[1245] In this way, the system of the present invention can analyze the user's voice instructions and provide appropriate responses in real time using the character's voice, thereby enabling attractive and sustainable learning and childcare support, especially for children.

[1246] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1247] Step 1: Input reception

[1248] The user stands in front of the device and says, "Doraemon, tell me what my homework is today." The device uses a microphone to wait for the user's voice input. When voice input is made, the device records this voice data. After recording is complete, the voice data is sent to the server.

[1249] input:

[1250] User voice data

[1251] output:

[1252] Audio data sent to the server

[1253] Specific behavior:

[1254] The device records audio through a microphone and transmits the recorded data to a server in real time.

[1255] Step 2: Speech recognition processing

[1256] The server receives the voice data sent from the device. The received voice data is passed to the IBM Watson speech recognition engine. The speech recognition engine analyzes the voice waveform and converts the voice data into text data.

[1257] input:

[1258] Audio data sent from the device

[1259] output:

[1260] Converting audio data into text data

[1261] Specific behavior:

[1262] The server passes the voice data to a voice recognition engine, which generates text data through waveform analysis.

[1263] Step 3: Context understanding and response generation

[1264] The server passes the text data obtained from the speech recognition engine to the Google Cloud Natural Language API. The NLP engine breaks the text down into tokens, performs grammatical analysis, and understands the user's intent. Specifically, it recognizes the intent as "Please tell me what my homework is." Based on this information, the NLP engine generates a response text such as "Today's homework is to do pages 10 to 15 of math."

[1265] input:

[1266] Text data obtained from the speech recognition engine

[1267] output:

[1268] Generated response text

[1269] Specific behavior:

[1270] The server passes the text data to an NLP engine, which analyzes the intent and generates a response text.

[1271] Step 4: Speech synthesis processing

[1272] The server passes the generated response text to the Amazon Polly speech synthesis engine, which converts the response text into audio data in the voice of the specified character.

[1273] input:

[1274] Generated response text

[1275] output:

[1276] Audio data converted into a character's voice

[1277] Specific behavior:

[1278] The server passes the response text to a speech synthesis engine, which generates voice data in the voice of the specified character.

[1279] Step 5: Respond to the user

[1280] The server sends the generated voice data to the device, which then plays it back through the speaker, allowing the user to hear Doraemon's voice reply, "Today's homework is to do pages 10 to 15 of math."

[1281] input:

[1282] Audio data sent from the server

[1283] output:

[1284] Audio that users can hear

[1285] Specific behavior:

[1286] The device receives the voice data and plays it over the speaker, and the user hears the response in Doraemon's voice.

[1287] Through the above steps, the system of the present invention analyzes the user's voice instructions and provides appropriate responses in real time using the character's voice.

[1288] (Application example 1)

[1289] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1290] In conventional factories, workers had to manually check instructions or ask other workers to understand work content and maintenance instructions, which resulted in a decrease in efficiency. Furthermore, systems that use voice input could only provide simple voice responses, making it difficult to provide the specific and immediate instructions workers needed. This significantly reduced work efficiency within the factory, and ultimately became a factor hindering productivity improvements.

[1291] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1292] In this invention, the server includes a voice input means that enables a user to input voice data, a transmission means that transmits voice data acquired by the voice input means to the server, a voice recognition means that converts the voice data received by the server by the transmission means into text data, a natural language processing means that analyzes the text data converted by the voice recognition means, understands the user's intention, and generates a response, a voice synthesis means that converts the response generated by the natural language processing means into voice data in the voice of a selected character, a playback means that transmits the voice data generated by the voice synthesis means to a terminal and provides the response to the user, and a means that recognizes voice input from a field worker and provides the character's response to the work instructions or maintenance questions. This enables work instructions and maintenance instructions to be given quickly and efficiently within a factory.

[1293] The "voice input means" is a device or system that allows a user to input voice.

[1294] The "transmitting means" is a device or system for transmitting the acquired voice data to the server.

[1295] A "voice recognition means" is a device or system for converting received voice data into text data.

[1296] A "natural language processing means" is a device or system that analyzes text data, understands the user's intent, and generates an appropriate response.

[1297] A "voice synthesis means" is a device or system that converts the generated response into voice data in the voice of a selected character.

[1298] A "playback means" is a device or system for providing the generated audio data to a user.

[1299] "Field worker" means a worker who performs work in a factory or work site.

[1300] "Work instructions" are instructions or guidance for carrying out a specific task or process.

[1301] "Maintenance questions" are questions about the maintenance and repair of machines and equipment.

[1302] A "character response" is a verbal response provided by the selected character.

[1303] This invention is a system for improving the efficiency of work instructions and maintenance in a factory. A user makes a voice inquiry, and an appropriate response is provided in the character's voice.

[1304] The main components of the system are:

[1305] 1. Voice input method

[1306] The microphones installed at each work station in the factory are used as devices for users to input voice, allowing workers to input voice in a hands-free manner.

[1307] 2. Transmission Method

[1308] The voice data acquired by the voice input means is transmitted to the server in real time via Wi-Fi or a wired network.

[1309] 3. Voice Recognition Methods

[1310] The server converts the received audio data into text data using the Google Cloud Speech-to-Text API, which prepares the audio data for analysis as text data.

[1311] 4. Natural Language Processing Methods

[1312] The text data is analyzed using the Google Cloud Natural Language API to understand the user's intent and generate an appropriate response, with the use of generative AI models to enable more accurate responses.

[1313] 5. Speech synthesis means

[1314] The generated response text is converted into audio data in a character's voice using the Google Cloud Text-to-Speech API. This character's voice is pre-set and can be selected from multiple options.

[1315] 6. Regeneration means

[1316] The resulting audio data is played back through the factory's speaker system to provide a response to the user, allowing workers to instantly obtain the information they need.

[1317] Specific examples

[1318] 1. User Input

[1319] The worker speaks into the microphone, asking, "What's the next step?" This voice is input into the system via the microphone.

[1320] 2. Voice Recognition

[1321] The input voice data is sent to the server and converted into text data such as "What's the next step?" using the Google Cloud Speech-to-Text API.

[1322] 3. Natural Language Processing

[1323] The text data is parsed by the Google Cloud Natural Language API, which understands that the request is for instructions on the next step, and the generative AI model generates a response such as, "The next step is to assemble the parts."

[1324] 4. Speech Synthesis

[1325] The generated response text is converted into audio data using the Google Cloud Text-to-Speech API, and a character's voice is generated saying, "The next step is assembling the parts."

[1326] 5. Responding to the User

[1327] The final audio data is played back through factory speakers and provided to workers.

[1328] Prompt Sentence Examples

[1329] User input: "What's next?"

[1330] This system will enable work instructions and maintenance instructions within the factory to be given quickly and efficiently, which is expected to improve work efficiency.

[1331] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1332] Step 1:

[1333] The user inputs voice using the voice input means. Specifically, a factory worker speaks into the microphone, saying, "What's the next process?" This voice data is acquired and sent to the server via the transmission means.

[1334] Input: Voice data (e.g., "What's the next step?")

[1335] Output: Audio data sent to the server

[1336] Step 2:

[1337] The server passes the received voice data to the speech recognition means and converts it into text data using the Google Cloud Speech-to-Text API, which converts the voice data into a string of characters.

[1338] Input: Audio data

[1339] Output: Text data (e.g. "What's the next step?")

[1340] How it works: Converts audio into text using the Google Cloud Speech-to-Text API.

[1341] Step 3:

[1342] The server passes the text data to a natural language processor, which uses the Google Cloud Natural Language API to analyze the text and understand the user's intent. A generative AI model is then used to generate an appropriate response to the user's question.

[1343] Input: Text data (e.g., "What's the next step?")

[1344] Output: Response text data (e.g. "The next step is assembling the parts.")

[1345] How it works: Uses the Google Cloud Natural Language API for text analysis and response generation.

[1346] Step 4:

[1347] The server passes the generated response text to the speech synthesis means, which converts it into audio data in the character's voice using the Google Cloud Text-to-Speech API, which converts the text into audio.

[1348] Input: Response text data (e.g., "The next step is assembling the parts.")

[1349] Output: Character voice data

[1350] How it works: Speech synthesis is performed using the Google Cloud Text-to-Speech API.

[1351] Step 5:

[1352] The server transmits the generated voice data to the terminal and uses a playback means to play back the response from the factory speaker system, allowing the worker to receive voice instructions regarding the next process.

[1353] Input: Character voice data

[1354] Output: Audio played through the speaker (e.g. "The next step is to assemble the parts.")

[1355] Action: Plays audio data on the device.

[1356] As described above, by executing each step consecutively, work instructions within the factory can be issued efficiently.

[1357] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1358] System Overview

[1359] This invention provides a system that allows users to input speech, converts the speech into text, generates appropriate responses based on information obtained from the text, and plays back the responses in a character's voice. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and generates responses based on those emotions, this system achieves more friendly and effective dialogue. It is designed to be used in education and childcare settings, particularly to keep children engaged.

[1360] Program processing overview

[1361] 1. Accepting user input

[1362] The device is always on standby for the user's (especially children's) voice. When the user speaks, the device records the voice and transmits it to the server in real time.

[1363] 2. Speech Recognition Processing

[1364] The server receives the voice data sent from the terminal. The received voice data is passed to a voice recognition engine and converted into text data. This text data is used in the next processing step.

[1365] 3. Emotion Recognition Processing

[1366] The server passes the text data obtained from the speech recognition engine to the emotion engine, which analyzes the text data and voice data and recognizes the user's emotional state (e.g., joy, sadness, anger, etc.).

[1367] 4. Context Understanding and Response Generation

[1368] The server passes the text data from the speech recognition engine to a natural language processing (NLP) engine, taking into account the user's emotional state obtained from the emotion engine. The NLP engine analyzes the text and understands the user's intent. For example, if a user says, "What is today's homework?", the NLP engine generates an appropriate response to the question: "Today's homework is to do pages 10 to 15 of math."

[1369] 5. Speech synthesis processing

[1370] The generated response text is passed to a speech synthesis engine, which converts the response text into audio data using the voice of a character selected by the user, while simultaneously adjusting the tone and expression of the voice to take into account the user's emotional state.

[1371] 6. Responding to the User

[1372] The generated voice data is sent from the server to the device. The device plays the received voice data and responds to the user (child) in the voice of the selected character. For example, the device could respond in Doraemon's voice, saying, "Today's homework is to do pages 10 to 15 of math," and if the user is feeling a little down, it could play an encouraging tone.

[1373] Specific examples

[1374] Conversation example 1: Helping with homework

[1375] 1. Accepting user input

[1376] The device waits for the user to say, "Doraemon, tell me what my homework is today." When the user speaks, the device records the voice and immediately sends it to the server.

[1377] 2. Speech Recognition Processing

[1378] The server receives the voice data "Doraemon, tell me today's homework," passes it to a voice recognition engine, and converts it into text data.

[1379] 3. Emotion Recognition Processing

[1380] The server passes the text data and voice data obtained from the voice recognition engine to the emotion engine, and recognizes the user's emotion (e.g., depressed).

[1381] 4. Context Understanding and Response Generation

[1382] The server passes the text data to the NLP engine based on the emotional information obtained from the emotion engine, understands the user's intention (please help me with my homework), and generates a response: "Today's homework is to do pages 10 to 15 of math."

[1383] 5. Speech synthesis processing

[1384] The server passes the response text to a speech synthesis engine, which converts it into audio data in Doraemon's voice, generating a cheerful tone to encourage the depressed user.

[1385] 6. Responding to the User

[1386] The server then sends the generated voice data to the device, which then plays it back, responding in Doraemon's cheerful voice, "Today's homework is to do pages 10 to 15 of math."

[1387] As described above, the present invention realizes a dialogue response system that takes emotions into consideration, thereby making it possible to provide users with friendly and effective educational and child-rearing support.

[1388] The processing flow will be explained below.

[1389] Step 1:

[1390] The device is always on standby for user voice input. When the user says, "Character (e.g., Doraemon), tell me what my homework is today," the device records the voice.

[1391] Step 2:

[1392] The device immediately sends the recorded audio data to a server, and the data is often encrypted before being sent.

[1393] Step 3:

[1394] The server receives the voice data sent from the device and then passes it to a voice recognition engine for analysis.

[1395] Step 4:

[1396] The speech recognition engine converts the received voice data into text data, generating text such as "Character, please tell me today's homework."

[1397] Step 5:

[1398] The server passes the text and voice data obtained from the speech recognition engine to the emotion engine, which analyzes the text and voice characteristics and recognizes the user's emotional state (e.g., joy, sadness, anger, etc.).

[1399] Step 6:

[1400] The server passes the text data from the speech recognition engine to the natural language processing (NLP) engine based on the user's emotional information obtained from the emotion engine. The NLP engine analyzes the text and understands the user's intent.

[1401] Step 7:

[1402] The NLP engine generates an appropriate response based on the user's intent and emotional information, in this case, a response text such as "Today's homework is to do pages 10 to 15 of math."

[1403] Step 8:

[1404] The server then passes the generated response text to the speech synthesis engine, which instructs it to generate voice data in the voice of a character selected by the user (e.g., Doraemon).The tone and expression of the voice are also adjusted according to the user's emotional state.

[1405] Step 9:

[1406] The speech synthesis engine converts the response text into voice data in the voice of the specified character, taking into account the user's emotions and using encouraging or gentle tones.

[1407] Step 10:

[1408] The server sends the generated voice data to the terminal, allowing the response content to be delivered to the user in real time.

[1409] Step 11:

[1410] The device plays back the received voice data and responds to the user in the character's voice, saying, "Today's homework is to do pages 10 to 15 of math." By playing back the voice in a tone that corresponds to the user's emotions, a more friendly dialogue is achieved.

[1411] Example 2

[1412] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1413] Conventional voice response systems generate responses without considering the user's emotions, which makes it difficult to realize more friendly and effective dialogue. Particularly in education and childcare settings, responses that respond to the user's emotions are required, but no system has been able to meet this demand. Furthermore, systems have been limited in their ability to provide responses in the voices of multiple characters, lacking the flexibility to allow users to choose from a variety of options.

[1414] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice input means, a transmission means, a voice recognition means, an emotion recognition means, a natural language processing means, a voice synthesis means, and a playback means. This makes it possible to recognize the user's emotional state and generate a response according to the emotion. Furthermore, by dynamically providing responses using the voices of multiple characters, it is possible to provide the user with a more familiar and effective dialogue experience.

[1415] "Voice input means" refers to a device or function that allows a user to input voice.

[1416] "Transmitting means" refers to a device or function for transmitting acquired voice data to a server.

[1417] "Speech recognition means" refers to a device or function for converting received voice data into text data.

[1418] "Emotion recognition means" refers to a device or function for analyzing the text data and voice data converted by the voice recognition means and recognizing the emotional state of the user.

[1419] "Natural language processing means" refers to a device or function that analyzes text data taking into account the emotional state recognized by the emotion recognition means, understands the user's intention, and generates a response.

[1420] "Speech synthesis means" refers to a device or function for converting a response generated by a natural language processing means into voice data in the voice of a selected character.

[1421] "Playback means" refers to a device or function for transmitting generated voice data to a terminal and providing a response to a user.

[1422] This invention provides a system that accepts user voice input, converts the speech into text, generates appropriate responses based on information obtained from the text, and plays the responses in a character's voice. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and generates responses based on those emotions, it achieves more friendly and effective dialogue. This system is specifically designed for use in education and childcare settings.

[1423] System Configuration

[1424] The system includes the following means:

[1425] 1. Voice input means: A device or function that allows a user to input voice. For example, when a user wants to ask a question, they can speak into the terminal.

[1426] 2. Transmission means: A device or function for transmitting acquired voice data to a server. The voice data acquired by the voice input means is transmitted to the server in real time.

[1427] 3. Speech recognition means: A device or function for converting received voice data into text data. Specifically, the voice data is converted into text data using the Google Cloud Speech-to-Text API.

[1428] 4. Emotion Recognition Means: A device or function for analyzing the text data and voice data converted by the Voice Recognition Means and recognizing the user's emotional state. The Microsoft Azure Emotion API is used to analyze the user's emotions.

[1429] 5. Natural language processing means: A device or function that analyzes text data taking into account the emotional state recognized by the emotion recognition means, understands the user's intention, and generates a response. OpenAI's GPT-4 is used to generate appropriate responses to user questions.

[1430] 6. Speech synthesis means: A device or function for converting the response generated by the natural language processing means into voice data in the voice of the selected character. Amazon Polly is used to convert into voice data in the voice of the selected character.

[1431] 7. Playback means: A device or function for transmitting the generated voice data to the terminal and providing a response to the user. The generated voice is played back through the terminal's speaker.

[1432] Specific processing operations

[1433] The specific processing operations of this system are shown below.

[1434] 1. Accepting user input

[1435] The device is always on standby for user voice input. For example, if a user says to the device, "Doraemon, tell me what my homework is today," the device records this voice and sends it to the server in real time.

[1436] 2. Speech Recognition Processing

[1437] The server receives the voice data sent from the device, passes it to the Google Cloud Speech-to-Text API, and converts it into text data, which is used for the next processing step.

[1438] 3. Emotion Recognition Processing

[1439] The server passes the text and voice data acquired by the speech recognition tool to the Microsoft Azure Emotion API, which analyzes the data and recognizes the user's emotional state. For example, it may recognize that the user is feeling a little depressed.

[1440] 4. Context Understanding and Response Generation

[1441] The server passes the text data from the speech recognition engine to OpenAI's GPT-4, taking into account the user's emotional state obtained from the emotion recognition engine. The NLP engine analyzes the text and understands the user's intent. For example, in response to the question "What is today's homework?", it generates a response such as "Today's homework is pages 10 to 15 of math."

[1442] 5. Speech synthesis processing

[1443] The generated response text is passed to Amazon Polly, which converts it into voice data in the voice of a character selected by the user, such as "Doraemon's voice."

[1444] 6. Responding to the User

[1445] The server sends the generated voice data to the device, which then plays the received voice data from its speaker and responds to the user in Doraemon's voice, saying, "Today's homework is to do pages 10 to 15 of math."

[1446] Specific examples

[1447] For example, if a user says, "Doraemon, tell me my homework today," the above process will result in a response in Doraemon's voice saying, "Today's homework is to do pages 10 to 15 of math." If the user is feeling down, the system can respond in a positive, encouraging tone.

[1448] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1449] Step 1: Accepting user input

[1450] The device is always on standby for user voice input. When the user says, "Doraemon, tell me what my homework is today," the device records the voice.

[1451] Input: User voice input

[1452] How it works: The device's built-in microphone captures and records audio. The voice recognition chip detects what the user is saying and starts the recording process.

[1453] Output: Recorded audio data

[1454] Step 2: Sending audio data

[1455] The recorded audio data is sent to the server in real time.

[1456] Input: Audio data recorded on the device

[1457] What it does: Uses the device's communications module to send audio data to a server over the internet.

[1458] Output: Audio data sent to the server

[1459] Step 3: Speech recognition processing

[1460] The server passes the received audio data to the Google Cloud Speech-to-Text API, which converts the audio data into text data, which is used for the next processing step.

[1461] Input: Audio data sent to the server

[1462] How it works: The server sends the audio data to the Google Cloud Speech-to-Text API, which uses a voice filtering function to remove background noise, then analyzes the audio data and converts it into text.

[1463] Output: Converted text data (e.g. "Doraemon, tell me today's homework.")

[1464] Step 4: Emotion recognition processing

[1465] The server passes the text and voice data acquired by the voice recognition means to the Microsoft Azure Emotion API, which analyzes the user's emotional state.

[1466] Input: Text data and audio data converted by speech recognition

[1467] How it works: The server sends the data to the Microsoft Azure Emotion API, which analyzes the linguistic features of the audio waveform data and text data to recognize the user's emotional state (e.g., depression).

[1468] Output: User's emotional state (e.g., "depressed")

[1469] Step 5: Context understanding and response generation

[1470] The server then passes the text data based on the emotion and speech recognition results to OpenAI's GPT-4, which analyzes the text, understands the user's intent, and generates an appropriate response.

[1471] Input: User's emotional state and text data obtained by speech recognition means

[1472] How it works: The server inputs emotional state and text data as prompts into GPT-4, which generates an appropriate response to the user's question (e.g., "Today's homework is to do pages 10 to 15 of math").

[1473] Output: The generated response text

[1474] Step 6: Speech synthesis processing

[1475] The generated response text is passed to Amazon Polly, which converts it into voice data in the voice of the character selected by the user.

[1476] Input: GPT-4 generated response text

[1477] How it works: The server sends the response text to Amazon Polly, which converts it into speech data in the voice of a selected character (e.g., Doraemon), while also adjusting the tone of the voice to take into account emotional information.

[1478] Output: Generated audio data

[1479] Step 7: Respond to the user

[1480] The generated voice data is transmitted from the server to the terminal, which then plays the received voice data and provides a response to the user.

[1481] Input: Audio data sent from the server

[1482] How it works: The device's speaker plays audio data, and responds in a cheerful, encouraging tone, for example, in Doraemon's voice, saying, "Today's homework is to do pages 10 to 15 of math."

[1483] Output: A spoken response to the user

[1484] (Application example 2)

[1485] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1486] Modern work environments, especially logistics centers, require efficient work performance and reduced employee fatigue. Conversational support systems are useful for enabling workers to work efficiently while taking appropriate breaks. However, conventional dialogue systems have limitations in recognizing users' emotions and are sometimes unable to provide appropriate responses. Furthermore, there is a lack of systems that can properly analyze users' voice input and provide appropriate instructions and support information in real time.

[1487] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a voice input means that enables a user to input voice; a transmission means that transmits voice data acquired by the voice input means to the server; a voice recognition means that converts the voice data received by the server by the transmission means into text data; a natural language processing means that analyzes the text data converted by the voice recognition means, understands the user's intention, and generates a response; a voice synthesis means that converts the response generated by the natural language processing means into voice data in the voice of a selected character; a playback means that transmits the voice data generated by the voice synthesis means to a terminal and provides the response to the user; and an emotion recognition means that analyzes the user's emotional state and adjusts the response based on the analysis results. This allows appropriate instructions and support to be provided in real time according to the user's emotional state, improving work efficiency and reducing worker fatigue.

[1488] The "voice input means" is a device that allows the user to input voice (for example, a microphone).

[1489] The "transmission means" is a means having a function of transmitting the acquired voice data to the server.

[1490] The "voice recognition means" is a system that converts received voice data into text data.

[1491] "Natural language processing means" is a function that analyzes text data, understands the user's intent, and generates an appropriate response.

[1492] The "voice synthesis means" is a system that converts the generated response text into voice data in the voice of a selected character.

[1493] The "playback means" is a function that transmits the generated voice data to the terminal and provides a response to the user.

[1494] The "emotion recognition means" is a system that analyzes the user's voice data and text data to analyze the user's emotional state.

[1495] This invention provides a system that allows a user to receive work instructions using voice input. The system picks up the user's voice using voice input means and transmits the voice data to a server via transmission means. The server converts the received voice data into text data using voice recognition means and analyzes it using natural language processing means to understand the user's instructions. It also analyzes the user's emotional state using emotion recognition means and generates a response in an appropriate tone. The response is converted into voice data in the voice of a specified character by voice synthesis means and transmitted to a terminal via playback means.

[1496] Hardware and software used

[1497] Audio input method: Microphone

[1498] Transmission method: Network module or communication adapter

[1499] Speech recognition method: Google Speech-to-Text API

[1500] Natural language processing tools: Various NLP libraries (e.g. spaCy, NLTK, etc.)

[1501] Emotion recognition: sentiment_analysis_spanish library

[1502] Speech synthesis method: Google Text-to-Speech API

[1503] Playback via: Speakers, smart glasses, and smartphones

[1504] System processing flow

[1505] The server acquires the user's speech input, performs speech recognition, and analyzes the resulting text data. The analyzed text data is passed to a natural language processing means, which understands the user's intention and generates an appropriate response. The emotion recognition means analyzes the user's emotional state and adjusts the content and tone of the response. The generated response text is converted into voice data by a voice synthesis means and transmitted to the terminal via a playback means.

[1506] Specific examples

[1507] When a worker puts on the smart glasses and says, "What are the next work instructions?", the voice is recognized by the system and sent to the server. The server converts the voice into text data, performs emotion recognition, and generates the next work instruction using natural language processing. The generated response is synthesized in the character's voice and provided to the worker through the smart glasses. For example, if the worker is tired, instructions such as "Next, please pick up container B. Please take a break while working" are provided in an encouraging tone.

[1508] Prompt Sentence Examples

[1509] "Please tell me the next work instructions."

[1510] Check the current progress

[1511] "Please tell me when to take a break."

[1512] In this way, appropriate instructions and support can be provided in real time according to the user's emotional state, improving work efficiency and reducing worker fatigue.

[1513] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1514] Step 1:

[1515] The user inputs the voice. The user inputs the voice through a microphone. For example, the user might say, "Please tell me the next work instruction." This voice data is generated and sent to the next step.

[1516] Step 2:

[1517] The terminal transmits the user's voice data to the server via a transmission means. The voice data is input and transferred to the server as output. During this process, data compression and encryption may be performed.

[1518] Step 3:

[1519] The server receives the voice data. It converts the received voice data into text data using a speech recognition tool. The input is voice data, and the output is text data. This conversion is specifically performed using the Google Speech-to-Text API.

[1520] Step 4:

[1521] The server passes the converted text data to a natural language processing means for analysis. The input is text data, and the output is response data that understands the user's intent. In natural language processing, a specific NLP library (e.g., spaCy or NLTK) is used to analyze the user's intent.

[1522] Step 5:

[1523] The server passes the text data and voice data to the emotion recognition means to recognize the user's emotional state. The input is text data and voice data, and the output is emotional state information. The sentiment_analysis_spanish library is used for emotion recognition.

[1524] Step 6:

[1525] The server generates an appropriate response taking into account the emotional state and passes the response text to a speech synthesis means. The input is the response text and emotional state information, and the output is audio data. The response text is converted into audio using the Google Text-to-Speech API.

[1526] Step 7:

[1527] The server transmits the generated voice data to the terminal. The terminal uses a playback means to play the voice data to the user. Specifically, the voice is played using smart glasses or a smartphone. The input is the voice data, and the output is the voice heard by the user.

[1528] Step 8:

[1529] The user receives the voice instructions and performs the next task. For example, if the instruction is "Next, pick up container B. Please take a break while working," the user will continue working in accordance with the instructions.

[1530] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1531] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1532] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1533] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1534] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1535] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1536] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1537] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1538] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1539] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1540] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1541] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1542] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1543] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1544] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1545] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1546] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1547] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1548] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1549] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1550] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1551] The following is further disclosed regarding the above embodiment.

[1552] (Claim 1)

[1553] a voice input means for enabling a user to provide voice input;

[1554] a transmitting means for transmitting the voice data acquired by the voice input means to a server;

[1555] a voice recognition means for converting the voice data received by the server via the transmission means into text data;

[1556] natural language processing means for analyzing the text data converted by the speech recognition means, understanding the user's intention, and generating a response;

[1557] a voice synthesis means for converting the response generated by the natural language processing means into voice data in the voice of a selected character;

[1558] a playback means for transmitting the voice data generated by the voice synthesis means to a terminal and providing a response to a user;

[1559] A system including:

[1560] (Claim 2)

[1561] 2. The system according to claim 1, further comprising means for taking into consideration a character setting selected by a user when generating an analysis result and a response by said natural language processing means.

[1562] (Claim 3)

[1563] 2. The system according to claim 1, wherein the voice data generated by the voice synthesis means can be provided in the voices of a plurality of different characters selected in advance by the user.

[1564] "Example 1"

[1565] (Claim 1)

[1566] a voice input means for enabling a user to provide voice input;

[1567] a transmitting means for transmitting the voice data acquired by the voice input means to a processing device;

[1568] a voice recognition means for converting the voice data received by the processing device through the transmission means into text data;

[1569] natural language processing means for analyzing the text data converted by the speech recognition means, understanding the user's intention, and generating a response;

[1570] a voice synthesis means for converting the response generated by the natural language processing means into voice data in the voice of a selected character;

[1571] a playback means for transmitting the voice data generated by the voice synthesis means to a receiving device and providing a response to a user;

[1572] A system including:

[1573] (Claim 2)

[1574] 2. The system according to claim 1, further comprising means for taking into consideration a character setting selected by a user when generating an analysis result and a response by said natural language processing means.

[1575] (Claim 3)

[1576] 2. The system according to claim 1, wherein the voice data generated by the voice synthesis means can be provided in the voices of a plurality of different characters selected in advance by the user.

[1577] "Application Example 1"

[1578] (Claim 1)

[1579] a voice input means for enabling a user to provide voice input;

[1580] a transmitting means for transmitting the voice data acquired by the voice input means to a server;

[1581] a voice recognition means for converting the voice data received by the server via the transmission means into text data;

[1582] natural language processing means for analyzing the text data converted by the speech recognition means, understanding the user's intention, and generating a response;

[1583] a voice synthesis means for converting the response generated by the natural language processing means into voice data in the voice of a selected character;

[1584] a playback means for transmitting the voice data generated by the voice synthesis means to a terminal and providing a response to a user;

[1585] a means for recognizing voice input from a field worker and providing character responses to the work instructions or maintenance questions;

[1586] A system including:

[1587] (Claim 2)

[1588] 2. The system according to claim 1, further comprising means for taking into consideration a character setting selected by a user when generating an analysis result and a response by said natural language processing means.

[1589] (Claim 3)

[1590] 2. The system according to claim 1, wherein the voice data generated by the voice synthesis means can be provided in the voices of a plurality of different characters selected in advance by the user.

[1591] "Example 2: Combining Emotion Engines"

[1592] (Claim 1)

[1593] a voice input means for enabling a user to provide voice input;

[1594] a transmitting means for transmitting the voice data acquired by the voice input means to a server;

[1595] a voice recognition means for converting the voice data received by the server via the transmission means into text data;

[1596] emotion recognition means for analyzing the text data and voice data converted by the voice recognition means and recognizing the emotional state of the user;

[1597] natural language processing means for analyzing text data in consideration of the emotional state recognized by the emotion recognition means, understanding the user's intention, and generating a response;

[1598] a voice synthesis means for converting the response generated by the natural language processing means into voice data in the voice of a selected character;

[1599] a playback means for transmitting the voice data generated by the voice synthesis means to a terminal and providing a response to a user;

[1600] A system including:

[1601] (Claim 2)

[1602] 10. The system of claim 1, further comprising: means for taking into account the emotional state of the user when generating analysis results and responses by the natural language processing means.

[1603] (Claim 3)

[1604] 2. The system according to claim 1, wherein the voice data generated by the voice synthesis means can be provided in the voices of a plurality of different characters selected in advance by the user.

[1605] "Application example 2 when combining emotion engines"

[1606] (Claim 1)

[1607] a voice input means for enabling a user to provide voice input;

[1608] a transmitting means for transmitting the voice data acquired by the voice input means to a server;

[1609] a voice recognition means for converting the voice data received by the server via the transmission means into text data;

[1610] natural language processing means for analyzing the text data converted by the speech recognition means, understanding the user's intention, and generating a response;

[1611] a voice synthesis means for converting the response generated by the natural language processing means into voice data in the voice of a selected character;

[1612] a playback means for transmitting the voice data generated by the voice synthesis means to a terminal and providing a response to a user;

[1613] emotion recognition means for analyzing an emotional state and adjusting a response based on the analysis;

[1614] A system including:

[1615] (Claim 2)

[1616] 2. The system of claim 1, further comprising means for taking into account a character setting selected by the user and an emotional state of the user when generating an analysis result and a response by the natural language processing means.

[1617] (Claim 3)

[1618] 2. The system according to claim 1, wherein the voice data generated by the voice synthesis means can be provided in tones corresponding to the voices and emotional states of a plurality of different characters selected in advance by the user. [Explanation of symbols]

[1619] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a voice input means for enabling a user to provide voice input; a transmitting means for transmitting the voice data acquired by the voice input means to a server; a voice recognition means for converting the voice data received by the server via the transmission means into text data; natural language processing means for analyzing the text data converted by the speech recognition means, understanding the user's intention, and generating a response; a voice synthesis means for converting the response generated by the natural language processing means into voice data in the voice of a selected character; a playback means for transmitting the voice data generated by the voice synthesis means to a terminal and providing a response to a user; A system including:

2. 2. The system according to claim 1, further comprising means for taking into consideration a character setting selected by a user when generating an analysis result and a response by said natural language processing means.

3. 2. The system according to claim 1, wherein the voice data generated by said voice synthesis means can be provided in the voices of a plurality of different characters selected in advance by the user.

Images