System

Abstract

To provide a system for supporting the intellectual training and growth of a child while reducing the labor of a parent.SOLUTION: The voice recognition system includes a means for acquiring voice uttered by a user, a voice recognition means for converting the acquired voice into text data, and a communication means for transmitting the converted text data. A system comprising: language analysis means for extracting a user's intention and emotional state; response generation means for generating an appropriate response based on the extracted intention and emotional state; communication means for receiving the generated response as text data; speech synthesis means for converting the received response text into voice data; and voice changer means for applying character voices to the converted voice data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, the number of single-parent households, where parents are forced to do housework and childcare alone, has been increasing. In such situations, parents often lack the time to spend with their children, and as a result, many parents try to temporarily solve this problem by letting their children watch videos. However, this can have a negative impact on children's eyesight and communication skills. Therefore, there is a need for a method to reduce the burden on parents while providing an environment where children can spend their time safely and healthily. [Means for solving the problem]

[0005] The present invention relates to an AI toy that allows parents to entertain their children while they are busy with housework and childcare. Specifically, it provides the following means:

[0006] A system including a means for acquiring voice uttered by a user, a voice recognition means for converting the acquired voice into text data, a communication means for transmitting the converted text data, a language analysis means for analyzing the transmitted text data and extracting the user's intention and emotional state, a response generation means for generating an appropriate response based on the extracted intention and emotional state, a communication means for receiving the generated response as text data, a voice synthesis means for converting the received response text into voice data, a voice changer means for applying a character voice to the converted voice data, and a voice output means for outputting voice data to which the character voice has been applied can support a child's intellectual development and growth while reducing the burden on parents.

[0007] The term "user" refers to the person who uses the system, and specifically in the present invention, it mainly refers to children.

[0008] "Speech recognition means" refers to a device or software that has the function of converting the voice spoken by the user into text data.

[0009] "Communication means" refers to a device or software that has the function of sending and receiving text data or other data over a network.

[0010] "Language analysis means" refers to a device or software that has the function of analyzing received text data and extracting the user's intentions and emotional state.

[0011] The "response generation means" refers to a device or software that has the function of generating an appropriate response based on the user's intention and emotional state extracted by the language analysis means.

[0012] "Speech synthesis means" refers to a device or software that has the function of converting text data into speech data.

[0013] "Voice changer means" refers to a device or software that has the function of applying a character voice to audio data.

[0014] "Audio output means" refers to a device or software that has the function of outputting converted audio data through a speaker or the like. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention relates to an AI toy that allows parents to entertain their children while they are busy with housework and childcare. The operation of the specific system and the processing of the related programs will be described in detail below.

[0037] This system mainly includes the following components: speech recognition means for acquiring voice uttered by the user, speech recognition means for converting the acquired voice into text data, communication means for transmitting the converted text data to a server, language analysis means for analyzing the transmitted text data and extracting the user's intentions and emotional state, response generation means for generating an appropriate response based on the analysis results, speech synthesis means for converting the generated response text into voice data, voice changer means for applying a character voice to the voice data, and voice output means for outputting the converted voice data.

[0038] A specific flow of program processing will be explained.

[0039] 1. User voice input

[0040] The user (child) speaks to the stuffed toy, for example, "Mommy, hold me!" This voice is captured by the device's built-in microphone.

[0041] 2. Voice Recognition

[0042] The device inputs the captured voice data into a voice recognition module, which converts the voice into text data such as "Mom, hold me!"

[0043] 3. Sending text data

[0044] The converted text data is transmitted to a server via the communication means of the terminal and over the Internet.

[0045] 4. Language Processing and Sentiment Analysis

[0046] The server inputs the received text data into a language analysis module, which analyzes the meaning and emotion of the text content. Through this analysis, for example, the intention of "I want to be held" and the "feeling of being pampered" can be extracted.

[0047] 5. Response Generation

[0048] Based on the results of the language analysis, the server generates an appropriate response text: "Okay, your stuffed animal is with you!"

[0049] 6. Sending Text Responses

[0050] The response text is sent from the server to the terminal.

[0051] 7. Speech Synthesis

[0052] The terminal inputs the received response text into a voice synthesis module, and converts the text data into voice data.

[0053] 8. Voice Change

[0054] The converted voice data is converted into a character voice by a voice changer means.

[0055] 9. Audio Output

[0056] Finally, the converted voice data is played back through the device's speaker, and the child hears, "Okay, your stuffed animal is with you!"

[0057] Through the above process, children can satisfy their emotions through interactions with stuffed toys while their parents are busy. To give an example, even if a child feels anxious about something, this system can provide an appropriate response immediately, reassuring the child. In this way, the present invention provides an effective means for reducing the burden on parents and supporting the healthy development of children.

[0058] The processing flow will be explained below.

[0059] Step 1:

[0060] The user (child) speaks to the stuffed animal. For example, they say, "Mommy, hold me!"

[0061] The device captures the user's voice through a built-in microphone.

[0062] Step 2:

[0063] The terminal inputs the acquired voice data into a voice recognition module.

[0064] The voice recognition module analyzes the voice data and converts it into text data such as "Mom, hold me!"

[0065] Step 3:

[0066] The terminal uses a communication means for transmitting the converted text data to the server.

[0067] The text data is sent to the server as an HTTP request.

[0068] Step 4:

[0069] The server inputs the received text data into a natural language processing (NLP) module.

[0070] The NLP module analyzes the text data and extracts the intent, "I want to be hugged."

[0071] Step 5:

[0072] The server passes the intent extracted by the NLP module to the sentiment analysis module.

[0073] The emotion analysis module analyzes the user's emotional state (e.g., affectionate feelings) from the text.

[0074] Step 6:

[0075] The server launches a response generation module based on the results of the language analysis and sentiment analysis.

[0076] The response generation module generates the appropriate response text: "Okay, your stuffed toy is with you!"

[0077] Step 7:

[0078] The server uses a communication means to transmit the generated response text to the terminal.

[0079] The response text is sent to the terminal as an HTTP response.

[0080] Step 8:

[0081] The terminal inputs the received response text into the speech synthesis module.

[0082] The speech synthesis module converts the response text into speech data.

[0083] Step 9:

[0084] The terminal passes the voice data to the voice changer means.

[0085] The voice changer converts audio data into the character voice of the stuffed animal.

[0086] Step 10:

[0087] The device transmits the converted audio data to the speaker.

[0088] A voice will play from the speaker saying, "Okay, your stuffed animal is with you!"

[0089] Through the above process, the user (child) can feel satisfied through interaction with the stuffed toy. The present invention provides an environment where children can play safely even while their parents are busy.

[0090] Example 1

[0091] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0092] Conventional systems lacked a means for parents to effectively communicate with their children when they were busy with housework and childcare. Furthermore, their limited ability to accurately analyze ambiguous words and emotions expressed by children and respond appropriately led to situations where children felt uneasy. Furthermore, the technology to apply character voices to voice data to provide appropriate responses for children was insufficient.

[0093] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0094] In this invention, the server includes means for acquiring voice uttered by the user, voice recognition means for converting the acquired voice into text data, communication means for transmitting the converted text data, language analysis means for analyzing the transmitted text data and extracting the user's intention and emotional state, response generation means for generating an appropriate response based on the extracted intention and emotional state, communication means for receiving the generated response as text data, voice synthesis means for converting the received response text into voice data, voice changer means for applying a character voice to the converted voice data, and voice output means for outputting the voice data to which the character voice has been applied. This allows children to communicate effectively and feel at ease even when their parents are busy.

[0095] A "user" is a subject who uses the system, specifically a person who gives instructions to the system by speaking.

[0096] "Speech recognition means" refers to a device or software that has the function of analyzing acquired voice data and converting it into text data.

[0097] "Communication means" refers to devices or software with Internet connection capabilities for sending and receiving data between a terminal and a server.

[0098] "Language analysis means" refers to a device or software that has the function of analyzing received text data and extracting the user's intentions and emotional state.

[0099] "Response generation means" refers to a device or software that has the function of generating an appropriate response based on the analyzed user's intentions and emotional state.

[0100] "Speech synthesis means" refers to a device or software that has the function of converting text data into voice data.

[0101] "Voice changer means" refers to a device or software that has the function of applying a character voice to audio data and changing the voice quality to a specific one.

[0102] "Audio output means" refers to a device such as a speaker that is used to play back the final converted audio data.

[0103] A "generative AI model" refers to an algorithm or computer program that uses artificial intelligence to generate text.

[0104] The present invention relates to an AI toy that can entertain children while parents are busy. The system uses multiple hardware and software components to implement a series of processes that captures user-spoken speech and converts it into an appropriate response. The specific components of the system and their operation are described in detail below.

[0105] The system consists of the following components: a voice recognition means for acquiring voice uttered by the user, a voice recognition means for converting the acquired voice into text data, a communication means for transmitting the converted text data to a server, a language analysis means for analyzing the transmitted text data and extracting the user's intentions and emotional state, a response generation means for generating an appropriate response based on the analysis results, a voice synthesis means for converting the generated response text into voice data, a voice changer means for applying a character voice to the voice data, and a voice output means for outputting the converted voice data.

[0106] Hardware and software used

[0107] Specifically, this system uses the following hardware and software.

[0108] 1. The speech recognition means uses the Google® Cloud Speech-to-Text API, which allows captured voice data to be converted into text data.

[0109] 2. The communication method is to connect to the Internet using a regular Wi-Fi module, which allows data to be sent and received.

[0110] 3. The language analysis means uses IBM Watson (registered trademark) NLU, which can analyze the user's intentions and emotional state from the transmitted text data.

[0111] 4. The response generation means uses OpenAI (registered trademark) GPT-3 (registered trademark). This allows it to generate appropriate responses based on the analyzed intent and emotional state. An example of a prompt sentence is, "What is an appropriate response when a child asks, 'Where is my stuffed animal?'"

[0112] 5. Amazon Polly is used as the speech synthesis means, which allows the generated text data to be converted into speech data.

[0113] 6. The voice changer uses Voicemod, which allows you to apply character voices to audio data.

[0114] 7. The built-in speaker is used as the audio output means, which allows the final audio data to be provided to the user.

[0115] Specific examples

[0116] A specific example of the system's operation is shown below.

[0117] The child turns to the stuffed animal and says, "Mommy, where is my stuffed animal?" The audio is captured by the device's built-in microphone.

[0118] The captured voice data is converted into text data using the Google Cloud Speech-to-Text API, generating the text "Mom, where's my stuffed animal?"

[0119] This text data is sent to the server using a Wi-Fi module.

[0120] The server uses IBM Watson NLU to analyze the intent of "I want to know where the stuffed animal is" from the text data.

[0121] The prompt for OpenAI GPT-3 was "What is the appropriate response when a child asks, 'Where is your stuffed animal?'" and an appropriate response was generated, resulting in the text data "Your stuffed animal is in the living room!"

[0122] This response text is sent from the server to the terminal.

[0123] The device converts the response text into voice data using Amazon Polly.

[0124] The converted voice data is then converted into a character voice using Voicemod.

[0125] Finally, the built-in speaker will play a voice saying, "The stuffed animal is in the living room!"

[0126] This allows children to feel safe and communicate effectively even while their parents are busy.

[0127] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0128] Step 1:

[0129] The device's built-in microphone captures the user's speech. This speech data becomes the input for the system. Speech data is generated when the user speaks a specific phrase, such as "Where is my stuffed animal?"

[0130] Step 2:

[0131] The device sends the captured voice data to the Google Cloud Speech-to-Text API, which converts the voice into text data. The input of this process is the voice data, and the output is the corresponding text data: "Where is the stuffed animal?"

[0132] Step 3:

[0133] The terminal transmits the converted text data to the server using a communication means (Wi-Fi module). At this time, the input is the converted text data, and the output is the data transmitted to the server.

[0134] Step 4:

[0135] The server receives the text data sent from the terminal. The input of this process is the text data from the terminal, and the output is the text data stored in the server.

[0136] Step 5:

[0137] The server inputs the received text data into IBM Watson NLU for language analysis, which extracts the user's intent and emotional state. The input for this process is text data, and the output is the analysis results. For example, the intent, "I want to know where the stuffed animal is," and the emotion, "confusion," are extracted.

[0138] Step 6:

[0139] The server uses OpenAI GPT-3 based on the analysis results to generate an appropriate response text. The prompt text used is something like, "What is the appropriate response when a child asks, 'Where is your stuffed animal?'" The input for this process is the analysis results, and the output is the response text, "Your stuffed animal is in the living room!"

[0140] Step 7:

[0141] The server transmits the generated response text to the terminal via a communication means, where the input is the generated response text and the output is data transmission to the terminal.

[0142] Step 8:

[0143] The terminal receives the response text sent from the server. The input of this process is the response text, and the output is text data stored in the terminal.

[0144] Step 9:

[0145] The device inputs the received response text into Amazon Polly and converts it into voice data. The input of this process is the response text, and the output is the generated voice data.

[0146] Step 10:

[0147] The device inputs the converted voice data into Voicemod, which converts it into a character voice. The input of this process is the generated voice data, and the output is the voice data with the character voice applied.

[0148] Step 11:

[0149] Finally, the device uses its built-in speaker to play back the voice data with the character voice applied. The input of this process is the voice data with the character voice, and the output is the actual voice that is played back. The user can hear the voice saying, "The stuffed animal is in the living room!"

[0150] (Application example 1)

[0151] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0152] The challenge is to solve the problem of children becoming bored or anxious while their parents are shopping in physical stores, and to provide a safe and enjoyable environment for children.To solve this problem, we aim to provide interactive entertainment for children and reduce the burden on parents.

[0153] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0154] In this invention, the server includes: means for acquiring voice uttered by a user; speech recognition means for converting the acquired voice into text data; communication means for transmitting the converted text data; language analysis means for analyzing the transmitted text data and extracting the user's intention and emotional state; response generation means for generating an appropriate response based on the extracted intention and emotional state; communication means for receiving the generated response as text data; speech synthesis means for converting the received response text into voice data; voice changer means for applying a character voice to the converted voice data; voice output means for outputting the voice data to which the character voice has been applied; and means for implementing interactions such as stories and games to provide entertainment for children. This allows children to enjoy interactive entertainment while parents can shop with peace of mind.

[0155] The "means for acquiring the voice uttered by the user" is a device for physically capturing the voice spoken by the user.

[0156] The "voice recognition means for converting captured voice into text data" is a technology for recognizing and converting captured voice into digital text data.

[0157] The "communication means for transmitting converted text data" is a method for transmitting the converted text data to a remote server or other device.

[0158] "Language analysis means that analyzes transmitted text data and extracts the user's intentions and emotional state" refers to software or algorithms that analyze received text data to understand the user's intentions and emotional state.

[0159] The "response generation means for generating an appropriate response based on the extracted intention and emotional state" is a system that generates an appropriate response based on the analysis results.

[0160] The "communication means for receiving the generated response as text data" is a method for receiving the generated response text data from the server back to the terminal.

[0161] The "voice synthesis means for converting received response text into voice data" is a technology for converting received text data into natural voice.

[0162] The "voice changer means for applying a character voice to converted voice data" is a technology for applying the voice of a specific character to generated voice data.

[0163] The "audio output means for outputting audio data to which a character voice has been applied" is a device that outputs the converted audio data to the outside through a speaker.

[0164] "Means for implementing interactions such as stories and games to provide entertainment for children" refers to equipment and technology that provides interactive content such as stories and games that children can enjoy.

[0165] The present invention relates to an interactive entertainment system that helps keep children entertained while their parents shop in a brick-and-mortar store. The system's primary purpose is to provide entertainment for children and make shopping safer and more convenient for parents.

[0166] The system includes the following elements:

[0167] 1. Voice acquisition means: A microphone attached to the terminal is used to acquire the voice spoken by the user (child).

[0168] 2. Speech recognition means: Speech recognition software (e.g., Google Speech Recognition API) is used to convert the acquired voice data into text data.

[0169] 3. Communication means: The communication module of the terminal is used to send the converted text data to the server.

[0170] 4. Language analysis means: The server uses natural language processing (NLP) algorithms (e.g., TextBlob) to analyze the received text data and extract the user's intent and emotional state.

[0171] 5. Response generation means: A generative AI model (e.g., GPT-3) that generates appropriate responses based on the extracted intent and emotional state is installed on the server.

[0172] 6. Communication means: The communication module of the server is used to send the generated response text data back to the terminal.

[0173] 7. Speech synthesis means: The terminal uses a speech synthesis tool (e.g., gTTS) to convert the received response text into voice data.

[0174] 8. Voice changer means: Using voice changer software to apply a character voice to the converted voice data.

[0175] 9. Audio output means: Audio data to which the character voice is applied is output through the device speaker.

[0176] Detailed explanation of the process

[0177] The process performed by the server is as follows: First, the microphone of the terminal captures the voice of the child speaking. The acquired voice data is converted into text data by voice recognition means. This converted text data is sent to the server via communication means. Next, the server analyzes the received text data using language analysis means to extract the user's intention and emotional state. Based on the analysis results, response generation means generates an appropriate response text. This generated response text is again sent to the terminal via communication means and converted into voice data by voice synthesis means. Furthermore, voice changer means is used to apply a character voice to this voice data. Finally, the converted voice data with the character voice is output through the terminal's speaker.

[0178] Specific examples

[0179] For example, while a parent is ordering coffee at a cafe in a brick-and-mortar store, a child might say to a robot, "I feel lonely." The robot's microphone captures the voice and converts it into text data using a speech recognition means. This text data is sent to a server, which then analyzes the emotion "I feel lonely" using a language analysis means. Based on the analysis results, a response text, "It's okay, let's play together!", is generated by a response generation means, which is then sent back to the terminal and converted into voice data by a voice synthesis means. After a character voice is applied by a voice changer means, this voice data is played back to the child through a speaker, reassuring the child.

[0180] Prompt Sentence Examples

[0181] python

[0182] if __name__ == "__main__":

[0183] print("System booting...")

[0184] Example

[0185] Child says "I'm lonely"

[0186] user_input = "lonely"

[0187] response = generate_response(user_input)

[0188] synthesize_speech(response)

[0189] Using this prompt, you can see how the system captures the child's speech and returns an appropriate response.

[0190] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0191] Step 1:

[0192] The user (child) speaks. The device's built-in microphone captures the voice data of the child. At this stage, the input is the user's voice, and the output is voice data.

[0193] Step 2:

[0194] The device sends the captured voice data to a voice recognition means, which (for example, Google Speech Recognition API) converts the voice data into text data. At this stage, the input is voice data and the output is text data.

[0195] Step 3:

[0196] The terminal sends the converted text data to the server using a communication means. The communication means delivers the text data to the server via the Internet. The input at this stage is text data, and the output is the transmission of text data to the server.

[0197] Step 4:

[0198] The text data received by the server is input to the language analysis means. The language analysis means (e.g., TextBlob) analyzes the text data and extracts the user's intention and emotional state. The input at this stage is text data, and the output is the user's intention and emotional state.

[0199] Step 5:

[0200] Based on the extracted intention and emotional state, the server generates an appropriate response using a response generation means (e.g., generative AI model: GPT-3). The input at this stage is the user's intention and emotional state, and the output is response text data.

[0201] Step 6:

[0202] The generated response text data is sent to the terminal using the communication means of the server. The input at this stage is the response text data, and the output is the transmission of the text data to the terminal.

[0203] Step 7:

[0204] The response text data received by the terminal is input to a speech synthesis means. The speech synthesis means (e.g., gTTS) converts the text data into speech data. The input at this stage is the response text data, and the output is the response speech data.

[0205] Step 8:

[0206] The terminal inputs the converted voice data to the voice changer means, which applies a character voice to the voice data. The input at this stage is the response voice data, and the output is the voice data after the character voice has been applied.

[0207] Step 9:

[0208] The terminal plays the voice data with the character voice applied from the speaker through the voice output means. The input at this stage is the voice data with the character voice applied, and the output is the actual voice. The process is completed when the child listens to this voice and responds interactively.

[0209] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0210] The present invention relates to an AI toy that allows parents to entertain their children while they are busy with housework and childcare. The operation of the specific system and the processing of the related programs will be described in detail below.

[0211] This system mainly includes the following components: speech recognition means for acquiring speech uttered by the user, speech recognition means for converting the acquired speech into text data, communication means for transmitting the converted text data to a server, language analysis means for analyzing the transmitted text data and extracting the user's intentions and emotional state, and an emotion engine for recognizing the emotional state, response generation means for generating an appropriate response based on the analysis results, speech synthesis means for converting the generated response text into speech data, voice changer means for applying a character voice to the speech data, and speech output means for outputting the converted speech data.

[0212] A specific flow of program processing will be explained.

[0213] 1. User voice input

[0214] The user (child) speaks to the stuffed toy, for example, "Mommy, hold me!" This voice is captured by the device's built-in microphone.

[0215] 2. Voice Recognition

[0216] The device inputs the captured voice data into a voice recognition module, which converts the voice into text data such as "Mom, hold me!"

[0217] 3. Sending text data

[0218] The converted text data is transmitted to a server via the communication means of the terminal and over the Internet.

[0219] 4. Language Processing and Sentiment Analysis

[0220] The server inputs the received text data into a language analysis module and analyzes the meaning of the text content.

[0221] At the same time, the server uses an emotion engine to analyze the user's emotional state from the text data. For example, the emotional state "feeling dependent" is extracted.

[0222] 5. Response Generation

[0223] Based on the results of language analysis and sentiment analysis, the server activates a response generation module, which generates an appropriate response based on the user's emotional state.

[0224] For example, an appropriate response text might be generated: "Okay, your stuffed animal is with you!"

[0225] 6. Sending Text Responses

[0226] The response text is sent from the server to the terminal.

[0227] 7. Speech Synthesis

[0228] The terminal inputs the received response text into a voice synthesis module, and converts the text data into voice data.

[0229] 8. Voice Change

[0230] The converted voice data is converted into a character voice by a voice changer means.

[0231] 9. Audio Output

[0232] Finally, the converted voice data is played back through the device's speaker, and the child hears, "Okay, your stuffed animal is with you!"

[0233] In this system, the emotion engine continuously monitors the user's emotional state and adjusts the response by comparing it with past emotional states. For example, if the user continuously displays sad emotions, the emotion engine can detect this and be programmed to provide a more specific response. In this way, the present invention can respond to a child's emotional needs while reducing the burden on parents. For example, if a child is feeling anxious, the stuffed toy can respond appropriately and calm the child. This provides a safe environment for the child even while the parent is busy.

[0234] The processing flow will be explained below.

[0235] Step 1:

[0236] The user (child) speaks to the stuffed animal, for example, "Mommy, hold me!" The device's built-in microphone captures this voice.

[0237] Step 2:

[0238] The device inputs the captured voice data into the voice recognition module, which analyzes the voice data and converts it into text data such as "Mama, hold me!"

[0239] Step 3:

[0240] The terminal uses a communication means to transmit the converted text data to the server, and the text data is transmitted to the server as an HTTP request.

[0241] Step 4:

[0242] The server inputs the received text data into a natural language processing (NLP) module, which analyzes the text data and extracts the intent, "I want a hug."

[0243] Step 5:

[0244] The server passes the intent extracted by the NLP module to the emotion engine, which analyzes the user's emotional state (e.g., dependent feelings) from the text data.

[0245] Step 6:

[0246] Based on the results of the language analysis and sentiment analysis, the server launches a response generation module, which generates an appropriate response text: "Okay, your stuffed toy is with you!"

[0247] Step 7:

[0248] The server sends the generated response text to the terminal, which then sends the response text to the terminal as an HTTP response.

[0249] Step 8:

[0250] The terminal inputs the received response text into a speech synthesis module, which converts the text data into speech data.

[0251] Step 9:

[0252] The terminal passes the converted voice data to a voice changer means, which converts the voice data into the voice of the character of the stuffed toy.

[0253] Step 10:

[0254] The device sends the converted character voice data to the speaker, which then plays back the voice, "Okay, your stuffed animal is with you!"

[0255] Through these specific steps, children can fulfill their emotions through interactions with stuffed toys. The present invention provides a safe environment for children even while their parents are busy, and the emotion engine has the function of continuously monitoring the user's emotional state and adjusting responses, thereby supporting the healthy development of children.

[0256] As a specific example, even if a child feels anxious about something, this system can immediately provide an appropriate response, putting the child at ease. For example, if a child says, "I'm scared," the system analyzes the emotion and generates a response such as, "It's okay, your stuffed toy is with you," providing reassurance to the child. In this way, the present invention reduces the burden on parents and enables appropriate responses to be made according to the emotional needs of children.

[0257] Example 2

[0258] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0259] In modern families, parents are often busy with work and housework, making it difficult for them to spend enough time with their children. There is also a need for systems that can respond appropriately when children feel anxious or lonely. Furthermore, conventional speech recognition and conversational systems lack the ability to accurately understand the user's emotional state and adjust responses accordingly, which means they are unable to adequately meet children's emotional needs.

[0260] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0261] In this invention, the server includes: means for acquiring voice uttered by the user; speech recognition means for converting the acquired voice into text data; communication means for transmitting the converted text data; language analysis means for analyzing the transmitted text data and extracting the user's intention and emotional state; response generation means for generating an appropriate response based on the extracted intention and emotional state; communication means for receiving the generated response as text data; speech synthesis means for converting the received response text into voice data; voice changer means for applying a character voice to the converted voice data; voice output means for outputting the voice data to which the character voice has been applied; and means for the response generation means to adjust the response by comparing it with the user's past emotional state. This makes it possible to provide an environment where parents can spend time with their children with peace of mind even when they are busy.

[0262] The "means for acquiring the voice uttered by the user" refers to hardware or software for capturing the voice spoken by the user (child).

[0263] The "voice recognition means for converting acquired voice into text data" is a technical means for converting voice data into text format.

[0264] The "communication means for transmitting the converted text data" refers to communication technology or equipment for transmitting the text data to a server or the like.

[0265] The "language analysis means for analyzing transmitted text data and extracting the user's intentions and emotional state" is a technology that has the function of analyzing the user's intentions and emotions from text data.

[0266] The "response generation means for generating an appropriate response based on the extracted intention and emotional state" is a technology for generating an appropriate reply according to the user's intention and emotion.

[0267] The "communication means for receiving the generated response as text data" refers to a communication technique or device for receiving the response text from the server.

[0268] The "voice synthesis means for converting the received response text into voice data" is a technical means for converting text data into voice data.

[0269] The "voice changer means for applying a character voice to converted voice data" is a technology for converting voice data into a specific character voice.

[0270] "Audio output means for outputting audio data to which a character voice has been applied" refers to technology such as a speaker for playing back audio data converted into a character voice.

[0271] "A means for generating a response that has the function of adjusting the response by comparing it with the user's past emotional state" refers to a technology that has the function of optimizing the current response by taking into account the user's past emotional data.

[0272] MODE FOR CARRYING OUT THE INVENTION

[0273] This invention relates to an AI toy that helps parents entertain their children while they are busy with housework and childcare. This system acquires speech uttered by the user (child) and generates and outputs an appropriate response. The detailed configuration and operation of the system are described below.

[0274] System Configuration

[0275] The system mainly includes the following hardware and software:

[0276] Voice acquisition means: A microphone (e.g., SHURE MV88) for acquiring the voice spoken by the user

[0277] Speech recognition means: A speech recognition module that converts captured speech into text data (e.g., Google Speech-to-Text API).

[0278] Communication means: Wi-Fi module (e.g., ESP32) for sending converted text data to a server

[0279] Language analysis means: A language analysis module (e.g., Google Natural Language API) that analyzes the transmitted text data to extract the user's intentions and emotional state.

[0280] Response generation means: A module that generates appropriate responses based on the extracted intentions and emotional states.

[0281] Speech synthesis means: A module that converts the generated response text into voice data (e.g., Amazon Polly)

[0282] Voice changer means: A device for applying a character voice to converted voice data (e.g., HT8955A)

[0283] Audio output means: a speaker for outputting audio data to which character voices are applied

[0284] Operation explanation

[0285] The server and the terminal work together to capture the user's voice and generate and output an appropriate response.

[0286] 1. Audio capture

[0287] The user speaks to the stuffed animal. For example, they say, "Mommy, hold me!" To capture this voice, a microphone (SHURE MV88) in the device captures the sound.

[0288] 2. Voice Recognition

[0289] The acquired voice data is input into a voice recognition module (Google Speech-to-Text API), and the voice data is converted into text data such as "Mom, hold me!"

[0290] 3. Data Transmission

[0291] The converted text data is sent to the server via the communication method (ESP32) using an HTTP POST request.

[0292] 4. Language and Sentiment Analysis

[0293] The server inputs the received text data into a language analysis module (Google Natural Language API) to analyze the meaning of the text content. At the same time, it uses an emotion engine to analyze the user's emotional state from the text data. For example, it can extract the emotional state of "feeling dependent."

[0294] 5. Response Generation

[0295] Based on the results of language analysis and sentiment analysis, the server activates a response generation module, which generates a response text such as "I understand, your stuffed toy is with you!"

[0296] 6. Response text reception and speech synthesis

[0297] The generated response text is sent from the server to the terminal, which inputs it into a speech synthesis module (Amazon Polly) to convert the text data into speech data.

[0298] 7. Voice Change and Audio Output

[0299] The converted voice data is converted into a character voice by the voice changer means (HT8955A) and then played back through the speaker, so that the user can hear the voice saying, "Okay, your stuffed toy is with you!"

[0300] Specific examples

[0301] The specific hardware and software used are as follows:

[0302] Microphone: SHURE MV88

[0303] Communication module: ESP32

[0304] Server: General cloud server

[0305] Speech Recognition Module: Google Speech-to-Text API

[0306] Language Analysis Module: Google Natural Language API

[0307] Speech synthesis module: Amazon Polly

[0308] Voice changer: HT8955A

[0309] Prompt Sentence Examples

[0310] "Analyze text data, such as a user saying, 'Hold me, Mommy!', recognize their emotional state, and generate an appropriate response."

[0311] Thus, the present invention can provide a sense of security to children even while parents are busy. In particular, the system has the ability to monitor the user's emotional state, compare it with past emotional states, and adjust responses to appropriately meet the emotional needs of children.

[0312] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0313] Step 1:

[0314] The user speaks to the stuffed toy. The user's voice (e.g., "Mommy, hold me!") is picked up by the device's microphone. The input is audio data, and the output is the captured raw audio signal. Specifically, the audio is picked up as an analog signal through the built-in microphone (e.g., SHURE MV88).

[0315] Step 2:

[0316] The device sends the captured voice signal to a voice recognition module, which (for example, Google Speech-to-Text API) converts the voice into text data. The input is voice data, and the output is text data such as "Mommy, hold me!". Specifically, the analog voice signal obtained from the microphone is digitized, then linked to the API and converted into text data.

[0317] Step 3:

[0318] The device sends the converted text data to the server via a communication method (for example, a Wi-Fi module such as ESP32). The input is the text data "Hold me, Mom!" and the output is an HTTP POST request to the server. Specifically, the text data is sent to the server via the Wi-Fi module over the Internet.

[0319] Step 4:

[0320] The server inputs the received text data into a language analysis module. The language analysis module (for example, Google Natural Language API) analyzes the meaning of the text content. The input is the text data "Mama, hold me!", and the output is the analyzed user intent. Specifically, the text data is sent via the API, and its content is analyzed.

[0321] Step 5:

[0322] At the same time, the server uses an emotion engine to analyze the user's emotional state from the text data. The input is the text data "Hold me, Mommy!", and the output is the emotional state "Feeling pampered." Specifically, the emotion engine runs an algorithm to identify the emotion based on the text analysis results.

[0323] Step 6:

[0324] The server launches a response generation module based on the results of language analysis and sentiment analysis. The generated response text is "I get it, your stuffed toy is with you!" The input is the analyzed intention and emotional state, and the output is the response text. Specifically, the response generation algorithm generates an appropriate response based on the analysis results.

[0325] Step 7:

[0326] The response text is sent from the server to the device. The input is the response text "Okay, your stuffed animal is with you!", and the output is the text data received by the device. Specifically, the server sends the text data as an HTTP response, and the device receives it.

[0327] Step 8:

[0328] The device inputs the received response text into a speech synthesis module and converts it into voice data (for example, Amazon Polly). The input is text data such as "I got it, your stuffed animal is with you!", and the output is voice data. Specifically, the speech synthesis module converts the text into a voice signal.

[0329] Step 9:

[0330] The converted voice data is then converted into a character voice by a voice changer (for example, HT8955A). The input is synthesized voice data, and the output is voice data of the character voice. Specifically, the pitch and tone of the voice data are adjusted to change it into the character voice.

[0331] Step 10:

[0332] The audio data converted into the character's voice is played through the device's speaker. The input is the character's voice data, and the output is the voice saying, "Okay, your stuffed animal is with you!". Specifically, the speaker plays the audio data, and the user can hear it.

[0333] In this way, the system acts as a way to keep the parents occupied while they are busy.

[0334] (Application example 2)

[0335] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0336] In recent years, there has been a demand for an environment where children can relax and enjoy themselves while their parents are busy shopping. However, physical stores often lack sufficient infrastructure and services to allow children to have fun while waiting. This can lead to children becoming bored or losing interest while their parents are shopping. There is also a lack of appropriate guides and instructions to ensure children's safety within the store. There is a demand for a solution to these problems and a way to provide an environment in physical stores where children can have fun while their parents can shop with peace of mind.

[0337] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0338] In this invention, the server includes means for acquiring voice uttered by the user, voice recognition means for converting the acquired voice into text data, and communication means for transmitting the converted text data, thereby enabling interface means for providing an interactive experience that attracts children's attention in public places and guide means for guiding them to spots and events within a store.

[0339] The "means for acquiring user-uttered speech" refers to a device or process that collects user-uttered speech using a speech input device.

[0340] The "speech recognition means for converting acquired speech into text data" is a system or algorithm that analyzes the speech of the user and converts it into corresponding text data.

[0341] The "communication means for transmitting converted text data" refers to a communication technology or protocol for transmitting the text data generated by the speech recognition module to a server or other device.

[0342] The "language analysis means that analyzes the transmitted text data and extracts the user's intention and emotional state" is software or an algorithm that analyzes the text data and identifies the user's intention and emotional state from its content.

[0343] The "response generation means for generating an appropriate response based on the extracted intention and emotional state" is a program or system for generating an appropriate reply or action according to the user's intention and emotion.

[0344] The "communication means for receiving the generated response as text data" is a technology or protocol for receiving the generated response text from a server or the like.

[0345] The "voice synthesis means for converting the received response text into voice data" refers to a synthesis technique or system for converting text data into natural voice.

[0346] The "voice changer means for applying a character voice to converted voice data" is a technology or module for adding a character voice to voice-synthesized data.

[0347] The "audio output means for outputting audio data to which character voices have been applied" refers to a device or system for reproducing the final audio data using a speaker or the like.

[0348] "Interface means for providing an interactive experience that attracts children in public places" refers to user interfaces and hardware that provide an interactive experience that is attractive to children.

[0349] The "guide means for providing information about spots and events within the store" is a system or device for providing information about various places and events within the store to children.

[0350] This invention relates to an interactive character experience for children in a brick-and-mortar store. The system is designed to keep children entertained while their parents shop. Below are detailed methods for implementing this system.

[0351] First, the system includes the following components: a microphone as a voice input device for capturing the user's (here, a child's) speech, a voice recognition module for converting the captured voice into text data, a communication means for sending the converted text data to a server, a language analysis module for analyzing the text data on the server side to extract the child's intentions and emotional state, and an emotion engine.

[0352] The server then has a response generation module that generates an appropriate response based on the analysis results. This response generation module generates a fun and engaging response for the child based on the extracted intention and emotional state. The generated response text is then sent to the terminal again via the communication means.

[0353] On the device side, a voice synthesis module operates to convert the received response text into voice data, and a voice changer is installed to apply a character voice to this voice data, and finally the voice data with the character voice applied is output through the speaker.

[0354] Robots, smartphones, tablets, and other devices can be used as interfaces to provide interactive experiences that will attract children's attention, especially in public places. Furthermore, interactive maps and augmented reality (AR) technology can be used as guides to guide children to in-store spots and events.

[0355] Specific examples

[0356] If a child asks "Where are the toys?" at the entrance to the store, the character will guide them by saying, "The toy section is on the third floor. Let's go together!" It can also pique the child's interest by asking questions along the way, such as "Do you know this candy?" This series of responses allows children to have fun walking around the store, while parents can continue shopping with peace of mind.

[0357] Prompt Sentence Examples

[0358] Text: "Where are your toys?"

[0359] Response Generation Prompt: "Create a response to guide a child through the store. Include the location of the toy department and an engaging quiz."

[0360] Example response:

[0361] "The toy section is on the third floor. Come with me! There'll be a quiz on the way, are you ready?"

[0362] Based on these components and specific examples, the present invention provides an environment in a physical store where children can have fun and not get bored.

[0363] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0364] Step 1:

[0365] Acquire the user's voice. In this step, a microphone is used to collect the user's (in this case, a child's) speech in real time. The input is the user's voice, and the output is the captured raw voice data.

[0366] Step 2:

[0367] The captured voice is converted into text data. The device activates a voice recognition module and converts the captured voice data into text data. Here, for example, the Google Speech-to-Text API is used. The input is voice data, and the output is the converted text data, "Where is your toy?"

[0368] Step 3:

[0369] The converted text data is sent to the server. The terminal sends the text data to the server using a communication means. The input is the text data, and the output is a transmission completion notification to the server.

[0370] Step 4:

[0371] Analyze the transmitted text data. The server analyzes the received text data and uses a language analysis module to extract the user's intent and emotional state. For example, it uses Natural Language Processing (NLP) technology. The input is the text data, and the output is the extracted intent "asking for the location of the toy section" and emotional state "interested."

[0372] Step 5:

[0373] Generate an appropriate response. The server invokes a response generation module to generate a corresponding response based on the extracted intent and emotional state. Here, a generative AI model is used to generate the response, "The toy department is on the third floor. Let's go together!" The input is the extracted intent and emotional state, and the output is the response text.

[0374] Step 6:

[0375] The generated response text is sent to the terminal. The server sends the generated response text to the terminal via a communication means. The input is the response text, and the output is a transmission completion notification to the terminal.

[0376] Step 7:

[0377] The device converts the received response text into voice data. The device activates a speech synthesis module to convert the received response text into natural-sounding voice, for example, using the Google Text-to-Speech API. The input is the response text, and the output is voice data.

[0378] Step 8:

[0379] A character voice is applied to the converted voice data. The terminal uses a voice changer means to add the character voice to the voice data. The input is the voice data, and the output is the voice data to which the character voice has been applied.

[0380] Step 9:

[0381] The voice data with the character voice applied is output. The device plays the final voice data through the speaker and provides a response to the user. The input is the voice data with the character voice applied, and the output is the voice played to the user.

[0382] This series of processing steps allows the user (child) to be guided around the store in an enjoyable and interactive manner, while the parent can continue shopping with peace of mind.

[0383] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0384] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0385] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0386] [Second embodiment]

[0387] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0388] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0389] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0390] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0391] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0392] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0393] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0394] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0395] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0396] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0397] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0398] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0399] The present invention relates to an AI toy that allows parents to entertain their children while they are busy with housework and childcare. The operation of the specific system and the processing of the related programs will be described in detail below.

[0400] This system mainly includes the following components: speech recognition means for acquiring voice uttered by the user, speech recognition means for converting the acquired voice into text data, communication means for transmitting the converted text data to a server, language analysis means for analyzing the transmitted text data and extracting the user's intentions and emotional state, response generation means for generating an appropriate response based on the analysis results, speech synthesis means for converting the generated response text into voice data, voice changer means for applying a character voice to the voice data, and voice output means for outputting the converted voice data.

[0401] A specific flow of program processing will be explained.

[0402] 1. User voice input

[0403] The user (child) speaks to the stuffed toy, for example, "Mommy, hold me!" This voice is captured by the device's built-in microphone.

[0404] 2. Voice Recognition

[0405] The device inputs the captured voice data into a voice recognition module, which converts the voice into text data such as "Mom, hold me!"

[0406] 3. Sending text data

[0407] The converted text data is transmitted to a server via the communication means of the terminal and over the Internet.

[0408] 4. Language Processing and Sentiment Analysis

[0409] The server inputs the received text data into a language analysis module, which analyzes the meaning and emotion of the text content. Through this analysis, for example, the intention of "I want to be held" and the "feeling of being pampered" can be extracted.

[0410] 5. Response Generation

[0411] Based on the results of the language analysis, the server generates an appropriate response text: "Okay, your stuffed animal is with you!"

[0412] 6. Sending Text Responses

[0413] The response text is sent from the server to the terminal.

[0414] 7. Speech Synthesis

[0415] The terminal inputs the received response text into a voice synthesis module, and converts the text data into voice data.

[0416] 8. Voice Change

[0417] The converted voice data is converted into a character voice by a voice changer means.

[0418] 9. Audio Output

[0419] Finally, the converted voice data is played back through the device's speaker, and the child hears, "Okay, your stuffed animal is with you!"

[0420] Through the above process, children can satisfy their emotions through interactions with stuffed toys while their parents are busy. To give an example, even if a child feels anxious about something, this system can provide an appropriate response immediately, reassuring the child. In this way, the present invention provides an effective means for reducing the burden on parents and supporting the healthy development of children.

[0421] The processing flow will be explained below.

[0422] Step 1:

[0423] The user (child) speaks to the stuffed animal. For example, they say, "Mommy, hold me!"

[0424] The device captures the user's voice through a built-in microphone.

[0425] Step 2:

[0426] The terminal inputs the acquired voice data into a voice recognition module.

[0427] The voice recognition module analyzes the voice data and converts it into text data such as "Mom, hold me!"

[0428] Step 3:

[0429] The terminal uses a communication means for transmitting the converted text data to the server.

[0430] The text data is sent to the server as an HTTP request.

[0431] Step 4:

[0432] The server inputs the received text data into a natural language processing (NLP) module.

[0433] The NLP module analyzes the text data and extracts the intent, "I want to be hugged."

[0434] Step 5:

[0435] The server passes the intent extracted by the NLP module to the sentiment analysis module.

[0436] The emotion analysis module analyzes the user's emotional state (e.g., affectionate feelings) from the text.

[0437] Step 6:

[0438] The server launches a response generation module based on the results of the language analysis and sentiment analysis.

[0439] The response generation module generates the appropriate response text: "Okay, your stuffed toy is with you!"

[0440] Step 7:

[0441] The server uses a communication means to transmit the generated response text to the terminal.

[0442] The response text is sent to the terminal as an HTTP response.

[0443] Step 8:

[0444] The terminal inputs the received response text into the speech synthesis module.

[0445] The speech synthesis module converts the response text into speech data.

[0446] Step 9:

[0447] The terminal passes the voice data to the voice changer means.

[0448] The voice changer converts audio data into the character voice of the stuffed animal.

[0449] Step 10:

[0450] The device transmits the converted audio data to the speaker.

[0451] A voice will play from the speaker saying, "Okay, your stuffed animal is with you!"

[0452] Through the above process, the user (child) can feel satisfied through interaction with the stuffed toy. The present invention provides an environment where children can play safely even while their parents are busy.

[0453] Example 1

[0454] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0455] Conventional systems lacked a means for parents to effectively communicate with their children when they were busy with housework and childcare. Furthermore, their limited ability to accurately analyze ambiguous words and emotions expressed by children and respond appropriately led to situations where children felt uneasy. Furthermore, the technology to apply character voices to voice data to provide appropriate responses for children was insufficient.

[0456] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0457] In this invention, the server includes means for acquiring voice uttered by the user, voice recognition means for converting the acquired voice into text data, communication means for transmitting the converted text data, language analysis means for analyzing the transmitted text data and extracting the user's intention and emotional state, response generation means for generating an appropriate response based on the extracted intention and emotional state, communication means for receiving the generated response as text data, voice synthesis means for converting the received response text into voice data, voice changer means for applying a character voice to the converted voice data, and voice output means for outputting the voice data to which the character voice has been applied. This allows children to communicate effectively and feel at ease even when their parents are busy.

[0458] A "user" is a subject who uses the system, specifically a person who gives instructions to the system by speaking.

[0459] "Speech recognition means" refers to a device or software that has the function of analyzing acquired voice data and converting it into text data.

[0460] "Communication means" refers to devices or software with Internet connection capabilities for sending and receiving data between a terminal and a server.

[0461] "Language analysis means" refers to a device or software that has the function of analyzing received text data and extracting the user's intentions and emotional state.

[0462] "Response generation means" refers to a device or software that has the function of generating an appropriate response based on the analyzed user's intentions and emotional state.

[0463] "Speech synthesis means" refers to a device or software that has the function of converting text data into voice data.

[0464] "Voice changer means" refers to a device or software that has the function of applying a character voice to audio data and changing the voice quality to a specific one.

[0465] "Audio output means" refers to a device such as a speaker that is used to play back the final converted audio data.

[0466] A "generative AI model" refers to an algorithm or computer program that uses artificial intelligence to generate text.

[0467] The present invention relates to an AI toy that can entertain children while parents are busy. The system uses multiple hardware and software components to implement a series of processes that captures user-spoken speech and converts it into an appropriate response. The specific components of the system and their operation are described in detail below.

[0468] The system consists of the following components: a voice recognition means for acquiring voice uttered by the user, a voice recognition means for converting the acquired voice into text data, a communication means for transmitting the converted text data to a server, a language analysis means for analyzing the transmitted text data and extracting the user's intentions and emotional state, a response generation means for generating an appropriate response based on the analysis results, a voice synthesis means for converting the generated response text into voice data, a voice changer means for applying a character voice to the voice data, and a voice output means for outputting the converted voice data.

[0469] Hardware and software used

[0470] Specifically, this system uses the following hardware and software.

[0471] 1. The speech recognition method uses the Google Cloud Speech-to-Text API, which allows the captured voice data to be converted into text data.

[0472] 2. The communication method is to connect to the Internet using a regular Wi-Fi module, which allows data to be sent and received.

[0473] 3. The language analysis method uses IBM Watson NLU, which allows the system to analyze the user's intent and emotional state from the text data sent.

[0474] 4. The response generation method uses OpenAI GPT-3, which can generate appropriate responses based on the analyzed intent and emotional state. An example of a prompt sentence is, "What is the appropriate response when a child asks, 'Where is my stuffed animal?'"

[0475] 5. Amazon Polly is used as the speech synthesis means, which allows the generated text data to be converted into speech data.

[0476] 6. The voice changer uses Voicemod, which allows you to apply character voices to audio data.

[0477] 7. The built-in speaker is used as the audio output means, which allows the final audio data to be provided to the user.

[0478] Specific examples

[0479] A specific example of the system's operation is shown below.

[0480] The child turns to the stuffed animal and says, "Mommy, where is my stuffed animal?" The audio is captured by the device's built-in microphone.

[0481] The captured voice data is converted into text data using the Google Cloud Speech-to-Text API, generating the text "Mom, where's my stuffed animal?"

[0482] This text data is sent to the server using a Wi-Fi module.

[0483] The server uses IBM Watson NLU to analyze the intent of "I want to know where the stuffed animal is" from the text data.

[0484] The prompt for OpenAI GPT-3 was "What is the appropriate response when a child asks, 'Where is your stuffed animal?'" and an appropriate response was generated, resulting in the text data "Your stuffed animal is in the living room!"

[0485] This response text is sent from the server to the terminal.

[0486] The device converts the response text into voice data using Amazon Polly.

[0487] The converted voice data is then converted into a character voice using Voicemod.

[0488] Finally, the built-in speaker will play a voice saying, "The stuffed animal is in the living room!"

[0489] This allows children to feel safe and communicate effectively even while their parents are busy.

[0490] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0491] Step 1:

[0492] The device's built-in microphone captures the user's speech. This speech data becomes the input for the system. Speech data is generated when the user speaks a specific phrase, such as "Where is my stuffed animal?"

[0493] Step 2:

[0494] The device sends the captured voice data to the Google Cloud Speech-to-Text API, which converts the voice into text data. The input of this process is the voice data, and the output is the corresponding text data: "Where is the stuffed animal?"

[0495] Step 3:

[0496] The terminal transmits the converted text data to the server using a communication means (Wi-Fi module). At this time, the input is the converted text data, and the output is the data transmitted to the server.

[0497] Step 4:

[0498] The server receives the text data sent from the terminal. The input of this process is the text data from the terminal, and the output is the text data stored in the server.

[0499] Step 5:

[0500] The server inputs the received text data into IBM Watson NLU for language analysis, which extracts the user's intent and emotional state. The input for this process is text data, and the output is the analysis results. For example, the intent, "I want to know where the stuffed animal is," and the emotion, "confusion," are extracted.

[0501] Step 6:

[0502] The server uses OpenAI GPT-3 based on the analysis results to generate an appropriate response text. The prompt text used is something like, "What is the appropriate response when a child asks, 'Where is your stuffed animal?'" The input for this process is the analysis results, and the output is the response text, "Your stuffed animal is in the living room!"

[0503] Step 7:

[0504] The server transmits the generated response text to the terminal via a communication means, where the input is the generated response text and the output is data transmission to the terminal.

[0505] Step 8:

[0506] The terminal receives the response text sent from the server. The input of this process is the response text, and the output is text data stored in the terminal.

[0507] Step 9:

[0508] The device inputs the received response text into Amazon Polly and converts it into voice data. The input of this process is the response text, and the output is the generated voice data.

[0509] Step 10:

[0510] The device inputs the converted voice data into Voicemod, which converts it into a character voice. The input of this process is the generated voice data, and the output is the voice data with the character voice applied.

[0511] Step 11:

[0512] Finally, the device uses its built-in speaker to play back the voice data with the character voice applied. The input of this process is the voice data with the character voice, and the output is the actual voice that is played back. The user can hear the voice saying, "The stuffed animal is in the living room!"

[0513] (Application example 1)

[0514] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0515] The challenge is to solve the problem of children becoming bored or anxious while their parents are shopping in physical stores, and to provide a safe and enjoyable environment for children.To solve this problem, we aim to provide interactive entertainment for children and reduce the burden on parents.

[0516] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0517] In this invention, the server includes: means for acquiring voice uttered by a user; speech recognition means for converting the acquired voice into text data; communication means for transmitting the converted text data; language analysis means for analyzing the transmitted text data and extracting the user's intention and emotional state; response generation means for generating an appropriate response based on the extracted intention and emotional state; communication means for receiving the generated response as text data; speech synthesis means for converting the received response text into voice data; voice changer means for applying a character voice to the converted voice data; voice output means for outputting the voice data to which the character voice has been applied; and means for implementing interactions such as stories and games to provide entertainment for children. This allows children to enjoy interactive entertainment while parents can shop with peace of mind.

[0518] The "means for acquiring the voice uttered by the user" is a device for physically capturing the voice spoken by the user.

[0519] The "voice recognition means for converting captured voice into text data" is a technology for recognizing and converting captured voice into digital text data.

[0520] The "communication means for transmitting converted text data" is a method for transmitting the converted text data to a remote server or other device.

[0521] "Language analysis means that analyzes transmitted text data and extracts the user's intentions and emotional state" refers to software or algorithms that analyze received text data to understand the user's intentions and emotional state.

[0522] The "response generation means for generating an appropriate response based on the extracted intention and emotional state" is a system that generates an appropriate response based on the analysis results.

[0523] The "communication means for receiving the generated response as text data" is a method for receiving the generated response text data from the server back to the terminal.

[0524] The "voice synthesis means for converting received response text into voice data" is a technology for converting received text data into natural voice.

[0525] The "voice changer means for applying a character voice to converted voice data" is a technology for applying the voice of a specific character to generated voice data.

[0526] The "audio output means for outputting audio data to which a character voice has been applied" is a device that outputs the converted audio data to the outside through a speaker.

[0527] "Means for implementing interactions such as stories and games to provide entertainment for children" refers to equipment and technology that provides interactive content such as stories and games that children can enjoy.

[0528] The present invention relates to an interactive entertainment system that helps keep children entertained while their parents shop in a brick-and-mortar store. The system's primary purpose is to provide entertainment for children and make shopping safer and more convenient for parents.

[0529] The system includes the following elements:

[0530] 1. Voice acquisition means: A microphone attached to the terminal is used to acquire the voice spoken by the user (child).

[0531] 2. Speech recognition means: Speech recognition software (e.g., Google Speech Recognition API) is used to convert the acquired voice data into text data.

[0532] 3. Communication means: The communication module of the terminal is used to send the converted text data to the server.

[0533] 4. Language analysis means: The server uses natural language processing (NLP) algorithms (e.g., TextBlob) to analyze the received text data and extract the user's intent and emotional state.

[0534] 5. Response generation means: A generative AI model (e.g., GPT-3) that generates appropriate responses based on the extracted intent and emotional state is installed on the server.

[0535] 6. Communication means: The communication module of the server is used to send the generated response text data back to the terminal.

[0536] 7. Speech synthesis means: The terminal uses a speech synthesis tool (e.g., gTTS) to convert the received response text into voice data.

[0537] 8. Voice changer means: Using voice changer software to apply a character voice to the converted voice data.

[0538] 9. Audio output means: Audio data to which the character voice is applied is output through the device speaker.

[0539] Detailed explanation of the process

[0540] The process performed by the server is as follows: First, the microphone of the terminal captures the voice of the child speaking. The acquired voice data is converted into text data by voice recognition means. This converted text data is sent to the server via communication means. Next, the server analyzes the received text data using language analysis means to extract the user's intention and emotional state. Based on the analysis results, response generation means generates an appropriate response text. This generated response text is again sent to the terminal via communication means and converted into voice data by voice synthesis means. Furthermore, voice changer means is used to apply a character voice to this voice data. Finally, the converted voice data with the character voice is output through the terminal's speaker.

[0541] Specific examples

[0542] For example, while a parent is ordering coffee at a cafe in a brick-and-mortar store, a child might say to a robot, "I feel lonely." The robot's microphone captures the voice and converts it into text data using a speech recognition means. This text data is sent to a server, which then analyzes the emotion "I feel lonely" using a language analysis means. Based on the analysis results, a response text, "It's okay, let's play together!", is generated by a response generation means, which is then sent back to the terminal and converted into voice data by a voice synthesis means. After a character voice is applied by a voice changer means, this voice data is played back to the child through a speaker, reassuring the child.

[0543] Prompt Sentence Examples

[0544] python

[0545] if __name__ == "__main__":

[0546] print("System booting...")

[0547] Example

[0548] Child says "I'm lonely"

[0549] user_input = "lonely"

[0550] response = generate_response(user_input)

[0551] synthesize_speech(response)

[0552] Using this prompt, you can see how the system captures the child's speech and returns an appropriate response.

[0553] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0554] Step 1:

[0555] The user (child) speaks. The device's built-in microphone captures the voice data of the child. At this stage, the input is the user's voice, and the output is voice data.

[0556] Step 2:

[0557] The device sends the captured voice data to a voice recognition means, which (for example, Google Speech Recognition API) converts the voice data into text data. At this stage, the input is voice data and the output is text data.

[0558] Step 3:

[0559] The terminal sends the converted text data to the server using a communication means. The communication means delivers the text data to the server via the Internet. The input at this stage is text data, and the output is the transmission of text data to the server.

[0560] Step 4:

[0561] The text data received by the server is input to the language analysis means. The language analysis means (e.g., TextBlob) analyzes the text data and extracts the user's intention and emotional state. The input at this stage is text data, and the output is the user's intention and emotional state.

[0562] Step 5:

[0563] Based on the extracted intention and emotional state, the server generates an appropriate response using a response generation means (e.g., generative AI model: GPT-3). The input at this stage is the user's intention and emotional state, and the output is response text data.

[0564] Step 6:

[0565] The generated response text data is sent to the terminal using the communication means of the server. The input at this stage is the response text data, and the output is the transmission of the text data to the terminal.

[0566] Step 7:

[0567] The response text data received by the terminal is input to a speech synthesis means. The speech synthesis means (e.g., gTTS) converts the text data into speech data. The input at this stage is the response text data, and the output is the response speech data.

[0568] Step 8:

[0569] The terminal inputs the converted voice data to the voice changer means, which applies a character voice to the voice data. The input at this stage is the response voice data, and the output is the voice data after the character voice has been applied.

[0570] Step 9:

[0571] The terminal plays the voice data with the character voice applied from the speaker through the voice output means. The input at this stage is the voice data with the character voice applied, and the output is the actual voice. The process is completed when the child listens to this voice and responds interactively.

[0572] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0573] The present invention relates to an AI toy that allows parents to entertain their children while they are busy with housework and childcare. The operation of the specific system and the processing of the related programs will be described in detail below.

[0574] This system mainly includes the following components: speech recognition means for acquiring speech uttered by the user, speech recognition means for converting the acquired speech into text data, communication means for transmitting the converted text data to a server, language analysis means for analyzing the transmitted text data and extracting the user's intentions and emotional state, and an emotion engine for recognizing the emotional state, response generation means for generating an appropriate response based on the analysis results, speech synthesis means for converting the generated response text into speech data, voice changer means for applying a character voice to the speech data, and speech output means for outputting the converted speech data.

[0575] A specific flow of program processing will be explained.

[0576] 1. User voice input

[0577] The user (child) speaks to the stuffed toy, for example, "Mommy, hold me!" This voice is captured by the device's built-in microphone.

[0578] 2. Voice Recognition

[0579] The device inputs the captured voice data into a voice recognition module, which converts the voice into text data such as "Mom, hold me!"

[0580] 3. Sending text data

[0581] The converted text data is transmitted to a server via the communication means of the terminal and over the Internet.

[0582] 4. Language Processing and Sentiment Analysis

[0583] The server inputs the received text data into a language analysis module and analyzes the meaning of the text content.

[0584] At the same time, the server uses an emotion engine to analyze the user's emotional state from the text data. For example, the emotional state "feeling dependent" is extracted.

[0585] 5. Response Generation

[0586] Based on the results of language analysis and sentiment analysis, the server activates a response generation module, which generates an appropriate response based on the user's emotional state.

[0587] For example, an appropriate response text might be generated: "Okay, your stuffed animal is with you!"

[0588] 6. Sending Text Responses

[0589] The response text is sent from the server to the terminal.

[0590] 7. Speech Synthesis

[0591] The terminal inputs the received response text into a voice synthesis module, and converts the text data into voice data.

[0592] 8. Voice Change

[0593] The converted voice data is converted into a character voice by a voice changer means.

[0594] 9. Audio Output

[0595] Finally, the converted voice data is played back through the device's speaker, and the child hears, "Okay, your stuffed animal is with you!"

[0596] In this system, the emotion engine continuously monitors the user's emotional state and adjusts the response by comparing it with past emotional states. For example, if the user continuously displays sad emotions, the emotion engine can detect this and be programmed to provide a more specific response. In this way, the present invention can respond to a child's emotional needs while reducing the burden on parents. For example, if a child is feeling anxious, the stuffed toy can respond appropriately and calm the child. This provides a safe environment for the child even while the parent is busy.

[0597] The processing flow will be explained below.

[0598] Step 1:

[0599] The user (child) speaks to the stuffed animal, for example, "Mommy, hold me!" The device's built-in microphone captures this voice.

[0600] Step 2:

[0601] The device inputs the captured voice data into the voice recognition module, which analyzes the voice data and converts it into text data such as "Mama, hold me!"

[0602] Step 3:

[0603] The terminal uses a communication means to transmit the converted text data to the server, and the text data is transmitted to the server as an HTTP request.

[0604] Step 4:

[0605] The server inputs the received text data into a natural language processing (NLP) module, which analyzes the text data and extracts the intent, "I want a hug."

[0606] Step 5:

[0607] The server passes the intent extracted by the NLP module to the emotion engine, which analyzes the user's emotional state (e.g., dependent feelings) from the text data.

[0608] Step 6:

[0609] Based on the results of the language analysis and sentiment analysis, the server launches a response generation module, which generates an appropriate response text: "Okay, your stuffed toy is with you!"

[0610] Step 7:

[0611] The server sends the generated response text to the terminal, which then sends the response text to the terminal as an HTTP response.

[0612] Step 8:

[0613] The terminal inputs the received response text into a speech synthesis module, which converts the text data into speech data.

[0614] Step 9:

[0615] The terminal passes the converted voice data to a voice changer means, which converts the voice data into the voice of the character of the stuffed toy.

[0616] Step 10:

[0617] The device sends the converted character voice data to the speaker, which then plays back the voice, "Okay, your stuffed animal is with you!"

[0618] Through these specific steps, children can fulfill their emotions through interactions with stuffed toys. The present invention provides a safe environment for children even while their parents are busy, and the emotion engine has the function of continuously monitoring the user's emotional state and adjusting responses, thereby supporting the healthy development of children.

[0619] As a specific example, even if a child feels anxious about something, this system can immediately provide an appropriate response, putting the child at ease. For example, if a child says, "I'm scared," the system analyzes the emotion and generates a response such as, "It's okay, your stuffed toy is with you," providing reassurance to the child. In this way, the present invention reduces the burden on parents and enables appropriate responses to be made according to the emotional needs of children.

[0620] Example 2

[0621] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0622] In modern families, parents are often busy with work and housework, making it difficult for them to spend enough time with their children. There is also a need for systems that can respond appropriately when children feel anxious or lonely. Furthermore, conventional speech recognition and conversational systems lack the ability to accurately understand the user's emotional state and adjust responses accordingly, which means they are unable to adequately meet children's emotional needs.

[0623] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0624] In this invention, the server includes: means for acquiring voice uttered by the user; speech recognition means for converting the acquired voice into text data; communication means for transmitting the converted text data; language analysis means for analyzing the transmitted text data and extracting the user's intention and emotional state; response generation means for generating an appropriate response based on the extracted intention and emotional state; communication means for receiving the generated response as text data; speech synthesis means for converting the received response text into voice data; voice changer means for applying a character voice to the converted voice data; voice output means for outputting the voice data to which the character voice has been applied; and means for the response generation means to adjust the response by comparing it with the user's past emotional state. This makes it possible to provide an environment where parents can spend time with their children with peace of mind even when they are busy.

[0625] The "means for acquiring the voice uttered by the user" refers to hardware or software for capturing the voice spoken by the user (child).

[0626] The "voice recognition means for converting acquired voice into text data" is a technical means for converting voice data into text format.

[0627] The "communication means for transmitting the converted text data" refers to communication technology or equipment for transmitting the text data to a server or the like.

[0628] The "language analysis means for analyzing transmitted text data and extracting the user's intentions and emotional state" is a technology that has the function of analyzing the user's intentions and emotions from text data.

[0629] The "response generation means for generating an appropriate response based on the extracted intention and emotional state" is a technology for generating an appropriate reply according to the user's intention and emotion.

[0630] The "communication means for receiving the generated response as text data" refers to a communication technique or device for receiving the response text from the server.

[0631] The "voice synthesis means for converting the received response text into voice data" is a technical means for converting text data into voice data.

[0632] The "voice changer means for applying a character voice to converted voice data" is a technology for converting voice data into a specific character voice.

[0633] "Audio output means for outputting audio data to which a character voice has been applied" refers to technology such as a speaker for playing back audio data converted into a character voice.

[0634] "A means for generating a response that has the function of adjusting the response by comparing it with the user's past emotional state" refers to a technology that has the function of optimizing the current response by taking into account the user's past emotional data.

[0635] MODE FOR CARRYING OUT THE INVENTION

[0636] This invention relates to an AI toy that helps parents entertain their children while they are busy with housework and childcare. This system acquires speech uttered by the user (child) and generates and outputs an appropriate response. The detailed configuration and operation of the system are described below.

[0637] System Configuration

[0638] The system mainly includes the following hardware and software:

[0639] Voice acquisition means: A microphone (e.g., SHURE MV88) for acquiring the voice spoken by the user

[0640] Speech recognition means: A speech recognition module that converts captured speech into text data (e.g., Google Speech-to-Text API).

[0641] Communication means: Wi-Fi module (e.g., ESP32) for sending converted text data to a server

[0642] Language analysis means: A language analysis module (e.g., Google Natural Language API) that analyzes the transmitted text data to extract the user's intentions and emotional state.

[0643] Response generation means: A module that generates appropriate responses based on the extracted intentions and emotional states.

[0644] Speech synthesis means: A module that converts the generated response text into voice data (e.g., Amazon Polly)

[0645] Voice changer means: A device for applying a character voice to converted voice data (e.g., HT8955A)

[0646] Audio output means: a speaker for outputting audio data to which character voices are applied

[0647] Operation explanation

[0648] The server and the terminal work together to capture the user's voice and generate and output an appropriate response.

[0649] 1. Audio capture

[0650] The user speaks to the stuffed animal. For example, they say, "Mommy, hold me!" To capture this voice, a microphone (SHURE MV88) in the device captures the sound.

[0651] 2. Voice Recognition

[0652] The acquired voice data is input into a voice recognition module (Google Speech-to-Text API), and the voice data is converted into text data such as "Mom, hold me!"

[0653] 3. Data Transmission

[0654] The converted text data is sent to the server via the communication method (ESP32) using an HTTP POST request.

[0655] 4. Language and Sentiment Analysis

[0656] The server inputs the received text data into a language analysis module (Google Natural Language API) to analyze the meaning of the text content. At the same time, it uses an emotion engine to analyze the user's emotional state from the text data. For example, it can extract the emotional state of "feeling dependent."

[0657] 5. Response Generation

[0658] Based on the results of language analysis and sentiment analysis, the server activates a response generation module, which generates a response text such as "I understand, your stuffed toy is with you!"

[0659] 6. Response text reception and speech synthesis

[0660] The generated response text is sent from the server to the terminal, which inputs it into a speech synthesis module (Amazon Polly) to convert the text data into speech data.

[0661] 7. Voice Change and Audio Output

[0662] The converted voice data is converted into a character voice by the voice changer means (HT8955A) and then played back through the speaker, so that the user can hear the voice saying, "Okay, your stuffed toy is with you!"

[0663] Specific examples

[0664] The specific hardware and software used are as follows:

[0665] Microphone: SHURE MV88

[0666] Communication module: ESP32

[0667] Server: General cloud server

[0668] Speech Recognition Module: Google Speech-to-Text API

[0669] Language Analysis Module: Google Natural Language API

[0670] Speech synthesis module: Amazon Polly

[0671] Voice changer: HT8955A

[0672] Prompt Sentence Examples

[0673] "Analyze text data, such as a user saying, 'Hold me, Mommy!', recognize their emotional state, and generate an appropriate response."

[0674] Thus, the present invention can provide a sense of security to children even while parents are busy. In particular, the system has the ability to monitor the user's emotional state, compare it with past emotional states, and adjust responses to appropriately meet the emotional needs of children.

[0675] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0676] Step 1:

[0677] The user speaks to the stuffed toy. The user's voice (e.g., "Mommy, hold me!") is picked up by the device's microphone. The input is audio data, and the output is the captured raw audio signal. Specifically, the audio is picked up as an analog signal through the built-in microphone (e.g., SHURE MV88).

[0678] Step 2:

[0679] The device sends the captured voice signal to a voice recognition module, which (for example, Google Speech-to-Text API) converts the voice into text data. The input is voice data, and the output is text data such as "Mommy, hold me!". Specifically, the analog voice signal obtained from the microphone is digitized, then linked to the API and converted into text data.

[0680] Step 3:

[0681] The device sends the converted text data to the server via a communication method (for example, a Wi-Fi module such as ESP32). The input is the text data "Hold me, Mom!" and the output is an HTTP POST request to the server. Specifically, the text data is sent to the server via the Wi-Fi module over the Internet.

[0682] Step 4:

[0683] The server inputs the received text data into a language analysis module. The language analysis module (for example, Google Natural Language API) analyzes the meaning of the text content. The input is the text data "Mama, hold me!", and the output is the analyzed user intent. Specifically, the text data is sent via the API, and its content is analyzed.

[0684] Step 5:

[0685] At the same time, the server uses an emotion engine to analyze the user's emotional state from the text data. The input is the text data "Hold me, Mommy!", and the output is the emotional state "Feeling pampered." Specifically, the emotion engine runs an algorithm to identify the emotion based on the text analysis results.

[0686] Step 6:

[0687] The server launches a response generation module based on the results of language analysis and sentiment analysis. The generated response text is "I get it, your stuffed toy is with you!" The input is the analyzed intention and emotional state, and the output is the response text. Specifically, the response generation algorithm generates an appropriate response based on the analysis results.

[0688] Step 7:

[0689] The response text is sent from the server to the device. The input is the response text "Okay, your stuffed animal is with you!", and the output is the text data received by the device. Specifically, the server sends the text data as an HTTP response, and the device receives it.

[0690] Step 8:

[0691] The device inputs the received response text into a speech synthesis module and converts it into voice data (for example, Amazon Polly). The input is text data such as "I got it, your stuffed animal is with you!", and the output is voice data. Specifically, the speech synthesis module converts the text into a voice signal.

[0692] Step 9:

[0693] The converted voice data is then converted into a character voice by a voice changer (for example, HT8955A). The input is synthesized voice data, and the output is voice data of the character voice. Specifically, the pitch and tone of the voice data are adjusted to change it into the character voice.

[0694] Step 10:

[0695] The audio data converted into the character's voice is played through the device's speaker. The input is the character's voice data, and the output is the voice saying, "Okay, your stuffed animal is with you!". Specifically, the speaker plays the audio data, and the user can hear it.

[0696] In this way, the system acts as a way to keep the parents occupied while they are busy.

[0697] (Application example 2)

[0698] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0699] In recent years, there has been a demand for an environment where children can relax and enjoy themselves while their parents are busy shopping. However, physical stores often lack sufficient infrastructure and services to allow children to have fun while waiting. This can lead to children becoming bored or losing interest while their parents are shopping. There is also a lack of appropriate guides and instructions to ensure children's safety within the store. There is a demand for a solution to these problems and a way to provide an environment in physical stores where children can have fun while their parents can shop with peace of mind.

[0700] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0701] In this invention, the server includes means for acquiring voice uttered by the user, voice recognition means for converting the acquired voice into text data, and communication means for transmitting the converted text data, thereby enabling interface means for providing an interactive experience that attracts children's attention in public places and guide means for guiding them to spots and events within a store.

[0702] The "means for acquiring user-uttered speech" refers to a device or process that collects user-uttered speech using a speech input device.

[0703] The "speech recognition means for converting acquired speech into text data" is a system or algorithm that analyzes the speech of the user and converts it into corresponding text data.

[0704] The "communication means for transmitting converted text data" refers to a communication technology or protocol for transmitting the text data generated by the speech recognition module to a server or other device.

[0705] The "language analysis means that analyzes the transmitted text data and extracts the user's intention and emotional state" is software or an algorithm that analyzes the text data and identifies the user's intention and emotional state from its content.

[0706] The "response generation means for generating an appropriate response based on the extracted intention and emotional state" is a program or system for generating an appropriate reply or action according to the user's intention and emotion.

[0707] The "communication means for receiving the generated response as text data" is a technology or protocol for receiving the generated response text from a server or the like.

[0708] The "voice synthesis means for converting the received response text into voice data" refers to a synthesis technique or system for converting text data into natural voice.

[0709] The "voice changer means for applying a character voice to converted voice data" is a technology or module for adding a character voice to voice-synthesized data.

[0710] The "audio output means for outputting audio data to which character voices have been applied" refers to a device or system for reproducing the final audio data using a speaker or the like.

[0711] "Interface means for providing an interactive experience that attracts children in public places" refers to user interfaces and hardware that provide an interactive experience that is attractive to children.

[0712] The "guide means for providing information about spots and events within the store" is a system or device for providing information about various places and events within the store to children.

[0713] This invention relates to an interactive character experience for children in a brick-and-mortar store. The system is designed to keep children entertained while their parents shop. Below are detailed methods for implementing this system.

[0714] First, the system includes the following components: a microphone as a voice input device for capturing the user's (here, a child's) speech, a voice recognition module for converting the captured voice into text data, a communication means for sending the converted text data to a server, a language analysis module for analyzing the text data on the server side to extract the child's intentions and emotional state, and an emotion engine.

[0715] The server then has a response generation module that generates an appropriate response based on the analysis results. This response generation module generates a fun and engaging response for the child based on the extracted intention and emotional state. The generated response text is then sent to the terminal again via the communication means.

[0716] On the device side, a voice synthesis module operates to convert the received response text into voice data, and a voice changer is installed to apply a character voice to this voice data, and finally the voice data with the character voice applied is output through the speaker.

[0717] Robots, smartphones, tablets, and other devices can be used as interfaces to provide interactive experiences that will attract children's attention, especially in public places. Furthermore, interactive maps and augmented reality (AR) technology can be used as guides to guide children to in-store spots and events.

[0718] Specific examples

[0719] If a child asks "Where are the toys?" at the entrance to the store, the character will guide them by saying, "The toy section is on the third floor. Let's go together!" It can also pique the child's interest by asking questions along the way, such as "Do you know this candy?" This series of responses allows children to have fun walking around the store, while parents can continue shopping with peace of mind.

[0720] Prompt Sentence Examples

[0721] Text: "Where are your toys?"

[0722] Response Generation Prompt: "Create a response to guide a child through the store. Include the location of the toy department and an engaging quiz."

[0723] Example response:

[0724] "The toy section is on the third floor. Come with me! There'll be a quiz on the way, are you ready?"

[0725] Based on these components and specific examples, the present invention provides an environment in a physical store where children can have fun and not get bored.

[0726] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0727] Step 1:

[0728] Acquire the user's voice. In this step, a microphone is used to collect the user's (in this case, a child's) speech in real time. The input is the user's voice, and the output is the captured raw voice data.

[0729] Step 2:

[0730] The captured voice is converted into text data. The device activates a voice recognition module and converts the captured voice data into text data. Here, for example, the Google Speech-to-Text API is used. The input is voice data, and the output is the converted text data, "Where is your toy?"

[0731] Step 3:

[0732] The converted text data is sent to the server. The terminal sends the text data to the server using a communication means. The input is the text data, and the output is a transmission completion notification to the server.

[0733] Step 4:

[0734] Analyze the transmitted text data. The server analyzes the received text data and uses a language analysis module to extract the user's intent and emotional state. For example, it uses Natural Language Processing (NLP) technology. The input is the text data, and the output is the extracted intent "asking for the location of the toy section" and emotional state "interested."

[0735] Step 5:

[0736] Generate an appropriate response. The server invokes a response generation module to generate a corresponding response based on the extracted intent and emotional state. Here, a generative AI model is used to generate the response, "The toy department is on the third floor. Let's go together!" The input is the extracted intent and emotional state, and the output is the response text.

[0737] Step 6:

[0738] The generated response text is sent to the terminal. The server sends the generated response text to the terminal via a communication means. The input is the response text, and the output is a transmission completion notification to the terminal.

[0739] Step 7:

[0740] The device converts the received response text into voice data. The device activates a speech synthesis module to convert the received response text into natural-sounding voice, for example, using the Google Text-to-Speech API. The input is the response text, and the output is voice data.

[0741] Step 8:

[0742] A character voice is applied to the converted voice data. The terminal uses a voice changer means to add the character voice to the voice data. The input is the voice data, and the output is the voice data to which the character voice has been applied.

[0743] Step 9:

[0744] The voice data with the character voice applied is output. The device plays the final voice data through the speaker and provides a response to the user. The input is the voice data with the character voice applied, and the output is the voice played to the user.

[0745] This series of processing steps allows the user (child) to be guided around the store in an enjoyable and interactive manner, while the parent can continue shopping with peace of mind.

[0746] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0747] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0748] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0749] [Third embodiment]

[0750] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0751] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0752] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0753] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0754] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0755] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0756] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0757] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0758] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0759] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0760] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0761] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0762] The present invention relates to an AI toy that allows parents to entertain their children while they are busy with housework and childcare. The operation of the specific system and the processing of the related programs will be described in detail below.

[0763] This system mainly includes the following components: speech recognition means for acquiring voice uttered by the user, speech recognition means for converting the acquired voice into text data, communication means for transmitting the converted text data to a server, language analysis means for analyzing the transmitted text data and extracting the user's intentions and emotional state, response generation means for generating an appropriate response based on the analysis results, speech synthesis means for converting the generated response text into voice data, voice changer means for applying a character voice to the voice data, and voice output means for outputting the converted voice data.

[0764] A specific flow of program processing will be explained.

[0765] 1. User voice input

[0766] The user (child) speaks to the stuffed toy, for example, "Mommy, hold me!" This voice is captured by the device's built-in microphone.

[0767] 2. Voice Recognition

[0768] The device inputs the captured voice data into a voice recognition module, which converts the voice into text data such as "Mom, hold me!"

[0769] 3. Sending text data

[0770] The converted text data is transmitted to a server via the communication means of the terminal and over the Internet.

[0771] 4. Language Processing and Sentiment Analysis

[0772] The server inputs the received text data into a language analysis module, which analyzes the meaning and emotion of the text content. Through this analysis, for example, the intention of "I want to be held" and the "feeling of being pampered" can be extracted.

[0773] 5. Response Generation

[0774] Based on the results of the language analysis, the server generates an appropriate response text: "Okay, your stuffed animal is with you!"

[0775] 6. Sending Text Responses

[0776] The response text is sent from the server to the terminal.

[0777] 7. Speech Synthesis

[0778] The terminal inputs the received response text into a voice synthesis module, and converts the text data into voice data.

[0779] 8. Voice Change

[0780] The converted voice data is converted into a character voice by a voice changer means.

[0781] 9. Audio Output

[0782] Finally, the converted voice data is played back through the device's speaker, and the child hears, "Okay, your stuffed animal is with you!"

[0783] Through the above process, children can satisfy their emotions through interactions with stuffed toys while their parents are busy. To give an example, even if a child feels anxious about something, this system can provide an appropriate response immediately, reassuring the child. In this way, the present invention provides an effective means for reducing the burden on parents and supporting the healthy development of children.

[0784] The processing flow will be explained below.

[0785] Step 1:

[0786] The user (child) speaks to the stuffed animal. For example, they say, "Mommy, hold me!"

[0787] The device captures the user's voice through a built-in microphone.

[0788] Step 2:

[0789] The terminal inputs the acquired voice data into a voice recognition module.

[0790] The voice recognition module analyzes the voice data and converts it into text data such as "Mom, hold me!"

[0791] Step 3:

[0792] The terminal uses a communication means for transmitting the converted text data to the server.

[0793] The text data is sent to the server as an HTTP request.

[0794] Step 4:

[0795] The server inputs the received text data into a natural language processing (NLP) module.

[0796] The NLP module analyzes the text data and extracts the intent, "I want to be hugged."

[0797] Step 5:

[0798] The server passes the intent extracted by the NLP module to the sentiment analysis module.

[0799] The emotion analysis module analyzes the user's emotional state (e.g., affectionate feelings) from the text.

[0800] Step 6:

[0801] The server launches a response generation module based on the results of the language analysis and sentiment analysis.

[0802] The response generation module generates the appropriate response text: "Okay, your stuffed toy is with you!"

[0803] Step 7:

[0804] The server uses a communication means to transmit the generated response text to the terminal.

[0805] The response text is sent to the terminal as an HTTP response.

[0806] Step 8:

[0807] The terminal inputs the received response text into the speech synthesis module.

[0808] The speech synthesis module converts the response text into speech data.

[0809] Step 9:

[0810] The terminal passes the voice data to the voice changer means.

[0811] The voice changer converts audio data into the character voice of the stuffed animal.

[0812] Step 10:

[0813] The device transmits the converted audio data to the speaker.

[0814] A voice will play from the speaker saying, "Okay, your stuffed animal is with you!"

[0815] Through the above process, the user (child) can feel satisfied through interaction with the stuffed toy. The present invention provides an environment where children can play safely even while their parents are busy.

[0816] Example 1

[0817] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0818] Conventional systems lacked a means for parents to effectively communicate with their children when they were busy with housework and childcare. Furthermore, their limited ability to accurately analyze ambiguous words and emotions expressed by children and respond appropriately led to situations where children felt uneasy. Furthermore, the technology to apply character voices to voice data to provide appropriate responses for children was insufficient.

[0819] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0820] In this invention, the server includes means for acquiring voice uttered by the user, voice recognition means for converting the acquired voice into text data, communication means for transmitting the converted text data, language analysis means for analyzing the transmitted text data and extracting the user's intention and emotional state, response generation means for generating an appropriate response based on the extracted intention and emotional state, communication means for receiving the generated response as text data, voice synthesis means for converting the received response text into voice data, voice changer means for applying a character voice to the converted voice data, and voice output means for outputting the voice data to which the character voice has been applied. This allows children to communicate effectively and feel at ease even when their parents are busy.

[0821] A "user" is a subject who uses the system, specifically a person who gives instructions to the system by speaking.

[0822] "Speech recognition means" refers to a device or software that has the function of analyzing acquired voice data and converting it into text data.

[0823] "Communication means" refers to devices or software with Internet connection capabilities for sending and receiving data between a terminal and a server.

[0824] "Language analysis means" refers to a device or software that has the function of analyzing received text data and extracting the user's intentions and emotional state.

[0825] "Response generation means" refers to a device or software that has the function of generating an appropriate response based on the analyzed user's intentions and emotional state.

[0826] "Speech synthesis means" refers to a device or software that has the function of converting text data into voice data.

[0827] "Voice changer means" refers to a device or software that has the function of applying a character voice to audio data and changing the voice quality to a specific one.

[0828] "Audio output means" refers to a device such as a speaker that is used to play back the final converted audio data.

[0829] A "generative AI model" refers to an algorithm or computer program that uses artificial intelligence to generate text.

[0830] The present invention relates to an AI toy that can entertain children while parents are busy. The system uses multiple hardware and software components to implement a series of processes that captures user-spoken speech and converts it into an appropriate response. The specific components of the system and their operation are described in detail below.

[0831] The system consists of the following components: a voice recognition means for acquiring voice uttered by the user, a voice recognition means for converting the acquired voice into text data, a communication means for transmitting the converted text data to a server, a language analysis means for analyzing the transmitted text data and extracting the user's intentions and emotional state, a response generation means for generating an appropriate response based on the analysis results, a voice synthesis means for converting the generated response text into voice data, a voice changer means for applying a character voice to the voice data, and a voice output means for outputting the converted voice data.

[0832] Hardware and software used

[0833] Specifically, this system uses the following hardware and software.

[0834] 1. The speech recognition method uses the Google Cloud Speech-to-Text API, which allows the captured voice data to be converted into text data.

[0835] 2. The communication method is to connect to the Internet using a regular Wi-Fi module, which allows data to be sent and received.

[0836] 3. The language analysis method uses IBM Watson NLU, which allows the system to analyze the user's intent and emotional state from the text data sent.

[0837] 4. The response generation method uses OpenAI GPT-3, which can generate appropriate responses based on the analyzed intent and emotional state. An example of a prompt sentence is, "What is the appropriate response when a child asks, 'Where is my stuffed animal?'"

[0838] 5. Amazon Polly is used as the speech synthesis means, which allows the generated text data to be converted into speech data.

[0839] 6. The voice changer uses Voicemod, which allows you to apply character voices to audio data.

[0840] 7. The built-in speaker is used as the audio output means, which allows the final audio data to be provided to the user.

[0841] Specific examples

[0842] A specific example of the system's operation is shown below.

[0843] The child turns to the stuffed animal and says, "Mommy, where is my stuffed animal?" The audio is captured by the device's built-in microphone.

[0844] The captured voice data is converted into text data using the Google Cloud Speech-to-Text API, generating the text "Mom, where's my stuffed animal?"

[0845] This text data is sent to the server using a Wi-Fi module.

[0846] The server uses IBM Watson NLU to analyze the intent of "I want to know where the stuffed animal is" from the text data.

[0847] The prompt for OpenAI GPT-3 was "What is the appropriate response when a child asks, 'Where is your stuffed animal?'" and an appropriate response was generated, resulting in the text data "Your stuffed animal is in the living room!"

[0848] This response text is sent from the server to the terminal.

[0849] The device converts the response text into voice data using Amazon Polly.

[0850] The converted voice data is then converted into a character voice using Voicemod.

[0851] Finally, the built-in speaker will play a voice saying, "The stuffed animal is in the living room!"

[0852] This allows children to feel safe and communicate effectively even while their parents are busy.

[0853] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0854] Step 1:

[0855] The device's built-in microphone captures the user's speech. This speech data becomes the input for the system. Speech data is generated when the user speaks a specific phrase, such as "Where is my stuffed animal?"

[0856] Step 2:

[0857] The device sends the captured voice data to the Google Cloud Speech-to-Text API, which converts the voice into text data. The input of this process is the voice data, and the output is the corresponding text data: "Where is the stuffed animal?"

[0858] Step 3:

[0859] The terminal transmits the converted text data to the server using a communication means (Wi-Fi module). At this time, the input is the converted text data, and the output is the data transmitted to the server.

[0860] Step 4:

[0861] The server receives the text data sent from the terminal. The input of this process is the text data from the terminal, and the output is the text data stored in the server.

[0862] Step 5:

[0863] The server inputs the received text data into IBM Watson NLU for language analysis, which extracts the user's intent and emotional state. The input for this process is text data, and the output is the analysis results. For example, the intent, "I want to know where the stuffed animal is," and the emotion, "confusion," are extracted.

[0864] Step 6:

[0865] The server uses OpenAI GPT-3 based on the analysis results to generate an appropriate response text. The prompt text used is something like, "What is the appropriate response when a child asks, 'Where is your stuffed animal?'" The input for this process is the analysis results, and the output is the response text, "Your stuffed animal is in the living room!"

[0866] Step 7:

[0867] The server transmits the generated response text to the terminal via a communication means, where the input is the generated response text and the output is data transmission to the terminal.

[0868] Step 8:

[0869] The terminal receives the response text sent from the server. The input of this process is the response text, and the output is text data stored in the terminal.

[0870] Step 9:

[0871] The device inputs the received response text into Amazon Polly and converts it into voice data. The input of this process is the response text, and the output is the generated voice data.

[0872] Step 10:

[0873] The device inputs the converted voice data into Voicemod, which converts it into a character voice. The input of this process is the generated voice data, and the output is the voice data with the character voice applied.

[0874] Step 11:

[0875] Finally, the device uses its built-in speaker to play back the voice data with the character voice applied. The input of this process is the voice data with the character voice, and the output is the actual voice that is played back. The user can hear the voice saying, "The stuffed animal is in the living room!"

[0876] (Application example 1)

[0877] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0878] The challenge is to solve the problem of children becoming bored or anxious while their parents are shopping in physical stores, and to provide a safe and enjoyable environment for children.To solve this problem, we aim to provide interactive entertainment for children and reduce the burden on parents.

[0879] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0880] In this invention, the server includes: means for acquiring voice uttered by a user; speech recognition means for converting the acquired voice into text data; communication means for transmitting the converted text data; language analysis means for analyzing the transmitted text data and extracting the user's intention and emotional state; response generation means for generating an appropriate response based on the extracted intention and emotional state; communication means for receiving the generated response as text data; speech synthesis means for converting the received response text into voice data; voice changer means for applying a character voice to the converted voice data; voice output means for outputting the voice data to which the character voice has been applied; and means for implementing interactions such as stories and games to provide entertainment for children. This allows children to enjoy interactive entertainment while parents can shop with peace of mind.

[0881] The "means for acquiring the voice uttered by the user" is a device for physically capturing the voice spoken by the user.

[0882] The "voice recognition means for converting captured voice into text data" is a technology for recognizing and converting captured voice into digital text data.

[0883] The "communication means for transmitting converted text data" is a method for transmitting the converted text data to a remote server or other device.

[0884] "Language analysis means that analyzes transmitted text data and extracts the user's intentions and emotional state" refers to software or algorithms that analyze received text data to understand the user's intentions and emotional state.

[0885] The "response generation means for generating an appropriate response based on the extracted intention and emotional state" is a system that generates an appropriate response based on the analysis results.

[0886] The "communication means for receiving the generated response as text data" is a method for receiving the generated response text data from the server back to the terminal.

[0887] The "voice synthesis means for converting received response text into voice data" is a technology for converting received text data into natural voice.

[0888] The "voice changer means for applying a character voice to converted voice data" is a technology for applying the voice of a specific character to generated voice data.

[0889] The "audio output means for outputting audio data to which a character voice has been applied" is a device that outputs the converted audio data to the outside through a speaker.

[0890] "Means for implementing interactions such as stories and games to provide entertainment for children" refers to equipment and technology that provides interactive content such as stories and games that children can enjoy.

[0891] The present invention relates to an interactive entertainment system that helps keep children entertained while their parents shop in a brick-and-mortar store. The system's primary purpose is to provide entertainment for children and make shopping safer and more convenient for parents.

[0892] The system includes the following elements:

[0893] 1. Voice acquisition means: A microphone attached to the terminal is used to acquire the voice spoken by the user (child).

[0894] 2. Speech recognition means: Speech recognition software (e.g., Google Speech Recognition API) is used to convert the acquired voice data into text data.

[0895] 3. Communication means: The communication module of the terminal is used to send the converted text data to the server.

[0896] 4. Language analysis means: The server uses natural language processing (NLP) algorithms (e.g., TextBlob) to analyze the received text data and extract the user's intent and emotional state.

[0897] 5. Response generation means: A generative AI model (e.g., GPT-3) that generates appropriate responses based on the extracted intent and emotional state is installed on the server.

[0898] 6. Communication means: The communication module of the server is used to send the generated response text data back to the terminal.

[0899] 7. Speech synthesis means: The terminal uses a speech synthesis tool (e.g., gTTS) to convert the received response text into voice data.

[0900] 8. Voice changer means: Using voice changer software to apply a character voice to the converted voice data.

[0901] 9. Audio output means: Audio data to which the character voice is applied is output through the device speaker.

[0902] Detailed explanation of the process

[0903] The process performed by the server is as follows: First, the microphone of the terminal captures the voice of the child speaking. The acquired voice data is converted into text data by voice recognition means. This converted text data is sent to the server via communication means. Next, the server analyzes the received text data using language analysis means to extract the user's intention and emotional state. Based on the analysis results, response generation means generates an appropriate response text. This generated response text is again sent to the terminal via communication means and converted into voice data by voice synthesis means. Furthermore, voice changer means is used to apply a character voice to this voice data. Finally, the converted voice data with the character voice is output through the terminal's speaker.

[0904] Specific examples

[0905] For example, while a parent is ordering coffee at a cafe in a brick-and-mortar store, a child might say to a robot, "I feel lonely." The robot's microphone captures the voice and converts it into text data using a speech recognition means. This text data is sent to a server, which then analyzes the emotion "I feel lonely" using a language analysis means. Based on the analysis results, a response text, "It's okay, let's play together!", is generated by a response generation means, which is then sent back to the terminal and converted into voice data by a voice synthesis means. After a character voice is applied by a voice changer means, this voice data is played back to the child through a speaker, reassuring the child.

[0906] Prompt Sentence Examples

[0907] python

[0908] if __name__ == "__main__":

[0909] print("System booting...")

[0910] Example

[0911] Child says "I'm lonely"

[0912] user_input = "lonely"

[0913] response = generate_response(user_input)

[0914] synthesize_speech(response)

[0915] Using this prompt, you can see how the system captures the child's speech and returns an appropriate response.

[0916] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0917] Step 1:

[0918] The user (child) speaks. The device's built-in microphone captures the voice data of the child. At this stage, the input is the user's voice, and the output is voice data.

[0919] Step 2:

[0920] The device sends the captured voice data to a voice recognition means, which (for example, Google Speech Recognition API) converts the voice data into text data. At this stage, the input is voice data and the output is text data.

[0921] Step 3:

[0922] The terminal sends the converted text data to the server using a communication means. The communication means delivers the text data to the server via the Internet. The input at this stage is text data, and the output is the transmission of text data to the server.

[0923] Step 4:

[0924] The text data received by the server is input to the language analysis means. The language analysis means (e.g., TextBlob) analyzes the text data and extracts the user's intention and emotional state. The input at this stage is text data, and the output is the user's intention and emotional state.

[0925] Step 5:

[0926] Based on the extracted intention and emotional state, the server generates an appropriate response using a response generation means (e.g., generative AI model: GPT-3). The input at this stage is the user's intention and emotional state, and the output is response text data.

[0927] Step 6:

[0928] The generated response text data is sent to the terminal using the communication means of the server. The input at this stage is the response text data, and the output is the transmission of the text data to the terminal.

[0929] Step 7:

[0930] The response text data received by the terminal is input to a speech synthesis means. The speech synthesis means (e.g., gTTS) converts the text data into speech data. The input at this stage is the response text data, and the output is the response speech data.

[0931] Step 8:

[0932] The terminal inputs the converted voice data to the voice changer means, which applies a character voice to the voice data. The input at this stage is the response voice data, and the output is the voice data after the character voice has been applied.

[0933] Step 9:

[0934] The terminal plays the voice data with the character voice applied from the speaker through the voice output means. The input at this stage is the voice data with the character voice applied, and the output is the actual voice. The process is completed when the child listens to this voice and responds interactively.

[0935] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0936] The present invention relates to an AI toy that allows parents to entertain their children while they are busy with housework and childcare. The operation of the specific system and the processing of the related programs will be described in detail below.

[0937] This system mainly includes the following components: speech recognition means for acquiring speech uttered by the user, speech recognition means for converting the acquired speech into text data, communication means for transmitting the converted text data to a server, language analysis means for analyzing the transmitted text data and extracting the user's intentions and emotional state, and an emotion engine for recognizing the emotional state, response generation means for generating an appropriate response based on the analysis results, speech synthesis means for converting the generated response text into speech data, voice changer means for applying a character voice to the speech data, and speech output means for outputting the converted speech data.

[0938] A specific flow of program processing will be explained.

[0939] 1. User voice input

[0940] The user (child) speaks to the stuffed toy, for example, "Mommy, hold me!" This voice is captured by the device's built-in microphone.

[0941] 2. Voice Recognition

[0942] The device inputs the captured voice data into a voice recognition module, which converts the voice into text data such as "Mom, hold me!"

[0943] 3. Sending text data

[0944] The converted text data is transmitted to a server via the communication means of the terminal and over the Internet.

[0945] 4. Language Processing and Sentiment Analysis

[0946] The server inputs the received text data into a language analysis module and analyzes the meaning of the text content.

[0947] At the same time, the server uses an emotion engine to analyze the user's emotional state from the text data. For example, the emotional state "feeling dependent" is extracted.

[0948] 5. Response Generation

[0949] Based on the results of language analysis and sentiment analysis, the server activates a response generation module, which generates an appropriate response based on the user's emotional state.

[0950] For example, an appropriate response text might be generated: "Okay, your stuffed animal is with you!"

[0951] 6. Sending Text Responses

[0952] The response text is sent from the server to the terminal.

[0953] 7. Speech Synthesis

[0954] The terminal inputs the received response text into a voice synthesis module, and converts the text data into voice data.

[0955] 8. Voice Change

[0956] The converted voice data is converted into a character voice by a voice changer means.

[0957] 9. Audio Output

[0958] Finally, the converted voice data is played back through the device's speaker, and the child hears, "Okay, your stuffed animal is with you!"

[0959] In this system, the emotion engine continuously monitors the user's emotional state and adjusts the response by comparing it with past emotional states. For example, if the user continuously displays sad emotions, the emotion engine can detect this and be programmed to provide a more specific response. In this way, the present invention can respond to a child's emotional needs while reducing the burden on parents. For example, if a child is feeling anxious, the stuffed toy can respond appropriately and calm the child. This provides a safe environment for the child even while the parent is busy.

[0960] The processing flow will be explained below.

[0961] Step 1:

[0962] The user (child) speaks to the stuffed animal, for example, "Mommy, hold me!" The device's built-in microphone captures this voice.

[0963] Step 2:

[0964] The device inputs the captured voice data into the voice recognition module, which analyzes the voice data and converts it into text data such as "Mama, hold me!"

[0965] Step 3:

[0966] The terminal uses a communication means to transmit the converted text data to the server, and the text data is transmitted to the server as an HTTP request.

[0967] Step 4:

[0968] The server inputs the received text data into a natural language processing (NLP) module, which analyzes the text data and extracts the intent, "I want a hug."

[0969] Step 5:

[0970] The server passes the intent extracted by the NLP module to the emotion engine, which analyzes the user's emotional state (e.g., dependent feelings) from the text data.

[0971] Step 6:

[0972] Based on the results of the language analysis and sentiment analysis, the server launches a response generation module, which generates an appropriate response text: "Okay, your stuffed toy is with you!"

[0973] Step 7:

[0974] The server sends the generated response text to the terminal, which then sends the response text to the terminal as an HTTP response.

[0975] Step 8:

[0976] The terminal inputs the received response text into a speech synthesis module, which converts the text data into speech data.

[0977] Step 9:

[0978] The terminal passes the converted voice data to a voice changer means, which converts the voice data into the voice of the character of the stuffed toy.

[0979] Step 10:

[0980] The device sends the converted character voice data to the speaker, which then plays back the voice, "Okay, your stuffed animal is with you!"

[0981] Through these specific steps, children can fulfill their emotions through interactions with stuffed toys. The present invention provides a safe environment for children even while their parents are busy, and the emotion engine has the function of continuously monitoring the user's emotional state and adjusting responses, thereby supporting the healthy development of children.

[0982] As a specific example, even if a child feels anxious about something, this system can immediately provide an appropriate response, putting the child at ease. For example, if a child says, "I'm scared," the system analyzes the emotion and generates a response such as, "It's okay, your stuffed toy is with you," providing reassurance to the child. In this way, the present invention reduces the burden on parents and enables appropriate responses to be made according to the emotional needs of children.

[0983] Example 2

[0984] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0985] In modern families, parents are often busy with work and housework, making it difficult for them to spend enough time with their children. There is also a need for systems that can respond appropriately when children feel anxious or lonely. Furthermore, conventional speech recognition and conversational systems lack the ability to accurately understand the user's emotional state and adjust responses accordingly, which means they are unable to adequately meet children's emotional needs.

[0986] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0987] In this invention, the server includes: means for acquiring voice uttered by the user; speech recognition means for converting the acquired voice into text data; communication means for transmitting the converted text data; language analysis means for analyzing the transmitted text data and extracting the user's intention and emotional state; response generation means for generating an appropriate response based on the extracted intention and emotional state; communication means for receiving the generated response as text data; speech synthesis means for converting the received response text into voice data; voice changer means for applying a character voice to the converted voice data; voice output means for outputting the voice data to which the character voice has been applied; and means for the response generation means to adjust the response by comparing it with the user's past emotional state. This makes it possible to provide an environment where parents can spend time with their children with peace of mind even when they are busy.

[0988] The "means for acquiring the voice uttered by the user" refers to hardware or software for capturing the voice spoken by the user (child).

[0989] The "voice recognition means for converting acquired voice into text data" is a technical means for converting voice data into text format.

[0990] The "communication means for transmitting the converted text data" refers to communication technology or equipment for transmitting the text data to a server or the like.

[0991] The "language analysis means for analyzing transmitted text data and extracting the user's intentions and emotional state" is a technology that has the function of analyzing the user's intentions and emotions from text data.

[0992] The "response generation means for generating an appropriate response based on the extracted intention and emotional state" is a technology for generating an appropriate reply according to the user's intention and emotion.

[0993] The "communication means for receiving the generated response as text data" refers to a communication technique or device for receiving the response text from the server.

[0994] The "voice synthesis means for converting the received response text into voice data" is a technical means for converting text data into voice data.

[0995] The "voice changer means for applying a character voice to converted voice data" is a technology for converting voice data into a specific character voice.

[0996] "Audio output means for outputting audio data to which a character voice has been applied" refers to technology such as a speaker for playing back audio data converted into a character voice.

[0997] "A means for generating a response that has the function of adjusting the response by comparing it with the user's past emotional state" refers to a technology that has the function of optimizing the current response by taking into account the user's past emotional data.

[0998] MODE FOR CARRYING OUT THE INVENTION

[0999] This invention relates to an AI toy that helps parents entertain their children while they are busy with housework and childcare. This system acquires speech uttered by the user (child) and generates and outputs an appropriate response. The detailed configuration and operation of the system are described below.

[1000] System Configuration

[1001] The system mainly includes the following hardware and software:

[1002] Voice acquisition means: A microphone (e.g., SHURE MV88) for acquiring the voice spoken by the user

[1003] Speech recognition means: A speech recognition module that converts captured speech into text data (e.g., Google Speech-to-Text API).

[1004] Communication means: Wi-Fi module (e.g., ESP32) for sending converted text data to a server

[1005] Language analysis means: A language analysis module (e.g., Google Natural Language API) that analyzes the transmitted text data to extract the user's intentions and emotional state.

[1006] Response generation means: A module that generates appropriate responses based on the extracted intentions and emotional states.

[1007] Speech synthesis means: A module that converts the generated response text into voice data (e.g., Amazon Polly)

[1008] Voice changer means: A device for applying a character voice to converted voice data (e.g., HT8955A)

[1009] Audio output means: a speaker for outputting audio data to which character voices are applied

[1010] Operation explanation

[1011] The server and the terminal work together to capture the user's voice and generate and output an appropriate response.

[1012] 1. Audio capture

[1013] The user speaks to the stuffed animal. For example, they say, "Mommy, hold me!" To capture this voice, a microphone (SHURE MV88) in the device captures the sound.

[1014] 2. Voice Recognition

[1015] The acquired voice data is input into a voice recognition module (Google Speech-to-Text API), and the voice data is converted into text data such as "Mom, hold me!"

[1016] 3. Data Transmission

[1017] The converted text data is sent to the server via the communication method (ESP32) using an HTTP POST request.

[1018] 4. Language and Sentiment Analysis

[1019] The server inputs the received text data into a language analysis module (Google Natural Language API) to analyze the meaning of the text content. At the same time, it uses an emotion engine to analyze the user's emotional state from the text data. For example, it can extract the emotional state of "feeling dependent."

[1020] 5. Response Generation

[1021] Based on the results of language analysis and sentiment analysis, the server activates a response generation module, which generates a response text such as "I understand, your stuffed toy is with you!"

[1022] 6. Response text reception and speech synthesis

[1023] The generated response text is sent from the server to the terminal, which inputs it into a speech synthesis module (Amazon Polly) to convert the text data into speech data.

[1024] 7. Voice Change and Audio Output

[1025] The converted voice data is converted into a character voice by the voice changer means (HT8955A) and then played back through the speaker, so that the user can hear the voice saying, "Okay, your stuffed toy is with you!"

[1026] Specific examples

[1027] The specific hardware and software used are as follows:

[1028] Microphone: SHURE MV88

[1029] Communication module: ESP32

[1030] Server: General cloud server

[1031] Speech Recognition Module: Google Speech-to-Text API

[1032] Language Analysis Module: Google Natural Language API

[1033] Speech synthesis module: Amazon Polly

[1034] Voice changer: HT8955A

[1035] Prompt Sentence Examples

[1036] "Analyze text data, such as a user saying, 'Hold me, Mommy!', recognize their emotional state, and generate an appropriate response."

[1037] Thus, the present invention can provide a sense of security to children even while parents are busy. In particular, the system has the ability to monitor the user's emotional state, compare it with past emotional states, and adjust responses to appropriately meet the emotional needs of children.

[1038] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1039] Step 1:

[1040] The user speaks to the stuffed toy. The user's voice (e.g., "Mommy, hold me!") is picked up by the device's microphone. The input is audio data, and the output is the captured raw audio signal. Specifically, the audio is picked up as an analog signal through the built-in microphone (e.g., SHURE MV88).

[1041] Step 2:

[1042] The device sends the captured voice signal to a voice recognition module, which (for example, Google Speech-to-Text API) converts the voice into text data. The input is voice data, and the output is text data such as "Mommy, hold me!". Specifically, the analog voice signal obtained from the microphone is digitized, then linked to the API and converted into text data.

[1043] Step 3:

[1044] The device sends the converted text data to the server via a communication method (for example, a Wi-Fi module such as ESP32). The input is the text data "Hold me, Mom!" and the output is an HTTP POST request to the server. Specifically, the text data is sent to the server via the Wi-Fi module over the Internet.

[1045] Step 4:

[1046] The server inputs the received text data into a language analysis module. The language analysis module (for example, Google Natural Language API) analyzes the meaning of the text content. The input is the text data "Mama, hold me!", and the output is the analyzed user intent. Specifically, the text data is sent via the API, and its content is analyzed.

[1047] Step 5:

[1048] At the same time, the server uses an emotion engine to analyze the user's emotional state from the text data. The input is the text data "Hold me, Mommy!", and the output is the emotional state "Feeling pampered." Specifically, the emotion engine runs an algorithm to identify the emotion based on the text analysis results.

[1049] Step 6:

[1050] The server launches a response generation module based on the results of language analysis and sentiment analysis. The generated response text is "I get it, your stuffed toy is with you!" The input is the analyzed intention and emotional state, and the output is the response text. Specifically, the response generation algorithm generates an appropriate response based on the analysis results.

[1051] Step 7:

[1052] The response text is sent from the server to the device. The input is the response text "Okay, your stuffed animal is with you!", and the output is the text data received by the device. Specifically, the server sends the text data as an HTTP response, and the device receives it.

[1053] Step 8:

[1054] The device inputs the received response text into a speech synthesis module and converts it into voice data (for example, Amazon Polly). The input is text data such as "I got it, your stuffed animal is with you!", and the output is voice data. Specifically, the speech synthesis module converts the text into a voice signal.

[1055] Step 9:

[1056] The converted voice data is then converted into a character voice by a voice changer (for example, HT8955A). The input is synthesized voice data, and the output is voice data of the character voice. Specifically, the pitch and tone of the voice data are adjusted to change it into the character voice.

[1057] Step 10:

[1058] The audio data converted into the character's voice is played through the device's speaker. The input is the character's voice data, and the output is the voice saying, "Okay, your stuffed animal is with you!". Specifically, the speaker plays the audio data, and the user can hear it.

[1059] In this way, the system acts as a way to keep the parents occupied while they are busy.

[1060] (Application example 2)

[1061] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1062] In recent years, there has been a demand for an environment where children can relax and enjoy themselves while their parents are busy shopping. However, physical stores often lack sufficient infrastructure and services to allow children to have fun while waiting. This can lead to children becoming bored or losing interest while their parents are shopping. There is also a lack of appropriate guides and instructions to ensure children's safety within the store. There is a demand for a solution to these problems and a way to provide an environment in physical stores where children can have fun while their parents can shop with peace of mind.

[1063] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1064] In this invention, the server includes means for acquiring voice uttered by the user, voice recognition means for converting the acquired voice into text data, and communication means for transmitting the converted text data, thereby enabling interface means for providing an interactive experience that attracts children's attention in public places and guide means for guiding them to spots and events within a store.

[1065] The "means for acquiring user-uttered speech" refers to a device or process that collects user-uttered speech using a speech input device.

[1066] The "speech recognition means for converting acquired speech into text data" is a system or algorithm that analyzes the speech of the user and converts it into corresponding text data.

[1067] The "communication means for transmitting converted text data" refers to a communication technology or protocol for transmitting the text data generated by the speech recognition module to a server or other device.

[1068] The "language analysis means that analyzes the transmitted text data and extracts the user's intention and emotional state" is software or an algorithm that analyzes the text data and identifies the user's intention and emotional state from its content.

[1069] The "response generation means for generating an appropriate response based on the extracted intention and emotional state" is a program or system for generating an appropriate reply or action according to the user's intention and emotion.

[1070] The "communication means for receiving the generated response as text data" is a technology or protocol for receiving the generated response text from a server or the like.

[1071] The "voice synthesis means for converting the received response text into voice data" refers to a synthesis technique or system for converting text data into natural voice.

[1072] The "voice changer means for applying a character voice to converted voice data" is a technology or module for adding a character voice to voice-synthesized data.

[1073] The "audio output means for outputting audio data to which character voices have been applied" refers to a device or system for reproducing the final audio data using a speaker or the like.

[1074] "Interface means for providing an interactive experience that attracts children in public places" refers to user interfaces and hardware that provide an interactive experience that is attractive to children.

[1075] The "guide means for providing information about spots and events within the store" is a system or device for providing information about various places and events within the store to children.

[1076] This invention relates to an interactive character experience for children in a brick-and-mortar store. The system is designed to keep children entertained while their parents shop. Below are detailed methods for implementing this system.

[1077] First, the system includes the following components: a microphone as a voice input device for capturing the user's (here, a child's) speech, a voice recognition module for converting the captured voice into text data, a communication means for sending the converted text data to a server, a language analysis module for analyzing the text data on the server side to extract the child's intentions and emotional state, and an emotion engine.

[1078] The server then has a response generation module that generates an appropriate response based on the analysis results. This response generation module generates a fun and engaging response for the child based on the extracted intention and emotional state. The generated response text is then sent to the terminal again via the communication means.

[1079] On the device side, a voice synthesis module operates to convert the received response text into voice data, and a voice changer is installed to apply a character voice to this voice data, and finally the voice data with the character voice applied is output through the speaker.

[1080] Robots, smartphones, tablets, and other devices can be used as interfaces to provide interactive experiences that will attract children's attention, especially in public places. Furthermore, interactive maps and augmented reality (AR) technology can be used as guides to guide children to in-store spots and events.

[1081] Specific examples

[1082] If a child asks "Where are the toys?" at the entrance to the store, the character will guide them by saying, "The toy section is on the third floor. Let's go together!" It can also pique the child's interest by asking questions along the way, such as "Do you know this candy?" This series of responses allows children to have fun walking around the store, while parents can continue shopping with peace of mind.

[1083] Prompt Sentence Examples

[1084] Text: "Where are your toys?"

[1085] Response Generation Prompt: "Create a response to guide a child through the store. Include the location of the toy department and an engaging quiz."

[1086] Example response:

[1087] "The toy section is on the third floor. Come with me! There'll be a quiz on the way, are you ready?"

[1088] Based on these components and specific examples, the present invention provides an environment in a physical store where children can have fun and not get bored.

[1089] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1090] Step 1:

[1091] Acquire the user's voice. In this step, a microphone is used to collect the user's (in this case, a child's) speech in real time. The input is the user's voice, and the output is the captured raw voice data.

[1092] Step 2:

[1093] The captured voice is converted into text data. The device activates a voice recognition module and converts the captured voice data into text data. Here, for example, the Google Speech-to-Text API is used. The input is voice data, and the output is the converted text data, "Where is your toy?"

[1094] Step 3:

[1095] The converted text data is sent to the server. The terminal sends the text data to the server using a communication means. The input is the text data, and the output is a transmission completion notification to the server.

[1096] Step 4:

[1097] Analyze the transmitted text data. The server analyzes the received text data and uses a language analysis module to extract the user's intent and emotional state. For example, it uses Natural Language Processing (NLP) technology. The input is the text data, and the output is the extracted intent "asking for the location of the toy section" and emotional state "interested."

[1098] Step 5:

[1099] Generate an appropriate response. The server invokes a response generation module to generate a corresponding response based on the extracted intent and emotional state. Here, a generative AI model is used to generate the response, "The toy department is on the third floor. Let's go together!" The input is the extracted intent and emotional state, and the output is the response text.

[1100] Step 6:

[1101] The generated response text is sent to the terminal. The server sends the generated response text to the terminal via a communication means. The input is the response text, and the output is a transmission completion notification to the terminal.

[1102] Step 7:

[1103] The device converts the received response text into voice data. The device activates a speech synthesis module to convert the received response text into natural-sounding voice, for example, using the Google Text-to-Speech API. The input is the response text, and the output is voice data.

[1104] Step 8:

[1105] A character voice is applied to the converted voice data. The terminal uses a voice changer means to add the character voice to the voice data. The input is the voice data, and the output is the voice data to which the character voice has been applied.

[1106] Step 9:

[1107] The voice data with the character voice applied is output. The device plays the final voice data through the speaker and provides a response to the user. The input is the voice data with the character voice applied, and the output is the voice played to the user.

[1108] This series of processing steps allows the user (child) to be guided around the store in an enjoyable and interactive manner, while the parent can continue shopping with peace of mind.

[1109] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1110] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1111] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1112] [Fourth embodiment]

[1113] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1114] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1115] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1116] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1117] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1118] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1119] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1120] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1121] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1122] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1123] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1124] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1125] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1126] The present invention relates to an AI toy that allows parents to entertain their children while they are busy with housework and childcare. The operation of the specific system and the processing of the related programs will be described in detail below.

[1127] This system mainly includes the following components: speech recognition means for acquiring voice uttered by the user, speech recognition means for converting the acquired voice into text data, communication means for transmitting the converted text data to a server, language analysis means for analyzing the transmitted text data and extracting the user's intentions and emotional state, response generation means for generating an appropriate response based on the analysis results, speech synthesis means for converting the generated response text into voice data, voice changer means for applying a character voice to the voice data, and voice output means for outputting the converted voice data.

[1128] A specific flow of program processing will be explained.

[1129] 1. User voice input

[1130] The user (child) speaks to the stuffed toy, for example, "Mommy, hold me!" This voice is captured by the device's built-in microphone.

[1131] 2. Voice Recognition

[1132] The device inputs the captured voice data into a voice recognition module, which converts the voice into text data such as "Mom, hold me!"

[1133] 3. Sending text data

[1134] The converted text data is transmitted to a server via the communication means of the terminal and over the Internet.

[1135] 4. Language Processing and Sentiment Analysis

[1136] The server inputs the received text data into a language analysis module, which analyzes the meaning and emotion of the text content. Through this analysis, for example, the intention of "I want to be held" and the "feeling of being pampered" can be extracted.

[1137] 5. Response Generation

[1138] Based on the results of the language analysis, the server generates an appropriate response text: "Okay, your stuffed animal is with you!"

[1139] 6. Sending Text Responses

[1140] The response text is sent from the server to the terminal.

[1141] 7. Speech Synthesis

[1142] The terminal inputs the received response text into a voice synthesis module, and converts the text data into voice data.

[1143] 8. Voice Change

[1144] The converted voice data is converted into a character voice by a voice changer means.

[1145] 9. Audio Output

[1146] Finally, the converted voice data is played back through the device's speaker, and the child hears, "Okay, your stuffed animal is with you!"

[1147] Through the above process, children can satisfy their emotions through interactions with stuffed toys while their parents are busy. To give an example, even if a child feels anxious about something, this system can provide an appropriate response immediately, reassuring the child. In this way, the present invention provides an effective means for reducing the burden on parents and supporting the healthy development of children.

[1148] The processing flow will be explained below.

[1149] Step 1:

[1150] The user (child) speaks to the stuffed animal. For example, they say, "Mommy, hold me!"

[1151] The device captures the user's voice through a built-in microphone.

[1152] Step 2:

[1153] The terminal inputs the acquired voice data into a voice recognition module.

[1154] The voice recognition module analyzes the voice data and converts it into text data such as "Mom, hold me!"

[1155] Step 3:

[1156] The terminal uses a communication means for transmitting the converted text data to the server.

[1157] The text data is sent to the server as an HTTP request.

[1158] Step 4:

[1159] The server inputs the received text data into a natural language processing (NLP) module.

[1160] The NLP module analyzes the text data and extracts the intent, "I want to be hugged."

[1161] Step 5:

[1162] The server passes the intent extracted by the NLP module to the sentiment analysis module.

[1163] The emotion analysis module analyzes the user's emotional state (e.g., affectionate feelings) from the text.

[1164] Step 6:

[1165] The server launches a response generation module based on the results of the language analysis and sentiment analysis.

[1166] The response generation module generates the appropriate response text: "Okay, your stuffed toy is with you!"

[1167] Step 7:

[1168] The server uses a communication means to transmit the generated response text to the terminal.

[1169] The response text is sent to the terminal as an HTTP response.

[1170] Step 8:

[1171] The terminal inputs the received response text into the speech synthesis module.

[1172] The speech synthesis module converts the response text into speech data.

[1173] Step 9:

[1174] The terminal passes the voice data to the voice changer means.

[1175] The voice changer converts audio data into the character voice of the stuffed animal.

[1176] Step 10:

[1177] The device transmits the converted audio data to the speaker.

[1178] A voice will play from the speaker saying, "Okay, your stuffed animal is with you!"

[1179] Through the above process, the user (child) can feel satisfied through interaction with the stuffed toy. The present invention provides an environment where children can play safely even while their parents are busy.

[1180] Example 1

[1181] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1182] Conventional systems lacked a means for parents to effectively communicate with their children when they were busy with housework and childcare. Furthermore, their limited ability to accurately analyze ambiguous words and emotions expressed by children and respond appropriately led to situations where children felt uneasy. Furthermore, the technology to apply character voices to voice data to provide appropriate responses for children was insufficient.

[1183] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1184] In this invention, the server includes means for acquiring voice uttered by the user, voice recognition means for converting the acquired voice into text data, communication means for transmitting the converted text data, language analysis means for analyzing the transmitted text data and extracting the user's intention and emotional state, response generation means for generating an appropriate response based on the extracted intention and emotional state, communication means for receiving the generated response as text data, voice synthesis means for converting the received response text into voice data, voice changer means for applying a character voice to the converted voice data, and voice output means for outputting the voice data to which the character voice has been applied. This allows children to communicate effectively and feel at ease even when their parents are busy.

[1185] A "user" is a subject who uses the system, specifically a person who gives instructions to the system by speaking.

[1186] "Speech recognition means" refers to a device or software that has the function of analyzing acquired voice data and converting it into text data.

[1187] "Communication means" refers to devices or software with Internet connection capabilities for sending and receiving data between a terminal and a server.

[1188] "Language analysis means" refers to a device or software that has the function of analyzing received text data and extracting the user's intentions and emotional state.

[1189] "Response generation means" refers to a device or software that has the function of generating an appropriate response based on the analyzed user's intentions and emotional state.

[1190] "Speech synthesis means" refers to a device or software that has the function of converting text data into voice data.

[1191] "Voice changer means" refers to a device or software that has the function of applying a character voice to audio data and changing the voice quality to a specific one.

[1192] "Audio output means" refers to a device such as a speaker that is used to play back the final converted audio data.

[1193] A "generative AI model" refers to an algorithm or computer program that uses artificial intelligence to generate text.

[1194] The present invention relates to an AI toy that can entertain children while parents are busy. The system uses multiple hardware and software components to implement a series of processes that captures user-spoken speech and converts it into an appropriate response. The specific components of the system and their operation are described in detail below.

[1195] The system consists of the following components: a voice recognition means for acquiring voice uttered by the user, a voice recognition means for converting the acquired voice into text data, a communication means for transmitting the converted text data to a server, a language analysis means for analyzing the transmitted text data and extracting the user's intentions and emotional state, a response generation means for generating an appropriate response based on the analysis results, a voice synthesis means for converting the generated response text into voice data, a voice changer means for applying a character voice to the voice data, and a voice output means for outputting the converted voice data.

[1196] Hardware and software used

[1197] Specifically, this system uses the following hardware and software.

[1198] 1. The speech recognition method uses the Google Cloud Speech-to-Text API, which allows the captured voice data to be converted into text data.

[1199] 2. The communication method is to connect to the Internet using a regular Wi-Fi module, which allows data to be sent and received.

[1200] 3. The language analysis method uses IBM Watson NLU, which allows the system to analyze the user's intent and emotional state from the text data sent.

[1201] 4. The response generation method uses OpenAI GPT-3, which can generate appropriate responses based on the analyzed intent and emotional state. An example of a prompt sentence is, "What is the appropriate response when a child asks, 'Where is my stuffed animal?'"

[1202] 5. Amazon Polly is used as the speech synthesis means, which allows the generated text data to be converted into speech data.

[1203] 6. The voice changer uses Voicemod, which allows you to apply character voices to audio data.

[1204] 7. The built-in speaker is used as the audio output means, which allows the final audio data to be provided to the user.

[1205] Specific examples

[1206] A specific example of the system's operation is shown below.

[1207] The child turns to the stuffed animal and says, "Mommy, where is my stuffed animal?" The audio is captured by the device's built-in microphone.

[1208] The captured voice data is converted into text data using the Google Cloud Speech-to-Text API, generating the text "Mom, where's my stuffed animal?"

[1209] This text data is sent to the server using a Wi-Fi module.

[1210] The server uses IBM Watson NLU to analyze the intent of "I want to know where the stuffed animal is" from the text data.

[1211] The prompt for OpenAI GPT-3 was "What is the appropriate response when a child asks, 'Where is your stuffed animal?'" and an appropriate response was generated, resulting in the text data "Your stuffed animal is in the living room!"

[1212] This response text is sent from the server to the terminal.

[1213] The device converts the response text into voice data using Amazon Polly.

[1214] The converted voice data is then converted into a character voice using Voicemod.

[1215] Finally, the built-in speaker will play a voice saying, "The stuffed animal is in the living room!"

[1216] This allows children to feel safe and communicate effectively even while their parents are busy.

[1217] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1218] Step 1:

[1219] The device's built-in microphone captures the user's speech. This speech data becomes the input for the system. Speech data is generated when the user speaks a specific phrase, such as "Where is my stuffed animal?"

[1220] Step 2:

[1221] The device sends the captured voice data to the Google Cloud Speech-to-Text API, which converts the voice into text data. The input of this process is the voice data, and the output is the corresponding text data: "Where is the stuffed animal?"

[1222] Step 3:

[1223] The terminal transmits the converted text data to the server using a communication means (Wi-Fi module). At this time, the input is the converted text data, and the output is the data transmitted to the server.

[1224] Step 4:

[1225] The server receives the text data sent from the terminal. The input of this process is the text data from the terminal, and the output is the text data stored in the server.

[1226] Step 5:

[1227] The server inputs the received text data into IBM Watson NLU for language analysis, which extracts the user's intent and emotional state. The input for this process is text data, and the output is the analysis results. For example, the intent, "I want to know where the stuffed animal is," and the emotion, "confusion," are extracted.

[1228] Step 6:

[1229] The server uses OpenAI GPT-3 based on the analysis results to generate an appropriate response text. The prompt text used is something like, "What is the appropriate response when a child asks, 'Where is your stuffed animal?'" The input for this process is the analysis results, and the output is the response text, "Your stuffed animal is in the living room!"

[1230] Step 7:

[1231] The server transmits the generated response text to the terminal via a communication means, where the input is the generated response text and the output is data transmission to the terminal.

[1232] Step 8:

[1233] The terminal receives the response text sent from the server. The input of this process is the response text, and the output is text data stored in the terminal.

[1234] Step 9:

[1235] The device inputs the received response text into Amazon Polly and converts it into voice data. The input of this process is the response text, and the output is the generated voice data.

[1236] Step 10:

[1237] The device inputs the converted voice data into Voicemod, which converts it into a character voice. The input of this process is the generated voice data, and the output is the voice data with the character voice applied.

[1238] Step 11:

[1239] Finally, the device uses its built-in speaker to play back the voice data with the character voice applied. The input of this process is the voice data with the character voice, and the output is the actual voice that is played back. The user can hear the voice saying, "The stuffed animal is in the living room!"

[1240] (Application example 1)

[1241] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1242] The challenge is to solve the problem of children becoming bored or anxious while their parents are shopping in physical stores, and to provide a safe and enjoyable environment for children.To solve this problem, we aim to provide interactive entertainment for children and reduce the burden on parents.

[1243] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1244] In this invention, the server includes: means for acquiring voice uttered by a user; speech recognition means for converting the acquired voice into text data; communication means for transmitting the converted text data; language analysis means for analyzing the transmitted text data and extracting the user's intention and emotional state; response generation means for generating an appropriate response based on the extracted intention and emotional state; communication means for receiving the generated response as text data; speech synthesis means for converting the received response text into voice data; voice changer means for applying a character voice to the converted voice data; voice output means for outputting the voice data to which the character voice has been applied; and means for implementing interactions such as stories and games to provide entertainment for children. This allows children to enjoy interactive entertainment while parents can shop with peace of mind.

[1245] The "means for acquiring the voice uttered by the user" is a device for physically capturing the voice spoken by the user.

[1246] The "voice recognition means for converting captured voice into text data" is a technology for recognizing and converting captured voice into digital text data.

[1247] The "communication means for transmitting converted text data" is a method for transmitting the converted text data to a remote server or other device.

[1248] "Language analysis means that analyzes transmitted text data and extracts the user's intentions and emotional state" refers to software or algorithms that analyze received text data to understand the user's intentions and emotional state.

[1249] The "response generation means for generating an appropriate response based on the extracted intention and emotional state" is a system that generates an appropriate response based on the analysis results.

[1250] The "communication means for receiving the generated response as text data" is a method for receiving the generated response text data from the server back to the terminal.

[1251] The "voice synthesis means for converting received response text into voice data" is a technology for converting received text data into natural voice.

[1252] The "voice changer means for applying a character voice to converted voice data" is a technology for applying the voice of a specific character to generated voice data.

[1253] The "audio output means for outputting audio data to which a character voice has been applied" is a device that outputs the converted audio data to the outside through a speaker.

[1254] "Means for implementing interactions such as stories and games to provide entertainment for children" refers to equipment and technology that provides interactive content such as stories and games that children can enjoy.

[1255] The present invention relates to an interactive entertainment system that helps keep children entertained while their parents shop in a brick-and-mortar store. The system's primary purpose is to provide entertainment for children and make shopping safer and more convenient for parents.

[1256] The system includes the following elements:

[1257] 1. Voice acquisition means: A microphone attached to the terminal is used to acquire the voice spoken by the user (child).

[1258] 2. Speech recognition means: Speech recognition software (e.g., Google Speech Recognition API) is used to convert the acquired voice data into text data.

[1259] 3. Communication means: The communication module of the terminal is used to send the converted text data to the server.

[1260] 4. Language analysis means: The server uses natural language processing (NLP) algorithms (e.g., TextBlob) to analyze the received text data and extract the user's intent and emotional state.

[1261] 5. Response generation means: A generative AI model (e.g., GPT-3) that generates appropriate responses based on the extracted intent and emotional state is installed on the server.

[1262] 6. Communication means: The communication module of the server is used to send the generated response text data back to the terminal.

[1263] 7. Speech synthesis means: The terminal uses a speech synthesis tool (e.g., gTTS) to convert the received response text into voice data.

[1264] 8. Voice changer means: Using voice changer software to apply a character voice to the converted voice data.

[1265] 9. Audio output means: Audio data to which the character voice is applied is output through the device speaker.

[1266] Detailed explanation of the process

[1267] The process performed by the server is as follows: First, the microphone of the terminal captures the voice of the child speaking. The acquired voice data is converted into text data by voice recognition means. This converted text data is sent to the server via communication means. Next, the server analyzes the received text data using language analysis means to extract the user's intention and emotional state. Based on the analysis results, response generation means generates an appropriate response text. This generated response text is again sent to the terminal via communication means and converted into voice data by voice synthesis means. Furthermore, voice changer means is used to apply a character voice to this voice data. Finally, the converted voice data with the character voice is output through the terminal's speaker.

[1268] Specific examples

[1269] For example, while a parent is ordering coffee at a cafe in a brick-and-mortar store, a child might say to a robot, "I feel lonely." The robot's microphone captures the voice and converts it into text data using a speech recognition means. This text data is sent to a server, which then analyzes the emotion "I feel lonely" using a language analysis means. Based on the analysis results, a response text, "It's okay, let's play together!", is generated by a response generation means, which is then sent back to the terminal and converted into voice data by a voice synthesis means. After a character voice is applied by a voice changer means, this voice data is played back to the child through a speaker, reassuring the child.

[1270] Prompt Sentence Examples

[1271] python

[1272] if __name__ == "__main__":

[1273] print("System booting...")

[1274] Example

[1275] Child says "I'm lonely"

[1276] user_input = "lonely"

[1277] response = generate_response(user_input)

[1278] synthesize_speech(response)

[1279] Using this prompt, you can see how the system captures the child's speech and returns an appropriate response.

[1280] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1281] Step 1:

[1282] The user (child) speaks. The device's built-in microphone captures the voice data of the child. At this stage, the input is the user's voice, and the output is voice data.

[1283] Step 2:

[1284] The device sends the captured voice data to a voice recognition means, which (for example, Google Speech Recognition API) converts the voice data into text data. At this stage, the input is voice data and the output is text data.

[1285] Step 3:

[1286] The terminal sends the converted text data to the server using a communication means. The communication means delivers the text data to the server via the Internet. The input at this stage is text data, and the output is the transmission of text data to the server.

[1287] Step 4:

[1288] The text data received by the server is input to the language analysis means. The language analysis means (e.g., TextBlob) analyzes the text data and extracts the user's intention and emotional state. The input at this stage is text data, and the output is the user's intention and emotional state.

[1289] Step 5:

[1290] Based on the extracted intention and emotional state, the server generates an appropriate response using a response generation means (e.g., generative AI model: GPT-3). The input at this stage is the user's intention and emotional state, and the output is response text data.

[1291] Step 6:

[1292] The generated response text data is sent to the terminal using the communication means of the server. The input at this stage is the response text data, and the output is the transmission of the text data to the terminal.

[1293] Step 7:

[1294] The response text data received by the terminal is input to a speech synthesis means. The speech synthesis means (e.g., gTTS) converts the text data into speech data. The input at this stage is the response text data, and the output is the response speech data.

[1295] Step 8:

[1296] The terminal inputs the converted voice data to the voice changer means, which applies a character voice to the voice data. The input at this stage is the response voice data, and the output is the voice data after the character voice has been applied.

[1297] Step 9:

[1298] The terminal plays the voice data with the character voice applied from the speaker through the voice output means. The input at this stage is the voice data with the character voice applied, and the output is the actual voice. The process is completed when the child listens to this voice and responds interactively.

[1299] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1300] The present invention relates to an AI toy that allows parents to entertain their children while they are busy with housework and childcare. The operation of the specific system and the processing of the related programs will be described in detail below.

[1301] This system mainly includes the following components: speech recognition means for acquiring speech uttered by the user, speech recognition means for converting the acquired speech into text data, communication means for transmitting the converted text data to a server, language analysis means for analyzing the transmitted text data and extracting the user's intentions and emotional state, and an emotion engine for recognizing the emotional state, response generation means for generating an appropriate response based on the analysis results, speech synthesis means for converting the generated response text into speech data, voice changer means for applying a character voice to the speech data, and speech output means for outputting the converted speech data.

[1302] A specific flow of program processing will be explained.

[1303] 1. User voice input

[1304] The user (child) speaks to the stuffed toy, for example, "Mommy, hold me!" This voice is captured by the device's built-in microphone.

[1305] 2. Voice Recognition

[1306] The device inputs the captured voice data into a voice recognition module, which converts the voice into text data such as "Mom, hold me!"

[1307] 3. Sending text data

[1308] The converted text data is transmitted to a server via the communication means of the terminal and over the Internet.

[1309] 4. Language Processing and Sentiment Analysis

[1310] The server inputs the received text data into a language analysis module and analyzes the meaning of the text content.

[1311] At the same time, the server uses an emotion engine to analyze the user's emotional state from the text data. For example, the emotional state "feeling dependent" is extracted.

[1312] 5. Response Generation

[1313] Based on the results of language analysis and sentiment analysis, the server activates a response generation module, which generates an appropriate response based on the user's emotional state.

[1314] For example, an appropriate response text might be generated: "Okay, your stuffed animal is with you!"

[1315] 6. Sending Text Responses

[1316] The response text is sent from the server to the terminal.

[1317] 7. Speech Synthesis

[1318] The terminal inputs the received response text into a voice synthesis module, and converts the text data into voice data.

[1319] 8. Voice Change

[1320] The converted voice data is converted into a character voice by a voice changer means.

[1321] 9. Audio Output

[1322] Finally, the converted voice data is played back through the device's speaker, and the child hears, "Okay, your stuffed animal is with you!"

[1323] In this system, the emotion engine continuously monitors the user's emotional state and adjusts the response by comparing it with past emotional states. For example, if the user continuously displays sad emotions, the emotion engine can detect this and be programmed to provide a more specific response. In this way, the present invention can respond to a child's emotional needs while reducing the burden on parents. For example, if a child is feeling anxious, the stuffed toy can respond appropriately and calm the child. This provides a safe environment for the child even while the parent is busy.

[1324] The processing flow will be explained below.

[1325] Step 1:

[1326] The user (child) speaks to the stuffed animal, for example, "Mommy, hold me!" The device's built-in microphone captures this voice.

[1327] Step 2:

[1328] The device inputs the captured voice data into the voice recognition module, which analyzes the voice data and converts it into text data such as "Mama, hold me!"

[1329] Step 3:

[1330] The terminal uses a communication means to transmit the converted text data to the server, and the text data is transmitted to the server as an HTTP request.

[1331] Step 4:

[1332] The server inputs the received text data into a natural language processing (NLP) module, which analyzes the text data and extracts the intent, "I want a hug."

[1333] Step 5:

[1334] The server passes the intent extracted by the NLP module to the emotion engine, which analyzes the user's emotional state (e.g., dependent feelings) from the text data.

[1335] Step 6:

[1336] Based on the results of the language analysis and sentiment analysis, the server launches a response generation module, which generates an appropriate response text: "Okay, your stuffed toy is with you!"

[1337] Step 7:

[1338] The server sends the generated response text to the terminal, which then sends the response text to the terminal as an HTTP response.

[1339] Step 8:

[1340] The terminal inputs the received response text into a speech synthesis module, which converts the text data into speech data.

[1341] Step 9:

[1342] The terminal passes the converted voice data to a voice changer means, which converts the voice data into the voice of the character of the stuffed toy.

[1343] Step 10:

[1344] The device sends the converted character voice data to the speaker, which then plays back the voice, "Okay, your stuffed animal is with you!"

[1345] Through these specific steps, children can fulfill their emotions through interactions with stuffed toys. The present invention provides a safe environment for children even while their parents are busy, and the emotion engine has the function of continuously monitoring the user's emotional state and adjusting responses, thereby supporting the healthy development of children.

[1346] As a specific example, even if a child feels anxious about something, this system can immediately provide an appropriate response, putting the child at ease. For example, if a child says, "I'm scared," the system analyzes the emotion and generates a response such as, "It's okay, your stuffed toy is with you," providing reassurance to the child. In this way, the present invention reduces the burden on parents and enables appropriate responses to be made according to the emotional needs of children.

[1347] Example 2

[1348] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1349] In modern families, parents are often busy with work and housework, making it difficult for them to spend enough time with their children. There is also a need for systems that can respond appropriately when children feel anxious or lonely. Furthermore, conventional speech recognition and conversational systems lack the ability to accurately understand the user's emotional state and adjust responses accordingly, which means they are unable to adequately meet children's emotional needs.

[1350] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1351] In this invention, the server includes: means for acquiring voice uttered by the user; speech recognition means for converting the acquired voice into text data; communication means for transmitting the converted text data; language analysis means for analyzing the transmitted text data and extracting the user's intention and emotional state; response generation means for generating an appropriate response based on the extracted intention and emotional state; communication means for receiving the generated response as text data; speech synthesis means for converting the received response text into voice data; voice changer means for applying a character voice to the converted voice data; voice output means for outputting the voice data to which the character voice has been applied; and means for the response generation means to adjust the response by comparing it with the user's past emotional state. This makes it possible to provide an environment where parents can spend time with their children with peace of mind even when they are busy.

[1352] The "means for acquiring the voice uttered by the user" refers to hardware or software for capturing the voice spoken by the user (child).

[1353] The "voice recognition means for converting acquired voice into text data" is a technical means for converting voice data into text format.

[1354] The "communication means for transmitting the converted text data" refers to communication technology or equipment for transmitting the text data to a server or the like.

[1355] The "language analysis means for analyzing transmitted text data and extracting the user's intentions and emotional state" is a technology that has the function of analyzing the user's intentions and emotions from text data.

[1356] The "response generation means for generating an appropriate response based on the extracted intention and emotional state" is a technology for generating an appropriate reply according to the user's intention and emotion.

[1357] The "communication means for receiving the generated response as text data" refers to a communication technique or device for receiving the response text from the server.

[1358] The "voice synthesis means for converting the received response text into voice data" is a technical means for converting text data into voice data.

[1359] The "voice changer means for applying a character voice to converted voice data" is a technology for converting voice data into a specific character voice.

[1360] "Audio output means for outputting audio data to which a character voice has been applied" refers to technology such as a speaker for playing back audio data converted into a character voice.

[1361] "A means for generating a response that has the function of adjusting the response by comparing it with the user's past emotional state" refers to a technology that has the function of optimizing the current response by taking into account the user's past emotional data.

[1362] MODE FOR CARRYING OUT THE INVENTION

[1363] This invention relates to an AI toy that helps parents entertain their children while they are busy with housework and childcare. This system acquires speech uttered by the user (child) and generates and outputs an appropriate response. The detailed configuration and operation of the system are described below.

[1364] System Configuration

[1365] The system mainly includes the following hardware and software:

[1366] Voice acquisition means: A microphone (e.g., SHURE MV88) for acquiring the voice spoken by the user

[1367] Speech recognition means: A speech recognition module that converts captured speech into text data (e.g., Google Speech-to-Text API).

[1368] Communication means: Wi-Fi module (e.g., ESP32) for sending converted text data to a server

[1369] Language analysis means: A language analysis module (e.g., Google Natural Language API) that analyzes the transmitted text data to extract the user's intentions and emotional state.

[1370] Response generation means: A module that generates appropriate responses based on the extracted intentions and emotional states.

[1371] Speech synthesis means: A module that converts the generated response text into voice data (e.g., Amazon Polly)

[1372] Voice changer means: A device for applying a character voice to converted voice data (e.g., HT8955A)

[1373] Audio output means: a speaker for outputting audio data to which character voices are applied

[1374] Operation explanation

[1375] The server and the terminal work together to capture the user's voice and generate and output an appropriate response.

[1376] 1. Audio capture

[1377] The user speaks to the stuffed animal. For example, they say, "Mommy, hold me!" To capture this voice, a microphone (SHURE MV88) in the device captures the sound.

[1378] 2. Voice Recognition

[1379] The acquired voice data is input into a voice recognition module (Google Speech-to-Text API), and the voice data is converted into text data such as "Mom, hold me!"

[1380] 3. Data Transmission

[1381] The converted text data is sent to the server via the communication method (ESP32) using an HTTP POST request.

[1382] 4. Language and Sentiment Analysis

[1383] The server inputs the received text data into a language analysis module (Google Natural Language API) to analyze the meaning of the text content. At the same time, it uses an emotion engine to analyze the user's emotional state from the text data. For example, it can extract the emotional state of "feeling dependent."

[1384] 5. Response Generation

[1385] Based on the results of language analysis and sentiment analysis, the server activates a response generation module, which generates a response text such as "I understand, your stuffed toy is with you!"

[1386] 6. Response text reception and speech synthesis

[1387] The generated response text is sent from the server to the terminal, which inputs it into a speech synthesis module (Amazon Polly) to convert the text data into speech data.

[1388] 7. Voice Change and Audio Output

[1389] The converted voice data is converted into a character voice by the voice changer means (HT8955A) and then played back through the speaker, so that the user can hear the voice saying, "Okay, your stuffed toy is with you!"

[1390] Specific examples

[1391] The specific hardware and software used are as follows:

[1392] Microphone: SHURE MV88

[1393] Communication module: ESP32

[1394] Server: General cloud server

[1395] Speech Recognition Module: Google Speech-to-Text API

[1396] Language Analysis Module: Google Natural Language API

[1397] Speech synthesis module: Amazon Polly

[1398] Voice changer: HT8955A

[1399] Prompt Sentence Examples

[1400] "Analyze text data, such as a user saying, 'Hold me, Mommy!', recognize their emotional state, and generate an appropriate response."

[1401] Thus, the present invention can provide a sense of security to children even while parents are busy. In particular, the system has the ability to monitor the user's emotional state, compare it with past emotional states, and adjust responses to appropriately meet the emotional needs of children.

[1402] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1403] Step 1:

[1404] The user speaks to the stuffed toy. The user's voice (e.g., "Mommy, hold me!") is picked up by the device's microphone. The input is audio data, and the output is the captured raw audio signal. Specifically, the audio is picked up as an analog signal through the built-in microphone (e.g., SHURE MV88).

[1405] Step 2:

[1406] The device sends the captured voice signal to a voice recognition module, which (for example, Google Speech-to-Text API) converts the voice into text data. The input is voice data, and the output is text data such as "Mommy, hold me!". Specifically, the analog voice signal obtained from the microphone is digitized, then linked to the API and converted into text data.

[1407] Step 3:

[1408] The device sends the converted text data to the server via a communication method (for example, a Wi-Fi module such as ESP32). The input is the text data "Hold me, Mom!" and the output is an HTTP POST request to the server. Specifically, the text data is sent to the server via the Wi-Fi module over the Internet.

[1409] Step 4:

[1410] The server inputs the received text data into a language analysis module. The language analysis module (for example, Google Natural Language API) analyzes the meaning of the text content. The input is the text data "Mama, hold me!", and the output is the analyzed user intent. Specifically, the text data is sent via the API, and its content is analyzed.

[1411] Step 5:

[1412] At the same time, the server uses an emotion engine to analyze the user's emotional state from the text data. The input is the text data "Hold me, Mommy!", and the output is the emotional state "Feeling pampered." Specifically, the emotion engine runs an algorithm to identify the emotion based on the text analysis results.

[1413] Step 6:

[1414] The server launches a response generation module based on the results of language analysis and sentiment analysis. The generated response text is "I get it, your stuffed toy is with you!" The input is the analyzed intention and emotional state, and the output is the response text. Specifically, the response generation algorithm generates an appropriate response based on the analysis results.

[1415] Step 7:

[1416] The response text is sent from the server to the device. The input is the response text "Okay, your stuffed animal is with you!", and the output is the text data received by the device. Specifically, the server sends the text data as an HTTP response, and the device receives it.

[1417] Step 8:

[1418] The device inputs the received response text into a speech synthesis module and converts it into voice data (for example, Amazon Polly). The input is text data such as "I got it, your stuffed animal is with you!", and the output is voice data. Specifically, the speech synthesis module converts the text into a voice signal.

[1419] Step 9:

[1420] The converted voice data is then converted into a character voice by a voice changer (for example, HT8955A). The input is synthesized voice data, and the output is voice data of the character voice. Specifically, the pitch and tone of the voice data are adjusted to change it into the character voice.

[1421] Step 10:

[1422] The audio data converted into the character's voice is played through the device's speaker. The input is the character's voice data, and the output is the voice saying, "Okay, your stuffed animal is with you!". Specifically, the speaker plays the audio data, and the user can hear it.

[1423] In this way, the system acts as a way to keep the parents occupied while they are busy.

[1424] (Application example 2)

[1425] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1426] In recent years, there has been a demand for an environment where children can relax and enjoy themselves while their parents are busy shopping. However, physical stores often lack sufficient infrastructure and services to allow children to have fun while waiting. This can lead to children becoming bored or losing interest while their parents are shopping. There is also a lack of appropriate guides and instructions to ensure children's safety within the store. There is a demand for a solution to these problems and a way to provide an environment in physical stores where children can have fun while their parents can shop with peace of mind.

[1427] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1428] In this invention, the server includes means for acquiring voice uttered by the user, voice recognition means for converting the acquired voice into text data, and communication means for transmitting the converted text data, thereby enabling interface means for providing an interactive experience that attracts children's attention in public places and guide means for guiding them to spots and events within a store.

[1429] The "means for acquiring user-uttered speech" refers to a device or process that collects user-uttered speech using a speech input device.

[1430] The "speech recognition means for converting acquired speech into text data" is a system or algorithm that analyzes the speech of the user and converts it into corresponding text data.

[1431] The "communication means for transmitting converted text data" refers to a communication technology or protocol for transmitting the text data generated by the speech recognition module to a server or other device.

[1432] The "language analysis means that analyzes the transmitted text data and extracts the user's intention and emotional state" is software or an algorithm that analyzes the text data and identifies the user's intention and emotional state from its content.

[1433] The "response generation means for generating an appropriate response based on the extracted intention and emotional state" is a program or system for generating an appropriate reply or action according to the user's intention and emotion.

[1434] The "communication means for receiving the generated response as text data" is a technology or protocol for receiving the generated response text from a server or the like.

[1435] The "voice synthesis means for converting the received response text into voice data" refers to a synthesis technique or system for converting text data into natural voice.

[1436] The "voice changer means for applying a character voice to converted voice data" is a technology or module for adding a character voice to voice-synthesized data.

[1437] The "audio output means for outputting audio data to which character voices have been applied" refers to a device or system for reproducing the final audio data using a speaker or the like.

[1438] "Interface means for providing an interactive experience that attracts children in public places" refers to user interfaces and hardware that provide an interactive experience that is attractive to children.

[1439] The "guide means for providing information about spots and events within the store" is a system or device for providing information about various places and events within the store to children.

[1440] This invention relates to an interactive character experience for children in a brick-and-mortar store. The system is designed to keep children entertained while their parents shop. Below are detailed methods for implementing this system.

[1441] First, the system includes the following components: a microphone as a voice input device for capturing the user's (here, a child's) speech, a voice recognition module for converting the captured voice into text data, a communication means for sending the converted text data to a server, a language analysis module for analyzing the text data on the server side to extract the child's intentions and emotional state, and an emotion engine.

[1442] The server then has a response generation module that generates an appropriate response based on the analysis results. This response generation module generates a fun and engaging response for the child based on the extracted intention and emotional state. The generated response text is then sent to the terminal again via the communication means.

[1443] On the device side, a voice synthesis module operates to convert the received response text into voice data, and a voice changer is installed to apply a character voice to this voice data, and finally the voice data with the character voice applied is output through the speaker.

[1444] Robots, smartphones, tablets, and other devices can be used as interfaces to provide interactive experiences that will attract children's attention, especially in public places. Furthermore, interactive maps and augmented reality (AR) technology can be used as guides to guide children to in-store spots and events.

[1445] Specific examples

[1446] If a child asks "Where are the toys?" at the entrance to the store, the character will guide them by saying, "The toy section is on the third floor. Let's go together!" It can also pique the child's interest by asking questions along the way, such as "Do you know this candy?" This series of responses allows children to have fun walking around the store, while parents can continue shopping with peace of mind.

[1447] Prompt Sentence Examples

[1448] Text: "Where are your toys?"

[1449] Response Generation Prompt: "Create a response to guide a child through the store. Include the location of the toy department and an engaging quiz."

[1450] Example response:

[1451] "The toy section is on the third floor. Come with me! There'll be a quiz on the way, are you ready?"

[1452] Based on these components and specific examples, the present invention provides an environment in a physical store where children can have fun and not get bored.

[1453] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1454] Step 1:

[1455] Acquire the user's voice. In this step, a microphone is used to collect the user's (in this case, a child's) speech in real time. The input is the user's voice, and the output is the captured raw voice data.

[1456] Step 2:

[1457] The captured voice is converted into text data. The device activates a voice recognition module and converts the captured voice data into text data. Here, for example, the Google Speech-to-Text API is used. The input is voice data, and the output is the converted text data, "Where is your toy?"

[1458] Step 3:

[1459] The converted text data is sent to the server. The terminal sends the text data to the server using a communication means. The input is the text data, and the output is a transmission completion notification to the server.

[1460] Step 4:

[1461] Analyze the transmitted text data. The server analyzes the received text data and uses a language analysis module to extract the user's intent and emotional state. For example, it uses Natural Language Processing (NLP) technology. The input is the text data, and the output is the extracted intent "asking for the location of the toy section" and emotional state "interested."

[1462] Step 5:

[1463] Generate an appropriate response. The server invokes a response generation module to generate a corresponding response based on the extracted intent and emotional state. Here, a generative AI model is used to generate the response, "The toy department is on the third floor. Let's go together!" The input is the extracted intent and emotional state, and the output is the response text.

[1464] Step 6:

[1465] The generated response text is sent to the terminal. The server sends the generated response text to the terminal via a communication means. The input is the response text, and the output is a transmission completion notification to the terminal.

[1466] Step 7:

[1467] The device converts the received response text into voice data. The device activates a speech synthesis module to convert the received response text into natural-sounding voice, for example, using the Google Text-to-Speech API. The input is the response text, and the output is voice data.

[1468] Step 8:

[1469] A character voice is applied to the converted voice data. The terminal uses a voice changer means to add the character voice to the voice data. The input is the voice data, and the output is the voice data to which the character voice has been applied.

[1470] Step 9:

[1471] The voice data with the character voice applied is output. The device plays the final voice data through the speaker and provides a response to the user. The input is the voice data with the character voice applied, and the output is the voice played to the user.

[1472] This series of processing steps allows the user (child) to be guided around the store in an enjoyable and interactive manner, while the parent can continue shopping with peace of mind.

[1473] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1474] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1475] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1476] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1477] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1478] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1479] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1480] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1481] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1482] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1483] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1484] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1485] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1486] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1487] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1488] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1489] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1490] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1491] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1492] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1493] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1494] The following is further disclosed regarding the above embodiment.

[1495] (Claim 1)

[1496] A means for acquiring a voice uttered by a user;

[1497] A speech recognition means for converting the acquired speech into text data;

[1498] a communication means for transmitting the converted text data;

[1499] a language analysis means for analyzing the transmitted text data and extracting the user's intention and emotional state;

[1500] a response generation means for generating an appropriate response based on the extracted intention and emotional state;

[1501] a communication means for receiving the generated response as text data;

[1502] a voice synthesis means for converting the received response text into voice data;

[1503] a voice changer means for applying a character voice to the converted voice data;

[1504] A system including an audio output means for outputting audio data to which a character voice is applied.

[1505] (Claim 2)

[1506] 2. The system according to claim 1, wherein said speech recognition means has a function of analyzing even ambiguous utterances of the user.

[1507] (Claim 3)

[1508] 2. The system according to claim 1, wherein the language analysis means has a function of analyzing an emotional state from voice data of a user.

[1509] "Example 1"

[1510] (Claim 1)

[1511] A means for acquiring a voice uttered by a user;

[1512] A speech recognition means for converting the acquired speech into text data;

[1513] a communication means for transmitting the converted text data;

[1514] a language analysis means for analyzing the transmitted text data and extracting the user's intention and emotional state;

[1515] a response generation means for generating an appropriate response based on the extracted intention and emotional state;

[1516] a communication means for receiving the generated response as text data;

[1517] a voice synthesis means for converting the received response text into voice data;

[1518] a voice changer means for applying a character voice to the converted voice data;

[1519] A system including an audio output means for outputting audio data to which a character voice is applied.

[1520] (Claim 2)

[1521] The system of claim 1, wherein the response generation means has the function of using a generative AI model to generate an appropriate response to a child's utterance.

[1522] (Claim 3)

[1523] 10. The system of claim 1, wherein the communication means has the capability of sending and receiving text data and responses via the Internet.

[1524] "Application Example 1"

[1525] (Claim 1)

[1526] A means for acquiring a voice uttered by a user;

[1527] A speech recognition means for converting the acquired speech into text data;

[1528] a communication means for transmitting the converted text data;

[1529] a language analysis means for analyzing the transmitted text data and extracting the user's intention and emotional state;

[1530] a response generation means for generating an appropriate response based on the extracted intention and emotional state;

[1531] a communication means for receiving the generated response as text data;

[1532] a voice synthesis means for converting the received response text into voice data;

[1533] a voice changer means for applying a character voice to the converted voice data;

[1534] an audio output means for outputting audio data to which the character voice is applied;

[1535] A system that includes a means for conducting interactions such as stories and games to provide entertainment for children.

[1536] (Claim 2)

[1537] 2. The system according to claim 1, wherein said speech recognition means has a function of analyzing even ambiguous utterances of the user.

[1538] (Claim 3)

[1539] 2. The system according to claim 1, wherein the language analysis means has a function of analyzing an emotional state from voice data of a user.

[1540] "Example 2: Combining Emotion Engines"

[1541] (Claim 1)

[1542] A means for acquiring a voice uttered by a user;

[1543] A speech recognition means for converting the acquired speech into text data;

[1544] a communication means for transmitting the converted text data;

[1545] a language analysis means for analyzing the transmitted text data and extracting the user's intention and emotional state;

[1546] a response generation means for generating an appropriate response based on the extracted intention and emotional state;

[1547] a communication means for receiving the generated response as text data;

[1548] a voice synthesis means for converting the received response text into voice data;

[1549] a voice changer means for applying a character voice to the converted voice data;

[1550] an audio output means for outputting audio data to which the character voice is applied;

[1551] The response generating means has a function of adjusting a response by comparing with a user's past emotional state;

[1552] A system including:

[1553] (Claim 2)

[1554] 2. The system according to claim 1, wherein said speech recognition means has a function of analyzing even ambiguous utterances of the user.

[1555] (Claim 3)

[1556] 2. The system according to claim 1, wherein the language analysis means has a function of analyzing an emotional state from voice data of a user.

[1557] "Application example 2 when combining emotion engines"

[1558] (Claim 1)

[1559] A means for acquiring a voice uttered by a user;

[1560] A speech recognition means for converting the acquired speech into text data;

[1561] a communication means for transmitting the converted text data;

[1562] a language analysis means for analyzing the transmitted text data and extracting the user's intention and emotional state;

[1563] a response generation means for generating an appropriate response based on the extracted intention and emotional state;

[1564] a communication means for receiving the generated response as text data;

[1565] a voice synthesis means for converting the received response text into voice data;

[1566] a voice changer means for applying a character voice to the converted voice data;

[1567] an audio output means for outputting audio data to which the character voice is applied;

[1568] an interface means for providing an engaging interactive experience for children in a public space;

[1569] The system includes a guide means for providing guidance to in-store spots and events.

[1570] (Claim 2)

[1571] 2. The system according to claim 1, wherein said speech recognition means has a function of analyzing even ambiguous utterances of the user.

[1572] (Claim 3)

[1573] 2. The system according to claim 1, wherein the language analysis means has a function of analyzing an emotional state from voice data of a user. [Explanation of symbols]

[1574] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for acquiring a voice uttered by a user; A speech recognition means for converting the acquired speech into text data; a communication means for transmitting the converted text data; a language analysis means for analyzing the transmitted text data and extracting the user's intention and emotional state; a response generation means for generating an appropriate response based on the extracted intention and emotional state; a communication means for receiving the generated response as text data; a voice synthesis means for converting the received response text into voice data; a voice changer means for applying a character voice to the converted voice data; A system including an audio output means for outputting audio data to which a character voice is applied.

2. 2. The system according to claim 1, wherein said speech recognition means has a function of analyzing even ambiguous utterances of the user.

3. 2. The system according to claim 1, wherein said language analysis means has a function of analyzing an emotional state from the user's voice data.

Images