system

Abstract

To provide a system that provides convenience and efficiency for both users and manufacturers. [Solution] The system includes a means for capturing a user's voice input using a microphone provided on a mobile terminal, a means for analyzing the captured voice input and converting it into text data, a means for selecting a generative artificial intelligence model based on the text data, a means for generating response data using the generative artificial intelligence model, and a means for converting the generated response data into voice data using a voice synthesis engine and transmitting it to the user.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, various devices utilizing generative AI are appearing on the market, but each has its own unique specifications, making it extremely cumbersome for users to gather information and operate these devices effectively. Furthermore, developing individual generative AI-enabled products requires significant cost and effort for device manufacturers. As a result, the user experience is fragmented, slowing the spread of generative AI technology. The present invention aims to solve these issues and provide a system that provides convenience and efficiency for both users and manufacturers. [Means for solving the problem]

[0005] The present invention provides a system including a means for capturing a user's voice input, a means for analyzing the captured voice input and converting it into text data, a means for selecting a generative artificial intelligence model based on the text data, a means for generating response data using the generative artificial intelligence model, and a means for transmitting the generated response data to the user. This system allows users to enjoy a consistent user experience, and device manufacturers can reduce development costs by standardizing the processes of selecting a generative artificial intelligence model and generating responses. In particular, the system is characterized by using a microphone on a mobile device to capture the user's voice input, converting the response data into voice data using a speech synthesis engine and playing it back to the user, generating a device design based on the user's customization requests, and generating responses that unify the user experience across multiple devices.

[0006] The "means for capturing user voice input" refers to a device or function that captures user voice using a microphone or other voice input device and stores it as data.

[0007] A "means for analyzing captured voice input and converting it into text data" is a device or function that uses voice recognition technology to convert captured voice data into text form.

[0008] "Means for selecting a generative AI model based on text data" refers to a device or function that identifies and selects an appropriate generative AI model based on the analyzed text data.

[0009] "Means for generating response data using a generative artificial intelligence model" refers to a device or function that uses a selected generative AI model to generate response data corresponding to a user request.

[0010] "Means for transmitting generated response data to a user" refers to a device or function that transmits the response data generated by the generative AI model in an appropriate format to the user's device.

[0011] A "mobile terminal microphone" is an audio input device built into a mobile device for capturing audio.

[0012] A "speech synthesis engine" is a device or function that converts text data into voice data and provides a voice response to a user.

[0013] A "means for generating device designs" is a device or function that uses generative AI to dynamically change the appearance or functionality of a device based on a user's customization requests.

[0014] A "means for generating responses that unify usage experience" is a device or function that generates unified responses to provide a consistent user experience across multiple devices. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention relates to a system that captures a user's voice input, converts it into text data, selects an appropriate generative AI model to generate response data, and provides it to the user. Below, the program processing of this system is explained in natural language.

[0037] server

[0038] The server receives a voice data request sent from the user's device. First, the server analyzes this voice data and converts it into text data using a speech recognition engine. Then, based on the converted text data, the server selects the necessary generative AI model. For example, if a user asks, "What's the weather like today?", the server selects the generative AI model that corresponds to the weather information.

[0039] Next, the server generates response data using the selected generative AI model. The generated response data is provided in text format, such as "Today's weather is sunny." Finally, the server sends this response data to the user's device.

[0040] Terminal

[0041] The device is responsible for capturing the user's voice input: it captures the user's voice as they speak into earphones or a smartphone microphone, and transmits this captured voice data to a server using a communication protocol such as HTTP requests or WebSocket.

[0042] When the device receives the response data from the server, it decodes the received data to obtain the response content in text format. It then uses a speech synthesis engine to convert this text into speech and play it back to the user. For example, when the user issues a voice command such as "Play the next song," the device receives a response from the server saying "Playing the next song," converts it into speech, and plays it back.

[0043] User

[0044] The user issues a voice command to the device. For example, they put on earphones and ask, "What time is it now?" This voice command is captured by the device and sent to the server. As a result of the server's processing, the user receives a voice response such as, "The current time is 3:00 PM."

[0045] Additionally, users can customize the design of their device using a smartphone app, which uses generative AI to generate new designs based on the user's customization requests and apply them to the device in real time.

[0046] This system allows users to enjoy a consistent experience and smooth operation across multiple devices, while also reducing development costs for device manufacturers by standardizing the process of selecting generative AI models and generating responses.

[0047] The processing flow will be explained below.

[0048] Step 1:

[0049] The device captures the user's voice input: as the user speaks voice commands into earphones or a smartphone microphone, the voice data is recorded in real time and stored in a buffer.

[0050] Step 2:

[0051] The device encodes the captured audio data and sends it to the server using a communication protocol such as an HTTP request or WebSocket.

[0052] Step 3:

[0053] The server receives the voice data sent from the terminal and stores it in the server's internal storage.

[0054] Step 4:

[0055] The server uses a voice recognition engine to convert the received voice data into text data, thereby obtaining the user's voice commands in text format.

[0056] Step 5:

[0057] The server selects an appropriate generative AI model based on the text data. It analyzes the text and identifies and selects the generative AI model that best suits the user's request.

[0058] Step 6:

[0059] The server generates response data using the selected generative AI model. For example, in response to the request "What's the weather like today?", it generates the response text "Today's weather is sunny."

[0060] Step 7:

[0061] The server transmits the generated response data to the terminal, which is then encoded in an appropriate format and transmitted to the terminal via a communication protocol.

[0062] Step 8:

[0063] The terminal receives the response data sent from the server, decodes the received data, and stores the response content in text format in the terminal.

[0064] Step 9:

[0065] The device converts the received text response into audio data using a speech synthesis engine, which analyzes the text and generates natural-sounding speech.

[0066] Step 10:

[0067] The device plays the generated voice data to the user. The user can hear the voice response through earphones or a speaker. For example, a voice such as "The current time is 3:00 PM" is played.

[0068] Step 11:

[0069] The user can then act on the generated voice response, for example by issuing the next voice command or customizing the device using the generative AI. This process is repeated, allowing the user to enjoy a consistent experience.

[0070] Example 1

[0071] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0072] Conventional voice recognition systems require a wide variety of technologies to properly convert voice input into text data and generate appropriate responses. This makes it difficult to provide accurate responses in real time to user voice commands. It also makes it difficult to provide a consistent user experience across multiple devices. Furthermore, it is not possible to customize the generated design or responses, resulting in a loss of user convenience.

[0073] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0074] In this invention, the server includes means for capturing a user's voice input, means for analyzing the captured voice input and converting it into text data, means for selecting an AI model generated based on the text data, means for generating response data using the generated AI model, means for transmitting the generated response data to the user, means for decoding the response data as text data, means for using a voice synthesis engine to convert the decoded text data into voice data, and means for playing back the converted voice data, thereby enabling accurate responses to the user's voice commands in real time and providing a unified user experience.

[0075] The "means for capturing user voice input" refers to a device or method for capturing voice uttered by a user, such as a microphone provided on a mobile terminal.

[0076] The "means for analyzing the captured voice input and converting it into text data" refers to a device or method for analyzing the captured voice using voice recognition technology and converting it into data in the form of a string of characters. Specifically, it includes a voice recognition engine.

[0077] The "means for selecting an AI model generated based on the text data" refers to a device or method for identifying and selecting an appropriate AI model based on the analyzed text data. Specifically, it includes an algorithm for selecting a generative AI model.

[0078] "Means for generating response data using the generated artificial intelligence model" refers to a device or method for generating a response to a user's query using a selected generative AI model.

[0079] The "means for transmitting the generated response data to the user" refers to a device or method for transferring the generated response data to the user's device. Specifically, it includes a data transmission technique using a communication protocol.

[0080] The "means for decoding the response data as text data" refers to a device or method for converting received response data into a character string format and analyzing it.

[0081] The "means for using a speech synthesis engine to convert the decoded text data into speech data" refers to a device or method for converting the decoded text data into speech format. Specifically, it includes a speech synthesis engine.

[0082] The "means for reproducing the converted audio data" refers to a device or method for allowing a user to listen to the converted audio data. Specifically, this applies to a speaker, earphones, etc.

[0083] The present invention relates to a system that captures a user's voice input, converts it into text data, selects an appropriate generative AI model, generates response data, and provides it to the user.

[0084] Server Roles

[0085] The server receives a voice data request sent from the user terminal. After receiving the voice data, the server converts the voice data into text data using a voice recognition engine (specifically, Google® Cloud Speech-to-Text API, etc.). For example, if a user asks, "What's the weather like today?", the voice data is converted into text data saying, "What's the weather like today?"

[0086] Based on the text data, the server selects an appropriate generative AI model by analyzing keywords and intent within the text data. For example, it selects a generative AI model (e.g., GPT-3 (registered trademark)) that corresponds to weather information.

[0087] Next, the server generates response data using the selected generative AI model. This generation process generates text data such as "Today's weather is sunny." Finally, the server sends the generated response data to the user's device. The communication protocol used is an HTTP POST request or WebSocket.

[0088] Device Role

[0089] The device has the function to capture the user's voice. Specifically, it records the user's voice using the smartphone microphone or earphone microphone, and sends the captured voice data to the server. The transmission method is an HTTP POST request or WebSocket.

[0090] When the device receives response data from the server, it decodes the data and obtains the response in text format. The obtained text data is converted into voice data using a speech synthesis engine (specifically, Amazon Polly, etc.). The response is provided to the user by playing this voice data through speakers or earphones.

[0091] User Roles

[0092] A user can issue a voice command to the device. For example, they can use earphones to ask, "What time is it now?" This voice command is captured by the device and sent to the server. As a result of the server's processing, the user can receive a voice response such as, "The current time is 3:00 PM."

[0093] Users can also customize the design of their device using a smartphone app, which uses generative AI to generate new designs based on the user's customization requests and apply them to the device in real time.

[0094] Specific operation example

[0095] Weather information inquiry

[0096] The user asks the device, "What's the weather like today?" The device captures this audio and sends the audio data to the server. The server uses the Google Cloud Speech-to-Text API to convert the audio into text, generating the text "What's the weather like today?" The server then selects a generative AI model (e.g., GPT-3) that corresponds to the weather information and generates response data saying, "The weather is sunny today." This response data is sent to the device, and when the device receives the text data, it uses a speech synthesis engine to convert it into audio data and play it back to the user.

[0097] Prompt Sentence Examples

[0098] The system prompts users with real-time responses based on their requests:

[0099] Received the user's voice saying "Play next song." Generate a text response saying "Play next song."

[0100] As described above, the system of the invention can efficiently process user voice input and provide appropriate responses. This system allows users to enjoy a consistent user experience and allows smooth operation across multiple devices. It also reduces development costs for device manufacturers by standardizing the process of selecting generative AI models and generating responses.

[0101] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0102] Step 1:

[0103] The user inputs a voice command into the device. For example, the user puts on earphones and says, "What's the weather like today?" The input data is voice data, and this is the starting point for the system's processing.

[0104] Step 2:

[0105] The device captures the user's voice. Specifically, it records the voice using the smartphone microphone or the earphone microphone. The input data is the voice data from the previous step, and the output data is the captured voice data.

[0106] Step 3:

[0107] The device sends the captured audio data to the server using a communication protocol such as an HTTP POST request or WebSocket. The input data is the captured audio data, and the output data is the audio data sent to the server.

[0108] Step 4:

[0109] The server analyzes the received voice data. Specifically, it converts the voice data into text data using a voice recognition engine (such as Google Cloud Speech-to-Text API). The input data is the voice data sent to the server, and the output data is text data. For example, the voice data "What's the weather like today?" is converted into the text data "What's the weather like today?"

[0110] Step 5:

[0111] The server selects an appropriate generative AI model based on the converted text data. The input data is the text data from the previous step, and it processes it to select a generative AI model (for example, a GPT-3 model for weather information). The output data is the selected generative AI model.

[0112] Step 6:

[0113] The server generates response data using the selected generative AI model. Specifically, it uses a generative AI such as the GPT-3 model to generate an optimal response to the text data. The input data is the selected generative AI model and the text data, and the output data is the generated response data. For example, in response to the question "What's the weather like today?", the generated response is "It's sunny today."

[0114] Step 7:

[0115] The server sends the generated response data to the user terminal. A communication protocol such as HTTP response or WebSocket is used for transmission. The input data is the generated response data, and the output data is the response data sent to the user terminal.

[0116] Step 8:

[0117] The terminal decodes the response data received from the server. The input data is the response data sent from the server, and the output data is the decoded text data.

[0118] Step 9:

[0119] The device converts the decoded text data into voice data using a speech synthesis engine. Specifically, a speech synthesis engine such as Amazon Polly is used. The input data is the decoded text data, and the output data is voice data. For example, the text "The weather is sunny today" is converted into voice data.

[0120] Step 10:

[0121] The device plays the converted audio data. Specifically, the audio is played back to the user through the smartphone's speaker or earphones. The input data is audio data, and the final output is the audio that the user hears.

[0122] Step 11:

[0123] The user receives the voice response played back from the terminal, and can hear a voice response such as "The weather is sunny today."

[0124] (Application example 1)

[0125] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0126] In factory operations, workers had to use conventional computers and touch panels to operate robots. This meant that operation took time at the workplace, resulting in reduced work efficiency. Furthermore, manual operation could lead to human error, which also led to reduced productivity and quality control issues. A system was needed to solve these issues and enable workers to operate robots more efficiently and reliably.

[0127] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0128] In this invention, the server includes means for capturing a user's voice input, means for analyzing the captured voice input and converting it into text data, means for selecting a generative AI model based on the text data, means for generating response data using the generative AI model, means for sending the generated response data to a user, means for causing a machine to execute the generated response data as an instruction, and means for a factory worker to give voice instructions and for the robot to execute the instructions. This allows the worker to instantly operate the robot using voice instructions, improving work efficiency and reducing operation errors.

[0129] "User voice input" refers to collecting the voice uttered by the user as data.

[0130] "Capture" is a method of obtaining audio as digital data.

[0131] "Text data" is information that is generated by analyzing voice input and expressed as a string of characters.

[0132] A "generative artificial intelligence model" is a machine learning model that generates appropriate responses to specific data or questions.

[0133] "Response data" refers to data corresponding to answers or instructions generated by a generative artificial intelligence model.

[0134] "Mobile terminal" refers to mobile devices in general, including smartphones, tablets, etc.

[0135] A "microphone" is a device that converts sound into an electrical signal.

[0136] A "headset" is a device that allows for the capture and playback of sound.

[0137] A "speech synthesis engine" is a technology that converts text data into voice data.

[0138] "Audio data" is information obtained by digitizing an audio signal.

[0139] A "machine" is a device designed to perform a specified task.

[0140] An "instruction" is a command to perform a particular action or operation.

[0141] A "factory worker" is a human worker who performs work in a factory.

[0142] A "robot" is an automated device for carrying out programmed instructions.

[0143] This invention is a system in which a factory worker gives voice instructions and a robot performs actions based on those instructions. The system includes the following components:

[0144] server

[0145] The server receives a voice data request sent from the user's device. First, the server analyzes this voice data and converts it into text data using a speech recognition engine (e.g., Google Speech-to-Text API). Next, based on the converted text data, the server selects the required generative AI model (e.g., OpenAI's GPT-4 (registered trademark)). For example, if a worker verbally instructs, "Tell me the next work step," the server selects an appropriate AI model and generates response data for the next step. This response data is provided in text format and is used as a command for the robot to execute. Finally, the server sends this response data to the user's device and the robot.

[0146] Terminal

[0147] The terminal is responsible for capturing the user's voice input. It captures the voice of the worker speaking into the microphone of a headset or mobile device and sends this captured voice data to the server. A communication protocol such as HTTP request or WebSocket is used for transmission. When the terminal receives response data from the server, it decodes the received data and obtains the response in text format. It then uses a speech synthesis engine (e.g., Google Cloud Text-to-Speech) to convert this text data into speech and play it back to the worker. For example, if a worker issues an instruction such as "Pick up a part from the shelf," the terminal converts the server's response, "Perform the next step," into speech and plays it back to the worker.

[0148] User

[0149] The user is a factory worker who issues a voice command. For example, the worker asks, "What is the next work process?" This voice command is captured by the terminal and sent to the server. As a result of the server's processing, the user receives a voice response such as, "The next work process is assembly." This simplifies operations for the user through voice instructions, reduces manual operation errors, and improves work efficiency.

[0150] Specific examples

[0151] When a worker asks "What's the next step?", the following prompt is input to the generative AI model:

[0152] The worker says, "Tell me the next step." Please explain the following steps.

[0153] As a result, the generative AI model generates a response saying, "The next work process is assembly. The robot will move the parts to the assembly station." The generated response data is converted into voice data by a speech synthesis engine and played back to the worker. At the same time, an instruction is given to the robot to move the parts to the assembly station as the next process.

[0154] This system allows workers in the factory to operate robots using voice commands, improving work efficiency and reducing operating errors.

[0155] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0156] Step 1:

[0157] The user issues instructions to the worker in the form of voice, speaking into a headset or a microphone on a mobile device. This voice input is captured by the device and obtained as digital data. As input, there is voice data. As output, this voice data is captured and ready to be sent to a server.

[0158] Step 2:

[0159] The device sends the captured audio data to the server. The communication protocol is HTTP request or WebSocket. The input is the captured audio data. The output is the server receiving the request for audio data.

[0160] Step 3:

[0161] The server analyzes the received voice data and converts it into text data using a speech recognition engine (Google Speech-to-Text API). The input is voice data, and the output is the voice data converted into text data.

[0162] Step 4:

[0163] The server selects an appropriate generative AI model (OpenAI's GPT-4) based on the converted text data. The input is the text data. The output is the selected generative AI model.

[0164] Step 5:

[0165] The server inputs a prompt statement to the selected generative AI model and generates response data. The inputs include the selected generative AI model and the prompt statement. Examples of prompt statements are as follows:

[0166] The worker says, "Tell me the next step." Please explain the following steps.

[0167] The output is the generated response data.

[0168] Step 6:

[0169] The server transmits the generated response data to the user terminal and the robot. The generated response data is the input. The user terminal and the robot receive the response data as the output.

[0170] Step 7:

[0171] The device decodes the received response data and converts it into voice data using a speech synthesis engine (Google Cloud Text-to-Speech). The input is the response data (in text format). The output is voice data.

[0172] Step 8:

[0173] The terminal plays the voice data and notifies the worker of the response. At the same time, instructions based on the response data are transmitted to the robot, which then carries out those instructions. The inputs are the voice data and instruction data for the robot. The outputs are a voice notification to the worker and the robot's execution of an action.

[0174] In this way, a system for operating robots in a factory using voice commands is realized, allowing workers to operate robots easily and reliably, improving work efficiency.

[0175] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0176] The present invention is a system that not only captures a user's voice input, converts it into text data, and selects an appropriate generative AI model to generate response data, but also combines it with an emotion engine that recognizes the user's emotions to provide more personalized responses. Below, the program processing of this system is explained in natural language.

[0177] server

[0178] The server receives a voice data request from the user terminal. First, the server analyzes the voice data and converts it into text data using a speech recognition engine. At the same time, it uses an emotion engine to recognize the user's emotional state from the voice data. For example, it determines whether the user is happy, angry, or sad based on characteristics such as the tone, speed, and accent of the voice.

[0179] Based on the text data and the recognized emotional state, the server selects an appropriate generative AI model. Depending on the user's request and emotional state, the generative AI model that generates the optimal response is identified and selected. For example, in response to the request "What's the weather today?", if the user is calm, a normal response is provided, but if the user is angry, a more considerate response is generated.

[0180] The generated response data is generated in the form of, for example, "The weather is sunny today. Have a nice day." The server transmits this response data to the user terminal.

[0181] Terminal

[0182] The device is responsible for capturing the user's voice input: it captures the user's voice as they speak into earphones or a smartphone microphone, and transmits this captured voice data to a server using a communication protocol such as HTTP requests or WebSocket.

[0183] When the device receives the response data from the server, it decodes the received data and obtains data that reflects the text response content and consideration according to the user's emotions. It then uses a speech synthesis engine to convert this text into speech and plays it back to the user. For example, when the user issues a voice command such as "Play the next song," the device receives a response from the server saying, "Playing the next song. Refresh your mood with good music," converts it into speech, and plays it back.

[0184] User

[0185] The user issues a voice command to the device. For example, they put on earphones and ask, "What time is it now?" This voice command is captured by the device and sent to the server. As a result of the server's processing, the user receives a voice response such as, "It is now 3:00 p.m. Have a nice afternoon."

[0186] Additionally, users can customize the device's design using a smartphone app. Generative AI generates new designs based on the user's customization requests and applies them to the device in real time. An emotion engine takes into account the user's current emotions and changes the color and theme of the design, providing a more personalized experience for the user.

[0187] This system allows users to enjoy a consistent experience and smooth operation across multiple devices, while also reducing development costs for device manufacturers by standardizing the process of selecting generative AI models and emotion engines and generating responses.

[0188] The processing flow will be explained below.

[0189] Step 1:

[0190] The device captures the user's voice input: as the user speaks voice commands into earphones or a smartphone microphone, the voice data is recorded in real time and stored in a buffer.

[0191] Step 2:

[0192] The device encodes the captured audio data and sends it to the server using a communication protocol such as an HTTP request or WebSocket.

[0193] Step 3:

[0194] The server receives the voice data sent from the terminal and stores it in the server's internal storage.

[0195] Step 4:

[0196] The server uses a voice recognition engine to convert the received voice data into text data, thereby obtaining the user's voice commands in text format.

[0197] Step 5:

[0198] The server inputs the text data into an emotion engine to recognize the user's emotional state. The emotion engine analyzes the tone, speed, and accent of the voice to determine whether the user is happy, angry, or sad.

[0199] Step 6:

[0200] The server selects an appropriate generative AI model based on the text data and the recognized emotional state. The server analyzes the text and identifies and selects the generative AI model that best suits the user's request and emotional state.

[0201] Step 7:

[0202] The server generates response data using the selected generative AI model. For example, in response to a request like "What's the weather today?", if the user is calm, it will provide a normal response, but if the user is angry, it will generate a more considerate response, such as "The weather is sunny today. Have a nice day."

[0203] Step 8:

[0204] The server transmits the generated response data to the terminal, which is then encoded in an appropriate format and transmitted to the terminal via a communication protocol.

[0205] Step 9:

[0206] The terminal receives the response data sent from the server, decodes the received data, and stores the response content in text format in the terminal.

[0207] Step 10:

[0208] The device converts the received text response into audio data using a speech synthesis engine, which analyzes the text and generates natural-sounding speech.

[0209] Step 11:

[0210] The device plays the generated voice data to the user. The user can hear the voice response through earphones or speakers. For example, a voice such as "It is now 3:00 p.m. Have a nice afternoon" may be played.

[0211] Step 12:

[0212] The user can then act on the generated voice response, for example by issuing the next voice command or customizing the device using generative AI and an emotion engine. This process is repeated, providing the user with a consistent experience.

[0213] Example 2

[0214] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0215] Conventional speech recognition systems simply convert a user's voice input into text and generate responses, but are unable to provide responses that take the user's emotions into account. This can lead to stress and an unpleasant experience when trying to obtain the information they need. Furthermore, because responses cannot be generated based on the user's emotions, responses are not personalized enough, which can result in a decrease in user satisfaction.

[0216] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for capturing a user's voice input, means for analyzing the captured voice input and converting it into text data, means for recognizing the user's emotional state from the voice input, means for selecting a generative AI model based on the text data and the user's emotional state, means for generating response data using the selected generative AI model, and means for transmitting the generated response data to the user. This makes it possible to provide a personalized response that takes the user's emotions into consideration.

[0217] "Means for capturing user voice input" refers to devices and techniques for recording the voices uttered by a user.

[0218] The "means for analyzing and converting into text data" refers to the technology and system for analyzing voice data and converting it into corresponding character string data.

[0219] "Means for recognizing a user's emotional state" refers to techniques and systems that detect and identify a user's emotional state (e.g., joy, anger, sadness, etc.) from voice data.

[0220] "Means for selecting a generative artificial intelligence model" refers to techniques and systems for selecting the most appropriate artificial intelligence model based on the analyzed data.

[0221] "Means for generating response data" refers to techniques and systems that use selected artificial intelligence models to automatically generate appropriate responses to user input.

[0222] "Means for transmitting to the user" refers to communication technology and systems for transmitting the generated response data to the user's terminal.

[0223] A "microphone installed in a mobile terminal" refers to a voice input device built into a portable device such as a mobile phone or smartphone.

[0224] A "speech synthesis engine" refers to technology and systems for converting text data into speech data.

[0225] This invention is a system that captures a user's voice input, converts it into text data, and then selects an appropriate generative AI model to generate response data.By combining this with an emotion engine that recognizes the user's emotions, it provides more personalized responses.

[0226] server

[0227] The server receives a voice data request sent from the user's device. First, the server analyzes this voice data and converts it into text data using a general-purpose voice recognition engine (such as Google's Speech-to-Text API). At the same time, it uses an emotion recognition engine (such as general emotion analysis software) to recognize the user's emotional state from the voice data. It determines the user's emotion based on characteristics such as the tone, speed, and accent of the voice. Based on this information, it selects the optimal generative artificial intelligence model and generates personalized response data using OpenAI's GPT-3 or similar. The generated response data is then sent to the user's device.

[0228] For example, if a user says, "What time is the next meeting?", the server converts the speech to text and then analyzes it with an emotion engine to recognize that the user is impatient. It then selects an appropriate generative AI model and provides a personalized response (e.g., "The next meeting is at 2 p.m. Take your time to prepare and you'll be fine.").

[0229] Example prompt sentence:

[0230] Generate a relaxing response for users who impatiently ask, "What time is the next meeting?"

[0231] Terminal

[0232] The terminal is responsible for capturing the user's voice input. It captures the user's speech into a microphone on a portable device (e.g., a smartphone or mobile phone) and sends this voice data to a server using a communication protocol (e.g., HTTP request or WebSocket). It decodes the received response data and obtains the response content in text format. It then converts this text into speech using a speech synthesis engine (e.g., Amazon's Polly) and plays it back to the user.

[0233] As a specific example, when a user issues a voice command such as "Play the next song," the device sends this voice data to the server, receives a response from the server saying, "Playing the next song. Refresh your mood with good music," converts it into voice, and plays it.

[0234] User

[0235] The user issues a voice command to the device. For example, they put on earphones and ask, "What time is it now?" The voice command is captured by the device and sent to the server. As a result of the server's processing, the user receives a voice response such as, "It is now 3:00 p.m. Have a nice afternoon."

[0236] Additionally, users can customize the device's design using a smartphone app: for example, an emotion engine can recognize when a user is relaxing and change the device's theme to softer colors.

[0237] This system allows users to enjoy a consistent experience and smooth operation across multiple devices, while also reducing development costs for device manufacturers by standardizing the process of selecting generative AI models and emotion engines and generating responses.

[0238] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0239] Step 1: Capture the user's voice input

[0240] Terminal

[0241] The terminal captures in real time the user's voice as they speak into a microphone installed on a portable device (e.g., a smartphone). The input is the user's voice data, and the output is a byte stream that converts this voice data into a digital format. This byte stream is stored in temporary storage.

[0242] Specific behavior:

[0243] A user issues the voice command, "What time is the next meeting?"

[0244] The device captures this voice command and converts it into digital audio data in .wav format.

[0245] Step 2: Send the audio data to the server

[0246] Terminal

[0247] The device sends the captured audio data to the server. The input is digital audio data, and the output is obtained by sending this data to the server using a communication protocol such as HTTP requests or WebSockets.

[0248] Specific behavior:

[0249] The device sends the saved audio data to the server in the body of an HTTP POST request.

[0250] Step 3: Convert the audio data to text

[0251] server

[0252] The server receives the voice data sent from the device and converts it into text data using a voice recognition engine (e.g., a general-purpose voice recognition engine). The input is the voice data sent to the server, and the output is the analyzed text data. This text data is stored in internal memory.

[0253] Specific behavior:

[0254] The server receives the voice data "What time is the next meeting?"

[0255] Call Google's Speech-to-Text API to generate text data such as "What time is the next meeting?"

[0256] Step 4: Recognize your emotional state

[0257] server

[0258] The server passes the received voice data to an emotion recognition engine to recognize the emotional state. The input is the voice data, and the output is the user's emotional state (e.g., happy, anxious).

[0259] Specific behavior:

[0260] The emotion engine analyzes the voice data and recognizes emotional states such as "anxious."

[0261] The recognized emotional state is kept in an internal memory.

[0262] Step 5: Select a generative AI model

[0263] server

[0264] The server generates prompt sentences and selects an appropriate generative AI model based on the text data and the recognized emotional state. The input is the text data and the emotional state, and the output is the selected generative AI model.

[0265] Specific behavior:

[0266] Prompt Sentence: Generate a relaxing response for a user who impatiently asks, "What time is the next meeting?"

[0267] Select the best generative artificial intelligence model, such as OpenAI's GPT-3.

[0268] Step 6: Generate response data

[0269] server

[0270] The server generates response data using the selected generative AI model. The input is the prompt sentence, and the output is the generated response data.

[0271] Specific behavior:

[0272] Enter a prompt into GPT-3 and it will generate a response like, "The next meeting is at 2 p.m. Take your time to prepare and you'll be fine."

[0273] Step 7: Send the response data to the user device

[0274] server

[0275] The server sends the generated response data to the user terminal. The input is the generated response data, and the output is the sent response data.

[0276] Specific behavior:

[0277] The server structures the response data in JSON format and sends it to the terminal as an HTTP response.

[0278] Step 8: Play the response aloud

[0279] Terminal

[0280] The device decodes the response data received from the server and converts it into voice data using a speech synthesis engine (e.g., Amazon's Polly). The input is the text response data received from the server, and the output is voice data. This voice data is then played back to the user through the speaker.

[0281] Specific behavior:

[0282] The device receives response data saying, "The next meeting is at 2:00 p.m. Take your time to prepare."

[0283] The text is converted into speech using a speech synthesis engine and played through speakers or earphones.

[0284] (Application example 2)

[0285] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0286] Conventional speech recognition systems simply convert a user's voice input into text and generate responses based on that text. However, they can only provide a uniform response without considering the user's emotions, making it difficult to provide a personalized experience for the user. Furthermore, especially in autonomous vehicles, there are many situations where responses that correspond to the passenger's emotional state are required. Therefore, there is a need to build a system that can provide more personalized responses based on the user's voice input and their emotional state.

[0287] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0288] In this invention, the server includes means for capturing a user's voice input, means for analyzing the captured voice input and converting it into text data, means for recognizing emotions based on the text data and voice characteristics, means for selecting a generative AI model based on the text data and the recognized emotional state, means for generating personalized response data using the generative AI model, means for transmitting the generated response data to the user, means for converting the generated response data into voice data, and means for playing the voice data to the user, thereby making it possible to provide a personalized experience adapted to the user's emotional state.

[0289] A "means for capturing user voice input" is a device or component for capturing voice uttered by a user and obtaining the voice data.

[0290] "Means for analyzing captured voice input and converting it into text data" refers to devices or software that analyzes captured voice data and converts it into corresponding text data.

[0291] "Means for recognizing emotions based on text data and voice characteristics" refers to devices and software for analyzing and recognizing a user's emotional state from the characteristics of text data and voice data.

[0292] The "means for selecting a generative artificial intelligence model based on text data and a recognized emotional state" refers to a device or software for automatically selecting an optimal artificial intelligence model based on the recognized text data and an emotional state.

[0293] "Means for generating personalized response data using a generative artificial intelligence model" refers to a device or software that uses a selected artificial intelligence model to generate response data in response to a user's request and emotional state.

[0294] The "means for transmitting the generated response data to the user" refers to a communication device or software for transferring the generated response data to the user device.

[0295] The "means for converting the generated response data into voice data" refers to a voice synthesizer or software for converting the generated text response into voice form.

[0296] The "means for reproducing audio data to the user" refers to a reproduction device or component that allows the user to listen to the audio data.

[0297] An "in-vehicle acoustic input device" is a microphone or other acoustic sensor installed inside a motor vehicle to capture sounds made by passengers.

[0298] "Playing back the voice data generated by the voice synthesizer through an audio playback device inside the vehicle" means letting the user hear the voice data generated by the voice synthesizer using a speaker system installed inside the vehicle.

[0299] The present invention is a system installed in an autonomous vehicle that captures a user's voice input, converts it into text data, recognizes emotions based on the text data and voice characteristics, and generates an optimal response. An embodiment of this system is described in detail below.

[0300] server

[0301] The server receives voice data sent from the user terminal. It converts the voice data into text data using a voice recognition engine (e.g., Google Cloud Speech-to-Text). At the same time, it recognizes the user's emotional state from the voice data using an emotion engine (e.g., IBM Watson (registered trademark) Tone Analyzer). Based on the recognized text data and emotional state, it selects a generative AI model (e.g., OpenAI GPT-3) and generates appropriate response data. The generated response data may be in the form of, for example, "Your next destination is Ginza. Enjoy a leisurely drive." The server then transmits this response data to the autonomous vehicle's terminal.

[0302] Terminal

[0303] The terminal captures the user's voice input using an audio input device (microphone) installed in the autonomous vehicle. This captured voice data is sent to the server using a communication protocol such as an HTTP request or WebSocket. When the terminal receives response data from the server, it decodes the received data and obtains the response content in text format. It then converts this text into speech using a speech synthesis engine (e.g., Amazon Polly) and plays it back to the user through the audio playback device (speaker) in the autonomous vehicle. For example, if the user issues a voice command such as "Play the next song," the terminal receives a response from the server saying, "Playing the next song. Relax," converts this into speech, and plays it back.

[0304] User

[0305] The user issues a voice command to the audio input device in the autonomous vehicle. For example, they might ask, "Set my next destination." This voice command is captured by the device and sent to the server. As a result of processing by the server, the user receives a voice response such as, "Your next destination is Ginza. Enjoy a leisurely drive." Additionally, the user can customize the in-vehicle infotainment system using a smartphone app. New settings based on the user's customization requests are generated using generative AI and applied to the vehicle's systems in real time. The emotion engine takes into account the user's current emotions and changes the music and lighting theme to provide a more personalized experience for the user.

[0306] Specific examples

[0307] When a passenger asks "Where's my next destination?", an example of the server's processing is as follows:

[0308] Input prompt: "Where's my next destination?"

[0309] The server converts the speech to text: "Where's my next destination?"

[0310] Sentiment analysis result: "Calm"

[0311] Generative AI model selection and response generation: "Your next destination is Ginza. Enjoy a leisurely drive."

[0312] Speech synthesis and response: "Your next destination is Ginza. Enjoy a leisurely drive." is played over the speaker in the self-driving vehicle.

[0313] This system will enable a more personalized experience inside self-driving vehicles, making travel smoother and more comfortable.

[0314] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0315] Step 1:

[0316] The terminal captures the user's voice input, i.e., the user speaks a voice command into a microphone in the car and acquires the voice data.

[0317] Input: User's voice command

[0318] Output: Captured audio data

[0319] Step 2:

[0320] The device sends the captured audio data to the server using HTTP requests and WebSockets as the communication protocol.

[0321] Input: Captured audio data

[0322] Output: Sending audio data to the server

[0323] Step 3:

[0324] The server converts the received voice data into text data using a speech recognition engine, such as Google Cloud Speech-to-Text.

[0325] Input: Incoming audio data

[0326] Output: Text data

[0327] Step 4:

[0328] The server analyzes the text data and voice characteristics using an emotion engine, such as IBM Watson Tone Analyzer, to recognize the user's emotional state.

[0329] Input: Text data and audio data features

[0330] Output: User's emotional state

[0331] Step 5:

[0332] The server selects the optimal generative AI model based on the recognized text data and emotional state, using OpenAI GPT-3 or other models.

[0333] Input: Text data, user's emotional state

[0334] Output: The selected generative AI model

[0335] Step 6:

[0336] The server generates personalized response data using the selected generative AI model.

[0337] Input: Text data, the user's emotional state, and a generative AI model

[0338] Output: Generated response data

[0339] Step 7:

[0340] The server transmits the generated response data to the terminal.

[0341] Input: Generated response data

[0342] Output: Send response data to the terminal

[0343] Step 8:

[0344] The device uses a speech synthesis engine, such as Amazon Polly, to convert the received response data from text to voice data.

[0345] Input: Received response data (text format)

[0346] Output: Audio data

[0347] Step 9:

[0348] The terminal reproduces the generated voice data through the speaker in the vehicle and provides it to the user.

[0349] Input: Generated audio data

[0350] Output: A voice response to the user

[0351] In this way, the system personalizes the user experience in an autonomous vehicle, ensuring a smooth and comfortable journey.

[0352] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0353] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0354] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0355] [Second embodiment]

[0356] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0357] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0358] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0359] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0360] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0361] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0362] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0363] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0364] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0365] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0366] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0367] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0368] The present invention relates to a system that captures a user's voice input, converts it into text data, selects an appropriate generative AI model to generate response data, and provides it to the user. Below, the program processing of this system is explained in natural language.

[0369] server

[0370] The server receives a voice data request sent from the user's device. First, the server analyzes this voice data and converts it into text data using a speech recognition engine. Then, based on the converted text data, the server selects the necessary generative AI model. For example, if a user asks, "What's the weather like today?", the server selects the generative AI model that corresponds to the weather information.

[0371] Next, the server generates response data using the selected generative AI model. The generated response data is provided in text format, such as "Today's weather is sunny." Finally, the server sends this response data to the user's device.

[0372] Terminal

[0373] The device is responsible for capturing the user's voice input: it captures the user's voice as they speak into earphones or a smartphone microphone, and transmits this captured voice data to a server using a communication protocol such as HTTP requests or WebSocket.

[0374] When the device receives the response data from the server, it decodes the received data to obtain the response content in text format. It then uses a speech synthesis engine to convert this text into speech and play it back to the user. For example, when the user issues a voice command such as "Play the next song," the device receives a response from the server saying "Playing the next song," converts it into speech, and plays it back.

[0375] User

[0376] The user issues a voice command to the device. For example, they put on earphones and ask, "What time is it now?" This voice command is captured by the device and sent to the server. As a result of the server's processing, the user receives a voice response such as, "The current time is 3:00 PM."

[0377] Additionally, users can customize the design of their device using a smartphone app, which uses generative AI to generate new designs based on the user's customization requests and apply them to the device in real time.

[0378] This system allows users to enjoy a consistent experience and smooth operation across multiple devices, while also reducing development costs for device manufacturers by standardizing the process of selecting generative AI models and generating responses.

[0379] The processing flow will be explained below.

[0380] Step 1:

[0381] The device captures the user's voice input: as the user speaks voice commands into earphones or a smartphone microphone, the voice data is recorded in real time and stored in a buffer.

[0382] Step 2:

[0383] The device encodes the captured audio data and sends it to the server using a communication protocol such as an HTTP request or WebSocket.

[0384] Step 3:

[0385] The server receives the voice data sent from the terminal and stores it in the server's internal storage.

[0386] Step 4:

[0387] The server uses a voice recognition engine to convert the received voice data into text data, thereby obtaining the user's voice commands in text format.

[0388] Step 5:

[0389] The server selects an appropriate generative AI model based on the text data. It analyzes the text and identifies and selects the generative AI model that best suits the user's request.

[0390] Step 6:

[0391] The server generates response data using the selected generative AI model. For example, in response to the request "What's the weather like today?", it generates the response text "Today's weather is sunny."

[0392] Step 7:

[0393] The server transmits the generated response data to the terminal, which is then encoded in an appropriate format and transmitted to the terminal via a communication protocol.

[0394] Step 8:

[0395] The terminal receives the response data sent from the server, decodes the received data, and stores the response content in text format in the terminal.

[0396] Step 9:

[0397] The device converts the received text response into audio data using a speech synthesis engine, which analyzes the text and generates natural-sounding speech.

[0398] Step 10:

[0399] The device plays the generated voice data to the user. The user can hear the voice response through earphones or a speaker. For example, a voice such as "The current time is 3:00 PM" is played.

[0400] Step 11:

[0401] The user can then act on the generated voice response, for example by issuing the next voice command or customizing the device using the generative AI. This process is repeated, allowing the user to enjoy a consistent experience.

[0402] Example 1

[0403] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0404] Conventional voice recognition systems require a wide variety of technologies to properly convert voice input into text data and generate appropriate responses. This makes it difficult to provide accurate responses in real time to user voice commands. It also makes it difficult to provide a consistent user experience across multiple devices. Furthermore, it is not possible to customize the generated design or responses, resulting in a loss of user convenience.

[0405] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0406] In this invention, the server includes means for capturing a user's voice input, means for analyzing the captured voice input and converting it into text data, means for selecting an AI model generated based on the text data, means for generating response data using the generated AI model, means for transmitting the generated response data to the user, means for decoding the response data as text data, means for using a voice synthesis engine to convert the decoded text data into voice data, and means for playing back the converted voice data, thereby enabling accurate responses to the user's voice commands in real time and providing a unified user experience.

[0407] The "means for capturing user voice input" refers to a device or method for capturing voice uttered by a user, such as a microphone provided on a mobile terminal.

[0408] The "means for analyzing the captured voice input and converting it into text data" refers to a device or method for analyzing the captured voice using voice recognition technology and converting it into data in the form of a string of characters. Specifically, it includes a voice recognition engine.

[0409] The "means for selecting an AI model generated based on the text data" refers to a device or method for identifying and selecting an appropriate AI model based on the analyzed text data. Specifically, it includes an algorithm for selecting a generative AI model.

[0410] "Means for generating response data using the generated artificial intelligence model" refers to a device or method for generating a response to a user's query using a selected generative AI model.

[0411] The "means for transmitting the generated response data to the user" refers to a device or method for transferring the generated response data to the user's device. Specifically, it includes a data transmission technique using a communication protocol.

[0412] The "means for decoding the response data as text data" refers to a device or method for converting received response data into a character string format and analyzing it.

[0413] The "means for using a speech synthesis engine to convert the decoded text data into speech data" refers to a device or method for converting the decoded text data into speech format. Specifically, it includes a speech synthesis engine.

[0414] The "means for reproducing the converted audio data" refers to a device or method for allowing a user to listen to the converted audio data. Specifically, this applies to a speaker, earphones, etc.

[0415] The present invention relates to a system that captures a user's voice input, converts it into text data, selects an appropriate generative AI model, generates response data, and provides it to the user.

[0416] Server Roles

[0417] The server receives a voice data request sent from the user device. After receiving the voice data, the server uses a speech recognition engine (specifically, Google Cloud Speech-to-Text API, etc.) to convert the voice data into text data. For example, if a user asks, "What's the weather like today?", the voice data is converted into text data such as, "What's the weather like today?"

[0418] Based on the text data, the server selects an appropriate generative AI model by analyzing keywords and intent within the text data. For example, it selects a generative AI model (e.g., GPT-3) that corresponds to weather information.

[0419] Next, the server generates response data using the selected generative AI model. This generation process generates text data such as "Today's weather is sunny." Finally, the server sends the generated response data to the user's device. The communication protocol used is an HTTP POST request or WebSocket.

[0420] Device Role

[0421] The device has the function to capture the user's voice. Specifically, it records the user's voice using the smartphone microphone or earphone microphone, and sends the captured voice data to the server. The transmission method is an HTTP POST request or WebSocket.

[0422] When the device receives response data from the server, it decodes the data and obtains the response in text format. The obtained text data is converted into voice data using a speech synthesis engine (specifically, Amazon Polly, etc.). The response is provided to the user by playing this voice data through speakers or earphones.

[0423] User Roles

[0424] A user can issue a voice command to the device. For example, they can use earphones to ask, "What time is it now?" This voice command is captured by the device and sent to the server. As a result of the server's processing, the user can receive a voice response such as, "The current time is 3:00 PM."

[0425] Users can also customize the design of their device using a smartphone app, which uses generative AI to generate new designs based on the user's customization requests and apply them to the device in real time.

[0426] Specific operation example

[0427] Weather information inquiry

[0428] The user asks the device, "What's the weather like today?" The device captures this audio and sends the audio data to the server. The server uses the Google Cloud Speech-to-Text API to convert the audio into text, generating the text "What's the weather like today?" The server then selects a generative AI model (e.g., GPT-3) that corresponds to the weather information and generates response data saying, "The weather is sunny today." This response data is sent to the device, and when the device receives the text data, it uses a speech synthesis engine to convert it into audio data and play it back to the user.

[0429] Prompt Sentence Examples

[0430] The system prompts users with real-time responses based on their requests:

[0431] Received the user's voice saying "Play next song." Generate a text response saying "Play next song."

[0432] As described above, the system of the invention can efficiently process user voice input and provide appropriate responses. This system allows users to enjoy a consistent user experience and allows smooth operation across multiple devices. It also reduces development costs for device manufacturers by standardizing the process of selecting generative AI models and generating responses.

[0433] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0434] Step 1:

[0435] The user inputs a voice command into the device. For example, the user puts on earphones and says, "What's the weather like today?" The input data is voice data, and this is the starting point for the system's processing.

[0436] Step 2:

[0437] The device captures the user's voice. Specifically, it records the voice using the smartphone microphone or the earphone microphone. The input data is the voice data from the previous step, and the output data is the captured voice data.

[0438] Step 3:

[0439] The device sends the captured audio data to the server using a communication protocol such as an HTTP POST request or WebSocket. The input data is the captured audio data, and the output data is the audio data sent to the server.

[0440] Step 4:

[0441] The server analyzes the received voice data. Specifically, it converts the voice data into text data using a voice recognition engine (such as Google Cloud Speech-to-Text API). The input data is the voice data sent to the server, and the output data is text data. For example, the voice data "What's the weather like today?" is converted into the text data "What's the weather like today?"

[0442] Step 5:

[0443] The server selects an appropriate generative AI model based on the converted text data. The input data is the text data from the previous step, and it processes it to select a generative AI model (for example, a GPT-3 model for weather information). The output data is the selected generative AI model.

[0444] Step 6:

[0445] The server generates response data using the selected generative AI model. Specifically, it uses a generative AI such as the GPT-3 model to generate an optimal response to the text data. The input data is the selected generative AI model and the text data, and the output data is the generated response data. For example, in response to the question "What's the weather like today?", the generated response is "It's sunny today."

[0446] Step 7:

[0447] The server sends the generated response data to the user terminal. A communication protocol such as HTTP response or WebSocket is used for transmission. The input data is the generated response data, and the output data is the response data sent to the user terminal.

[0448] Step 8:

[0449] The terminal decodes the response data received from the server. The input data is the response data sent from the server, and the output data is the decoded text data.

[0450] Step 9:

[0451] The device converts the decoded text data into voice data using a speech synthesis engine. Specifically, a speech synthesis engine such as Amazon Polly is used. The input data is the decoded text data, and the output data is voice data. For example, the text "The weather is sunny today" is converted into voice data.

[0452] Step 10:

[0453] The device plays the converted audio data. Specifically, the audio is played back to the user through the smartphone's speaker or earphones. The input data is audio data, and the final output is the audio that the user hears.

[0454] Step 11:

[0455] The user receives the voice response played back from the terminal, and can hear a voice response such as "The weather is sunny today."

[0456] (Application example 1)

[0457] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0458] In factory operations, workers had to use conventional computers and touch panels to operate robots. This meant that operation took time at the workplace, resulting in reduced work efficiency. Furthermore, manual operation could lead to human error, which also led to reduced productivity and quality control issues. A system was needed to solve these issues and enable workers to operate robots more efficiently and reliably.

[0459] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0460] In this invention, the server includes means for capturing a user's voice input, means for analyzing the captured voice input and converting it into text data, means for selecting a generative AI model based on the text data, means for generating response data using the generative AI model, means for sending the generated response data to a user, means for causing a machine to execute the generated response data as an instruction, and means for a factory worker to give voice instructions and for the robot to execute the instructions. This allows the worker to instantly operate the robot using voice instructions, improving work efficiency and reducing operation errors.

[0461] "User voice input" refers to collecting the voice uttered by the user as data.

[0462] "Capture" is a method of obtaining audio as digital data.

[0463] "Text data" is information that is generated by analyzing voice input and expressed as a string of characters.

[0464] A "generative artificial intelligence model" is a machine learning model that generates appropriate responses to specific data or questions.

[0465] "Response data" refers to data corresponding to answers or instructions generated by a generative artificial intelligence model.

[0466] "Mobile terminal" refers to mobile devices in general, including smartphones, tablets, etc.

[0467] A "microphone" is a device that converts sound into an electrical signal.

[0468] A "headset" is a device that allows for the capture and playback of sound.

[0469] A "speech synthesis engine" is a technology that converts text data into voice data.

[0470] "Audio data" is information obtained by digitizing an audio signal.

[0471] A "machine" is a device designed to perform a specified task.

[0472] An "instruction" is a command to perform a particular action or operation.

[0473] A "factory worker" is a human worker who performs work in a factory.

[0474] A "robot" is an automated device for carrying out programmed instructions.

[0475] This invention is a system in which a factory worker gives voice instructions and a robot performs actions based on those instructions. The system includes the following components:

[0476] server

[0477] The server receives a voice data request sent from the user's device. First, the server analyzes this voice data and converts it into text data using a speech recognition engine (e.g., Google Speech-to-Text API). Next, based on the converted text data, it selects the required generative AI model (e.g., OpenAI's GPT-4). For example, if a worker verbally instructs, "Tell me the next work step," the server selects an appropriate AI model and generates response data for the next step. This response data is provided in text format and is used as a command for the robot to execute. Finally, the server sends this response data to the user's device and the robot.

[0478] Terminal

[0479] The terminal is responsible for capturing the user's voice input. It captures the voice of the worker speaking into the microphone of a headset or mobile device and sends this captured voice data to the server. A communication protocol such as HTTP request or WebSocket is used for transmission. When the terminal receives response data from the server, it decodes the received data and obtains the response in text format. It then uses a speech synthesis engine (e.g., Google Cloud Text-to-Speech) to convert this text data into speech and play it back to the worker. For example, if a worker issues an instruction such as "Pick up a part from the shelf," the terminal converts the server's response, "Perform the next step," into speech and plays it back to the worker.

[0480] User

[0481] The user is a factory worker who issues a voice command. For example, the worker asks, "What is the next work process?" This voice command is captured by the terminal and sent to the server. As a result of the server's processing, the user receives a voice response such as, "The next work process is assembly." This simplifies operations for the user through voice instructions, reduces manual operation errors, and improves work efficiency.

[0482] Specific examples

[0483] When a worker asks "What's the next step?", the following prompt is input to the generative AI model:

[0484] The worker says, "Tell me the next step." Please explain the following steps.

[0485] As a result, the generative AI model generates a response saying, "The next work process is assembly. The robot will move the parts to the assembly station." The generated response data is converted into voice data by a speech synthesis engine and played back to the worker. At the same time, an instruction is given to the robot to move the parts to the assembly station as the next process.

[0486] This system allows workers in the factory to operate robots using voice commands, improving work efficiency and reducing operating errors.

[0487] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0488] Step 1:

[0489] The user issues instructions to the worker in the form of voice, speaking into a headset or a microphone on a mobile device. This voice input is captured by the device and obtained as digital data. As input, there is voice data. As output, this voice data is captured and ready to be sent to a server.

[0490] Step 2:

[0491] The device sends the captured audio data to the server. The communication protocol is HTTP request or WebSocket. The input is the captured audio data. The output is the server receiving the request for audio data.

[0492] Step 3:

[0493] The server analyzes the received voice data and converts it into text data using a speech recognition engine (Google Speech-to-Text API). The input is voice data, and the output is the voice data converted into text data.

[0494] Step 4:

[0495] The server selects an appropriate generative AI model (OpenAI's GPT-4) based on the converted text data. The input is the text data. The output is the selected generative AI model.

[0496] Step 5:

[0497] The server inputs a prompt statement to the selected generative AI model and generates response data. The inputs include the selected generative AI model and the prompt statement. Examples of prompt statements are as follows:

[0498] The worker says, "Tell me the next step." Please explain the following steps.

[0499] The output is the generated response data.

[0500] Step 6:

[0501] The server transmits the generated response data to the user terminal and the robot. The generated response data is the input. The user terminal and the robot receive the response data as the output.

[0502] Step 7:

[0503] The device decodes the received response data and converts it into voice data using a speech synthesis engine (Google Cloud Text-to-Speech). The input is the response data (in text format). The output is voice data.

[0504] Step 8:

[0505] The terminal plays the voice data and notifies the worker of the response. At the same time, instructions based on the response data are transmitted to the robot, which then carries out those instructions. The inputs are the voice data and instruction data for the robot. The outputs are a voice notification to the worker and the robot's execution of an action.

[0506] In this way, a system for operating robots in a factory using voice commands is realized, allowing workers to operate robots easily and reliably, improving work efficiency.

[0507] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0508] The present invention is a system that not only captures a user's voice input, converts it into text data, and selects an appropriate generative AI model to generate response data, but also combines it with an emotion engine that recognizes the user's emotions to provide more personalized responses. Below, the program processing of this system is explained in natural language.

[0509] server

[0510] The server receives a voice data request from the user terminal. First, the server analyzes the voice data and converts it into text data using a speech recognition engine. At the same time, it uses an emotion engine to recognize the user's emotional state from the voice data. For example, it determines whether the user is happy, angry, or sad based on characteristics such as the tone, speed, and accent of the voice.

[0511] Based on the text data and the recognized emotional state, the server selects an appropriate generative AI model. Depending on the user's request and emotional state, the generative AI model that generates the optimal response is identified and selected. For example, in response to the request "What's the weather today?", if the user is calm, a normal response is provided, but if the user is angry, a more considerate response is generated.

[0512] The generated response data is generated in the form of, for example, "The weather is sunny today. Have a nice day." The server transmits this response data to the user terminal.

[0513] Terminal

[0514] The device is responsible for capturing the user's voice input: it captures the user's voice as they speak into earphones or a smartphone microphone, and transmits this captured voice data to a server using a communication protocol such as HTTP requests or WebSocket.

[0515] When the device receives the response data from the server, it decodes the received data and obtains data that reflects the text response content and consideration according to the user's emotions. It then uses a speech synthesis engine to convert this text into speech and plays it back to the user. For example, when the user issues a voice command such as "Play the next song," the device receives a response from the server saying, "Playing the next song. Refresh your mood with good music," converts it into speech, and plays it back.

[0516] User

[0517] The user issues a voice command to the device. For example, they put on earphones and ask, "What time is it now?" This voice command is captured by the device and sent to the server. As a result of the server's processing, the user receives a voice response such as, "It is now 3:00 p.m. Have a nice afternoon."

[0518] Additionally, users can customize the device's design using a smartphone app. Generative AI generates new designs based on the user's customization requests and applies them to the device in real time. An emotion engine takes into account the user's current emotions and changes the color and theme of the design, providing a more personalized experience for the user.

[0519] This system allows users to enjoy a consistent experience and smooth operation across multiple devices, while also reducing development costs for device manufacturers by standardizing the process of selecting generative AI models and emotion engines and generating responses.

[0520] The processing flow will be explained below.

[0521] Step 1:

[0522] The device captures the user's voice input: as the user speaks voice commands into earphones or a smartphone microphone, the voice data is recorded in real time and stored in a buffer.

[0523] Step 2:

[0524] The device encodes the captured audio data and sends it to the server using a communication protocol such as an HTTP request or WebSocket.

[0525] Step 3:

[0526] The server receives the voice data sent from the terminal and stores it in the server's internal storage.

[0527] Step 4:

[0528] The server uses a voice recognition engine to convert the received voice data into text data, thereby obtaining the user's voice commands in text format.

[0529] Step 5:

[0530] The server inputs the text data into an emotion engine to recognize the user's emotional state. The emotion engine analyzes the tone, speed, and accent of the voice to determine whether the user is happy, angry, or sad.

[0531] Step 6:

[0532] The server selects an appropriate generative AI model based on the text data and the recognized emotional state. The server analyzes the text and identifies and selects the generative AI model that best suits the user's request and emotional state.

[0533] Step 7:

[0534] The server generates response data using the selected generative AI model. For example, in response to a request like "What's the weather today?", if the user is calm, it will provide a normal response, but if the user is angry, it will generate a more considerate response, such as "The weather is sunny today. Have a nice day."

[0535] Step 8:

[0536] The server transmits the generated response data to the terminal, which is then encoded in an appropriate format and transmitted to the terminal via a communication protocol.

[0537] Step 9:

[0538] The terminal receives the response data sent from the server, decodes the received data, and stores the response content in text format in the terminal.

[0539] Step 10:

[0540] The device converts the received text response into audio data using a speech synthesis engine, which analyzes the text and generates natural-sounding speech.

[0541] Step 11:

[0542] The device plays the generated voice data to the user. The user can hear the voice response through earphones or speakers. For example, a voice such as "It is now 3:00 p.m. Have a nice afternoon" may be played.

[0543] Step 12:

[0544] The user can then act on the generated voice response, for example by issuing the next voice command or customizing the device using generative AI and an emotion engine. This process is repeated, providing the user with a consistent experience.

[0545] Example 2

[0546] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0547] Conventional speech recognition systems simply convert a user's voice input into text and generate responses, but are unable to provide responses that take the user's emotions into account. This can lead to stress and an unpleasant experience when trying to obtain the information they need. Furthermore, because responses cannot be generated based on the user's emotions, responses are not personalized enough, which can result in a decrease in user satisfaction.

[0548] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for capturing a user's voice input, means for analyzing the captured voice input and converting it into text data, means for recognizing the user's emotional state from the voice input, means for selecting a generative AI model based on the text data and the user's emotional state, means for generating response data using the selected generative AI model, and means for transmitting the generated response data to the user. This makes it possible to provide a personalized response that takes the user's emotions into consideration.

[0549] "Means for capturing user voice input" refers to devices and techniques for recording the voices uttered by a user.

[0550] The "means for analyzing and converting into text data" refers to the technology and system for analyzing voice data and converting it into corresponding character string data.

[0551] "Means for recognizing a user's emotional state" refers to techniques and systems that detect and identify a user's emotional state (e.g., joy, anger, sadness, etc.) from voice data.

[0552] "Means for selecting a generative artificial intelligence model" refers to techniques and systems for selecting the most appropriate artificial intelligence model based on the analyzed data.

[0553] "Means for generating response data" refers to techniques and systems that use selected artificial intelligence models to automatically generate appropriate responses to user input.

[0554] "Means for transmitting to the user" refers to communication technology and systems for transmitting the generated response data to the user's terminal.

[0555] A "microphone installed in a mobile terminal" refers to a voice input device built into a portable device such as a mobile phone or smartphone.

[0556] A "speech synthesis engine" refers to technology and systems for converting text data into speech data.

[0557] This invention is a system that captures a user's voice input, converts it into text data, and then selects an appropriate generative AI model to generate response data.By combining this with an emotion engine that recognizes the user's emotions, it provides more personalized responses.

[0558] server

[0559] The server receives a voice data request sent from the user's device. First, the server analyzes this voice data and converts it into text data using a general-purpose voice recognition engine (such as Google's Speech-to-Text API). At the same time, it uses an emotion recognition engine (such as general emotion analysis software) to recognize the user's emotional state from the voice data. It determines the user's emotion based on characteristics such as the tone, speed, and accent of the voice. Based on this information, it selects the optimal generative artificial intelligence model and generates personalized response data using OpenAI's GPT-3 or similar. The generated response data is then sent to the user's device.

[0560] For example, if a user says, "What time is the next meeting?", the server converts the speech to text and then analyzes it with an emotion engine to recognize that the user is impatient. It then selects an appropriate generative AI model and provides a personalized response (e.g., "The next meeting is at 2 p.m. Take your time to prepare and you'll be fine.").

[0561] Example prompt sentence:

[0562] Generate a relaxing response for users who impatiently ask, "What time is the next meeting?"

[0563] Terminal

[0564] The terminal is responsible for capturing the user's voice input. It captures the user's speech into a microphone on a portable device (e.g., a smartphone or mobile phone) and sends this voice data to a server using a communication protocol (e.g., HTTP request or WebSocket). It decodes the received response data and obtains the response content in text format. It then converts this text into speech using a speech synthesis engine (e.g., Amazon's Polly) and plays it back to the user.

[0565] As a specific example, when a user issues a voice command such as "Play the next song," the device sends this voice data to the server, receives a response from the server saying, "Playing the next song. Refresh your mood with good music," converts it into voice, and plays it.

[0566] User

[0567] The user issues a voice command to the device. For example, they put on earphones and ask, "What time is it now?" The voice command is captured by the device and sent to the server. As a result of the server's processing, the user receives a voice response such as, "It is now 3:00 p.m. Have a nice afternoon."

[0568] Additionally, users can customize the device's design using a smartphone app: for example, an emotion engine can recognize when a user is relaxing and change the device's theme to softer colors.

[0569] This system allows users to enjoy a consistent experience and smooth operation across multiple devices, while also reducing development costs for device manufacturers by standardizing the process of selecting generative AI models and emotion engines and generating responses.

[0570] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0571] Step 1: Capture the user's voice input

[0572] Terminal

[0573] The terminal captures in real time the user's voice as they speak into a microphone installed on a portable device (e.g., a smartphone). The input is the user's voice data, and the output is a byte stream that converts this voice data into a digital format. This byte stream is stored in temporary storage.

[0574] Specific behavior:

[0575] A user issues the voice command, "What time is the next meeting?"

[0576] The device captures this voice command and converts it into digital audio data in .wav format.

[0577] Step 2: Send the audio data to the server

[0578] Terminal

[0579] The device sends the captured audio data to the server. The input is digital audio data, and the output is obtained by sending this data to the server using a communication protocol such as HTTP requests or WebSockets.

[0580] Specific behavior:

[0581] The device sends the saved audio data to the server in the body of an HTTP POST request.

[0582] Step 3: Convert the audio data to text

[0583] server

[0584] The server receives the voice data sent from the device and converts it into text data using a voice recognition engine (e.g., a general-purpose voice recognition engine). The input is the voice data sent to the server, and the output is the analyzed text data. This text data is stored in internal memory.

[0585] Specific behavior:

[0586] The server receives the voice data "What time is the next meeting?"

[0587] Call Google's Speech-to-Text API to generate text data such as "What time is the next meeting?"

[0588] Step 4: Recognize your emotional state

[0589] server

[0590] The server passes the received voice data to an emotion recognition engine to recognize the emotional state. The input is the voice data, and the output is the user's emotional state (e.g., happy, anxious).

[0591] Specific behavior:

[0592] The emotion engine analyzes the voice data and recognizes emotional states such as "anxious."

[0593] The recognized emotional state is kept in an internal memory.

[0594] Step 5: Select a generative AI model

[0595] server

[0596] The server generates prompt sentences and selects an appropriate generative AI model based on the text data and the recognized emotional state. The input is the text data and the emotional state, and the output is the selected generative AI model.

[0597] Specific behavior:

[0598] Prompt Sentence: Generate a relaxing response for a user who impatiently asks, "What time is the next meeting?"

[0599] Select the best generative artificial intelligence model, such as OpenAI's GPT-3.

[0600] Step 6: Generate response data

[0601] server

[0602] The server generates response data using the selected generative AI model. The input is the prompt sentence, and the output is the generated response data.

[0603] Specific behavior:

[0604] Enter a prompt into GPT-3 and it will generate a response like, "The next meeting is at 2 p.m. Take your time to prepare and you'll be fine."

[0605] Step 7: Send the response data to the user device

[0606] server

[0607] The server sends the generated response data to the user terminal. The input is the generated response data, and the output is the sent response data.

[0608] Specific behavior:

[0609] The server structures the response data in JSON format and sends it to the terminal as an HTTP response.

[0610] Step 8: Play the response aloud

[0611] Terminal

[0612] The device decodes the response data received from the server and converts it into voice data using a speech synthesis engine (e.g., Amazon's Polly). The input is the text response data received from the server, and the output is voice data. This voice data is then played back to the user through the speaker.

[0613] Specific behavior:

[0614] The device receives response data saying, "The next meeting is at 2:00 p.m. Take your time to prepare."

[0615] The text is converted into speech using a speech synthesis engine and played through speakers or earphones.

[0616] (Application example 2)

[0617] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0618] Conventional speech recognition systems simply convert a user's voice input into text and generate responses based on that text. However, they can only provide a uniform response without considering the user's emotions, making it difficult to provide a personalized experience for the user. Furthermore, especially in autonomous vehicles, there are many situations where responses that correspond to the passenger's emotional state are required. Therefore, there is a need to build a system that can provide more personalized responses based on the user's voice input and their emotional state.

[0619] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0620] In this invention, the server includes means for capturing a user's voice input, means for analyzing the captured voice input and converting it into text data, means for recognizing emotions based on the text data and voice characteristics, means for selecting a generative AI model based on the text data and the recognized emotional state, means for generating personalized response data using the generative AI model, means for transmitting the generated response data to the user, means for converting the generated response data into voice data, and means for playing the voice data to the user, thereby making it possible to provide a personalized experience adapted to the user's emotional state.

[0621] A "means for capturing user voice input" is a device or component for capturing voice uttered by a user and obtaining the voice data.

[0622] "Means for analyzing captured voice input and converting it into text data" refers to devices or software that analyzes captured voice data and converts it into corresponding text data.

[0623] "Means for recognizing emotions based on text data and voice characteristics" refers to devices and software for analyzing and recognizing a user's emotional state from the characteristics of text data and voice data.

[0624] The "means for selecting a generative artificial intelligence model based on text data and a recognized emotional state" refers to a device or software for automatically selecting an optimal artificial intelligence model based on the recognized text data and an emotional state.

[0625] "Means for generating personalized response data using a generative artificial intelligence model" refers to a device or software that uses a selected artificial intelligence model to generate response data in response to a user's request and emotional state.

[0626] The "means for transmitting the generated response data to the user" refers to a communication device or software for transferring the generated response data to the user device.

[0627] The "means for converting the generated response data into voice data" refers to a voice synthesizer or software for converting the generated text response into voice form.

[0628] The "means for reproducing audio data to the user" refers to a reproduction device or component that allows the user to listen to the audio data.

[0629] An "in-vehicle acoustic input device" is a microphone or other acoustic sensor installed inside a motor vehicle to capture sounds made by passengers.

[0630] "Playing back the voice data generated by the voice synthesizer through an audio playback device inside the vehicle" means letting the user hear the voice data generated by the voice synthesizer using a speaker system installed inside the vehicle.

[0631] The present invention is a system installed in an autonomous vehicle that captures a user's voice input, converts it into text data, recognizes emotions based on the text data and voice characteristics, and generates an optimal response. An embodiment of this system is described in detail below.

[0632] server

[0633] The server receives voice data sent from the user's device. It converts the voice data into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text). At the same time, it recognizes the user's emotional state from the voice data using an emotion engine (e.g., IBM Watson Tone Analyzer). Based on the recognized text data and emotional state, it selects a generative AI model (e.g., OpenAI GPT-3) and generates appropriate response data. The generated response data might be in the form of, for example, "Your next destination is Ginza. Enjoy a leisurely drive." The server then sends this response data to the autonomous vehicle's device.

[0634] Terminal

[0635] The terminal captures the user's voice input using an audio input device (microphone) installed in the autonomous vehicle. This captured voice data is sent to the server using a communication protocol such as an HTTP request or WebSocket. When the terminal receives response data from the server, it decodes the received data and obtains the response content in text format. It then converts this text into speech using a speech synthesis engine (e.g., Amazon Polly) and plays it back to the user through the audio playback device (speaker) in the autonomous vehicle. For example, if the user issues a voice command such as "Play the next song," the terminal receives a response from the server saying, "Playing the next song. Relax," converts this into speech, and plays it back.

[0636] User

[0637] The user issues a voice command to the audio input device in the autonomous vehicle. For example, they might ask, "Set my next destination." This voice command is captured by the device and sent to the server. As a result of processing by the server, the user receives a voice response such as, "Your next destination is Ginza. Enjoy a leisurely drive." Additionally, the user can customize the in-vehicle infotainment system using a smartphone app. New settings based on the user's customization requests are generated using generative AI and applied to the vehicle's systems in real time. The emotion engine takes into account the user's current emotions and changes the music and lighting theme to provide a more personalized experience for the user.

[0638] Specific examples

[0639] When a passenger asks "Where's my next destination?", an example of the server's processing is as follows:

[0640] Input prompt: "Where's my next destination?"

[0641] The server converts the speech to text: "Where's my next destination?"

[0642] Sentiment analysis result: "Calm"

[0643] Generative AI model selection and response generation: "Your next destination is Ginza. Enjoy a leisurely drive."

[0644] Speech synthesis and response: "Your next destination is Ginza. Enjoy a leisurely drive." is played over the speaker in the self-driving vehicle.

[0645] This system will enable a more personalized experience inside self-driving vehicles, making travel smoother and more comfortable.

[0646] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0647] Step 1:

[0648] The terminal captures the user's voice input, i.e., the user speaks a voice command into a microphone in the car and acquires the voice data.

[0649] Input: User's voice command

[0650] Output: Captured audio data

[0651] Step 2:

[0652] The device sends the captured audio data to the server using HTTP requests and WebSockets as the communication protocol.

[0653] Input: Captured audio data

[0654] Output: Sending audio data to the server

[0655] Step 3:

[0656] The server converts the received voice data into text data using a speech recognition engine, such as Google Cloud Speech-to-Text.

[0657] Input: Incoming audio data

[0658] Output: Text data

[0659] Step 4:

[0660] The server analyzes the text data and voice characteristics using an emotion engine, such as IBM Watson Tone Analyzer, to recognize the user's emotional state.

[0661] Input: Text data and audio data features

[0662] Output: User's emotional state

[0663] Step 5:

[0664] The server selects the optimal generative AI model based on the recognized text data and emotional state, using OpenAI GPT-3 or other models.

[0665] Input: Text data, user's emotional state

[0666] Output: The selected generative AI model

[0667] Step 6:

[0668] The server generates personalized response data using the selected generative AI model.

[0669] Input: Text data, the user's emotional state, and a generative AI model

[0670] Output: Generated response data

[0671] Step 7:

[0672] The server transmits the generated response data to the terminal.

[0673] Input: Generated response data

[0674] Output: Send response data to the terminal

[0675] Step 8:

[0676] The device uses a speech synthesis engine, such as Amazon Polly, to convert the received response data from text to voice data.

[0677] Input: Received response data (text format)

[0678] Output: Audio data

[0679] Step 9:

[0680] The terminal reproduces the generated voice data through the speaker in the vehicle and provides it to the user.

[0681] Input: Generated audio data

[0682] Output: A voice response to the user

[0683] In this way, the system personalizes the user experience in an autonomous vehicle, ensuring a smooth and comfortable journey.

[0684] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0685] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0686] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0687] [Third embodiment]

[0688] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0689] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0690] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0691] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0692] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0693] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0694] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0695] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0696] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0697] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0698] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0699] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0700] The present invention relates to a system that captures a user's voice input, converts it into text data, selects an appropriate generative AI model to generate response data, and provides it to the user. Below, the program processing of this system is explained in natural language.

[0701] server

[0702] The server receives a voice data request sent from the user's device. First, the server analyzes this voice data and converts it into text data using a speech recognition engine. Then, based on the converted text data, the server selects the necessary generative AI model. For example, if a user asks, "What's the weather like today?", the server selects the generative AI model that corresponds to the weather information.

[0703] Next, the server generates response data using the selected generative AI model. The generated response data is provided in text format, such as "Today's weather is sunny." Finally, the server sends this response data to the user's device.

[0704] Terminal

[0705] The device is responsible for capturing the user's voice input: it captures the user's voice as they speak into earphones or a smartphone microphone, and transmits this captured voice data to a server using a communication protocol such as HTTP requests or WebSocket.

[0706] When the device receives the response data from the server, it decodes the received data to obtain the response content in text format. It then uses a speech synthesis engine to convert this text into speech and play it back to the user. For example, when the user issues a voice command such as "Play the next song," the device receives a response from the server saying "Playing the next song," converts it into speech, and plays it back.

[0707] User

[0708] The user issues a voice command to the device. For example, they put on earphones and ask, "What time is it now?" This voice command is captured by the device and sent to the server. As a result of the server's processing, the user receives a voice response such as, "The current time is 3:00 PM."

[0709] Additionally, users can customize the design of their device using a smartphone app, which uses generative AI to generate new designs based on the user's customization requests and apply them to the device in real time.

[0710] This system allows users to enjoy a consistent experience and smooth operation across multiple devices, while also reducing development costs for device manufacturers by standardizing the process of selecting generative AI models and generating responses.

[0711] The processing flow will be explained below.

[0712] Step 1:

[0713] The device captures the user's voice input: as the user speaks voice commands into earphones or a smartphone microphone, the voice data is recorded in real time and stored in a buffer.

[0714] Step 2:

[0715] The device encodes the captured audio data and sends it to the server using a communication protocol such as an HTTP request or WebSocket.

[0716] Step 3:

[0717] The server receives the voice data sent from the terminal and stores it in the server's internal storage.

[0718] Step 4:

[0719] The server uses a voice recognition engine to convert the received voice data into text data, thereby obtaining the user's voice commands in text format.

[0720] Step 5:

[0721] The server selects an appropriate generative AI model based on the text data. It analyzes the text and identifies and selects the generative AI model that best suits the user's request.

[0722] Step 6:

[0723] The server generates response data using the selected generative AI model. For example, in response to the request "What's the weather like today?", it generates the response text "Today's weather is sunny."

[0724] Step 7:

[0725] The server transmits the generated response data to the terminal, which is then encoded in an appropriate format and transmitted to the terminal via a communication protocol.

[0726] Step 8:

[0727] The terminal receives the response data sent from the server, decodes the received data, and stores the response content in text format in the terminal.

[0728] Step 9:

[0729] The device converts the received text response into audio data using a speech synthesis engine, which analyzes the text and generates natural-sounding speech.

[0730] Step 10:

[0731] The device plays the generated voice data to the user. The user can hear the voice response through earphones or a speaker. For example, a voice such as "The current time is 3:00 PM" is played.

[0732] Step 11:

[0733] The user can then act on the generated voice response, for example by issuing the next voice command or customizing the device using the generative AI. This process is repeated, allowing the user to enjoy a consistent experience.

[0734] Example 1

[0735] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0736] Conventional voice recognition systems require a wide variety of technologies to properly convert voice input into text data and generate appropriate responses. This makes it difficult to provide accurate responses in real time to user voice commands. It also makes it difficult to provide a consistent user experience across multiple devices. Furthermore, it is not possible to customize the generated design or responses, resulting in a loss of user convenience.

[0737] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0738] In this invention, the server includes means for capturing a user's voice input, means for analyzing the captured voice input and converting it into text data, means for selecting an AI model generated based on the text data, means for generating response data using the generated AI model, means for transmitting the generated response data to the user, means for decoding the response data as text data, means for using a voice synthesis engine to convert the decoded text data into voice data, and means for playing back the converted voice data, thereby enabling accurate responses to the user's voice commands in real time and providing a unified user experience.

[0739] The "means for capturing user voice input" refers to a device or method for capturing voice uttered by a user, such as a microphone provided on a mobile terminal.

[0740] The "means for analyzing the captured voice input and converting it into text data" refers to a device or method for analyzing the captured voice using voice recognition technology and converting it into data in the form of a string of characters. Specifically, it includes a voice recognition engine.

[0741] The "means for selecting an AI model generated based on the text data" refers to a device or method for identifying and selecting an appropriate AI model based on the analyzed text data. Specifically, it includes an algorithm for selecting a generative AI model.

[0742] "Means for generating response data using the generated artificial intelligence model" refers to a device or method for generating a response to a user's query using a selected generative AI model.

[0743] The "means for transmitting the generated response data to the user" refers to a device or method for transferring the generated response data to the user's device. Specifically, it includes a data transmission technique using a communication protocol.

[0744] The "means for decoding the response data as text data" refers to a device or method for converting received response data into a character string format and analyzing it.

[0745] The "means for using a speech synthesis engine to convert the decoded text data into speech data" refers to a device or method for converting the decoded text data into speech format. Specifically, it includes a speech synthesis engine.

[0746] The "means for reproducing the converted audio data" refers to a device or method for allowing a user to listen to the converted audio data. Specifically, this applies to a speaker, earphones, etc.

[0747] The present invention relates to a system that captures a user's voice input, converts it into text data, selects an appropriate generative AI model, generates response data, and provides it to the user.

[0748] Server Roles

[0749] The server receives a voice data request sent from the user device. After receiving the voice data, the server uses a speech recognition engine (specifically, Google Cloud Speech-to-Text API, etc.) to convert the voice data into text data. For example, if a user asks, "What's the weather like today?", the voice data is converted into text data such as, "What's the weather like today?"

[0750] Based on the text data, the server selects an appropriate generative AI model by analyzing keywords and intent within the text data. For example, it selects a generative AI model (e.g., GPT-3) that corresponds to weather information.

[0751] Next, the server generates response data using the selected generative AI model. This generation process generates text data such as "Today's weather is sunny." Finally, the server sends the generated response data to the user's device. The communication protocol used is an HTTP POST request or WebSocket.

[0752] Device Role

[0753] The device has the function to capture the user's voice. Specifically, it records the user's voice using the smartphone microphone or earphone microphone, and sends the captured voice data to the server. The transmission method is an HTTP POST request or WebSocket.

[0754] When the device receives response data from the server, it decodes the data and obtains the response in text format. The obtained text data is converted into voice data using a speech synthesis engine (specifically, Amazon Polly, etc.). The response is provided to the user by playing this voice data through speakers or earphones.

[0755] User Roles

[0756] A user can issue a voice command to the device. For example, they can use earphones to ask, "What time is it now?" This voice command is captured by the device and sent to the server. As a result of the server's processing, the user can receive a voice response such as, "The current time is 3:00 PM."

[0757] Users can also customize the design of their device using a smartphone app, which uses generative AI to generate new designs based on the user's customization requests and apply them to the device in real time.

[0758] Specific operation example

[0759] Weather information inquiry

[0760] The user asks the device, "What's the weather like today?" The device captures this audio and sends the audio data to the server. The server uses the Google Cloud Speech-to-Text API to convert the audio into text, generating the text "What's the weather like today?" The server then selects a generative AI model (e.g., GPT-3) that corresponds to the weather information and generates response data saying, "The weather is sunny today." This response data is sent to the device, and when the device receives the text data, it uses a speech synthesis engine to convert it into audio data and play it back to the user.

[0761] Prompt Sentence Examples

[0762] The system prompts users with real-time responses based on their requests:

[0763] Received the user's voice saying "Play next song." Generate a text response saying "Play next song."

[0764] As described above, the system of the invention can efficiently process user voice input and provide appropriate responses. This system allows users to enjoy a consistent user experience and allows smooth operation across multiple devices. It also reduces development costs for device manufacturers by standardizing the process of selecting generative AI models and generating responses.

[0765] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0766] Step 1:

[0767] The user inputs a voice command into the device. For example, the user puts on earphones and says, "What's the weather like today?" The input data is voice data, and this is the starting point for the system's processing.

[0768] Step 2:

[0769] The device captures the user's voice. Specifically, it records the voice using the smartphone microphone or the earphone microphone. The input data is the voice data from the previous step, and the output data is the captured voice data.

[0770] Step 3:

[0771] The device sends the captured audio data to the server using a communication protocol such as an HTTP POST request or WebSocket. The input data is the captured audio data, and the output data is the audio data sent to the server.

[0772] Step 4:

[0773] The server analyzes the received voice data. Specifically, it converts the voice data into text data using a voice recognition engine (such as Google Cloud Speech-to-Text API). The input data is the voice data sent to the server, and the output data is text data. For example, the voice data "What's the weather like today?" is converted into the text data "What's the weather like today?"

[0774] Step 5:

[0775] The server selects an appropriate generative AI model based on the converted text data. The input data is the text data from the previous step, and it processes the data to select a generative AI model (for example, a GPT-3 model for weather information). The output data is the selected generative AI model.

[0776] Step 6:

[0777] The server generates response data using the selected generative AI model. Specifically, it uses a generative AI such as the GPT-3 model to generate an optimal response to the text data. The input data is the selected generative AI model and the text data, and the output data is the generated response data. For example, in response to the question "What's the weather like today?", the generated response is "It's sunny today."

[0778] Step 7:

[0779] The server sends the generated response data to the user terminal. A communication protocol such as HTTP response or WebSocket is used for transmission. The input data is the generated response data, and the output data is the response data sent to the user terminal.

[0780] Step 8:

[0781] The terminal decodes the response data received from the server. The input data is the response data sent from the server, and the output data is the decoded text data.

[0782] Step 9:

[0783] The device converts the decoded text data into voice data using a speech synthesis engine. Specifically, a speech synthesis engine such as Amazon Polly is used. The input data is the decoded text data, and the output data is voice data. For example, the text "The weather is sunny today" is converted into voice data.

[0784] Step 10:

[0785] The device plays the converted audio data. Specifically, the audio is played back to the user through the smartphone's speaker or earphones. The input data is audio data, and the final output is the audio that the user hears.

[0786] Step 11:

[0787] The user receives the voice response played back from the terminal, and can hear a voice response such as "The weather is sunny today."

[0788] (Application example 1)

[0789] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0790] In factory operations, workers had to use conventional computers and touch panels to operate robots. This meant that operation took time at the workplace, resulting in reduced work efficiency. Furthermore, manual operation could lead to human error, which also led to reduced productivity and quality control issues. A system was needed to solve these issues and enable workers to operate robots more efficiently and reliably.

[0791] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0792] In this invention, the server includes means for capturing a user's voice input, means for analyzing the captured voice input and converting it into text data, means for selecting a generative AI model based on the text data, means for generating response data using the generative AI model, means for sending the generated response data to a user, means for causing a machine to execute the generated response data as an instruction, and means for a factory worker to give voice instructions and for the robot to execute the instructions. This allows the worker to instantly operate the robot using voice instructions, improving work efficiency and reducing operation errors.

[0793] "User voice input" refers to collecting the voice uttered by the user as data.

[0794] "Capture" is a method of obtaining audio as digital data.

[0795] "Text data" is information that is generated by analyzing voice input and expressed as a string of characters.

[0796] A "generative artificial intelligence model" is a machine learning model that generates appropriate responses to specific data or questions.

[0797] "Response data" refers to data corresponding to answers or instructions generated by a generative artificial intelligence model.

[0798] "Mobile terminal" refers to mobile devices in general, including smartphones, tablets, etc.

[0799] A "microphone" is a device that converts sound into an electrical signal.

[0800] A "headset" is a device that allows for the capture and playback of sound.

[0801] A "speech synthesis engine" is a technology that converts text data into voice data.

[0802] "Audio data" is information obtained by digitizing an audio signal.

[0803] A "machine" is a device designed to perform a specified task.

[0804] An "instruction" is a command to perform a particular action or operation.

[0805] A "factory worker" is a human worker who performs work in a factory.

[0806] A "robot" is an automated device for carrying out programmed instructions.

[0807] This invention is a system in which a factory worker gives voice instructions and a robot performs actions based on those instructions. The system includes the following components:

[0808] server

[0809] The server receives a voice data request sent from the user's device. First, the server analyzes this voice data and converts it into text data using a speech recognition engine (e.g., Google Speech-to-Text API). Next, based on the converted text data, it selects the required generative AI model (e.g., OpenAI's GPT-4). For example, if a worker verbally instructs, "Tell me the next work step," the server selects an appropriate AI model and generates response data for the next step. This response data is provided in text format and is used as a command for the robot to execute. Finally, the server sends this response data to the user's device and the robot.

[0810] Terminal

[0811] The terminal is responsible for capturing the user's voice input. It captures the voice of the worker speaking into the microphone of a headset or mobile device and sends this captured voice data to the server. A communication protocol such as HTTP request or WebSocket is used for transmission. When the terminal receives response data from the server, it decodes the received data and obtains the response in text format. It then uses a speech synthesis engine (e.g., Google Cloud Text-to-Speech) to convert this text data into speech and play it back to the worker. For example, if a worker issues an instruction such as "Pick up a part from the shelf," the terminal converts the server's response, "Perform the next step," into speech and plays it back to the worker.

[0812] User

[0813] The user is a factory worker who issues a voice command. For example, the worker asks, "What is the next work process?" This voice command is captured by the terminal and sent to the server. As a result of the server's processing, the user receives a voice response such as, "The next work process is assembly." This simplifies operations for the user through voice instructions, reduces manual operation errors, and improves work efficiency.

[0814] Specific examples

[0815] When a worker asks "What's the next step?", the following prompt is input to the generative AI model:

[0816] The worker says, "Tell me the next step." Please explain the following steps.

[0817] As a result, the generative AI model generates a response saying, "The next work process is assembly. The robot will move the parts to the assembly station." The generated response data is converted into voice data by a speech synthesis engine and played back to the worker. At the same time, an instruction is given to the robot to move the parts to the assembly station as the next process.

[0818] This system allows workers in the factory to operate robots using voice commands, improving work efficiency and reducing operating errors.

[0819] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0820] Step 1:

[0821] The user issues instructions to the worker in the form of voice, speaking into a headset or a microphone on a mobile device. This voice input is captured by the device and obtained as digital data. As input, there is voice data. As output, this voice data is captured and ready to be sent to a server.

[0822] Step 2:

[0823] The device sends the captured audio data to the server. The communication protocol is HTTP request or WebSocket. The input is the captured audio data. The output is the server receiving the request for audio data.

[0824] Step 3:

[0825] The server analyzes the received voice data and converts it into text data using a speech recognition engine (Google Speech-to-Text API). The input is voice data, and the output is the voice data converted into text data.

[0826] Step 4:

[0827] The server selects an appropriate generative AI model (OpenAI's GPT-4) based on the converted text data. The input is the text data. The output is the selected generative AI model.

[0828] Step 5:

[0829] The server inputs a prompt statement to the selected generative AI model and generates response data. The inputs include the selected generative AI model and the prompt statement. Examples of prompt statements are as follows:

[0830] The worker says, "Tell me the next step." Please explain the following steps.

[0831] The output is the generated response data.

[0832] Step 6:

[0833] The server transmits the generated response data to the user terminal and the robot. The generated response data is the input. The user terminal and the robot receive the response data as the output.

[0834] Step 7:

[0835] The device decodes the received response data and converts it into voice data using a speech synthesis engine (Google Cloud Text-to-Speech). The input is the response data (in text format). The output is voice data.

[0836] Step 8:

[0837] The terminal plays the voice data and notifies the worker of the response. At the same time, instructions based on the response data are transmitted to the robot, which then carries out those instructions. The inputs are the voice data and instruction data for the robot. The outputs are a voice notification to the worker and the robot's execution of an action.

[0838] In this way, a system for operating robots in a factory using voice commands is realized, allowing workers to operate robots easily and reliably, improving work efficiency.

[0839] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0840] The present invention is a system that not only captures a user's voice input, converts it into text data, and selects an appropriate generative AI model to generate response data, but also combines it with an emotion engine that recognizes the user's emotions to provide more personalized responses. Below, the program processing of this system is explained in natural language.

[0841] server

[0842] The server receives a voice data request from the user terminal. First, the server analyzes the voice data and converts it into text data using a speech recognition engine. At the same time, it uses an emotion engine to recognize the user's emotional state from the voice data. For example, it determines whether the user is happy, angry, or sad based on characteristics such as the tone, speed, and accent of the voice.

[0843] Based on the text data and the recognized emotional state, the server selects an appropriate generative AI model. Depending on the user's request and emotional state, the generative AI model that generates the optimal response is identified and selected. For example, in response to the request "What's the weather today?", if the user is calm, a normal response is provided, but if the user is angry, a more considerate response is generated.

[0844] The generated response data is generated in the form of, for example, "The weather is sunny today. Have a nice day." The server transmits this response data to the user terminal.

[0845] Terminal

[0846] The device is responsible for capturing the user's voice input: it captures the user's voice as they speak into earphones or a smartphone microphone, and transmits this captured voice data to a server using a communication protocol such as HTTP requests or WebSocket.

[0847] When the device receives the response data from the server, it decodes the received data and obtains data that reflects the text response content and consideration according to the user's emotions. It then uses a speech synthesis engine to convert this text into speech and plays it back to the user. For example, when the user issues a voice command such as "Play the next song," the device receives a response from the server saying, "Playing the next song. Refresh your mood with good music," converts it into speech, and plays it back.

[0848] User

[0849] The user issues a voice command to the device. For example, they put on earphones and ask, "What time is it now?" This voice command is captured by the device and sent to the server. As a result of the server's processing, the user receives a voice response such as, "It is now 3:00 p.m. Have a nice afternoon."

[0850] Additionally, users can customize the device's design using a smartphone app. Generative AI generates new designs based on the user's customization requests and applies them to the device in real time. An emotion engine takes into account the user's current emotions and changes the color and theme of the design, providing a more personalized experience for the user.

[0851] This system allows users to enjoy a consistent experience and smooth operation across multiple devices, while also reducing development costs for device manufacturers by standardizing the process of selecting generative AI models and emotion engines and generating responses.

[0852] The processing flow will be explained below.

[0853] Step 1:

[0854] The device captures the user's voice input: as the user speaks voice commands into earphones or a smartphone microphone, the voice data is recorded in real time and stored in a buffer.

[0855] Step 2:

[0856] The device encodes the captured audio data and sends it to the server using a communication protocol such as an HTTP request or WebSocket.

[0857] Step 3:

[0858] The server receives the voice data sent from the terminal and stores it in the server's internal storage.

[0859] Step 4:

[0860] The server uses a voice recognition engine to convert the received voice data into text data, thereby obtaining the user's voice commands in text format.

[0861] Step 5:

[0862] The server inputs the text data into an emotion engine to recognize the user's emotional state. The emotion engine analyzes the tone, speed, and accent of the voice to determine whether the user is happy, angry, or sad.

[0863] Step 6:

[0864] The server selects an appropriate generative AI model based on the text data and the recognized emotional state. The server analyzes the text and identifies and selects the generative AI model that best suits the user's request and emotional state.

[0865] Step 7:

[0866] The server generates response data using the selected generative AI model. For example, in response to a request like "What's the weather today?", if the user is calm, it will provide a normal response, but if the user is angry, it will generate a more considerate response, such as "The weather is sunny today. Have a nice day."

[0867] Step 8:

[0868] The server transmits the generated response data to the terminal, which is then encoded in an appropriate format and transmitted to the terminal via a communication protocol.

[0869] Step 9:

[0870] The terminal receives the response data sent from the server, decodes the received data, and stores the response content in text format in the terminal.

[0871] Step 10:

[0872] The device converts the received text response into audio data using a speech synthesis engine, which analyzes the text and generates natural-sounding speech.

[0873] Step 11:

[0874] The device plays the generated voice data to the user. The user can hear the voice response through earphones or speakers. For example, a voice such as "It is now 3:00 p.m. Have a nice afternoon" may be played.

[0875] Step 12:

[0876] The user can then act on the generated voice response, for example by issuing the next voice command or customizing the device using generative AI and an emotion engine. This process is repeated, providing the user with a consistent experience.

[0877] Example 2

[0878] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0879] Conventional speech recognition systems simply convert a user's voice input into text and generate responses, but are unable to provide responses that take the user's emotions into account. This can lead to stress and an unpleasant experience when trying to obtain the information they need. Furthermore, because responses cannot be generated based on the user's emotions, responses are not personalized enough, which can result in a decrease in user satisfaction.

[0880] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for capturing a user's voice input, means for analyzing the captured voice input and converting it into text data, means for recognizing the user's emotional state from the voice input, means for selecting a generative AI model based on the text data and the user's emotional state, means for generating response data using the selected generative AI model, and means for transmitting the generated response data to the user. This makes it possible to provide a personalized response that takes the user's emotions into consideration.

[0881] "Means for capturing user voice input" refers to devices and techniques for recording the voices uttered by a user.

[0882] The "means for analyzing and converting into text data" refers to the technology and system for analyzing voice data and converting it into corresponding character string data.

[0883] "Means for recognizing a user's emotional state" refers to techniques and systems that detect and identify a user's emotional state (e.g., joy, anger, sadness, etc.) from voice data.

[0884] "Means for selecting a generative artificial intelligence model" refers to techniques and systems for selecting the most appropriate artificial intelligence model based on the analyzed data.

[0885] "Means for generating response data" refers to techniques and systems that use selected artificial intelligence models to automatically generate appropriate responses to user input.

[0886] "Means for transmitting to the user" refers to communication technology and systems for transmitting the generated response data to the user's terminal.

[0887] A "microphone installed in a mobile terminal" refers to a voice input device built into a portable device such as a mobile phone or smartphone.

[0888] A "speech synthesis engine" refers to technology and systems for converting text data into speech data.

[0889] This invention is a system that captures a user's voice input, converts it into text data, and then selects an appropriate generative AI model to generate response data.By combining this with an emotion engine that recognizes the user's emotions, it provides more personalized responses.

[0890] server

[0891] The server receives a voice data request sent from the user's device. First, the server analyzes this voice data and converts it into text data using a general-purpose voice recognition engine (such as Google's Speech-to-Text API). At the same time, it uses an emotion recognition engine (such as general emotion analysis software) to recognize the user's emotional state from the voice data. It determines the user's emotion based on characteristics such as the tone, speed, and accent of the voice. Based on this information, it selects the optimal generative artificial intelligence model and generates personalized response data using OpenAI's GPT-3 or similar. The generated response data is then sent to the user's device.

[0892] For example, if a user says, "What time is the next meeting?", the server converts the speech to text and then analyzes it with an emotion engine to recognize that the user is impatient. It then selects an appropriate generative AI model and provides a personalized response (e.g., "The next meeting is at 2 p.m. Take your time to prepare and you'll be fine.").

[0893] Example prompt sentence:

[0894] Generate a relaxing response for users who impatiently ask, "What time is the next meeting?"

[0895] Terminal

[0896] The terminal is responsible for capturing the user's voice input. It captures the user's speech into a microphone on a portable device (e.g., a smartphone or mobile phone) and sends this voice data to a server using a communication protocol (e.g., HTTP request or WebSocket). It decodes the received response data and obtains the response content in text format. It then converts this text into speech using a speech synthesis engine (e.g., Amazon's Polly) and plays it back to the user.

[0897] As a specific example, when a user issues a voice command such as "Play the next song," the device sends this voice data to the server, receives a response from the server saying, "Playing the next song. Refresh your mood with good music," converts it into voice, and plays it.

[0898] User

[0899] The user issues a voice command to the device. For example, they put on earphones and ask, "What time is it now?" The voice command is captured by the device and sent to the server. As a result of the server's processing, the user receives a voice response such as, "It is now 3:00 p.m. Have a nice afternoon."

[0900] Additionally, users can customize the device's design using a smartphone app: for example, an emotion engine can recognize when a user is relaxing and change the device's theme to softer colors.

[0901] This system allows users to enjoy a consistent experience and smooth operation across multiple devices, while also reducing development costs for device manufacturers by standardizing the process of selecting generative AI models and emotion engines and generating responses.

[0902] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0903] Step 1: Capture the user's voice input

[0904] Terminal

[0905] The terminal captures in real time the user's voice as they speak into a microphone installed on a portable device (e.g., a smartphone). The input is the user's voice data, and the output is a byte stream that converts this voice data into a digital format. This byte stream is stored in temporary storage.

[0906] Specific behavior:

[0907] A user issues the voice command, "What time is the next meeting?"

[0908] The device captures this voice command and converts it into digital audio data in .wav format.

[0909] Step 2: Send the audio data to the server

[0910] Terminal

[0911] The device sends the captured audio data to the server. The input is digital audio data, and the output is obtained by sending this data to the server using a communication protocol such as HTTP requests or WebSockets.

[0912] Specific behavior:

[0913] The device sends the saved audio data to the server in the body of an HTTP POST request.

[0914] Step 3: Convert the audio data to text

[0915] server

[0916] The server receives the voice data sent from the device and converts it into text data using a voice recognition engine (e.g., a general-purpose voice recognition engine). The input is the voice data sent to the server, and the output is the analyzed text data. This text data is stored in internal memory.

[0917] Specific behavior:

[0918] The server receives the voice data "What time is the next meeting?"

[0919] Call Google's Speech-to-Text API to generate text data such as "What time is the next meeting?"

[0920] Step 4: Recognize your emotional state

[0921] server

[0922] The server passes the received voice data to an emotion recognition engine to recognize the emotional state. The input is the voice data, and the output is the user's emotional state (e.g., happy, anxious).

[0923] Specific behavior:

[0924] The emotion engine analyzes the voice data and recognizes emotional states such as "anxious."

[0925] The recognized emotional state is kept in an internal memory.

[0926] Step 5: Select a generative AI model

[0927] server

[0928] The server generates prompt sentences and selects an appropriate generative AI model based on the text data and the recognized emotional state. The input is the text data and the emotional state, and the output is the selected generative AI model.

[0929] Specific behavior:

[0930] Prompt Sentence: Generate a relaxing response for a user who impatiently asks, "What time is the next meeting?"

[0931] Select the best generative artificial intelligence model, such as OpenAI's GPT-3.

[0932] Step 6: Generate response data

[0933] server

[0934] The server generates response data using the selected generative AI model. The input is the prompt sentence, and the output is the generated response data.

[0935] Specific behavior:

[0936] Enter a prompt into GPT-3 and it will generate a response like, "The next meeting is at 2 p.m. Take your time to prepare and you'll be fine."

[0937] Step 7: Send the response data to the user device

[0938] server

[0939] The server sends the generated response data to the user terminal. The input is the generated response data, and the output is the sent response data.

[0940] Specific behavior:

[0941] The server structures the response data in JSON format and sends it to the terminal as an HTTP response.

[0942] Step 8: Play the response aloud

[0943] Terminal

[0944] The device decodes the response data received from the server and converts it into voice data using a speech synthesis engine (e.g., Amazon's Polly). The input is the text response data received from the server, and the output is voice data. This voice data is then played back to the user through the speaker.

[0945] Specific behavior:

[0946] The device receives response data saying, "The next meeting is at 2:00 p.m. Take your time to prepare."

[0947] The text is converted into speech using a speech synthesis engine and played through speakers or earphones.

[0948] (Application example 2)

[0949] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0950] Conventional speech recognition systems simply convert a user's voice input into text and generate responses based on that text. However, they can only provide a uniform response without considering the user's emotions, making it difficult to provide a personalized experience for the user. Furthermore, especially in autonomous vehicles, there are many situations where responses that correspond to the passenger's emotional state are required. Therefore, there is a need to build a system that can provide more personalized responses based on the user's voice input and their emotional state.

[0951] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0952] In this invention, the server includes means for capturing a user's voice input, means for analyzing the captured voice input and converting it into text data, means for recognizing emotions based on the text data and voice characteristics, means for selecting a generative AI model based on the text data and the recognized emotional state, means for generating personalized response data using the generative AI model, means for transmitting the generated response data to the user, means for converting the generated response data into voice data, and means for playing the voice data to the user, thereby making it possible to provide a personalized experience adapted to the user's emotional state.

[0953] A "means for capturing user voice input" is a device or component for capturing voice uttered by a user and obtaining the voice data.

[0954] "Means for analyzing captured voice input and converting it into text data" refers to devices or software that analyzes captured voice data and converts it into corresponding text data.

[0955] "Means for recognizing emotions based on text data and voice characteristics" refers to devices and software for analyzing and recognizing a user's emotional state from the characteristics of text data and voice data.

[0956] The "means for selecting a generative artificial intelligence model based on text data and a recognized emotional state" refers to a device or software for automatically selecting an optimal artificial intelligence model based on the recognized text data and an emotional state.

[0957] "Means for generating personalized response data using a generative artificial intelligence model" refers to a device or software that uses a selected artificial intelligence model to generate response data in response to a user's request and emotional state.

[0958] The "means for transmitting the generated response data to the user" refers to a communication device or software for transferring the generated response data to the user device.

[0959] The "means for converting the generated response data into voice data" refers to a voice synthesizer or software for converting the generated text response into voice form.

[0960] The "means for reproducing audio data to the user" refers to a reproduction device or component that allows the user to listen to the audio data.

[0961] An "in-vehicle acoustic input device" is a microphone or other acoustic sensor installed inside a motor vehicle to capture sounds made by passengers.

[0962] "Playing back the voice data generated by the voice synthesizer through an audio playback device inside the vehicle" means letting the user hear the voice data generated by the voice synthesizer using a speaker system installed inside the vehicle.

[0963] The present invention is a system installed in an autonomous vehicle that captures a user's voice input, converts it into text data, recognizes emotions based on the text data and voice characteristics, and generates an optimal response. An embodiment of this system is described in detail below.

[0964] server

[0965] The server receives voice data sent from the user's device. It converts the voice data into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text). At the same time, it recognizes the user's emotional state from the voice data using an emotion engine (e.g., IBM Watson Tone Analyzer). Based on the recognized text data and emotional state, it selects a generative AI model (e.g., OpenAI GPT-3) and generates appropriate response data. The generated response data might be in the form of, for example, "Your next destination is Ginza. Enjoy a leisurely drive." The server then sends this response data to the autonomous vehicle's device.

[0966] Terminal

[0967] The terminal captures the user's voice input using an audio input device (microphone) installed in the autonomous vehicle. This captured voice data is sent to the server using a communication protocol such as an HTTP request or WebSocket. When the terminal receives response data from the server, it decodes the received data and obtains the response content in text format. It then converts this text into speech using a speech synthesis engine (e.g., Amazon Polly) and plays it back to the user through the audio playback device (speaker) in the autonomous vehicle. For example, if the user issues a voice command such as "Play the next song," the terminal receives a response from the server saying, "Playing the next song. Relax," converts this into speech, and plays it back.

[0968] User

[0969] The user issues a voice command to the audio input device in the autonomous vehicle. For example, they might ask, "Set my next destination." This voice command is captured by the device and sent to the server. As a result of processing by the server, the user receives a voice response such as, "Your next destination is Ginza. Enjoy a leisurely drive." Additionally, the user can customize the in-vehicle infotainment system using a smartphone app. New settings based on the user's customization requests are generated using generative AI and applied to the vehicle's systems in real time. The emotion engine takes into account the user's current emotions and changes the music and lighting theme to provide a more personalized experience for the user.

[0970] Specific examples

[0971] When a passenger asks "Where's my next destination?", an example of the server's processing is as follows:

[0972] Input prompt: "Where's my next destination?"

[0973] The server converts the speech to text: "Where's my next destination?"

[0974] Sentiment analysis result: "Calm"

[0975] Generative AI model selection and response generation: "Your next destination is Ginza. Enjoy a leisurely drive."

[0976] Speech synthesis and response: "Your next destination is Ginza. Enjoy a leisurely drive." is played over the speaker in the self-driving vehicle.

[0977] This system will enable a more personalized experience inside self-driving vehicles, making travel smoother and more comfortable.

[0978] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0979] Step 1:

[0980] The terminal captures the user's voice input, i.e., the user speaks a voice command into a microphone in the car and acquires the voice data.

[0981] Input: User's voice command

[0982] Output: Captured audio data

[0983] Step 2:

[0984] The device sends the captured audio data to the server using HTTP requests and WebSockets as the communication protocol.

[0985] Input: Captured audio data

[0986] Output: Sending audio data to the server

[0987] Step 3:

[0988] The server converts the received voice data into text data using a speech recognition engine, such as Google Cloud Speech-to-Text.

[0989] Input: Incoming audio data

[0990] Output: Text data

[0991] Step 4:

[0992] The server analyzes the text data and voice characteristics using an emotion engine, such as IBM Watson Tone Analyzer, to recognize the user's emotional state.

[0993] Input: Text data and audio data features

[0994] Output: User's emotional state

[0995] Step 5:

[0996] The server selects the optimal generative AI model based on the recognized text data and emotional state, using OpenAI GPT-3 or other models.

[0997] Input: Text data, user's emotional state

[0998] Output: The selected generative AI model

[0999] Step 6:

[1000] The server generates personalized response data using the selected generative AI model.

[1001] Input: Text data, the user's emotional state, and a generative AI model

[1002] Output: Generated response data

[1003] Step 7:

[1004] The server transmits the generated response data to the terminal.

[1005] Input: Generated response data

[1006] Output: Send response data to the terminal

[1007] Step 8:

[1008] The device uses a speech synthesis engine, such as Amazon Polly, to convert the received response data from text to voice data.

[1009] Input: Received response data (text format)

[1010] Output: Audio data

[1011] Step 9:

[1012] The terminal reproduces the generated voice data through the speaker in the vehicle and provides it to the user.

[1013] Input: Generated audio data

[1014] Output: A voice response to the user

[1015] In this way, the system personalizes the user experience in an autonomous vehicle, ensuring a smooth and comfortable journey.

[1016] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1017] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1018] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1019] [Fourth embodiment]

[1020] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1021] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1022] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1023] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1024] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1025] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1026] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1027] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1028] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1029] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1031] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1033] The present invention relates to a system that captures a user's voice input, converts it into text data, selects an appropriate generative AI model to generate response data, and provides it to the user. Below, the program processing of this system is explained in natural language.

[1034] server

[1035] The server receives a voice data request sent from the user's device. First, the server analyzes this voice data and converts it into text data using a speech recognition engine. Then, based on the converted text data, the server selects the necessary generative AI model. For example, if a user asks, "What's the weather like today?", the server selects the generative AI model that corresponds to the weather information.

[1036] Next, the server generates response data using the selected generative AI model. The generated response data is provided in text format, such as "Today's weather is sunny." Finally, the server sends this response data to the user's device.

[1037] Terminal

[1038] The device is responsible for capturing the user's voice input: it captures the user's voice as they speak into earphones or a smartphone microphone, and transmits this captured voice data to a server using a communication protocol such as HTTP requests or WebSocket.

[1039] When the device receives the response data from the server, it decodes the received data to obtain the response content in text format. It then uses a speech synthesis engine to convert this text into speech and play it back to the user. For example, when the user issues a voice command such as "Play the next song," the device receives a response from the server saying "Playing the next song," converts it into speech, and plays it back.

[1040] User

[1041] The user issues a voice command to the device. For example, they put on earphones and ask, "What time is it now?" This voice command is captured by the device and sent to the server. As a result of the server's processing, the user receives a voice response such as, "The current time is 3:00 PM."

[1042] Additionally, users can customize the design of their device using a smartphone app, which uses generative AI to generate new designs based on the user's customization requests and apply them to the device in real time.

[1043] This system allows users to enjoy a consistent experience and smooth operation across multiple devices, while also reducing development costs for device manufacturers by standardizing the process of selecting generative AI models and generating responses.

[1044] The processing flow will be explained below.

[1045] Step 1:

[1046] The device captures the user's voice input: as the user speaks voice commands into earphones or a smartphone microphone, the voice data is recorded in real time and stored in a buffer.

[1047] Step 2:

[1048] The device encodes the captured audio data and sends it to the server using a communication protocol such as an HTTP request or WebSocket.

[1049] Step 3:

[1050] The server receives the voice data sent from the terminal and stores it in the server's internal storage.

[1051] Step 4:

[1052] The server uses a voice recognition engine to convert the received voice data into text data, thereby obtaining the user's voice commands in text format.

[1053] Step 5:

[1054] The server selects an appropriate generative AI model based on the text data. It analyzes the text and identifies and selects the generative AI model that best suits the user's request.

[1055] Step 6:

[1056] The server generates response data using the selected generative AI model. For example, in response to the request "What's the weather like today?", it generates the response text "Today's weather is sunny."

[1057] Step 7:

[1058] The server transmits the generated response data to the terminal, which is then encoded in an appropriate format and transmitted to the terminal via a communication protocol.

[1059] Step 8:

[1060] The terminal receives the response data sent from the server, decodes the received data, and stores the response content in text format in the terminal.

[1061] Step 9:

[1062] The device converts the received text response into audio data using a speech synthesis engine, which analyzes the text and generates natural-sounding speech.

[1063] Step 10:

[1064] The device plays the generated voice data to the user. The user can hear the voice response through earphones or a speaker. For example, a voice such as "The current time is 3:00 PM" is played.

[1065] Step 11:

[1066] The user can then act on the generated voice response, for example by issuing the next voice command or customizing the device using the generative AI. This process is repeated, allowing the user to enjoy a consistent experience.

[1067] Example 1

[1068] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1069] Conventional voice recognition systems require a wide variety of technologies to properly convert voice input into text data and generate appropriate responses. This makes it difficult to provide accurate responses in real time to user voice commands. It also makes it difficult to provide a consistent user experience across multiple devices. Furthermore, it is not possible to customize the generated design or responses, resulting in a loss of user convenience.

[1070] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1071] In this invention, the server includes means for capturing a user's voice input, means for analyzing the captured voice input and converting it into text data, means for selecting an AI model generated based on the text data, means for generating response data using the generated AI model, means for transmitting the generated response data to the user, means for decoding the response data as text data, means for using a voice synthesis engine to convert the decoded text data into voice data, and means for playing back the converted voice data, thereby enabling accurate responses to the user's voice commands in real time and providing a unified user experience.

[1072] The "means for capturing user voice input" refers to a device or method for capturing voice uttered by a user, such as a microphone provided on a mobile terminal.

[1073] The "means for analyzing the captured voice input and converting it into text data" refers to a device or method for analyzing the captured voice using voice recognition technology and converting it into data in the form of a string of characters. Specifically, it includes a voice recognition engine.

[1074] The "means for selecting an AI model generated based on the text data" refers to a device or method for identifying and selecting an appropriate AI model based on the analyzed text data. Specifically, it includes an algorithm for selecting a generative AI model.

[1075] "Means for generating response data using the generated artificial intelligence model" refers to a device or method for generating a response to a user's query using a selected generative AI model.

[1076] The "means for transmitting the generated response data to the user" refers to a device or method for transferring the generated response data to the user's device. Specifically, it includes a data transmission technique using a communication protocol.

[1077] The "means for decoding the response data as text data" refers to a device or method for converting received response data into a character string format and analyzing it.

[1078] The "means for using a speech synthesis engine to convert the decoded text data into speech data" refers to a device or method for converting the decoded text data into speech format. Specifically, it includes a speech synthesis engine.

[1079] The "means for reproducing the converted audio data" refers to a device or method for allowing a user to listen to the converted audio data. Specifically, this applies to a speaker, earphones, etc.

[1080] The present invention relates to a system that captures a user's voice input, converts it into text data, selects an appropriate generative AI model, generates response data, and provides it to the user.

[1081] Server Roles

[1082] The server receives a voice data request sent from the user device. After receiving the voice data, the server uses a speech recognition engine (specifically, Google Cloud Speech-to-Text API, etc.) to convert the voice data into text data. For example, if a user asks, "What's the weather like today?", the voice data is converted into text data such as, "What's the weather like today?"

[1083] Based on the text data, the server selects an appropriate generative AI model by analyzing keywords and intent within the text data. For example, it selects a generative AI model (e.g., GPT-3) that corresponds to weather information.

[1084] Next, the server generates response data using the selected generative AI model. This generation process generates text data such as "Today's weather is sunny." Finally, the server sends the generated response data to the user's device. The communication protocol used is an HTTP POST request or WebSocket.

[1085] Device Role

[1086] The device has the function to capture the user's voice. Specifically, it records the user's voice using the smartphone microphone or earphone microphone, and sends the captured voice data to the server. The transmission method is an HTTP POST request or WebSocket.

[1087] When the device receives response data from the server, it decodes the data and obtains the response in text format. The obtained text data is converted into voice data using a speech synthesis engine (specifically, Amazon Polly, etc.). The response is provided to the user by playing this voice data through speakers or earphones.

[1088] User Roles

[1089] A user can issue a voice command to the device. For example, they can use earphones to ask, "What time is it now?" This voice command is captured by the device and sent to the server. As a result of the server's processing, the user can receive a voice response such as, "The current time is 3:00 PM."

[1090] Users can also customize the design of their device using a smartphone app, which uses generative AI to generate new designs based on the user's customization requests and apply them to the device in real time.

[1091] Specific operation example

[1092] Weather information inquiry

[1093] The user asks the device, "What's the weather like today?" The device captures this audio and sends the audio data to the server. The server uses the Google Cloud Speech-to-Text API to convert the audio into text, generating the text "What's the weather like today?" The server then selects a generative AI model (e.g., GPT-3) that corresponds to the weather information and generates response data saying, "The weather is sunny today." This response data is sent to the device, and when the device receives the text data, it uses a speech synthesis engine to convert it into audio data and play it back to the user.

[1094] Prompt Sentence Examples

[1095] The system prompts users with real-time responses based on their requests:

[1096] Received the user's voice saying "Play next song." Generate a text response saying "Play next song."

[1097] As described above, the system of the invention can efficiently process user voice input and provide appropriate responses. This system allows users to enjoy a consistent user experience and allows smooth operation across multiple devices. It also reduces development costs for device manufacturers by standardizing the process of selecting generative AI models and generating responses.

[1098] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1099] Step 1:

[1100] The user inputs a voice command into the device. For example, the user puts on earphones and says, "What's the weather like today?" The input data is voice data, and this is the starting point for the system's processing.

[1101] Step 2:

[1102] The device captures the user's voice. Specifically, it records the voice using the smartphone microphone or the earphone microphone. The input data is the voice data from the previous step, and the output data is the captured voice data.

[1103] Step 3:

[1104] The device sends the captured audio data to the server using a communication protocol such as an HTTP POST request or WebSocket. The input data is the captured audio data, and the output data is the audio data sent to the server.

[1105] Step 4:

[1106] The server analyzes the received voice data. Specifically, it converts the voice data into text data using a voice recognition engine (such as Google Cloud Speech-to-Text API). The input data is the voice data sent to the server, and the output data is text data. For example, the voice data "What's the weather like today?" is converted into the text data "What's the weather like today?"

[1107] Step 5:

[1108] The server selects an appropriate generative AI model based on the converted text data. The input data is the text data from the previous step, and it processes it to select a generative AI model (for example, a GPT-3 model for weather information). The output data is the selected generative AI model.

[1109] Step 6:

[1110] The server generates response data using the selected generative AI model. Specifically, it uses a generative AI such as the GPT-3 model to generate an optimal response to the text data. The input data is the selected generative AI model and the text data, and the output data is the generated response data. For example, in response to the question "What's the weather like today?", the generated response is "It's sunny today."

[1111] Step 7:

[1112] The server sends the generated response data to the user terminal. A communication protocol such as HTTP response or WebSocket is used for transmission. The input data is the generated response data, and the output data is the response data sent to the user terminal.

[1113] Step 8:

[1114] The terminal decodes the response data received from the server. The input data is the response data sent from the server, and the output data is the decoded text data.

[1115] Step 9:

[1116] The device converts the decoded text data into voice data using a speech synthesis engine. Specifically, a speech synthesis engine such as Amazon Polly is used. The input data is the decoded text data, and the output data is voice data. For example, the text "The weather is sunny today" is converted into voice data.

[1117] Step 10:

[1118] The device plays the converted audio data. Specifically, the audio is played back to the user through the smartphone's speaker or earphones. The input data is audio data, and the final output is the audio that the user hears.

[1119] Step 11:

[1120] The user receives the voice response played back from the terminal, and can hear a voice response such as "The weather is sunny today."

[1121] (Application example 1)

[1122] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1123] In factory operations, workers had to use conventional computers and touch panels to operate robots. This meant that operation took time at the workplace, resulting in reduced work efficiency. Furthermore, manual operation could lead to human error, which also led to reduced productivity and quality control issues. A system was needed to solve these issues and enable workers to operate robots more efficiently and reliably.

[1124] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1125] In this invention, the server includes means for capturing a user's voice input, means for analyzing the captured voice input and converting it into text data, means for selecting a generative AI model based on the text data, means for generating response data using the generative AI model, means for sending the generated response data to a user, means for causing a machine to execute the generated response data as an instruction, and means for a factory worker to give voice instructions and for the robot to execute the instructions. This allows the worker to instantly operate the robot using voice instructions, improving work efficiency and reducing operation errors.

[1126] "User voice input" refers to collecting the voice uttered by the user as data.

[1127] "Capture" is a method of obtaining audio as digital data.

[1128] "Text data" is information that is generated by analyzing voice input and expressed as a string of characters.

[1129] A "generative artificial intelligence model" is a machine learning model that generates appropriate responses to specific data or questions.

[1130] "Response data" refers to data corresponding to answers or instructions generated by a generative artificial intelligence model.

[1131] "Mobile terminal" refers to mobile devices in general, including smartphones, tablets, etc.

[1132] A "microphone" is a device that converts sound into an electrical signal.

[1133] A "headset" is a device that allows for the capture and playback of sound.

[1134] A "speech synthesis engine" is a technology that converts text data into voice data.

[1135] "Audio data" is information obtained by digitizing an audio signal.

[1136] A "machine" is a device designed to perform a specified task.

[1137] An "instruction" is a command to perform a particular action or operation.

[1138] A "factory worker" is a human worker who performs work in a factory.

[1139] A "robot" is an automated device for carrying out programmed instructions.

[1140] This invention is a system in which a factory worker gives voice instructions and a robot performs actions based on those instructions. The system includes the following components:

[1141] server

[1142] The server receives a voice data request sent from the user's device. First, the server analyzes this voice data and converts it into text data using a speech recognition engine (e.g., Google Speech-to-Text API). Next, based on the converted text data, it selects the required generative AI model (e.g., OpenAI's GPT-4). For example, if a worker verbally instructs, "Tell me the next work step," the server selects an appropriate AI model and generates response data for the next step. This response data is provided in text format and is used as a command for the robot to execute. Finally, the server sends this response data to the user's device and the robot.

[1143] Terminal

[1144] The terminal is responsible for capturing the user's voice input. It captures the voice of the worker speaking into the microphone of a headset or mobile device and sends this captured voice data to the server. A communication protocol such as HTTP request or WebSocket is used for transmission. When the terminal receives response data from the server, it decodes the received data and obtains the response in text format. It then uses a speech synthesis engine (e.g., Google Cloud Text-to-Speech) to convert this text data into speech and play it back to the worker. For example, if a worker issues an instruction such as "Pick up a part from the shelf," the terminal converts the server's response, "Perform the next step," into speech and plays it back to the worker.

[1145] User

[1146] The user is a factory worker who issues a voice command. For example, the worker asks, "What is the next work process?" This voice command is captured by the terminal and sent to the server. As a result of the server's processing, the user receives a voice response such as, "The next work process is assembly." This simplifies operations for the user through voice instructions, reduces manual operation errors, and improves work efficiency.

[1147] Specific examples

[1148] When a worker asks "What's the next step?", the following prompt is input to the generative AI model:

[1149] The worker says, "Tell me the next step." Please explain the following steps.

[1150] As a result, the generative AI model generates a response saying, "The next work process is assembly. The robot will move the parts to the assembly station." The generated response data is converted into voice data by a speech synthesis engine and played back to the worker. At the same time, an instruction is given to the robot to move the parts to the assembly station as the next process.

[1151] This system allows workers in the factory to operate robots using voice commands, improving work efficiency and reducing operating errors.

[1152] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1153] Step 1:

[1154] The user issues instructions to the worker in the form of voice, speaking into a headset or a microphone on a mobile device. This voice input is captured by the device and obtained as digital data. As input, there is voice data. As output, this voice data is captured and ready to be sent to a server.

[1155] Step 2:

[1156] The device sends the captured audio data to the server. The communication protocol is HTTP request or WebSocket. The input is the captured audio data. The output is the server receiving the request for audio data.

[1157] Step 3:

[1158] The server analyzes the received voice data and converts it into text data using a speech recognition engine (Google Speech-to-Text API). The input is voice data, and the output is the voice data converted into text data.

[1159] Step 4:

[1160] The server selects an appropriate generative AI model (OpenAI's GPT-4) based on the converted text data. The input is the text data. The output is the selected generative AI model.

[1161] Step 5:

[1162] The server inputs a prompt statement to the selected generative AI model and generates response data. The inputs include the selected generative AI model and the prompt statement. Examples of prompt statements are as follows:

[1163] The worker says, "Tell me the next step." Please explain the following steps.

[1164] The output is the generated response data.

[1165] Step 6:

[1166] The server transmits the generated response data to the user terminal and the robot. The generated response data is the input. The user terminal and the robot receive the response data as the output.

[1167] Step 7:

[1168] The device decodes the received response data and converts it into voice data using a speech synthesis engine (Google Cloud Text-to-Speech). The input is the response data (in text format). The output is voice data.

[1169] Step 8:

[1170] The terminal plays the voice data and notifies the worker of the response. At the same time, instructions based on the response data are transmitted to the robot, which then carries out those instructions. The inputs are the voice data and instruction data for the robot. The outputs are a voice notification to the worker and the robot's execution of an action.

[1171] In this way, a system for operating robots in a factory using voice commands is realized, allowing workers to operate robots easily and reliably, improving work efficiency.

[1172] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1173] The present invention is a system that not only captures a user's voice input, converts it into text data, and selects an appropriate generative AI model to generate response data, but also combines it with an emotion engine that recognizes the user's emotions to provide more personalized responses. Below, the program processing of this system is explained in natural language.

[1174] server

[1175] The server receives a voice data request from the user terminal. First, the server analyzes the voice data and converts it into text data using a speech recognition engine. At the same time, it uses an emotion engine to recognize the user's emotional state from the voice data. For example, it determines whether the user is happy, angry, or sad based on characteristics such as the tone, speed, and accent of the voice.

[1176] Based on the text data and the recognized emotional state, the server selects an appropriate generative AI model. Depending on the user's request and emotional state, the generative AI model that generates the optimal response is identified and selected. For example, in response to the request "What's the weather today?", if the user is calm, a normal response is provided, but if the user is angry, a more considerate response is generated.

[1177] The generated response data is generated in the form of, for example, "The weather is sunny today. Have a nice day." The server transmits this response data to the user terminal.

[1178] Terminal

[1179] The device is responsible for capturing the user's voice input: it captures the user's voice as they speak into earphones or a smartphone microphone, and transmits this captured voice data to a server using a communication protocol such as HTTP requests or WebSocket.

[1180] When the device receives the response data from the server, it decodes the received data and obtains data that reflects the text response content and consideration according to the user's emotions. It then uses a speech synthesis engine to convert this text into speech and plays it back to the user. For example, when the user issues a voice command such as "Play the next song," the device receives a response from the server saying, "Playing the next song. Refresh your mood with good music," converts it into speech, and plays it back.

[1181] User

[1182] The user issues a voice command to the device. For example, they put on earphones and ask, "What time is it now?" This voice command is captured by the device and sent to the server. As a result of the server's processing, the user receives a voice response such as, "It is now 3:00 p.m. Have a nice afternoon."

[1183] Additionally, users can customize the device's design using a smartphone app. Generative AI generates new designs based on the user's customization requests and applies them to the device in real time. An emotion engine takes into account the user's current emotions and changes the color and theme of the design, providing a more personalized experience for the user.

[1184] This system allows users to enjoy a consistent experience and smooth operation across multiple devices, while also reducing development costs for device manufacturers by standardizing the process of selecting generative AI models and emotion engines and generating responses.

[1185] The processing flow will be explained below.

[1186] Step 1:

[1187] The device captures the user's voice input: as the user speaks voice commands into earphones or a smartphone microphone, the voice data is recorded in real time and stored in a buffer.

[1188] Step 2:

[1189] The device encodes the captured audio data and sends it to the server using a communication protocol such as an HTTP request or WebSocket.

[1190] Step 3:

[1191] The server receives the voice data sent from the terminal and stores it in the server's internal storage.

[1192] Step 4:

[1193] The server uses a voice recognition engine to convert the received voice data into text data, thereby obtaining the user's voice commands in text format.

[1194] Step 5:

[1195] The server inputs the text data into an emotion engine to recognize the user's emotional state. The emotion engine analyzes the tone, speed, and accent of the voice to determine whether the user is happy, angry, or sad.

[1196] Step 6:

[1197] The server selects an appropriate generative AI model based on the text data and the recognized emotional state. The server analyzes the text and identifies and selects the generative AI model that best suits the user's request and emotional state.

[1198] Step 7:

[1199] The server generates response data using the selected generative AI model. For example, in response to a request like "What's the weather today?", if the user is calm, it will provide a normal response, but if the user is angry, it will generate a more considerate response, such as "The weather is sunny today. Have a nice day."

[1200] Step 8:

[1201] The server transmits the generated response data to the terminal, which is then encoded in an appropriate format and transmitted to the terminal via a communication protocol.

[1202] Step 9:

[1203] The terminal receives the response data sent from the server, decodes the received data, and stores the response content in text format in the terminal.

[1204] Step 10:

[1205] The device converts the received text response into audio data using a speech synthesis engine, which analyzes the text and generates natural-sounding speech.

[1206] Step 11:

[1207] The device plays the generated voice data to the user. The user can hear the voice response through earphones or speakers. For example, a voice such as "It is now 3:00 p.m. Have a nice afternoon" may be played.

[1208] Step 12:

[1209] The user can then act on the generated voice response, for example by issuing the next voice command or customizing the device using generative AI and an emotion engine. This process is repeated, providing the user with a consistent experience.

[1210] Example 2

[1211] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1212] Conventional speech recognition systems simply convert a user's voice input into text and generate responses, but are unable to provide responses that take the user's emotions into account. This can lead to stress and an unpleasant experience when trying to obtain the information they need. Furthermore, because responses cannot be generated based on the user's emotions, responses are not personalized enough, which can result in a decrease in user satisfaction.

[1213] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for capturing a user's voice input, means for analyzing the captured voice input and converting it into text data, means for recognizing the user's emotional state from the voice input, means for selecting a generative AI model based on the text data and the user's emotional state, means for generating response data using the selected generative AI model, and means for transmitting the generated response data to the user. This makes it possible to provide a personalized response that takes the user's emotions into consideration.

[1214] "Means for capturing user voice input" refers to devices and techniques for recording the voices uttered by a user.

[1215] The "means for analyzing and converting into text data" refers to the technology and system for analyzing voice data and converting it into corresponding character string data.

[1216] "Means for recognizing a user's emotional state" refers to techniques and systems that detect and identify a user's emotional state (e.g., joy, anger, sadness, etc.) from voice data.

[1217] "Means for selecting a generative artificial intelligence model" refers to techniques and systems for selecting the most appropriate artificial intelligence model based on the analyzed data.

[1218] "Means for generating response data" refers to techniques and systems that use selected artificial intelligence models to automatically generate appropriate responses to user input.

[1219] "Means for transmitting to the user" refers to communication technology and systems for transmitting the generated response data to the user's terminal.

[1220] A "microphone installed in a mobile terminal" refers to a voice input device built into a portable device such as a mobile phone or smartphone.

[1221] A "speech synthesis engine" refers to technology and systems for converting text data into speech data.

[1222] This invention is a system that captures a user's voice input, converts it into text data, and then selects an appropriate generative AI model to generate response data.By combining this with an emotion engine that recognizes the user's emotions, it provides more personalized responses.

[1223] server

[1224] The server receives a voice data request sent from the user's device. First, the server analyzes this voice data and converts it into text data using a general-purpose voice recognition engine (such as Google's Speech-to-Text API). At the same time, it uses an emotion recognition engine (such as general emotion analysis software) to recognize the user's emotional state from the voice data. It determines the user's emotion based on characteristics such as the tone, speed, and accent of the voice. Based on this information, it selects the optimal generative artificial intelligence model and generates personalized response data using OpenAI's GPT-3 or similar. The generated response data is then sent to the user's device.

[1225] For example, if a user says, "What time is the next meeting?", the server converts the speech to text and then analyzes it with an emotion engine to recognize that the user is impatient. It then selects an appropriate generative AI model and provides a personalized response (e.g., "The next meeting is at 2 p.m. Take your time to prepare and you'll be fine.").

[1226] Example prompt sentence:

[1227] Generate a relaxing response for users who impatiently ask, "What time is the next meeting?"

[1228] Terminal

[1229] The terminal is responsible for capturing the user's voice input. It captures the user's speech into a microphone on a portable device (e.g., a smartphone or mobile phone) and sends this voice data to a server using a communication protocol (e.g., HTTP request or WebSocket). It decodes the received response data and obtains the response content in text format. It then converts this text into speech using a speech synthesis engine (e.g., Amazon's Polly) and plays it back to the user.

[1230] As a specific example, when a user issues a voice command such as "Play the next song," the device sends this voice data to the server, receives a response from the server saying, "Playing the next song. Refresh your mood with good music," converts it into voice, and plays it.

[1231] User

[1232] The user issues a voice command to the device. For example, they put on earphones and ask, "What time is it now?" The voice command is captured by the device and sent to the server. As a result of the server's processing, the user receives a voice response such as, "It is now 3:00 p.m. Have a nice afternoon."

[1233] Additionally, users can customize the device's design using a smartphone app: for example, an emotion engine can recognize when a user is relaxing and change the device's theme to softer colors.

[1234] This system allows users to enjoy a consistent experience and smooth operation across multiple devices, while also reducing development costs for device manufacturers by standardizing the process of selecting generative AI models and emotion engines and generating responses.

[1235] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1236] Step 1: Capture the user's voice input

[1237] Terminal

[1238] The terminal captures in real time the user's voice as they speak into a microphone installed on a portable device (e.g., a smartphone). The input is the user's voice data, and the output is a byte stream that converts this voice data into a digital format. This byte stream is stored in temporary storage.

[1239] Specific behavior:

[1240] A user issues the voice command, "What time is the next meeting?"

[1241] The device captures this voice command and converts it into digital audio data in .wav format.

[1242] Step 2: Send the audio data to the server

[1243] Terminal

[1244] The device sends the captured audio data to the server. The input is digital audio data, and the output is obtained by sending this data to the server using a communication protocol such as HTTP requests or WebSockets.

[1245] Specific behavior:

[1246] The device sends the saved audio data to the server in the body of an HTTP POST request.

[1247] Step 3: Convert the audio data to text

[1248] server

[1249] The server receives the voice data sent from the device and converts it into text data using a voice recognition engine (e.g., a general-purpose voice recognition engine). The input is the voice data sent to the server, and the output is the analyzed text data. This text data is stored in internal memory.

[1250] Specific behavior:

[1251] The server receives the voice data "What time is the next meeting?"

[1252] Call Google's Speech-to-Text API to generate text data such as "What time is the next meeting?"

[1253] Step 4: Recognize your emotional state

[1254] server

[1255] The server passes the received voice data to an emotion recognition engine to recognize the emotional state. The input is the voice data, and the output is the user's emotional state (e.g., happy, anxious).

[1256] Specific behavior:

[1257] The emotion engine analyzes the voice data and recognizes emotional states such as "anxious."

[1258] The recognized emotional state is kept in an internal memory.

[1259] Step 5: Select a generative AI model

[1260] server

[1261] The server generates prompt sentences and selects an appropriate generative AI model based on the text data and the recognized emotional state. The input is the text data and the emotional state, and the output is the selected generative AI model.

[1262] Specific behavior:

[1263] Prompt Sentence: Generate a relaxing response for a user who impatiently asks, "What time is the next meeting?"

[1264] Select the best generative artificial intelligence model, such as OpenAI's GPT-3.

[1265] Step 6: Generate response data

[1266] server

[1267] The server generates response data using the selected generative AI model. The input is the prompt sentence, and the output is the generated response data.

[1268] Specific behavior:

[1269] Enter a prompt into GPT-3 and it will generate a response like, "The next meeting is at 2 p.m. Take your time to prepare and you'll be fine."

[1270] Step 7: Send the response data to the user device

[1271] server

[1272] The server sends the generated response data to the user terminal. The input is the generated response data, and the output is the sent response data.

[1273] Specific behavior:

[1274] The server structures the response data in JSON format and sends it to the terminal as an HTTP response.

[1275] Step 8: Play the response aloud

[1276] Terminal

[1277] The device decodes the response data received from the server and converts it into voice data using a speech synthesis engine (e.g., Amazon's Polly). The input is the text response data received from the server, and the output is voice data. This voice data is then played back to the user through the speaker.

[1278] Specific behavior:

[1279] The device receives response data saying, "The next meeting is at 2:00 p.m. Take your time to prepare."

[1280] The text is converted into speech using a speech synthesis engine and played through speakers or earphones.

[1281] (Application example 2)

[1282] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1283] Conventional speech recognition systems simply convert a user's voice input into text and generate responses based on that text. However, they can only provide a uniform response without considering the user's emotions, making it difficult to provide a personalized experience for the user. Furthermore, especially in autonomous vehicles, there are many situations where responses that correspond to the passenger's emotional state are required. Therefore, there is a need to build a system that can provide more personalized responses based on the user's voice input and their emotional state.

[1284] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1285] In this invention, the server includes means for capturing a user's voice input, means for analyzing the captured voice input and converting it into text data, means for recognizing emotions based on the text data and voice characteristics, means for selecting a generative AI model based on the text data and the recognized emotional state, means for generating personalized response data using the generative AI model, means for transmitting the generated response data to the user, means for converting the generated response data into voice data, and means for playing the voice data to the user, thereby making it possible to provide a personalized experience adapted to the user's emotional state.

[1286] A "means for capturing user voice input" is a device or component for capturing voice uttered by a user and obtaining the voice data.

[1287] "Means for analyzing captured voice input and converting it into text data" refers to devices or software that analyzes captured voice data and converts it into corresponding text data.

[1288] "Means for recognizing emotions based on text data and voice characteristics" refers to devices and software for analyzing and recognizing a user's emotional state from the characteristics of text data and voice data.

[1289] The "means for selecting a generative artificial intelligence model based on text data and a recognized emotional state" refers to a device or software for automatically selecting an optimal artificial intelligence model based on the recognized text data and an emotional state.

[1290] "Means for generating personalized response data using a generative artificial intelligence model" refers to a device or software that uses a selected artificial intelligence model to generate response data in response to a user's request and emotional state.

[1291] The "means for transmitting the generated response data to the user" refers to a communication device or software for transferring the generated response data to the user device.

[1292] The "means for converting the generated response data into voice data" refers to a voice synthesizer or software for converting the generated text response into voice form.

[1293] The "means for reproducing audio data to the user" refers to a reproduction device or component that allows the user to listen to the audio data.

[1294] An "in-vehicle acoustic input device" is a microphone or other acoustic sensor installed inside a motor vehicle to capture sounds made by passengers.

[1295] "Playing back the voice data generated by the voice synthesizer through an audio playback device inside the vehicle" means letting the user hear the voice data generated by the voice synthesizer using a speaker system installed inside the vehicle.

[1296] The present invention is a system installed in an autonomous vehicle that captures a user's voice input, converts it into text data, recognizes emotions based on the text data and voice characteristics, and generates an optimal response. An embodiment of this system is described in detail below.

[1297] server

[1298] The server receives voice data sent from the user's device. It converts the voice data into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text). At the same time, it recognizes the user's emotional state from the voice data using an emotion engine (e.g., IBM Watson Tone Analyzer). Based on the recognized text data and emotional state, it selects a generative AI model (e.g., OpenAI GPT-3) and generates appropriate response data. The generated response data might be in the form of, for example, "Your next destination is Ginza. Enjoy a leisurely drive." The server then sends this response data to the autonomous vehicle's device.

[1299] Terminal

[1300] The terminal captures the user's voice input using an audio input device (microphone) installed in the autonomous vehicle. This captured voice data is sent to the server using a communication protocol such as an HTTP request or WebSocket. When the terminal receives response data from the server, it decodes the received data and obtains the response content in text format. It then converts this text into speech using a speech synthesis engine (e.g., Amazon Polly) and plays it back to the user through the audio playback device (speaker) in the autonomous vehicle. For example, if the user issues a voice command such as "Play the next song," the terminal receives a response from the server saying, "Playing the next song. Relax," converts this into speech, and plays it back.

[1301] User

[1302] The user issues a voice command to the audio input device in the autonomous vehicle. For example, they might ask, "Set my next destination." This voice command is captured by the device and sent to the server. As a result of processing by the server, the user receives a voice response such as, "Your next destination is Ginza. Enjoy a leisurely drive." Additionally, the user can customize the in-vehicle infotainment system using a smartphone app. New settings based on the user's customization requests are generated using generative AI and applied to the vehicle's systems in real time. The emotion engine takes into account the user's current emotions and changes the music and lighting theme to provide a more personalized experience for the user.

[1303] Specific examples

[1304] When a passenger asks "Where's my next destination?", an example of the server's processing is as follows:

[1305] Input prompt: "Where's my next destination?"

[1306] The server converts the speech to text: "Where's my next destination?"

[1307] Sentiment analysis result: "Calm"

[1308] Generative AI model selection and response generation: "Your next destination is Ginza. Enjoy a leisurely drive."

[1309] Speech synthesis and response: "Your next destination is Ginza. Enjoy a leisurely drive." is played over the speaker in the self-driving vehicle.

[1310] This system will enable a more personalized experience inside self-driving vehicles, making travel smoother and more comfortable.

[1311] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1312] Step 1:

[1313] The terminal captures the user's voice input, i.e., the user speaks a voice command into a microphone in the car and acquires the voice data.

[1314] Input: User's voice command

[1315] Output: Captured audio data

[1316] Step 2:

[1317] The device sends the captured audio data to the server using HTTP requests and WebSockets as the communication protocol.

[1318] Input: Captured audio data

[1319] Output: Sending audio data to the server

[1320] Step 3:

[1321] The server converts the received voice data into text data using a speech recognition engine, such as Google Cloud Speech-to-Text.

[1322] Input: Incoming audio data

[1323] Output: Text data

[1324] Step 4:

[1325] The server analyzes the text data and voice characteristics using an emotion engine, such as IBM Watson Tone Analyzer, to recognize the user's emotional state.

[1326] Input: Text data and audio data features

[1327] Output: User's emotional state

[1328] Step 5:

[1329] The server selects the optimal generative AI model based on the recognized text data and emotional state, using OpenAI GPT-3 or other models.

[1330] Input: Text data, user's emotional state

[1331] Output: The selected generative AI model

[1332] Step 6:

[1333] The server generates personalized response data using the selected generative AI model.

[1334] Input: Text data, the user's emotional state, and a generative AI model

[1335] Output: Generated response data

[1336] Step 7:

[1337] The server transmits the generated response data to the terminal.

[1338] Input: Generated response data

[1339] Output: Send response data to the terminal

[1340] Step 8:

[1341] The device uses a speech synthesis engine, such as Amazon Polly, to convert the received response data from text to voice data.

[1342] Input: Received response data (text format)

[1343] Output: Audio data

[1344] Step 9:

[1345] The terminal reproduces the generated voice data through the speaker in the vehicle and provides it to the user.

[1346] Input: Generated audio data

[1347] Output: A voice response to the user

[1348] In this way, the system personalizes the user experience in an autonomous vehicle, ensuring a smooth and comfortable journey.

[1349] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1350] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1351] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1352] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1353] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1354] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1355] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1356] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1357] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1358] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1359] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1360] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1361] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1362] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1363] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1364] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1365] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1366] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1367] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1368] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1369] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1370] The following is further disclosed regarding the above embodiment.

[1371] (Claim 1)

[1372] means for capturing a user's voice input;

[1373] means for analyzing and converting the captured voice input into text data;

[1374] means for selecting a generative artificial intelligence model based on the text data;

[1375] means for generating response data using the generative artificial intelligence model;

[1376] means for transmitting the generated response data to a user;

[1377] A system including:

[1378] (Claim 2)

[1379] 10. The system of claim 1, wherein the means for capturing the user's voice input uses a microphone located on the mobile terminal.

[1380] (Claim 3)

[1381] 2. The system according to claim 1, wherein the means for transmitting the response data to the user converts the response data into voice data using a voice synthesis engine and plays the voice data to the user.

[1382] (Claim 4)

[1383] 10. The system of claim 1, wherein the generative artificial intelligence model includes means for generating a design for a device based on a user's customization requests.

[1384] (Claim 5)

[1385] 10. The system of claim 1, wherein the generative artificial intelligence model includes means for generating responses based on a user's voice commands to unify the experience across multiple devices.

[1386] "Example 1"

[1387] (Claim 1)

[1388] means for capturing a user's voice input;

[1389] means for analyzing and converting the captured voice input into text data;

[1390] means for selecting an artificial intelligence model generated based on the text data;

[1391] means for generating response data using the generated artificial intelligence model;

[1392] means for transmitting the generated response data to a user;

[1393] means for decoding the response data as text data;

[1394] means for using a speech synthesis engine to convert the decoded text data into speech data;

[1395] means for reproducing the converted audio data;

[1396] A system including:

[1397] (Claim 2)

[1398] 10. The system of claim 1, wherein the means for capturing the user's voice input uses a microphone located on the mobile terminal.

[1399] (Claim 3)

[1400] 2. The system according to claim 1, wherein the means for transmitting the response data to the user converts the response data into voice data using a voice synthesis engine and plays the voice data to the user.

[1401] "Application Example 1"

[1402] (Claim 1)

[1403] means for capturing a user's voice input;

[1404] means for analyzing and converting the captured voice input into text data;

[1405] means for selecting a generative artificial intelligence model based on the text data;

[1406] means for generating response data using the generative artificial intelligence model;

[1407] means for transmitting the generated response data to a user;

[1408] means for causing a machine to execute the generated response data as instructions;

[1409] A means for factory workers to give voice instructions and for the robot to execute the instructions;

[1410] A system including:

[1411] (Claim 2)

[1412] 10. The system of claim 1, wherein the means for capturing the user's voice input uses a microphone and a headset provided on the mobile terminal.

[1413] (Claim 3)

[1414] 2. The system according to claim 1, wherein the means for transmitting the response data to the user converts the response data into voice data using a voice synthesis engine and plays the voice data to the user.

[1415] "Example 2: Combining Emotion Engines"

[1416] (Claim 1)

[1417] means for capturing a user's voice input;

[1418] means for analyzing and converting the captured voice input into text data;

[1419] means for recognizing a user's emotional state from said speech input;

[1420] means for selecting a generative artificial intelligence model based on the text data and the user's emotional state;

[1421] means for generating response data using the selected generative artificial intelligence model;

[1422] means for transmitting the generated response data to a user;

[1423] A system including:

[1424] (Claim 2)

[1425] 10. The system of claim 1, wherein the means for capturing the user's voice input uses a microphone located on the mobile terminal.

[1426] (Claim 3)

[1427] 2. The system according to claim 1, wherein the means for transmitting the response data to the user converts the response data into voice data using a voice synthesis engine and plays the voice data to the user.

[1428] "Application example 2 when combining emotion engines"

[1429] (Claim 1)

[1430] means for capturing a user's voice input;

[1431] means for analyzing and converting the captured voice input into text data;

[1432] means for recognizing emotions based on the text data and voice characteristics;

[1433] means for selecting a generative artificial intelligence model based on the text data and the recognized emotional state;

[1434] means for generating personalized response data using the generative artificial intelligence model;

[1435] means for transmitting the generated response data to a user;

[1436] means for converting the generated response data into voice data;

[1437] means for playing the audio data to a user;

[1438] A system including:

[1439] (Claim 2)

[1440] 10. The system of claim 1, wherein the means for capturing the user's voice input uses an audio input device located within the vehicle.

[1441] (Claim 3)

[1442] 2. The system according to claim 1, wherein the means for transmitting the response data to the user reproduces the response data as voice data generated by a voice synthesizer through an audio reproduction device in the vehicle. [Explanation of symbols]

[1443] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for capturing a user's voice input; means for analyzing and converting the captured voice input into text data; means for selecting a generative artificial intelligence model based on the text data; means for generating response data using the generative artificial intelligence model; means for transmitting the generated response data to a user; A system including:

2. 10. The system of claim 1, wherein the means for capturing the user's voice input uses a microphone located on the mobile terminal.

3. 2. The system according to claim 1, wherein the means for transmitting the response data to the user converts the response data into voice data using a voice synthesis engine and plays the voice data to the user.

4. The system of claim 1 , wherein the generative artificial intelligence model includes means for generating a design for a device based on a user's customization requests.

5. The system of claim 1 , wherein the generative artificial intelligence model includes means for generating responses based on a user's voice commands to unify the experience across multiple devices.

Images