System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring an audio input; means for converting the acquired audio input into text data; means for analyzing the text data and understanding a user's intention; means for acquiring and generating information based on the user's intention; means for receiving the generated information as text data and converting the text data into audio speech synthesis; and means for providing the audio output to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Users who have never used generative AI, especially those unfamiliar with PC operation such as the elderly and children, have difficulty using generative AI. Conventional systems require keyboard input and are difficult to operate intuitively, which means that many potential users cannot enjoy the convenience of generative AI. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following system. By providing a means for acquiring voice input, questions and instructions can be asked or given by voice. A speech recognition means, which converts the acquired voice input into text data, analyzes the content of the user's voice input and records it as text information. Next, a natural language processing means, which analyzes the text data and understands the user's intent, is used to accurately grasp the user's questions and requests. This natural language processing means uses a machine learning algorithm to improve the accuracy of intent analysis. Furthermore, by using a generation means, which acquires and generates information based on the user's intent, information that meets the user's requests is provided. The generation means acquires information from an external database in real time, providing constantly up-to-date information. Finally, a speech synthesis means, which receives the generated information as text data and converts it into voice data, provides answers in a voice format that is easy for the user to hear. This realizes a system that allows even people who are unfamiliar with using computers to easily enjoy the convenience of generation AI.

[0006] The "means for acquiring voice input" is a device or function that records the user's speech as voice data via a microphone.

[0007] "Speech recognition means" refers to a device or software that analyzes acquired voice data and converts it into text data.

[0008] "Natural language processing means" refers to algorithms and software that analyze text data and understand the user's intent.

[0009] The "generation means" is a device or software that acquires necessary information based on the user's intentions and generates an appropriate answer.

[0010] The "voice synthesis means" is a device or software that converts the generated text data into voice data and provides it to the user.

[0011] "User" refers to a person who uses this system to ask questions or give instructions.

[0012] "Server" refers to a central processing unit or system that processes data, generates information, or analyzes information.

[0013] A "terminal" is a device that interfaces with a user, receives voice input, and provides voice responses. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention is a system including means for acquiring a voice input, means for speech recognition, means for natural language processing, means for generating, means for synthesizing speech, and means for providing a voice response. The following describes an embodiment of the system.

[0036] Acquiring voice input

[0037] The user speaks a question or instruction to the terminal. For example, they might say, "Tell me what the weather will be tomorrow." This speech is picked up through the terminal's built-in microphone or a connected external microphone. The means for acquiring speech input stores the speech data in a buffer and sends it to the speech recognition means, which is the next step.

[0038] Voice Recognition

[0039] The acquired voice data is sent to the terminal's voice recognition means. This means converts the acquired voice data into text data. Existing technology (e.g., voice recognition API) can be used for the voice recognition means. In this step, text data such as "Tell me what the weather will be like tomorrow" is generated.

[0040] Natural Language Processing

[0041] The text data generated by the speech recognition means is sent from the terminal to the server. The server analyzes the text data using natural language processing means to understand the user's intent. This natural language processing means can use a machine learning algorithm (e.g., a natural language processing API). In this step, the text data is interpreted as a "weather information request."

[0042] Information Acquisition and Generation

[0043] After the user's intent is interpreted, the server's generator retrieves and generates information. This generator retrieves information in real time from external databases or APIs (e.g., weather information APIs). Based on the retrieved information, the generator generates a natural and relevant response. In this step, a response such as "Tomorrow will be sunny" is generated.

[0044] Audio conversion

[0045] The text data generated by the generation means is sent to the terminal. The terminal converts the text data into voice data using a voice synthesis means. This means can use existing technology (e.g., a voice synthesis API). In this step, voice data saying "Tomorrow will be sunny" is generated.

[0046] Providing voice responses

[0047] Finally, the generated voice data is provided to the user through the speaker of the terminal, and the terminal plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand manner.

[0048] Specific examples

[0049] As an example, the flow when a user asks "What's the weather going to be like tomorrow?" is shown below. The user's voice input is acquired by the terminal and converted into text data "What's the weather going to be like tomorrow?" by the speech recognition means. This text data is sent to the server and interpreted as a "weather information request" by the natural language processing means. Next, the generation means calls the weather information API to acquire tomorrow's weather information, and generates a response such as "It will be sunny tomorrow." Finally, this response is converted into voice data by the voice synthesis means and provided to the user through the terminal's speaker.

[0050] In this way, the system of the present invention utilizes generative AI to provide the user with the necessary information through voice dialogue, providing an environment that is easy to use even for elderly people and children who are unfamiliar with using computers.

[0051] The processing flow will be explained below.

[0052] Step 1:

[0053] The user speaks to the device, saying, "Tell me what the weather will be tomorrow." The device picks up the voice data through its built-in microphone or a connected external microphone.

[0054] Step 2:

[0055] The acquired voice data is sent to the device's voice recognition means. This means converts the voice data into text data. Specifically, it calls a voice recognition API to generate text data such as "Tell me what the weather will be like tomorrow."

[0056] Step 3:

[0057] The text data generated by the speech recognition means is transmitted from the terminal to the server, which then passes the received text data to the natural language processing means.

[0058] Step 4:

[0059] The server's natural language processing means analyzes the text data and understands the user's intent. Specifically, it uses a machine learning algorithm to interpret it as a "weather information request."

[0060] Step 5:

[0061] After the user's intention is interpreted, the server's generating means operates. The server sends a request to an external weather information API to obtain weather information in real time.

[0062] Step 6:

[0063] Based on the data obtained from the weather information API, the server's generation means generates a response such as "It will be sunny tomorrow." The generated response is sent to the terminal as text data.

[0064] Step 7:

[0065] The device passes the received response to the speech synthesis means. This means converts text data into speech data. Specifically, it calls the speech synthesis API to generate speech data saying, "Tomorrow will be sunny."

[0066] Step 8:

[0067] The generated voice data is provided to the user through the speaker of the terminal, and the terminal plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand format.

[0068] By following these steps, users can obtain the necessary information using the generation AI while interacting with the system via voice.

[0069] Example 1

[0070] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0071] Many users today desire intuitive voice-based interfaces, but existing systems have issues with speech recognition accuracy and response speed. It is also difficult to perform the entire process from voice input to data search and response generation in real time, making them difficult to use, especially for users who are unfamiliar with technology, such as the elderly and children. Furthermore, current technologies require complex integration of different APIs and buffer management, making it difficult to achieve stable operation.

[0072] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0073] In this invention, the server includes means for acquiring voice input, speech recognition means for converting the acquired voice input into text data, natural language processing means for analyzing the text data and understanding the user's intention, information generation means for acquiring and generating information based on the user's intention, speech synthesis means for receiving the generated information as text data and converting it into voice data, and means for providing the voice data to the user. This enables the user to intuitively interact with the system using voice and obtain necessary information in real time.

[0074] "Means for acquiring voice input" refers to a device or method for acquiring voice uttered by a user and recording it as digital data.

[0075] "Speech recognition means" refers to technology or devices that convert acquired voice data into text data, and includes speech recognition APIs, etc.

[0076] "Natural language processing means" refers to technology or devices that analyze text data and understand the user's intentions and requests, and includes natural language processing APIs and machine learning algorithms.

[0077] "Information generation means" refers to technology or devices that acquire necessary information from external databases, APIs, etc. based on the user's intentions, and generate a response sentence to convey that information to the user.

[0078] "Speech synthesis means" refers to technology or devices that convert generated text data into voice data for output as voice, and includes speech synthesis APIs, etc.

[0079] The "means for providing audio data to a user" refers to a technique or device for making the generated audio data audible to a user through a speaker or other audio output device.

[0080] A "buffer" is a storage device that temporarily stores data to facilitate transmission and processing.

[0081] An "external database" is a database or API that exists outside the system and provides specific information.

[0082] A "machine learning algorithm" is a technique or method for training a model using large amounts of data to make appropriate predictions and classifications for new data.

[0083] MODE FOR CARRYING OUT THE INVENTION

[0084] The present invention is a system that allows a user to obtain information through a voice interface. The system comprises the following elements: means for obtaining voice input, means for speech recognition, means for natural language processing, means for generating information, means for speech synthesis, and means for providing voice data.

[0085] Acquiring voice input

[0086] The user speaks a question or instruction to the terminal. For example, the user might say, "Tell me what the weather will be like tomorrow." The terminal captures this voice using a built-in microphone or a connected external microphone and stores it in a buffer as digital voice data. The voice input means then transmits this voice data to the next step.

[0087] Voice Recognition

[0088] The device sends the acquired voice data to the voice recognition means. The device converts the voice data into text data using a voice recognition API (for example, Google Cloud Speech-to-Text). At this stage, the voice is converted into text data such as "Tell me what the weather will be tomorrow."

[0089] Natural Language Processing

[0090] The text data is sent from the device to the server. The server analyzes the text data using natural language processing means to understand the user's intent. This means can use a natural language processing API (e.g., OpenAI GPT-3). Through this step, the text data is interpreted as a "weather information request."

[0091] Information Acquisition and Generation

[0092] After the user's intent is interpreted, the server's information generation means retrieves and generates information in real time from external databases or APIs (e.g., weather information APIs). Based on the retrieved information, the generation means creates a natural and relevant response. At this stage, a response such as "Tomorrow will be sunny" is generated.

[0093] Audio conversion

[0094] The generated text data is sent from the server to the terminal. The terminal converts the text data into voice data using a voice synthesis means. This means can use a voice synthesis API (for example, Amazon Polly or Google Text-to-Speech). In this step, voice data saying "Tomorrow will be sunny" is generated.

[0095] Providing voice responses

[0096] Finally, the generated voice data is provided to the user through the device's speaker. The device plays back the voice data and conveys information to the user in an intuitive and easy-to-understand manner.

[0097] Specific examples

[0098] For example, if a user asks "What's the weather going to be like tomorrow?", the processing flow is as follows:

[0099] 1. The user says, "What's the weather going to be like tomorrow?"

[0100] 2. The device captures audio using the built-in microphone and stores it in a buffer.

[0101] 3. The device uses a voice recognition API to convert the voice into text data: "Tell me what the weather will be like tomorrow."

[0102] 4. The device sends this text data to the server.

[0103] 5. The server uses a natural language processing API to analyze the text data and understand the user's intent.

[0104] 6. The server retrieves weather information using an external database or API.

[0105] 7. The server generates a text response saying "Tomorrow will be sunny."

[0106] 8. The server sends the generated text data to the terminal.

[0107] 9. The device uses the speech synthesis API to generate voice data saying, "Tomorrow will be sunny."

[0108] 10. The device speaker plays the audio data and provides it to the user.

[0109] In this way, the system's specific processing flow allows intuitive and efficient processing of a series of processes from voice input to information acquisition and voice response. Examples of prompt sentences include the following:

[0110] "If a user asks about tomorrow's weather, explain how the system processes voice input and responds."

[0111] Using such prompts allows us to effectively leverage generative AI models to get fast and accurate responses.

[0112] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0113] Step 1:

[0114] The user speaks a question or instruction to the device. For example, they might say, "Tell me what the weather will be tomorrow." This voice is picked up as user input through the device's built-in microphone or an external microphone. The device stores this voice input in a buffer as digital voice data. This data becomes the input for the next process.

[0115] Step 2:

[0116] The device retrieves the voice data stored in the buffer and sends it to the voice recognition means. Specifically, it calls a voice recognition API (for example, Google Cloud Speech-to-Text) and converts this voice data into text data. At this stage, the input is digital voice data, and the output is text data such as "What's the weather going to be like tomorrow?"

[0117] Step 3:

[0118] The device sends the generated text data to the server. The server passes this text data to a natural language processing means. The server analyzes the text data using a natural language processing API (e.g., OpenAI GPT-3) to understand the user's intent. In this step, the input is the text data "Tell me what the weather will be tomorrow," and the output is data interpreted as a "weather information request."

[0119] Step 4:

[0120] The server uses information generation means to obtain and generate the necessary information based on the interpreted user intent. Specifically, it obtains weather data in real time from an external database or API (for example, a weather information API). In this process, the input is a "weather information request," and the output is a response such as "It will be sunny tomorrow." Natural language processing is performed based on the obtained weather information to generate the response.

[0121] Step 5:

[0122] The server sends the generated response sentence to the terminal as text data. The terminal receives this text data and converts it into voice data using a voice synthesis means. A voice synthesis API (for example, Amazon Polly or Google Text-to-Speech) is used as the voice synthesis means. In this step, the input is the text data "Tomorrow will be sunny," and the output is voice data.

[0123] Step 6:

[0124] The terminal plays the generated voice data through a speaker and provides it to the user. The user listens to this voice response and accepts it as an answer to their question. At this final stage, the input is voice data and the output is voice information that the user hears.

[0125] In this way, each processing step of the system is executed successively, thereby achieving a series of data processing steps from voice input to voice response.

[0126] (Application example 1)

[0127] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0128] Autonomous vehicles require systems that allow users to give voice instructions and easily access navigation and other information services. Current systems have difficulty accurately understanding the user's intentions and obtaining the necessary information in real time to provide it via voice. Furthermore, improving the accuracy and response speed of the voice interface for users while driving is also a challenge.

[0129] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0130] In this invention, the server includes means for acquiring voice input, speech recognition means for converting the acquired voice input into text data, natural language processing means for analyzing the text data and understanding the user's intention, generation means for acquiring and generating information based on the user's intention, speech synthesis means for receiving the generated information as text data and converting it into voice data, and means for providing the voice data to the user. This enables the user to give voice instructions to operate an autonomous vehicle and use navigation and other information services with high accuracy in real time.

[0131] "Means for acquiring voice input" refers to devices or software that collect the voice uttered by the user as data.

[0132] "Speech recognition means" refers to a device or software that converts acquired voice data into text data.

[0133] "Natural language processing means" refers to devices or software that analyze text data and understand the user's intent.

[0134] "Generation means" refers to a device or software that acquires necessary information based on the user's intentions and generates that information.

[0135] "Speech synthesis means" refers to a device or software that converts the generated information from text data into speech data.

[0136] The "means for providing audio data to a user" refers to a device or software for playing back the generated audio data and providing it to a user.

[0137] "Navigation and information services" refers to functions that provide route guidance and various information for autonomous vehicles.

[0138] "Multiple means for responding to voice-based operational instructions" refers to multiple devices and software that allow the user to give voice instructions and the system to respond accordingly.

[0139] A "machine learning algorithm" is a technology that learns patterns and features from data and automatically analyzes them.

[0140] An "external database" is a collection of data that exists outside the system and is referenced to provide required information.

[0141] "Acquiring information in real time" means instantly acquiring and providing information requested by a user.

[0142] The present invention provides a system for providing navigation and information services to an autonomous vehicle by allowing a user to give voice instructions for operation. The system includes a means for acquiring voice input, a speech recognition means, a natural language processing means, a generation means, a speech synthesis means, and a means for providing voice data to the user.

[0143] Acquiring voice input

[0144] The user speaks a question or instruction into a microphone installed in the vehicle. For example, the user might say, "Find a nearby restaurant." This speech is picked up by the vehicle's built-in microphone. The means for acquiring speech input stores this speech data in a buffer and sends it to the speech recognition means, which is the next step.

[0145] Voice Recognition

[0146] The acquired voice data is sent to the vehicle's voice recognition means. This means converts the acquired voice data into text data. Existing technology (e.g., SpeechRecognition library) can be used for the voice recognition means. In this step, the text data "Find nearby restaurants" is generated.

[0147] Natural Language Processing

[0148] The text data generated by the speech recognition means is sent from the vehicle's computer system to a server. The server uses natural language processing means to analyze the text data and understand the user's intent. This natural language processing means can use machine learning algorithms (e.g., Dialogflow API). In this step, the text data is interpreted as a "request for restaurant location information."

[0149] Information Acquisition and Generation

[0150] After the user's intent is interpreted, the server's generator retrieves and generates relevant information. This generator retrieves information in real time from external databases or APIs (e.g., Google Maps API). Based on the retrieved information, the generator generates a natural and relevant response. In this step, a response such as "The nearest restaurant is ____" is generated.

[0151] Conversion to audio data

[0152] The text data generated by the generation means is sent to the vehicle's computer system. The vehicle converts the text data into voice data using a voice synthesis means. This means can use existing technology (e.g., Google Cloud Text-to-Speech API). In this step, voice data such as "The nearest restaurant is ____" is generated.

[0153] Providing voice responses

[0154] Finally, the generated voice data is provided to the user through the vehicle's speakers, and the vehicle's system plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand manner.

[0155] As a specific example, when a user utters "Tell me where the nearest gas station is," the user's voice input is acquired by the vehicle and converted into text data "Tell me where the nearest gas station is" by speech recognition means. This text data is sent to the server and interpreted as a "request for gas station location information" by natural language processing means. Next, generation means refers to an external database to acquire information about nearby gas stations, and generates a response sentence such as "The nearest gas station is XX." Finally, this response sentence is converted into voice data by speech synthesis means and provided to the user through the vehicle's speaker.

[0156] Prompt Sentence Examples

[0157] "Find a nearby restaurant"

[0158] Increase the temperature inside the car

[0159] "Tell me where the nearest gas station is."

[0160] As a result, the system of the present invention provides an environment in which users can intuitively use various functions of an autonomous vehicle through voice control.

[0161] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0162] Step 1:

[0163] A user speaks a question or command into a microphone in an autonomous vehicle. For example, they say, "Find a nearby restaurant." The input is the user's voice, and the output is the voice data captured by the vehicle's microphone.

[0164] Step 2:

[0165] The terminal stores the acquired voice data in a buffer and sends it to the voice recognition means. The input is the voice data acquired by the microphone, and the output is the voice data sent to the voice recognition means.

[0166] Step 3:

[0167] The device's speech recognition means converts the voice data into text data. Existing speech recognition technology (e.g., the SpeechRecognition library) is used here. The input is voice data, and the output is text data such as "Find nearby restaurants."

[0168] Step 4:

[0169] The terminal sends text data to the server. The input is the text data, and the output is the text data sent to the server.

[0170] Step 5:

[0171] The server's natural language processing means analyzes the text data and understands the user's intent. At this stage, machine learning algorithms (e.g., Dialogflow API) are used. The input is text data, and the output is an interpretation result that includes the user's intent, such as "request restaurant location information."

[0172] Step 6:

[0173] The server's generation means retrieves relevant information based on the user's intent in real time from external databases or APIs (e.g., Google Maps API). The input is the result of interpreting the user's intent, and the output is relevant information, such as "location information of the nearest restaurant."

[0174] Step 7:

[0175] The server's generation means generates a natural and relevant response based on the acquired information. The input is the acquired information, and the output is a response sentence such as "The nearest restaurant is ____."

[0176] Step 8:

[0177] The server sends the generated response to the terminal. The input is the response, and the output is the response sent to the terminal.

[0178] Step 9:

[0179] The device's speech synthesis means converts the response sentence into voice data. At this stage, existing speech synthesis technology (e.g., Google Cloud Text-to-Speech API) is used. The input is text data, and the output is voice data such as "The nearest restaurant is ____."

[0180] Step 10:

[0181] The terminal provides voice data to the user through a speaker, where the input is the voice data and the output is the voice information provided to the user.

[0182] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0183] The present invention is a system including a means for acquiring a voice input, a voice recognition means, a natural language processing means, a generation means, an emotion engine, a voice synthesis means, and a means for providing a voice response. An embodiment of the system is shown below.

[0184] Acquiring voice input

[0185] The user speaks a question or instruction to the device. For example, "Tell me what the weather will be tomorrow." This speech is picked up through the device's built-in microphone or a connected external microphone. The means for acquiring speech input stores the speech data in a buffer and sends it to the speech recognition means and emotion engine, which are the next steps.

[0186] Voice Recognition

[0187] The acquired voice data is sent to the device's voice recognition means. This means converts the voice data into text data. Existing technology (e.g., voice recognition API) can be used for the voice recognition means. In this step, the text data "Tell me what the weather will be tomorrow" is generated.

[0188] Emotion recognition

[0189] The voice data is also sent to the emotion engine. The device's emotion engine analyzes the voice tone and speech patterns to recognize the user's emotions. For example, when a user says, "Tell me the weather tomorrow," the tone, speed, and strength of the voice are analyzed to determine emotions such as "excited" or "calm."

[0190] Natural Language Processing

[0191] The text data generated by the speech recognition means is sent from the terminal to the server. The server passes the received text data to the natural language processing means. The server's natural language processing means analyzes the text data and understands the user's intent. Specifically, it uses a machine learning algorithm to interpret it as a "weather information request." The natural language processing means also uses an emotion engine to analyze the user's emotions.

[0192] Information Acquisition and Generation

[0193] After the user's intent is interpreted, the server's generation means retrieves and generates information. This retrieves information in real time from external databases and APIs (e.g., weather information APIs). Based on the retrieved information, the generation means generates a natural and relevant response. In this process, it uses data from the emotion engine to adjust the response. For example, if the user is excited, it generates a response that includes calming words. In this step, the response generated is, "Tomorrow is sunny. It looks like it's going to be a very pleasant day."

[0194] Audio conversion

[0195] The text data generated by the generation means is sent to the terminal. The terminal converts the text data into voice data using a voice synthesis means. This means can use existing technology (e.g., a voice synthesis API). In this step, voice data is generated that says, "Tomorrow is sunny. It looks like it's going to be a very pleasant day."

[0196] Providing voice responses

[0197] Finally, the generated voice data is provided to the user through the speaker of the terminal, and the terminal plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand manner.

[0198] Specific examples

[0199] As an example, the flow when a user asks, "What's the weather going to be like tomorrow?" is shown below. The user's voice input is acquired by the terminal and converted into text data "What's the weather going to be like tomorrow?" by the speech recognition means. At the same time, the emotion engine analyzes the voice tone and recognizes that the user is excited. This text data is sent to the server and interpreted as a "weather information request" by the natural language processing means. Next, the generation means calls the weather information API to acquire tomorrow's weather information and generates a response such as "It's going to be sunny tomorrow. It looks like it's going to be a very pleasant day." Finally, this response is converted into voice data by the speech synthesis means and provided to the user through the terminal's speaker.

[0200] In this way, the system of the present invention improves the usability of the AI ​​generator by recognizing emotions in response to voice input and providing an appropriate response according to the user's emotional state, thereby providing an environment that can be easily used by elderly people and children who are unfamiliar with using computers.

[0201] The processing flow will be explained below.

[0202] Step 1:

[0203] The user speaks to the device, saying, "Tell me the weather tomorrow." The device acquires voice data through the built-in microphone or a connected external microphone. The means for acquiring voice input stores the voice data in a buffer and sends it to the speech recognition means and emotion engine, which are the next steps.

[0204] Step 2:

[0205] The acquired voice data is sent to the terminal's voice recognition means. This means converts the voice data into text data. Existing technology (e.g., voice recognition API) can be used for the voice recognition means. In this step, the text data "Tell me what the weather will be like tomorrow" is generated.

[0206] Step 3:

[0207] At the same time, the voice data is also sent to the device's emotion engine, which analyzes the voice tone and speech patterns to recognize the user's emotions. For example, when a user says, "Tell me the weather tomorrow," the engine analyzes the tone, speed, and volume of the voice to determine emotions such as "excited" or "calm."

[0208] Step 4:

[0209] The text data generated by the speech recognition means is sent from the terminal to the server. The server passes the received text data to the natural language processing means. The server's natural language processing means analyzes the text data and understands the user's intent. Specifically, it uses a machine learning algorithm to interpret the text data as a "weather information request."

[0210] Step 5:

[0211] At the same time as the results of natural language processing, emotional information analyzed by the device's emotion engine is also sent to the server, allowing the server to simultaneously grasp the user's intentions and emotional state.

[0212] Step 6:

[0213] After the user's intent is interpreted and their emotional state is grasped, the server's generation means acquires and generates information. This means acquires information in real time from external databases or APIs (e.g., weather information APIs). Based on the acquired information, the generation means utilizes data from the emotion engine to adjust the response. For example, if the user is excited, the server generates a response that includes calming words such as, "Tomorrow is sunny. It looks like it's going to be a very pleasant day."

[0214] Step 7:

[0215] The text data generated by the generation means is sent to the terminal. The terminal converts the text data into voice data using a voice synthesis means. This means can use existing technology (e.g., a voice synthesis API). In this step, voice data is generated that says, "Tomorrow will be sunny. It looks like it will be a very pleasant day."

[0216] Step 8:

[0217] The generated voice data is provided to the user through the device's speaker. The device then plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand manner, using the tone and speed corresponding to the information provided by the emotion engine.

[0218] Through these steps, users can use the AI ​​to interact with the system through voice and obtain optimal information according to their emotional state. This system provides an environment that is easy to use, even for elderly people and children who are unfamiliar with using computers.

[0219] Example 2

[0220] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0221] Existing speech recognition systems can only provide simple responses to user voice commands and are unable to provide adaptive responses that take the user's emotional state into account. Intuitive and easy-to-understand responses are particularly required for people who are unfamiliar with PC operation, such as the elderly and children. Given this background, there is a need for a system that can recognize a user's emotions and provide natural responses in response to them.

[0222] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring a voice input, a voice recognition means for converting the acquired voice input into text data, an emotion recognition means for recognizing a user's emotion from the analyzed voice data, a natural language processing means for analyzing the text data and understanding the user's intention, a generation means for acquiring and generating information based on the user's intention and emotion, a voice synthesis means for receiving the generated information as text data and converting it into voice data, and a means for providing the voice data to the user. This makes it possible to provide an appropriate and natural voice response according to the user's emotional state.

[0223] The "means for acquiring voice input" refers to a means for collecting voice data from a user using a device such as a microphone and converting it into a format that can be processed within the system.

[0224] The "voice recognition means" is a means for converting acquired voice input into text data and generating character information from voice information.

[0225] The "emotion recognition means" is a means for analyzing voice data and estimating and recognizing the user's emotions from the tone of the sound and speech patterns.

[0226] "Natural language processing means" refers to means that use machine learning algorithms to analyze text data and understand user intent and requirements.

[0227] The "generation means" is a means for obtaining necessary information from an external database or API based on the user's intentions and emotions, and generating a response based on this information.

[0228] The "voice synthesis means" is a means for converting the generated text data into voice data and outputting it in a reproducible voice format.

[0229] The "means for providing the user with voice data" refers to a means for letting the user hear the generated voice data through a speaker or the like.

[0230] This invention is a system for interacting with a user through voice input and providing an appropriate response based on the user's intentions and emotions. Specifically, it includes a means for acquiring voice input, a voice recognition means, an emotion recognition means, a natural language processing means, a generation means, a voice synthesis means, and a means for providing voice data to the user.

[0231] Acquiring voice input

[0232] A user speaks questions or commands into a device such as a smartphone or tablet that supports voice recognition. For example, they might say, "Tell me what the weather will be tomorrow." The device captures the voice through its built-in or external microphone and stores the voice data in a buffer.

[0233] Voice Recognition

[0234] The captured voice data is sent to the device's voice recognition means, which then converts the voice data into text data using existing technologies such as Google's Speech-to-Text API. For example, the text data generated would be "What's the weather going to be like tomorrow?"

[0235] Emotion recognition

[0236] The voice data is simultaneously sent to the device's emotion recognition means, which analyzes the voice tone and speech patterns to recognize the user's emotions. Specifically, the user's emotional state, such as "excited" or "calm," is determined from the voice pitch and speed.

[0237] Natural Language Processing

[0238] The text data generated by the speech recognition means is sent from the terminal to the server. The server passes the text data to the natural language processing means, which uses a machine learning algorithm to analyze the user's intent. This results in the text data being interpreted as a "weather information request."

[0239] Information Acquisition and Generation

[0240] The server's generation means retrieves the necessary information in real time from an external database or API (e.g., a weather information API) based on the user's intentions and emotions. Based on the retrieved information, the generation means generates a natural and relevant response. Since the user's emotional data is also taken into account, the generated response might be something like, "Tomorrow is sunny. It looks like it's going to be a very pleasant day."

[0241] Audio conversion

[0242] The generated text data is sent to the device and converted into voice data by the device's voice synthesis means, using IBM Watson's Text-to-Speech API or similar.

[0243] Providing voice responses

[0244] The generated voice data is provided to the user through the device's speaker, allowing the user to receive an intuitive and easy-to-understand voice response.

[0245] Specific examples

[0246] The following describes the process when a user asks, "What's the weather going to be like tomorrow?" The user's voice input is acquired by the device and converted into text data "What's the weather going to be like tomorrow?" by the voice recognition means. At the same time, the emotion recognition means analyzes the voice tone and recognizes that the user is excited. This text data is sent to the server and interpreted as a "weather information request" by the natural language processing means. Next, the generation means calls a weather information API to acquire tomorrow's weather information and generates a response sentence such as "Tomorrow is sunny. It looks like it's going to be a very pleasant day." Finally, this response sentence is converted into voice data by the voice synthesis means and provided to the user through the device's speaker. This process allows the user to receive an appropriate voice response that takes into account their emotions.

[0247] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0248] Step 1: Getting voice input

[0249] The user speaks a question or command to the device. For example, "Tell me what the weather will be tomorrow." The device captures this voice through its built-in microphone or an external microphone. The captured voice data is stored in a buffer.

[0250] Input: User's voice commands

[0251] Operation: Pick up sound with the device's microphone and save it in a buffer

[0252] Output: Audio data

[0253] Step 2: Voice Recognition

[0254] The acquired voice data is sent to the device's voice recognition means. The device then converts the voice data into text data using a method such as Google's Speech-to-Text API. For example, the generated text data would be, "Tell me what the weather will be like tomorrow."

[0255] Input: Audio data

[0256] Operation: Converts speech to text using the Speech-to-Text API

[0257] Output: Text data "What's the weather going to be like tomorrow?"

[0258] Step 3: Recognize emotions

[0259] The voice data is simultaneously transmitted to the device's emotion recognition means, which analyzes the voice tone and speech pattern to recognize the user's emotion. For example, the device may analyze the voice pitch and speed to determine that the user is "excited."

[0260] Input: Audio data

[0261] Operation: Voice tone and speech pattern analysis

[0262] Output: Emotion data (e.g. "Excited")

[0263] Step 4: Natural Language Processing

[0264] The text data generated by the speech recognition means is sent from the terminal to the server. The server passes the text data to the natural language processing means, which uses a machine learning algorithm to analyze the user's intent. As a result, the text data is interpreted as a "weather information request."

[0265] Input: Text data "What's the weather going to be like tomorrow?"

[0266] Operation: Intention analysis using natural language processing (machine learning algorithms)

[0267] Output: Intent data (e.g., "Request weather information")

[0268] Step 5: Obtaining and generating information

[0269] The server's generation means retrieves information from an external database or API based on the user's intent and emotional data. For example, it retrieves tomorrow's weather information from a weather information API. Based on the retrieved information, the generation means generates a natural and relevant response. Since the user's emotional data is also taken into consideration, a response such as "Tomorrow will be sunny. It looks like it will be a very pleasant day" is generated.

[0270] Input: Intent data (weather information request), emotion data

[0271] Operation: Retrieving information from external databases or APIs and generating appropriate responses

[0272] Output: Response text data (e.g. "Tomorrow is sunny. It looks like it's going to be a very pleasant day.")

[0273] Step 6: Audio conversion

[0274] The generated text data is sent to the device and converted into voice data by the device's voice synthesis means, using IBM Watson's Text-to-Speech API or similar.

[0275] Input: Response text data

[0276] Operation: Convert text data to audio data (using the Text-to-Speech API)

[0277] Output: Speech data (e.g. "Tomorrow is sunny. It looks like it's going to be a very pleasant day.")

[0278] Step 7: Provide a voice response

[0279] The generated voice data is provided to the user through the device's speaker. The device plays back the voice data and conveys information to the user in an intuitive and easy-to-understand format.

[0280] Input: Audio data

[0281] Operation: Play audio data through the speaker

[0282] Output: A spoken response to the user (e.g., "Tomorrow is sunny. It's going to be a very pleasant day.")

[0283] In this way, the system can perform emotion recognition from voice input and provide natural responses according to the user's emotional state.

[0284] (Application example 2)

[0285] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0286] Conventional speech recognition and response systems simply convert text into speech and provide information without considering the user's emotions. This results in a lack of appropriate information provided based on the user's emotions, leading to a decline in user satisfaction, especially in stressful environments or places where frequent guidance is required. Furthermore, improved customer service is required in brick-and-mortar stores, and voice guidance can improve the quality of customer service by providing prompt and accurate guidance. To solve these issues, a system that recognizes the user's emotions and adjusts responses accordingly is needed.

[0287]

[0288] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0289] In this invention, the server includes means for acquiring voice input, speech recognition means for converting the acquired voice input into text data, natural language processing means for analyzing the text data and understanding the user's intention, generation means for acquiring and generating information based on the user's intention, speech synthesis means for receiving the generated information as text data and converting it into voice data, means for providing the voice data to the user, emotion analysis means for recognizing the user's emotion, and emotion adjustment means for adjusting responses based on the user's emotional state. This makes it possible to provide information that is appropriately adjusted according to the user's emotion, thereby improving the quality of customer service, particularly in providing guidance and information in physical stores.

[0290] "Audio input" refers to audio signals emitted by a user to the system through a microphone.

[0291] "Speech recognition means" refers to technology that converts voice signals into text data.

[0292] "Natural language processing means" refers to technology for analyzing text data and understanding the user's intent.

[0293] "Generation means" refers to a technology that acquires information based on the user's intentions and generates text data based on that information.

[0294] "Speech synthesis means" refers to a technology that converts generated text data into voice data.

[0295] "Means for providing audio data to a user" refers to technology for transmitting generated audio data to a user through a speaker or the like.

[0296] "Emotion analysis means" refers to technology that recognizes a user's emotions by analyzing the user's tone of voice and speech patterns.

[0297] "Emotion adjustment means" refers to a technique for adjusting the content of a response based on the recognized emotions of a user.

[0298] A system for implementing the present invention includes means for obtaining speech input, speech recognition means, natural language processing means, generation means, emotion analysis means, emotion modulation means, speech synthesis means, and means for providing a speech response.

[0299] Hardware and software used

[0300] Hardware:

[0301] Smart glasses (with built-in microphone, speaker, and camera)

[0302] software:

[0303] Speech recognition API (e.g., converting voice data into text using a speech recognition API)

[0304] Natural language processing APIs (e.g., using natural language processing APIs to analyze text data)

[0305] Speech synthesis API (e.g., converting text data into speech data using a speech synthesis API)

[0306] Emotion analysis API (analyzes voice tone and speech patterns)

[0307] System processing overview

[0308] Acquiring voice input

[0309] The user speaks a question into the microphone of the smart glasses. An example question is "Where is the bread section?" This voice signal is picked up by the microphone of the smart glasses.

[0310] Voice Recognition

[0311] The acquired voice signal is converted into text data using a speech recognition API, which generates the text data "Where is the bread section?"

[0312] emotion recognition

[0313] At the same time, an emotion analysis API is used to analyze voice tone and speech patterns to recognize the user's emotions, for example, to determine whether the user is excited or calm.

[0314] Natural Language Processing

[0315] The generated text data is sent to a natural language processing API, which analyzes the user's intent. In this case, the intent is understood to be "directions to the bread section."

[0316] Information Acquisition and Generation

[0317] Once the user's intention is analyzed, the generation means obtains related information from the cloud server of the smart glasses and generates a response. For example, specific location information of the bread section is obtained, and a response sentence such as "The bread section is at the end of the aisle on the left" is generated as text data.

[0318] emotional regulation

[0319] The emotion adjustment means adjusts the response sentence based on the recognized emotion of the user, for example, adding calming words if the user is excited.

[0320] Audio conversion

[0321] The generated response is converted into voice data using a speech synthesis API, which generates the voice data, "The bread section is down the aisle on the left, straight ahead."

[0322] Providing voice responses

[0323] Finally, the generated audio data is provided to the user through the speakers of the smart glasses.

[0324] Specific examples

[0325] For example, if a user asks, "What are the current sales?" the same series of processes are performed, providing the following concrete example:

[0326] Question: "What are the current sales?"

[0327] Example prompt: Tell me about this week's sales.

[0328] Response: "All drinks are 10% off this week. Give them a try! It's a great deal."

[0329] This provides a highly convenient shopping experience, allowing users to easily obtain information in-store and find the products they need.

[0330] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0331] Step 1:

[0332] Acquiring voice input

[0333] The user speaks a question into the microphone of the smart glasses. An example question here is "Where is the bread section?" The microphone built into the smart glasses picks up the voice signal and stores it in a buffer as voice data. This voice data is then sent to the next step of speech recognition.

[0334] Input: User speech ("Where is the bread section?")

[0335] Output: Audio data (audio signal)

[0336] Step 2:

[0337] Voice Recognition

[0338] The device's speech recognition API receives the voice data stored in the buffer. The speech recognition API analyzes the voice signal, breaks it down into phonemes, and generates corresponding text data. The generated text data is "Where is the bread section?"

[0339] Input: Audio data

[0340] Output: Text data ("Where is the bread section?")

[0341] Step 3:

[0342] emotion recognition

[0343] The device's emotion analysis API receives the voice data. The emotion analysis API analyzes the voice tone and speech patterns to recognize the user's emotional state. For example, it analyzes the voice tone, speaking speed, and strength to determine whether the user is in a "calm" state.

[0344] Input: Audio data

[0345] Output: Emotion data ("Calm")

[0346] Step 4:

[0347] Natural Language Processing

[0348] The text data is sent to the server's natural language processing API. The natural language processing API analyzes the text data and understands the user's intent. As a result of the analysis, it is determined that the user is asking where the bread section is. This analysis result is passed to the generation means.

[0349] Input: Text data ("Where is the bread section?")

[0350] Output: Intent data ("Where is the bread section?")

[0351] Step 5:

[0352] Information Acquisition and Generation

[0353] The server's generation means retrieves location information for the bread section from an external database in real time. For example, it executes a query to the database and retrieves information such as "The bread section is located at the end of the aisle on the left." A response sentence is generated based on the retrieved information. The generated response sentence is "The bread section is located at the end of the aisle on the left."

[0354] Input: Intent data ("location of the bread section")

[0355] Output: Response ("The bread section is down the left aisle, straight ahead.")

[0356] Step 6:

[0357] emotional regulation

[0358] The emotion adjustment means adjusts the generated response sentence based on the recognized emotion data ("calm"). If the emotion data is "calm", it is determined that no additional adjustment is necessary for the response sentence, and the response sentence is used as is.

[0359] Input: Response ("The bread section is down the left aisle, straight ahead."), Emotion ("Calm")

[0360] Output: Adjusted response ("The bakery is down the left aisle, straight ahead.")

[0361] Step 7:

[0362] Audio conversion

[0363] The generated text data is sent to the device's speech synthesis API and converted into voice data. The speech synthesis API converts the text data into voice signals, generating voice data such as, "The bread section is down the aisle on the left, straight ahead."

[0364] Input: Adjusted response ("The bakery is down the left aisle, straight ahead.")

[0365] Output: Speech data ("The bread section is down the left aisle, straight ahead.")

[0366] Step 8:

[0367] Providing voice responses

[0368] Finally, the generated voice data is provided to the user through the smart glasses speaker, and the user receives the voice data stating, "The bread section is down the left aisle, straight ahead."

[0369] Input: Speech data ("The bread section is down the left aisle, straight ahead.")

[0370] Output: Voice guidance ("The bread section is down the left aisle, straight ahead.")

[0371] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0372] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0373] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0374] [Second embodiment]

[0375] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0376] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0377] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0378] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0379] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0380] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0381] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0382] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0383] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0384] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0385] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0386] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0387] The present invention is a system including means for acquiring a voice input, means for speech recognition, means for natural language processing, means for generating, means for synthesizing speech, and means for providing a voice response. The following describes an embodiment of the system.

[0388] Acquiring voice input

[0389] The user speaks a question or instruction to the terminal. For example, they might say, "Tell me what the weather will be tomorrow." This speech is picked up through the terminal's built-in microphone or a connected external microphone. The means for acquiring speech input stores the speech data in a buffer and sends it to the speech recognition means, which is the next step.

[0390] Voice Recognition

[0391] The acquired voice data is sent to the terminal's voice recognition means. This means converts the acquired voice data into text data. Existing technology (e.g., voice recognition API) can be used for the voice recognition means. In this step, text data such as "Tell me what the weather will be like tomorrow" is generated.

[0392] Natural Language Processing

[0393] The text data generated by the speech recognition means is sent from the terminal to the server. The server analyzes the text data using natural language processing means to understand the user's intent. This natural language processing means can use a machine learning algorithm (e.g., a natural language processing API). In this step, the text data is interpreted as a "weather information request."

[0394] Information Acquisition and Generation

[0395] After the user's intent is interpreted, the server's generator retrieves and generates information. This generator retrieves information in real time from external databases or APIs (e.g., weather information APIs). Based on the retrieved information, the generator generates a natural and relevant response. In this step, a response such as "Tomorrow will be sunny" is generated.

[0396] Audio conversion

[0397] The text data generated by the generation means is sent to the terminal. The terminal converts the text data into voice data using a voice synthesis means. This means can use existing technology (e.g., a voice synthesis API). In this step, voice data saying "Tomorrow will be sunny" is generated.

[0398] Providing voice responses

[0399] Finally, the generated voice data is provided to the user through the speaker of the terminal, and the terminal plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand manner.

[0400] Specific examples

[0401] As an example, the flow when a user asks "What's the weather going to be like tomorrow?" is shown below. The user's voice input is acquired by the terminal and converted into text data "What's the weather going to be like tomorrow?" by the speech recognition means. This text data is sent to the server and interpreted as a "weather information request" by the natural language processing means. Next, the generation means calls the weather information API to acquire tomorrow's weather information, and generates a response such as "It will be sunny tomorrow." Finally, this response is converted into voice data by the voice synthesis means and provided to the user through the terminal's speaker.

[0402] In this way, the system of the present invention utilizes generative AI to provide the user with the necessary information through voice dialogue, providing an environment that is easy to use even for elderly people and children who are unfamiliar with using computers.

[0403] The processing flow will be explained below.

[0404] Step 1:

[0405] The user speaks to the device, saying, "Tell me what the weather will be tomorrow." The device picks up the voice data through its built-in microphone or a connected external microphone.

[0406] Step 2:

[0407] The acquired voice data is sent to the device's voice recognition means. This means converts the voice data into text data. Specifically, it calls a voice recognition API to generate text data such as "Tell me what the weather will be like tomorrow."

[0408] Step 3:

[0409] The text data generated by the speech recognition means is transmitted from the terminal to the server, which then passes the received text data to the natural language processing means.

[0410] Step 4:

[0411] The server's natural language processing means analyzes the text data and understands the user's intent. Specifically, it uses a machine learning algorithm to interpret it as a "weather information request."

[0412] Step 5:

[0413] After the user's intention is interpreted, the server's generating means operates. The server sends a request to an external weather information API to obtain weather information in real time.

[0414] Step 6:

[0415] Based on the data obtained from the weather information API, the server's generation means generates a response such as "It will be sunny tomorrow." The generated response is sent to the terminal as text data.

[0416] Step 7:

[0417] The device passes the received response to the speech synthesis means. This means converts text data into speech data. Specifically, it calls the speech synthesis API to generate speech data saying, "Tomorrow will be sunny."

[0418] Step 8:

[0419] The generated voice data is provided to the user through the speaker of the terminal, and the terminal plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand format.

[0420] By following these steps, users can obtain the necessary information using the generation AI while interacting with the system via voice.

[0421] Example 1

[0422] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0423] Many users today desire intuitive voice-based interfaces, but existing systems have issues with speech recognition accuracy and response speed. It is also difficult to perform the entire process from voice input to data search and response generation in real time, making them difficult to use, especially for users who are unfamiliar with technology, such as the elderly and children. Furthermore, current technologies require complex integration of different APIs and buffer management, making it difficult to achieve stable operation.

[0424] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0425] In this invention, the server includes means for acquiring voice input, speech recognition means for converting the acquired voice input into text data, natural language processing means for analyzing the text data and understanding the user's intention, information generation means for acquiring and generating information based on the user's intention, speech synthesis means for receiving the generated information as text data and converting it into voice data, and means for providing the voice data to the user. This enables the user to intuitively interact with the system using voice and obtain necessary information in real time.

[0426] "Means for acquiring voice input" refers to a device or method for acquiring voice uttered by a user and recording it as digital data.

[0427] "Speech recognition means" refers to technology or devices that convert acquired voice data into text data, and includes speech recognition APIs, etc.

[0428] "Natural language processing means" refers to technology or devices that analyze text data and understand the user's intentions and requests, and includes natural language processing APIs and machine learning algorithms.

[0429] "Information generation means" refers to technology or devices that acquire necessary information from external databases, APIs, etc. based on the user's intentions, and generate a response sentence to convey that information to the user.

[0430] "Speech synthesis means" refers to technology or devices that convert generated text data into voice data for output as voice, and includes speech synthesis APIs, etc.

[0431] The "means for providing audio data to a user" refers to a technique or device for making the generated audio data audible to a user through a speaker or other audio output device.

[0432] A "buffer" is a storage device that temporarily stores data to facilitate transmission and processing.

[0433] An "external database" is a database or API that exists outside the system and provides specific information.

[0434] A "machine learning algorithm" is a technique or method for training a model using large amounts of data to make appropriate predictions and classifications for new data.

[0435] MODE FOR CARRYING OUT THE INVENTION

[0436] The present invention is a system that allows a user to obtain information through a voice interface. The system comprises the following elements: means for obtaining voice input, means for speech recognition, means for natural language processing, means for generating information, means for speech synthesis, and means for providing voice data.

[0437] Acquiring voice input

[0438] The user speaks a question or instruction to the terminal. For example, the user might say, "Tell me what the weather will be like tomorrow." The terminal captures this voice using a built-in microphone or a connected external microphone and stores it in a buffer as digital voice data. The voice input means then transmits this voice data to the next step.

[0439] Voice Recognition

[0440] The device sends the acquired voice data to the voice recognition means. The device converts the voice data into text data using a voice recognition API (for example, Google Cloud Speech-to-Text). At this stage, the voice is converted into text data such as "Tell me what the weather will be tomorrow."

[0441] Natural Language Processing

[0442] The text data is sent from the device to the server. The server analyzes the text data using natural language processing means to understand the user's intent. This means can use a natural language processing API (e.g., OpenAI GPT-3). Through this step, the text data is interpreted as a "weather information request."

[0443] Information Acquisition and Generation

[0444] After the user's intent is interpreted, the server's information generation means retrieves and generates information in real time from external databases or APIs (e.g., weather information APIs). Based on the retrieved information, the generation means creates a natural and relevant response. At this stage, a response such as "Tomorrow will be sunny" is generated.

[0445] Audio conversion

[0446] The generated text data is sent from the server to the terminal. The terminal converts the text data into voice data using a voice synthesis means. This means can use a voice synthesis API (for example, Amazon Polly or Google Text-to-Speech). In this step, voice data saying "Tomorrow will be sunny" is generated.

[0447] Providing voice responses

[0448] Finally, the generated voice data is provided to the user through the device's speaker. The device plays back the voice data and conveys information to the user in an intuitive and easy-to-understand manner.

[0449] Specific examples

[0450] For example, if a user asks "What's the weather going to be like tomorrow?", the processing flow is as follows:

[0451] 1. The user says, "What's the weather going to be like tomorrow?"

[0452] 2. The device captures audio using the built-in microphone and stores it in a buffer.

[0453] 3. The device uses a voice recognition API to convert the voice into text data: "Tell me what the weather will be like tomorrow."

[0454] 4. The device sends this text data to the server.

[0455] 5. The server uses a natural language processing API to analyze the text data and understand the user's intent.

[0456] 6. The server retrieves weather information using an external database or API.

[0457] 7. The server generates a text response saying "Tomorrow will be sunny."

[0458] 8. The server sends the generated text data to the terminal.

[0459] 9. The device uses the speech synthesis API to generate voice data saying, "Tomorrow will be sunny."

[0460] 10. The device speaker plays the audio data and provides it to the user.

[0461] In this way, the system's specific processing flow allows intuitive and efficient processing of a series of processes from voice input to information acquisition and voice response. Examples of prompt sentences include the following:

[0462] "If a user asks about tomorrow's weather, explain how the system processes voice input and responds."

[0463] Using such prompts allows us to effectively leverage generative AI models to get fast and accurate responses.

[0464] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0465] Step 1:

[0466] The user speaks a question or instruction to the device. For example, they might say, "Tell me what the weather will be tomorrow." This voice is picked up as user input through the device's built-in microphone or an external microphone. The device stores this voice input in a buffer as digital voice data. This data becomes the input for the next process.

[0467] Step 2:

[0468] The device retrieves the voice data stored in the buffer and sends it to the voice recognition means. Specifically, it calls a voice recognition API (for example, Google Cloud Speech-to-Text) and converts this voice data into text data. At this stage, the input is digital voice data, and the output is text data such as "What's the weather going to be like tomorrow?"

[0469] Step 3:

[0470] The device sends the generated text data to the server. The server passes this text data to a natural language processing means. The server analyzes the text data using a natural language processing API (e.g., OpenAI GPT-3) to understand the user's intent. In this step, the input is the text data "Tell me what the weather will be tomorrow," and the output is data interpreted as a "weather information request."

[0471] Step 4:

[0472] The server uses information generation means to obtain and generate the necessary information based on the interpreted user intent. Specifically, it obtains weather data in real time from an external database or API (for example, a weather information API). In this process, the input is a "weather information request," and the output is a response such as "It will be sunny tomorrow." Natural language processing is performed based on the obtained weather information to generate the response.

[0473] Step 5:

[0474] The server sends the generated response sentence to the terminal as text data. The terminal receives this text data and converts it into voice data using a voice synthesis means. A voice synthesis API (for example, Amazon Polly or Google Text-to-Speech) is used as the voice synthesis means. In this step, the input is the text data "Tomorrow will be sunny," and the output is voice data.

[0475] Step 6:

[0476] The terminal plays the generated voice data through a speaker and provides it to the user. The user listens to this voice response and accepts it as an answer to their question. At this final stage, the input is voice data and the output is voice information that the user hears.

[0477] In this way, each processing step of the system is executed successively, thereby achieving a series of data processing steps from voice input to voice response.

[0478] (Application example 1)

[0479] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0480] Autonomous vehicles require systems that allow users to give voice instructions and easily access navigation and other information services. Current systems have difficulty accurately understanding the user's intentions and obtaining the necessary information in real time to provide it via voice. Furthermore, improving the accuracy and response speed of the voice interface for users while driving is also a challenge.

[0481] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0482] In this invention, the server includes means for acquiring voice input, speech recognition means for converting the acquired voice input into text data, natural language processing means for analyzing the text data and understanding the user's intention, generation means for acquiring and generating information based on the user's intention, speech synthesis means for receiving the generated information as text data and converting it into voice data, and means for providing the voice data to the user. This enables the user to give voice instructions to operate an autonomous vehicle and use navigation and other information services with high accuracy in real time.

[0483] "Means for acquiring voice input" refers to devices or software that collect the voice uttered by the user as data.

[0484] "Speech recognition means" refers to a device or software that converts acquired voice data into text data.

[0485] "Natural language processing means" refers to devices or software that analyze text data and understand the user's intent.

[0486] "Generation means" refers to a device or software that acquires necessary information based on the user's intentions and generates that information.

[0487] "Speech synthesis means" refers to a device or software that converts the generated information from text data into speech data.

[0488] The "means for providing audio data to a user" refers to a device or software for playing back the generated audio data and providing it to a user.

[0489] "Navigation and information services" refers to functions that provide route guidance and various information for autonomous vehicles.

[0490] "Multiple means for responding to voice-based operational instructions" refers to multiple devices and software that allow the user to give voice instructions and the system to respond accordingly.

[0491] A "machine learning algorithm" is a technology that learns patterns and features from data and automatically analyzes them.

[0492] An "external database" is a collection of data that exists outside the system and is referenced to provide required information.

[0493] "Acquiring information in real time" means instantly acquiring and providing information requested by a user.

[0494] The present invention provides a system for providing navigation and information services to an autonomous vehicle by allowing a user to give voice instructions for operation. The system includes a means for acquiring voice input, a speech recognition means, a natural language processing means, a generation means, a speech synthesis means, and a means for providing voice data to the user.

[0495] Acquiring voice input

[0496] The user speaks a question or instruction into a microphone installed in the vehicle. For example, the user might say, "Find a nearby restaurant." This speech is picked up by the vehicle's built-in microphone. The means for acquiring speech input stores this speech data in a buffer and sends it to the speech recognition means, which is the next step.

[0497] Voice Recognition

[0498] The acquired voice data is sent to the vehicle's voice recognition means. This means converts the acquired voice data into text data. Existing technology (e.g., SpeechRecognition library) can be used for the voice recognition means. In this step, the text data "Find nearby restaurants" is generated.

[0499] Natural Language Processing

[0500] The text data generated by the speech recognition means is sent from the vehicle's computer system to a server. The server uses natural language processing means to analyze the text data and understand the user's intent. This natural language processing means can use machine learning algorithms (e.g., Dialogflow API). In this step, the text data is interpreted as a "request for restaurant location information."

[0501] Information Acquisition and Generation

[0502] After the user's intent is interpreted, the server's generator retrieves and generates relevant information. This generator retrieves information in real time from external databases or APIs (e.g., Google Maps API). Based on the retrieved information, the generator generates a natural and relevant response. In this step, a response such as "The nearest restaurant is ____" is generated.

[0503] Conversion to audio data

[0504] The text data generated by the generation means is sent to the vehicle's computer system. The vehicle converts the text data into voice data using a voice synthesis means. This means can use existing technology (e.g., Google Cloud Text-to-Speech API). In this step, voice data such as "The nearest restaurant is ____" is generated.

[0505] Providing voice responses

[0506] Finally, the generated voice data is provided to the user through the vehicle's speakers, and the vehicle's system plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand manner.

[0507] As a specific example, when a user utters "Tell me where the nearest gas station is," the user's voice input is acquired by the vehicle and converted into text data "Tell me where the nearest gas station is" by speech recognition means. This text data is sent to the server and interpreted as a "request for gas station location information" by natural language processing means. Next, generation means refers to an external database to acquire information about nearby gas stations, and generates a response sentence such as "The nearest gas station is XX." Finally, this response sentence is converted into voice data by speech synthesis means and provided to the user through the vehicle's speaker.

[0508] Prompt Sentence Examples

[0509] "Find a nearby restaurant"

[0510] Increase the temperature inside the car

[0511] "Tell me where the nearest gas station is."

[0512] As a result, the system of the present invention provides an environment in which users can intuitively use various functions of an autonomous vehicle through voice control.

[0513] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0514] Step 1:

[0515] A user speaks a question or command into a microphone in an autonomous vehicle. For example, they say, "Find a nearby restaurant." The input is the user's voice, and the output is the voice data captured by the vehicle's microphone.

[0516] Step 2:

[0517] The terminal stores the acquired voice data in a buffer and sends it to the voice recognition means. The input is the voice data acquired by the microphone, and the output is the voice data sent to the voice recognition means.

[0518] Step 3:

[0519] The device's speech recognition means converts the voice data into text data. Existing speech recognition technology (e.g., the SpeechRecognition library) is used here. The input is voice data, and the output is text data such as "Find nearby restaurants."

[0520] Step 4:

[0521] The terminal sends text data to the server. The input is the text data, and the output is the text data sent to the server.

[0522] Step 5:

[0523] The server's natural language processing means analyzes the text data and understands the user's intent. At this stage, machine learning algorithms (e.g., Dialogflow API) are used. The input is text data, and the output is an interpretation result that includes the user's intent, such as "request restaurant location information."

[0524] Step 6:

[0525] The server's generation means retrieves relevant information based on the user's intent in real time from external databases or APIs (e.g., Google Maps API). The input is the result of interpreting the user's intent, and the output is relevant information, such as "location information of the nearest restaurant."

[0526] Step 7:

[0527] The server's generation means generates a natural and relevant response based on the acquired information. The input is the acquired information, and the output is a response sentence such as "The nearest restaurant is ____."

[0528] Step 8:

[0529] The server sends the generated response to the terminal. The input is the response, and the output is the response sent to the terminal.

[0530] Step 9:

[0531] The device's speech synthesis means converts the response sentence into voice data. At this stage, existing speech synthesis technology (e.g., Google Cloud Text-to-Speech API) is used. The input is text data, and the output is voice data such as "The nearest restaurant is ____."

[0532] Step 10:

[0533] The terminal provides voice data to the user through a speaker, where the input is the voice data and the output is the voice information provided to the user.

[0534] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0535] The present invention is a system including a means for acquiring a voice input, a voice recognition means, a natural language processing means, a generation means, an emotion engine, a voice synthesis means, and a means for providing a voice response. An embodiment of the system is shown below.

[0536] Acquiring voice input

[0537] The user speaks a question or instruction to the device. For example, "Tell me what the weather will be tomorrow." This speech is picked up through the device's built-in microphone or a connected external microphone. The means for acquiring speech input stores the speech data in a buffer and sends it to the speech recognition means and emotion engine, which are the next steps.

[0538] Voice Recognition

[0539] The acquired voice data is sent to the device's voice recognition means. This means converts the voice data into text data. Existing technology (e.g., voice recognition API) can be used for the voice recognition means. In this step, the text data "Tell me what the weather will be tomorrow" is generated.

[0540] Emotion recognition

[0541] The voice data is also sent to the emotion engine. The device's emotion engine analyzes the voice tone and speech patterns to recognize the user's emotions. For example, when a user says, "Tell me the weather tomorrow," the tone, speed, and strength of the voice are analyzed to determine emotions such as "excited" or "calm."

[0542] Natural Language Processing

[0543] The text data generated by the speech recognition means is sent from the terminal to the server. The server passes the received text data to the natural language processing means. The server's natural language processing means analyzes the text data and understands the user's intent. Specifically, it uses a machine learning algorithm to interpret it as a "weather information request." The natural language processing means also uses an emotion engine to analyze the user's emotions.

[0544] Information Acquisition and Generation

[0545] After the user's intent is interpreted, the server's generation means retrieves and generates information. This retrieves information in real time from external databases and APIs (e.g., weather information APIs). Based on the retrieved information, the generation means generates a natural and relevant response. In this process, it uses data from the emotion engine to adjust the response. For example, if the user is excited, it generates a response that includes calming words. In this step, the response generated is, "Tomorrow is sunny. It looks like it's going to be a very pleasant day."

[0546] Audio conversion

[0547] The text data generated by the generation means is sent to the terminal. The terminal converts the text data into voice data using a voice synthesis means. This means can use existing technology (e.g., a voice synthesis API). In this step, voice data is generated that says, "Tomorrow is sunny. It looks like it's going to be a very pleasant day."

[0548] Providing voice responses

[0549] Finally, the generated voice data is provided to the user through the speaker of the terminal, and the terminal plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand manner.

[0550] Specific examples

[0551] As an example, the flow when a user asks, "What's the weather going to be like tomorrow?" is shown below. The user's voice input is acquired by the terminal and converted into text data "What's the weather going to be like tomorrow?" by the speech recognition means. At the same time, the emotion engine analyzes the voice tone and recognizes that the user is excited. This text data is sent to the server and interpreted as a "weather information request" by the natural language processing means. Next, the generation means calls the weather information API to acquire tomorrow's weather information and generates a response such as "It's going to be sunny tomorrow. It looks like it's going to be a very pleasant day." Finally, this response is converted into voice data by the speech synthesis means and provided to the user through the terminal's speaker.

[0552] In this way, the system of the present invention improves the usability of the AI ​​generator by recognizing emotions in response to voice input and providing an appropriate response according to the user's emotional state, thereby providing an environment that can be easily used by elderly people and children who are unfamiliar with using computers.

[0553] The processing flow will be explained below.

[0554] Step 1:

[0555] The user speaks to the device, saying, "Tell me the weather tomorrow." The device acquires voice data through the built-in microphone or a connected external microphone. The means for acquiring voice input stores the voice data in a buffer and sends it to the speech recognition means and emotion engine, which are the next steps.

[0556] Step 2:

[0557] The acquired voice data is sent to the terminal's voice recognition means. This means converts the voice data into text data. Existing technology (e.g., voice recognition API) can be used for the voice recognition means. In this step, the text data "Tell me what the weather will be like tomorrow" is generated.

[0558] Step 3:

[0559] At the same time, the voice data is also sent to the device's emotion engine, which analyzes the voice tone and speech patterns to recognize the user's emotions. For example, when a user says, "Tell me the weather tomorrow," the engine analyzes the tone, speed, and volume of the voice to determine emotions such as "excited" or "calm."

[0560] Step 4:

[0561] The text data generated by the speech recognition means is sent from the terminal to the server. The server passes the received text data to the natural language processing means. The server's natural language processing means analyzes the text data and understands the user's intent. Specifically, it uses a machine learning algorithm to interpret the text data as a "weather information request."

[0562] Step 5:

[0563] At the same time as the results of natural language processing, emotional information analyzed by the device's emotion engine is also sent to the server, allowing the server to simultaneously grasp the user's intentions and emotional state.

[0564] Step 6:

[0565] After the user's intent is interpreted and their emotional state is grasped, the server's generation means acquires and generates information. This means acquires information in real time from external databases or APIs (e.g., weather information APIs). Based on the acquired information, the generation means utilizes data from the emotion engine to adjust the response. For example, if the user is excited, the server generates a response that includes calming words such as, "Tomorrow is sunny. It looks like it's going to be a very pleasant day."

[0566] Step 7:

[0567] The text data generated by the generation means is sent to the terminal. The terminal converts the text data into voice data using a voice synthesis means. This means can use existing technology (e.g., a voice synthesis API). In this step, voice data is generated that says, "Tomorrow will be sunny. It looks like it will be a very pleasant day."

[0568] Step 8:

[0569] The generated voice data is provided to the user through the device's speaker. The device then plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand manner, using the tone and speed corresponding to the information provided by the emotion engine.

[0570] Through these steps, users can use the AI ​​to interact with the system through voice and obtain optimal information according to their emotional state. This system provides an environment that is easy to use, even for elderly people and children who are unfamiliar with using computers.

[0571] Example 2

[0572] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0573] Existing speech recognition systems can only provide simple responses to user voice commands and are unable to provide adaptive responses that take the user's emotional state into account. Intuitive and easy-to-understand responses are particularly required for people who are unfamiliar with PC operation, such as the elderly and children. Given this background, there is a need for a system that can recognize a user's emotions and provide natural responses in response to them.

[0574] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring a voice input, a voice recognition means for converting the acquired voice input into text data, an emotion recognition means for recognizing a user's emotion from the analyzed voice data, a natural language processing means for analyzing the text data and understanding the user's intention, a generation means for acquiring and generating information based on the user's intention and emotion, a voice synthesis means for receiving the generated information as text data and converting it into voice data, and a means for providing the voice data to the user. This makes it possible to provide an appropriate and natural voice response according to the user's emotional state.

[0575] The "means for acquiring voice input" refers to a means for collecting voice data from a user using a device such as a microphone and converting it into a format that can be processed within the system.

[0576] The "voice recognition means" is a means for converting acquired voice input into text data and generating character information from voice information.

[0577] The "emotion recognition means" is a means for analyzing voice data and estimating and recognizing the user's emotions from the tone of the sound and speech patterns.

[0578] "Natural language processing means" refers to means that use machine learning algorithms to analyze text data and understand user intent and requirements.

[0579] The "generation means" is a means for obtaining necessary information from an external database or API based on the user's intentions and emotions, and generating a response based on this information.

[0580] The "voice synthesis means" is a means for converting the generated text data into voice data and outputting it in a reproducible voice format.

[0581] The "means for providing the user with voice data" refers to a means for letting the user hear the generated voice data through a speaker or the like.

[0582] This invention is a system for interacting with a user through voice input and providing an appropriate response based on the user's intentions and emotions. Specifically, it includes a means for acquiring voice input, a voice recognition means, an emotion recognition means, a natural language processing means, a generation means, a voice synthesis means, and a means for providing voice data to the user.

[0583] Acquiring voice input

[0584] A user speaks questions or commands into a device such as a smartphone or tablet that supports voice recognition. For example, they might say, "Tell me what the weather will be tomorrow." The device captures the voice through its built-in or external microphone and stores the voice data in a buffer.

[0585] Voice Recognition

[0586] The captured voice data is sent to the device's voice recognition means, which then converts the voice data into text data using existing technologies such as Google's Speech-to-Text API. For example, the text data generated would be "What's the weather going to be like tomorrow?"

[0587] Emotion recognition

[0588] The voice data is simultaneously sent to the device's emotion recognition means, which analyzes the voice tone and speech patterns to recognize the user's emotions. Specifically, the user's emotional state, such as "excited" or "calm," is determined from the voice pitch and speed.

[0589] Natural Language Processing

[0590] The text data generated by the speech recognition means is sent from the terminal to the server. The server passes the text data to the natural language processing means, which uses a machine learning algorithm to analyze the user's intent. This results in the text data being interpreted as a "weather information request."

[0591] Information Acquisition and Generation

[0592] The server's generation means retrieves the necessary information in real time from an external database or API (e.g., a weather information API) based on the user's intentions and emotions. Based on the retrieved information, the generation means generates a natural and relevant response. Since the user's emotional data is also taken into account, the generated response might be something like, "Tomorrow is sunny. It looks like it's going to be a very pleasant day."

[0593] Audio conversion

[0594] The generated text data is sent to the device and converted into voice data by the device's voice synthesis means, using IBM Watson's Text-to-Speech API or similar.

[0595] Providing voice responses

[0596] The generated voice data is provided to the user through the device's speaker, allowing the user to receive an intuitive and easy-to-understand voice response.

[0597] Specific examples

[0598] The following describes the process when a user asks, "What's the weather going to be like tomorrow?" The user's voice input is acquired by the device and converted into text data "What's the weather going to be like tomorrow?" by the voice recognition means. At the same time, the emotion recognition means analyzes the voice tone and recognizes that the user is excited. This text data is sent to the server and interpreted as a "weather information request" by the natural language processing means. Next, the generation means calls a weather information API to acquire tomorrow's weather information and generates a response sentence such as "Tomorrow is sunny. It looks like it's going to be a very pleasant day." Finally, this response sentence is converted into voice data by the voice synthesis means and provided to the user through the device's speaker. This process allows the user to receive an appropriate voice response that takes into account their emotions.

[0599] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0600] Step 1: Getting voice input

[0601] The user speaks a question or command to the device. For example, "Tell me what the weather will be tomorrow." The device captures this voice through its built-in microphone or an external microphone. The captured voice data is stored in a buffer.

[0602] Input: User's voice commands

[0603] Operation: Pick up sound with the device's microphone and save it in a buffer

[0604] Output: Audio data

[0605] Step 2: Voice Recognition

[0606] The acquired voice data is sent to the device's voice recognition means. The device then converts the voice data into text data using a method such as Google's Speech-to-Text API. For example, the generated text data would be, "Tell me what the weather will be like tomorrow."

[0607] Input: Audio data

[0608] Operation: Converts speech to text using the Speech-to-Text API

[0609] Output: Text data "What's the weather going to be like tomorrow?"

[0610] Step 3: Recognize emotions

[0611] The voice data is simultaneously transmitted to the device's emotion recognition means, which analyzes the voice tone and speech pattern to recognize the user's emotion. For example, the device may analyze the voice pitch and speed to determine that the user is "excited."

[0612] Input: Audio data

[0613] Operation: Voice tone and speech pattern analysis

[0614] Output: Emotion data (e.g. "Excited")

[0615] Step 4: Natural Language Processing

[0616] The text data generated by the speech recognition means is sent from the terminal to the server. The server passes the text data to the natural language processing means, which uses a machine learning algorithm to analyze the user's intent. As a result, the text data is interpreted as a "weather information request."

[0617] Input: Text data "What's the weather going to be like tomorrow?"

[0618] Operation: Intention analysis using natural language processing (machine learning algorithms)

[0619] Output: Intent data (e.g., "Request weather information")

[0620] Step 5: Obtaining and generating information

[0621] The server's generation means retrieves information from an external database or API based on the user's intent and emotional data. For example, it retrieves tomorrow's weather information from a weather information API. Based on the retrieved information, the generation means generates a natural and relevant response. Since the user's emotional data is also taken into consideration, a response such as "Tomorrow will be sunny. It looks like it will be a very pleasant day" is generated.

[0622] Input: Intent data (weather information request), emotion data

[0623] Operation: Retrieving information from external databases or APIs and generating appropriate responses

[0624] Output: Response text data (e.g. "Tomorrow is sunny. It looks like it's going to be a very pleasant day.")

[0625] Step 6: Audio conversion

[0626] The generated text data is sent to the device and converted into voice data by the device's voice synthesis means, using IBM Watson's Text-to-Speech API or similar.

[0627] Input: Response text data

[0628] Operation: Convert text data to audio data (using the Text-to-Speech API)

[0629] Output: Speech data (e.g. "Tomorrow is sunny. It looks like it's going to be a very pleasant day.")

[0630] Step 7: Provide a voice response

[0631] The generated voice data is provided to the user through the device's speaker. The device plays back the voice data and conveys information to the user in an intuitive and easy-to-understand format.

[0632] Input: Audio data

[0633] Operation: Play audio data through the speaker

[0634] Output: A spoken response to the user (e.g., "Tomorrow is sunny. It's going to be a very pleasant day.")

[0635] In this way, the system can perform emotion recognition from voice input and provide natural responses according to the user's emotional state.

[0636] (Application example 2)

[0637] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0638] Conventional speech recognition and response systems simply convert text into speech and provide information without considering the user's emotions. This results in a lack of appropriate information provided based on the user's emotions, leading to a decline in user satisfaction, especially in stressful environments or places where frequent guidance is required. Furthermore, improved customer service is required in brick-and-mortar stores, and voice guidance can improve the quality of customer service by providing prompt and accurate guidance. To solve these issues, a system that recognizes the user's emotions and adjusts responses accordingly is needed.

[0639]

[0640] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0641] In this invention, the server includes means for acquiring voice input, speech recognition means for converting the acquired voice input into text data, natural language processing means for analyzing the text data and understanding the user's intention, generation means for acquiring and generating information based on the user's intention, speech synthesis means for receiving the generated information as text data and converting it into voice data, means for providing the voice data to the user, emotion analysis means for recognizing the user's emotion, and emotion adjustment means for adjusting responses based on the user's emotional state. This makes it possible to provide information that is appropriately adjusted according to the user's emotion, thereby improving the quality of customer service, particularly in providing guidance and information in physical stores.

[0642] "Audio input" refers to audio signals emitted by a user to the system through a microphone.

[0643] "Speech recognition means" refers to technology that converts voice signals into text data.

[0644] "Natural language processing means" refers to technology for analyzing text data and understanding the user's intent.

[0645] "Generation means" refers to a technology that acquires information based on the user's intentions and generates text data based on that information.

[0646] "Speech synthesis means" refers to a technology that converts generated text data into voice data.

[0647] "Means for providing audio data to a user" refers to technology for transmitting generated audio data to a user through a speaker or the like.

[0648] "Emotion analysis means" refers to technology that recognizes a user's emotions by analyzing the user's tone of voice and speech patterns.

[0649] "Emotion adjustment means" refers to a technique for adjusting the content of a response based on the recognized emotions of a user.

[0650] A system for implementing the present invention includes means for obtaining speech input, speech recognition means, natural language processing means, generation means, emotion analysis means, emotion modulation means, speech synthesis means, and means for providing a speech response.

[0651] Hardware and software used

[0652] Hardware:

[0653] Smart glasses (with built-in microphone, speaker, and camera)

[0654] software:

[0655] Speech recognition API (e.g., converting voice data into text using a speech recognition API)

[0656] Natural language processing APIs (e.g., using natural language processing APIs to analyze text data)

[0657] Speech synthesis API (e.g., converting text data into speech data using a speech synthesis API)

[0658] Emotion analysis API (analyzes voice tone and speech patterns)

[0659] System processing overview

[0660] Acquiring voice input

[0661] The user speaks a question into the microphone of the smart glasses. An example question is "Where is the bread section?" This voice signal is picked up by the microphone of the smart glasses.

[0662] Voice Recognition

[0663] The acquired voice signal is converted into text data using a speech recognition API, which generates the text data "Where is the bread section?"

[0664] emotion recognition

[0665] At the same time, an emotion analysis API is used to analyze voice tone and speech patterns to recognize the user's emotions, for example, to determine whether the user is excited or calm.

[0666] Natural Language Processing

[0667] The generated text data is sent to a natural language processing API, which analyzes the user's intent. In this case, the intent is understood to be "directions to the bread section."

[0668] Information Acquisition and Generation

[0669] Once the user's intention is analyzed, the generation means obtains related information from the cloud server of the smart glasses and generates a response. For example, specific location information of the bread section is obtained, and a response sentence such as "The bread section is at the end of the aisle on the left" is generated as text data.

[0670] emotional regulation

[0671] The emotion adjustment means adjusts the response sentence based on the recognized emotion of the user, for example, adding calming words if the user is excited.

[0672] Audio conversion

[0673] The generated response is converted into voice data using a speech synthesis API, which generates the voice data, "The bread section is down the aisle on the left, straight ahead."

[0674] Providing voice responses

[0675] Finally, the generated audio data is provided to the user through the speakers of the smart glasses.

[0676] Specific examples

[0677] For example, if a user asks, "What are the current sales?" the same series of processes are performed, providing the following concrete example:

[0678] Question: "What are the current sales?"

[0679] Example prompt: Tell me about this week's sales.

[0680] Response: "All drinks are 10% off this week. Give them a try! It's a great deal."

[0681] This provides a highly convenient shopping experience, allowing users to easily obtain information in-store and find the products they need.

[0682] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0683] Step 1:

[0684] Acquiring voice input

[0685] The user speaks a question into the microphone of the smart glasses. An example question here is "Where is the bread section?" The microphone built into the smart glasses picks up the voice signal and stores it in a buffer as voice data. This voice data is then sent to the next step of speech recognition.

[0686] Input: User speech ("Where is the bread section?")

[0687] Output: Audio data (audio signal)

[0688] Step 2:

[0689] Voice Recognition

[0690] The device's speech recognition API receives the voice data stored in the buffer. The speech recognition API analyzes the voice signal, breaks it down into phonemes, and generates corresponding text data. The generated text data is "Where is the bread section?"

[0691] Input: Audio data

[0692] Output: Text data ("Where is the bread section?")

[0693] Step 3:

[0694] emotion recognition

[0695] The device's emotion analysis API receives the voice data. The emotion analysis API analyzes the voice tone and speech patterns to recognize the user's emotional state. For example, it analyzes the voice tone, speaking speed, and strength to determine whether the user is in a "calm" state.

[0696] Input: Audio data

[0697] Output: Emotion data ("Calm")

[0698] Step 4:

[0699] Natural Language Processing

[0700] The text data is sent to the server's natural language processing API. The natural language processing API analyzes the text data and understands the user's intent. As a result of the analysis, it is determined that the user is asking where the bread section is. This analysis result is passed to the generation means.

[0701] Input: Text data ("Where is the bread section?")

[0702] Output: Intent data ("Where is the bread section?")

[0703] Step 5:

[0704] Information Acquisition and Generation

[0705] The server's generation means retrieves location information for the bread section from an external database in real time. For example, it executes a query to the database and retrieves information such as "The bread section is located at the end of the aisle on the left." A response sentence is generated based on the retrieved information. The generated response sentence is "The bread section is located at the end of the aisle on the left."

[0706] Input: Intent data ("location of the bread section")

[0707] Output: Response ("The bread section is down the left aisle, straight ahead.")

[0708] Step 6:

[0709] emotional regulation

[0710] The emotion adjustment means adjusts the generated response sentence based on the recognized emotion data ("calm"). If the emotion data is "calm", it is determined that no additional adjustment is necessary for the response sentence, and the response sentence is used as is.

[0711] Input: Response ("The bread section is down the left aisle, straight ahead."), Emotion ("Calm")

[0712] Output: Adjusted response ("The bakery is down the left aisle, straight ahead.")

[0713] Step 7:

[0714] Audio conversion

[0715] The generated text data is sent to the device's speech synthesis API and converted into voice data. The speech synthesis API converts the text data into voice signals, generating voice data such as, "The bread section is down the aisle on the left, straight ahead."

[0716] Input: Adjusted response ("The bakery is down the left aisle, straight ahead.")

[0717] Output: Speech data ("The bread section is down the left aisle, straight ahead.")

[0718] Step 8:

[0719] Providing voice responses

[0720] Finally, the generated voice data is provided to the user through the smart glasses speaker, and the user receives the voice data stating, "The bread section is down the left aisle, straight ahead."

[0721] Input: Speech data ("The bread section is down the left aisle, straight ahead.")

[0722] Output: Voice guidance ("The bread section is down the left aisle, straight ahead.")

[0723] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0724] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0725] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0726] [Third embodiment]

[0727] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0728] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0729] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0730] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0731] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0732] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0733] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0734] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0735] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0736] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0737] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0738] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0739] The present invention is a system including means for acquiring a voice input, means for speech recognition, means for natural language processing, means for generating, means for synthesizing speech, and means for providing a voice response. The following describes an embodiment of the system.

[0740] Acquiring voice input

[0741] The user speaks a question or instruction to the terminal. For example, they might say, "Tell me what the weather will be tomorrow." This speech is picked up through the terminal's built-in microphone or a connected external microphone. The means for acquiring speech input stores the speech data in a buffer and sends it to the speech recognition means, which is the next step.

[0742] Voice Recognition

[0743] The acquired voice data is sent to the terminal's voice recognition means. This means converts the acquired voice data into text data. Existing technology (e.g., voice recognition API) can be used for the voice recognition means. In this step, text data such as "Tell me what the weather will be like tomorrow" is generated.

[0744] Natural Language Processing

[0745] The text data generated by the speech recognition means is sent from the terminal to the server. The server analyzes the text data using natural language processing means to understand the user's intent. This natural language processing means can use a machine learning algorithm (e.g., a natural language processing API). In this step, the text data is interpreted as a "weather information request."

[0746] Information Acquisition and Generation

[0747] After the user's intent is interpreted, the server's generator retrieves and generates information. This generator retrieves information in real time from external databases or APIs (e.g., weather information APIs). Based on the retrieved information, the generator generates a natural and relevant response. In this step, a response such as "Tomorrow will be sunny" is generated.

[0748] Audio conversion

[0749] The text data generated by the generation means is sent to the terminal. The terminal converts the text data into voice data using a voice synthesis means. This means can use existing technology (e.g., a voice synthesis API). In this step, voice data saying "Tomorrow will be sunny" is generated.

[0750] Providing voice responses

[0751] Finally, the generated voice data is provided to the user through the speaker of the terminal, and the terminal plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand manner.

[0752] Specific examples

[0753] As an example, the flow when a user asks "What's the weather going to be like tomorrow?" is shown below. The user's voice input is acquired by the terminal and converted into text data "What's the weather going to be like tomorrow?" by the speech recognition means. This text data is sent to the server and interpreted as a "weather information request" by the natural language processing means. Next, the generation means calls the weather information API to acquire tomorrow's weather information, and generates a response such as "It will be sunny tomorrow." Finally, this response is converted into voice data by the voice synthesis means and provided to the user through the terminal's speaker.

[0754] In this way, the system of the present invention utilizes generative AI to provide the user with the necessary information through voice dialogue, providing an environment that is easy to use even for elderly people and children who are unfamiliar with using computers.

[0755] The processing flow will be explained below.

[0756] Step 1:

[0757] The user speaks to the device, saying, "Tell me what the weather will be tomorrow." The device picks up the voice data through its built-in microphone or a connected external microphone.

[0758] Step 2:

[0759] The acquired voice data is sent to the device's voice recognition means. This means converts the voice data into text data. Specifically, it calls a voice recognition API to generate text data such as "Tell me what the weather will be like tomorrow."

[0760] Step 3:

[0761] The text data generated by the speech recognition means is transmitted from the terminal to the server, which then passes the received text data to the natural language processing means.

[0762] Step 4:

[0763] The server's natural language processing means analyzes the text data and understands the user's intent. Specifically, it uses a machine learning algorithm to interpret it as a "weather information request."

[0764] Step 5:

[0765] After the user's intention is interpreted, the server's generating means operates. The server sends a request to an external weather information API to obtain weather information in real time.

[0766] Step 6:

[0767] Based on the data obtained from the weather information API, the server's generation means generates a response such as "It will be sunny tomorrow." The generated response is sent to the terminal as text data.

[0768] Step 7:

[0769] The device passes the received response to the speech synthesis means. This means converts text data into speech data. Specifically, it calls the speech synthesis API to generate speech data saying, "Tomorrow will be sunny."

[0770] Step 8:

[0771] The generated voice data is provided to the user through the speaker of the terminal, and the terminal plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand format.

[0772] By following these steps, users can obtain the necessary information using the generation AI while interacting with the system via voice.

[0773] Example 1

[0774] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0775] Many users today desire intuitive voice-based interfaces, but existing systems have issues with speech recognition accuracy and response speed. It is also difficult to perform the entire process from voice input to data search and response generation in real time, making them difficult to use, especially for users who are unfamiliar with technology, such as the elderly and children. Furthermore, current technologies require complex integration of different APIs and buffer management, making it difficult to achieve stable operation.

[0776] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0777] In this invention, the server includes means for acquiring voice input, speech recognition means for converting the acquired voice input into text data, natural language processing means for analyzing the text data and understanding the user's intention, information generation means for acquiring and generating information based on the user's intention, speech synthesis means for receiving the generated information as text data and converting it into voice data, and means for providing the voice data to the user. This enables the user to intuitively interact with the system using voice and obtain necessary information in real time.

[0778] "Means for acquiring voice input" refers to a device or method for acquiring voice uttered by a user and recording it as digital data.

[0779] "Speech recognition means" refers to technology or devices that convert acquired voice data into text data, and includes speech recognition APIs, etc.

[0780] "Natural language processing means" refers to technology or devices that analyze text data and understand the user's intentions and requests, and includes natural language processing APIs and machine learning algorithms.

[0781] "Information generation means" refers to technology or devices that acquire necessary information from external databases, APIs, etc. based on the user's intentions, and generate a response sentence to convey that information to the user.

[0782] "Speech synthesis means" refers to technology or devices that convert generated text data into voice data for output as voice, and includes speech synthesis APIs, etc.

[0783] The "means for providing audio data to a user" refers to a technique or device for making the generated audio data audible to a user through a speaker or other audio output device.

[0784] A "buffer" is a storage device that temporarily stores data to facilitate transmission and processing.

[0785] An "external database" is a database or API that exists outside the system and provides specific information.

[0786] A "machine learning algorithm" is a technique or method for training a model using large amounts of data to make appropriate predictions and classifications for new data.

[0787] MODE FOR CARRYING OUT THE INVENTION

[0788] The present invention is a system that allows a user to obtain information through a voice interface. The system comprises the following elements: means for obtaining voice input, means for speech recognition, means for natural language processing, means for generating information, means for speech synthesis, and means for providing voice data.

[0789] Acquiring voice input

[0790] The user speaks a question or instruction to the terminal. For example, the user might say, "Tell me what the weather will be like tomorrow." The terminal captures this voice using a built-in microphone or a connected external microphone and stores it in a buffer as digital voice data. The voice input means then transmits this voice data to the next step.

[0791] Voice Recognition

[0792] The device sends the acquired voice data to the voice recognition means. The device converts the voice data into text data using a voice recognition API (for example, Google Cloud Speech-to-Text). At this stage, the voice is converted into text data such as "Tell me what the weather will be tomorrow."

[0793] Natural Language Processing

[0794] The text data is sent from the device to the server. The server analyzes the text data using natural language processing means to understand the user's intent. This means can use a natural language processing API (e.g., OpenAI GPT-3). Through this step, the text data is interpreted as a "weather information request."

[0795] Information Acquisition and Generation

[0796] After the user's intent is interpreted, the server's information generation means retrieves and generates information in real time from external databases or APIs (e.g., weather information APIs). Based on the retrieved information, the generation means creates a natural and relevant response. At this stage, a response such as "Tomorrow will be sunny" is generated.

[0797] Audio conversion

[0798] The generated text data is sent from the server to the terminal. The terminal converts the text data into voice data using a voice synthesis means. This means can use a voice synthesis API (for example, Amazon Polly or Google Text-to-Speech). In this step, voice data saying "Tomorrow will be sunny" is generated.

[0799] Providing voice responses

[0800] Finally, the generated voice data is provided to the user through the device's speaker. The device plays back the voice data and conveys information to the user in an intuitive and easy-to-understand manner.

[0801] Specific examples

[0802] For example, if a user asks "What's the weather going to be like tomorrow?", the processing flow is as follows:

[0803] 1. The user says, "What's the weather going to be like tomorrow?"

[0804] 2. The device captures audio using the built-in microphone and stores it in a buffer.

[0805] 3. The device uses a voice recognition API to convert the voice into text data: "Tell me what the weather will be like tomorrow."

[0806] 4. The device sends this text data to the server.

[0807] 5. The server uses a natural language processing API to analyze the text data and understand the user's intent.

[0808] 6. The server retrieves weather information using an external database or API.

[0809] 7. The server generates a text response saying "Tomorrow will be sunny."

[0810] 8. The server sends the generated text data to the terminal.

[0811] 9. The device uses the speech synthesis API to generate voice data saying, "Tomorrow will be sunny."

[0812] 10. The device speaker plays the audio data and provides it to the user.

[0813] In this way, the system's specific processing flow allows intuitive and efficient processing of a series of processes from voice input to information acquisition and voice response. Examples of prompt sentences include the following:

[0814] "If a user asks about tomorrow's weather, explain how the system processes voice input and responds."

[0815] Using such prompts allows us to effectively leverage generative AI models to get fast and accurate responses.

[0816] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0817] Step 1:

[0818] The user speaks a question or instruction to the device. For example, they might say, "Tell me what the weather will be tomorrow." This voice is picked up as user input through the device's built-in microphone or an external microphone. The device stores this voice input in a buffer as digital voice data. This data becomes the input for the next process.

[0819] Step 2:

[0820] The device retrieves the voice data stored in the buffer and sends it to the voice recognition means. Specifically, it calls a voice recognition API (for example, Google Cloud Speech-to-Text) and converts this voice data into text data. At this stage, the input is digital voice data, and the output is text data such as "What's the weather going to be like tomorrow?"

[0821] Step 3:

[0822] The device sends the generated text data to the server. The server passes this text data to a natural language processing means. The server analyzes the text data using a natural language processing API (e.g., OpenAI GPT-3) to understand the user's intent. In this step, the input is the text data "Tell me what the weather will be tomorrow," and the output is data interpreted as a "weather information request."

[0823] Step 4:

[0824] The server uses information generation means to obtain and generate the necessary information based on the interpreted user intent. Specifically, it obtains weather data in real time from an external database or API (for example, a weather information API). In this process, the input is a "weather information request," and the output is a response such as "It will be sunny tomorrow." Natural language processing is performed based on the obtained weather information to generate the response.

[0825] Step 5:

[0826] The server sends the generated response sentence to the terminal as text data. The terminal receives this text data and converts it into voice data using a voice synthesis means. A voice synthesis API (for example, Amazon Polly or Google Text-to-Speech) is used as the voice synthesis means. In this step, the input is the text data "Tomorrow will be sunny," and the output is voice data.

[0827] Step 6:

[0828] The terminal plays the generated voice data through a speaker and provides it to the user. The user listens to this voice response and accepts it as an answer to their question. At this final stage, the input is voice data and the output is voice information that the user hears.

[0829] In this way, each processing step of the system is executed successively, thereby achieving a series of data processing steps from voice input to voice response.

[0830] (Application example 1)

[0831] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0832] Autonomous vehicles require systems that allow users to give voice instructions and easily access navigation and other information services. Current systems have difficulty accurately understanding the user's intentions and obtaining the necessary information in real time to provide it via voice. Furthermore, improving the accuracy and response speed of the voice interface for users while driving is also a challenge.

[0833] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0834] In this invention, the server includes means for acquiring voice input, speech recognition means for converting the acquired voice input into text data, natural language processing means for analyzing the text data and understanding the user's intention, generation means for acquiring and generating information based on the user's intention, speech synthesis means for receiving the generated information as text data and converting it into voice data, and means for providing the voice data to the user. This enables the user to give voice instructions to operate an autonomous vehicle and use navigation and other information services with high accuracy in real time.

[0835] "Means for acquiring voice input" refers to devices or software that collect the voice uttered by the user as data.

[0836] "Speech recognition means" refers to a device or software that converts acquired voice data into text data.

[0837] "Natural language processing means" refers to devices or software that analyze text data and understand the user's intent.

[0838] "Generation means" refers to a device or software that acquires necessary information based on the user's intentions and generates that information.

[0839] "Speech synthesis means" refers to a device or software that converts the generated information from text data into speech data.

[0840] The "means for providing audio data to a user" refers to a device or software for playing back the generated audio data and providing it to a user.

[0841] "Navigation and information services" refers to functions that provide route guidance and various information for autonomous vehicles.

[0842] "Multiple means for responding to voice-based operational instructions" refers to multiple devices and software that allow the user to give voice instructions and the system to respond accordingly.

[0843] A "machine learning algorithm" is a technology that learns patterns and features from data and automatically analyzes them.

[0844] An "external database" is a collection of data that exists outside the system and is referenced to provide required information.

[0845] "Acquiring information in real time" means instantly acquiring and providing information requested by a user.

[0846] The present invention provides a system for providing navigation and information services to an autonomous vehicle by allowing a user to give voice instructions for operation. The system includes a means for acquiring voice input, a speech recognition means, a natural language processing means, a generation means, a speech synthesis means, and a means for providing voice data to the user.

[0847] Acquiring voice input

[0848] The user speaks a question or instruction into a microphone installed in the vehicle. For example, the user might say, "Find a nearby restaurant." This speech is picked up by the vehicle's built-in microphone. The means for acquiring speech input stores this speech data in a buffer and sends it to the speech recognition means, which is the next step.

[0849] Voice Recognition

[0850] The acquired voice data is sent to the vehicle's voice recognition means. This means converts the acquired voice data into text data. Existing technology (e.g., SpeechRecognition library) can be used for the voice recognition means. In this step, the text data "Find nearby restaurants" is generated.

[0851] Natural Language Processing

[0852] The text data generated by the speech recognition means is sent from the vehicle's computer system to a server. The server uses natural language processing means to analyze the text data and understand the user's intent. This natural language processing means can use machine learning algorithms (e.g., Dialogflow API). In this step, the text data is interpreted as a "request for restaurant location information."

[0853] Information Acquisition and Generation

[0854] After the user's intent is interpreted, the server's generator retrieves and generates relevant information. This generator retrieves information in real time from external databases or APIs (e.g., Google Maps API). Based on the retrieved information, the generator generates a natural and relevant response. In this step, a response such as "The nearest restaurant is ____" is generated.

[0855] Conversion to audio data

[0856] The text data generated by the generation means is sent to the vehicle's computer system. The vehicle converts the text data into voice data using a voice synthesis means. This means can use existing technology (e.g., Google Cloud Text-to-Speech API). In this step, voice data such as "The nearest restaurant is ____" is generated.

[0857] Providing voice responses

[0858] Finally, the generated voice data is provided to the user through the vehicle's speakers, and the vehicle's system plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand manner.

[0859] As a specific example, when a user utters "Tell me where the nearest gas station is," the user's voice input is acquired by the vehicle and converted into text data "Tell me where the nearest gas station is" by speech recognition means. This text data is sent to the server and interpreted as a "request for gas station location information" by natural language processing means. Next, generation means refers to an external database to acquire information about nearby gas stations, and generates a response sentence such as "The nearest gas station is XX." Finally, this response sentence is converted into voice data by speech synthesis means and provided to the user through the vehicle's speaker.

[0860] Prompt Sentence Examples

[0861] "Find a nearby restaurant"

[0862] Increase the temperature inside the car

[0863] "Tell me where the nearest gas station is."

[0864] As a result, the system of the present invention provides an environment in which users can intuitively use various functions of an autonomous vehicle through voice control.

[0865] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0866] Step 1:

[0867] A user speaks a question or command into a microphone in an autonomous vehicle. For example, they say, "Find a nearby restaurant." The input is the user's voice, and the output is the voice data captured by the vehicle's microphone.

[0868] Step 2:

[0869] The terminal stores the acquired voice data in a buffer and sends it to the voice recognition means. The input is the voice data acquired by the microphone, and the output is the voice data sent to the voice recognition means.

[0870] Step 3:

[0871] The device's speech recognition means converts the voice data into text data. Existing speech recognition technology (e.g., the SpeechRecognition library) is used here. The input is voice data, and the output is text data such as "Find nearby restaurants."

[0872] Step 4:

[0873] The terminal sends text data to the server. The input is the text data, and the output is the text data sent to the server.

[0874] Step 5:

[0875] The server's natural language processing means analyzes the text data and understands the user's intent. At this stage, machine learning algorithms (e.g., Dialogflow API) are used. The input is text data, and the output is an interpretation result that includes the user's intent, such as "request restaurant location information."

[0876] Step 6:

[0877] The server's generation means retrieves relevant information based on the user's intent in real time from external databases or APIs (e.g., Google Maps API). The input is the result of interpreting the user's intent, and the output is relevant information, such as "location information of the nearest restaurant."

[0878] Step 7:

[0879] The server's generation means generates a natural and relevant response based on the acquired information. The input is the acquired information, and the output is a response sentence such as "The nearest restaurant is ____."

[0880] Step 8:

[0881] The server sends the generated response to the terminal. The input is the response, and the output is the response sent to the terminal.

[0882] Step 9:

[0883] The device's speech synthesis means converts the response sentence into voice data. At this stage, existing speech synthesis technology (e.g., Google Cloud Text-to-Speech API) is used. The input is text data, and the output is voice data such as "The nearest restaurant is ____."

[0884] Step 10:

[0885] The terminal provides voice data to the user through a speaker, where the input is the voice data and the output is the voice information provided to the user.

[0886] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0887] The present invention is a system including a means for acquiring a voice input, a voice recognition means, a natural language processing means, a generation means, an emotion engine, a voice synthesis means, and a means for providing a voice response. An embodiment of the system is shown below.

[0888] Acquiring voice input

[0889] The user speaks a question or instruction to the device. For example, "Tell me what the weather will be tomorrow." This speech is picked up through the device's built-in microphone or a connected external microphone. The means for acquiring speech input stores the speech data in a buffer and sends it to the speech recognition means and emotion engine, which are the next steps.

[0890] Voice Recognition

[0891] The acquired voice data is sent to the device's voice recognition means. This means converts the voice data into text data. Existing technology (e.g., voice recognition API) can be used for the voice recognition means. In this step, the text data "Tell me what the weather will be tomorrow" is generated.

[0892] Emotion recognition

[0893] The voice data is also sent to the emotion engine. The device's emotion engine analyzes the voice tone and speech patterns to recognize the user's emotions. For example, when a user says, "Tell me the weather tomorrow," the tone, speed, and strength of the voice are analyzed to determine emotions such as "excited" or "calm."

[0894] Natural Language Processing

[0895] The text data generated by the speech recognition means is sent from the terminal to the server. The server passes the received text data to the natural language processing means. The server's natural language processing means analyzes the text data and understands the user's intent. Specifically, it uses a machine learning algorithm to interpret it as a "weather information request." The natural language processing means also uses an emotion engine to analyze the user's emotions.

[0896] Information Acquisition and Generation

[0897] After the user's intent is interpreted, the server's generation means retrieves and generates information. This retrieves information in real time from external databases and APIs (e.g., weather information APIs). Based on the retrieved information, the generation means generates a natural and relevant response. In this process, it uses data from the emotion engine to adjust the response. For example, if the user is excited, it generates a response that includes calming words. In this step, the response generated is, "Tomorrow is sunny. It looks like it's going to be a very pleasant day."

[0898] Audio conversion

[0899] The text data generated by the generation means is sent to the terminal. The terminal converts the text data into voice data using a voice synthesis means. This means can use existing technology (e.g., a voice synthesis API). In this step, voice data is generated that says, "Tomorrow is sunny. It looks like it's going to be a very pleasant day."

[0900] Providing voice responses

[0901] Finally, the generated voice data is provided to the user through the speaker of the terminal, and the terminal plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand manner.

[0902] Specific examples

[0903] As an example, the flow when a user asks, "What's the weather going to be like tomorrow?" is shown below. The user's voice input is acquired by the terminal and converted into text data "What's the weather going to be like tomorrow?" by the speech recognition means. At the same time, the emotion engine analyzes the voice tone and recognizes that the user is excited. This text data is sent to the server and interpreted as a "weather information request" by the natural language processing means. Next, the generation means calls the weather information API to acquire tomorrow's weather information and generates a response such as "It's going to be sunny tomorrow. It looks like it's going to be a very pleasant day." Finally, this response is converted into voice data by the speech synthesis means and provided to the user through the terminal's speaker.

[0904] In this way, the system of the present invention improves the usability of the AI ​​generator by recognizing emotions in response to voice input and providing an appropriate response according to the user's emotional state, thereby providing an environment that can be easily used by elderly people and children who are unfamiliar with using computers.

[0905] The processing flow will be explained below.

[0906] Step 1:

[0907] The user speaks to the device, saying, "Tell me the weather tomorrow." The device acquires voice data through the built-in microphone or a connected external microphone. The means for acquiring voice input stores the voice data in a buffer and sends it to the speech recognition means and emotion engine, which are the next steps.

[0908] Step 2:

[0909] The acquired voice data is sent to the terminal's voice recognition means. This means converts the voice data into text data. Existing technology (e.g., voice recognition API) can be used for the voice recognition means. In this step, the text data "Tell me what the weather will be like tomorrow" is generated.

[0910] Step 3:

[0911] At the same time, the voice data is also sent to the device's emotion engine, which analyzes the voice tone and speech patterns to recognize the user's emotions. For example, when a user says, "Tell me the weather tomorrow," the engine analyzes the tone, speed, and volume of the voice to determine emotions such as "excited" or "calm."

[0912] Step 4:

[0913] The text data generated by the speech recognition means is sent from the terminal to the server. The server passes the received text data to the natural language processing means. The server's natural language processing means analyzes the text data and understands the user's intent. Specifically, it uses a machine learning algorithm to interpret the text data as a "weather information request."

[0914] Step 5:

[0915] At the same time as the results of natural language processing, emotional information analyzed by the device's emotion engine is also sent to the server, allowing the server to simultaneously grasp the user's intentions and emotional state.

[0916] Step 6:

[0917] After the user's intent is interpreted and their emotional state is grasped, the server's generation means acquires and generates information. This means acquires information in real time from external databases or APIs (e.g., weather information APIs). Based on the acquired information, the generation means utilizes data from the emotion engine to adjust the response. For example, if the user is excited, the server generates a response that includes calming words such as, "Tomorrow is sunny. It looks like it's going to be a very pleasant day."

[0918] Step 7:

[0919] The text data generated by the generation means is sent to the terminal. The terminal converts the text data into voice data using a voice synthesis means. This means can use existing technology (e.g., a voice synthesis API). In this step, voice data is generated that says, "Tomorrow will be sunny. It looks like it will be a very pleasant day."

[0920] Step 8:

[0921] The generated voice data is provided to the user through the device's speaker. The device then plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand manner, using the tone and speed corresponding to the information provided by the emotion engine.

[0922] Through these steps, users can use the AI ​​to interact with the system through voice and obtain optimal information according to their emotional state. This system provides an environment that is easy to use, even for elderly people and children who are unfamiliar with using computers.

[0923] Example 2

[0924] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0925] Existing speech recognition systems can only provide simple responses to user voice commands and are unable to provide adaptive responses that take the user's emotional state into account. Intuitive and easy-to-understand responses are particularly required for people who are unfamiliar with PC operation, such as the elderly and children. Given this background, there is a need for a system that can recognize a user's emotions and provide natural responses in response to them.

[0926] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring a voice input, a voice recognition means for converting the acquired voice input into text data, an emotion recognition means for recognizing a user's emotion from the analyzed voice data, a natural language processing means for analyzing the text data and understanding the user's intention, a generation means for acquiring and generating information based on the user's intention and emotion, a voice synthesis means for receiving the generated information as text data and converting it into voice data, and a means for providing the voice data to the user. This makes it possible to provide an appropriate and natural voice response according to the user's emotional state.

[0927] The "means for acquiring voice input" refers to a means for collecting voice data from a user using a device such as a microphone and converting it into a format that can be processed within the system.

[0928] The "voice recognition means" is a means for converting acquired voice input into text data and generating character information from voice information.

[0929] The "emotion recognition means" is a means for analyzing voice data and estimating and recognizing the user's emotions from the tone of the sound and speech patterns.

[0930] "Natural language processing means" refers to means that use machine learning algorithms to analyze text data and understand user intent and requirements.

[0931] The "generation means" is a means for obtaining necessary information from an external database or API based on the user's intentions and emotions, and generating a response based on this information.

[0932] The "voice synthesis means" is a means for converting the generated text data into voice data and outputting it in a reproducible voice format.

[0933] The "means for providing the user with voice data" refers to a means for letting the user hear the generated voice data through a speaker or the like.

[0934] This invention is a system for interacting with a user through voice input and providing an appropriate response based on the user's intentions and emotions. Specifically, it includes a means for acquiring voice input, a voice recognition means, an emotion recognition means, a natural language processing means, a generation means, a voice synthesis means, and a means for providing voice data to the user.

[0935] Acquiring voice input

[0936] A user speaks questions or commands into a device such as a smartphone or tablet that supports voice recognition. For example, they might say, "Tell me what the weather will be tomorrow." The device captures the voice through its built-in or external microphone and stores the voice data in a buffer.

[0937] Voice Recognition

[0938] The captured voice data is sent to the device's voice recognition means, which then converts the voice data into text data using existing technologies such as Google's Speech-to-Text API. For example, the text data generated would be "What's the weather going to be like tomorrow?"

[0939] Emotion recognition

[0940] The voice data is simultaneously sent to the device's emotion recognition means, which analyzes the voice tone and speech patterns to recognize the user's emotions. Specifically, the user's emotional state, such as "excited" or "calm," is determined from the voice pitch and speed.

[0941] Natural Language Processing

[0942] The text data generated by the speech recognition means is sent from the terminal to the server. The server passes the text data to the natural language processing means, which uses a machine learning algorithm to analyze the user's intent. This results in the text data being interpreted as a "weather information request."

[0943] Information Acquisition and Generation

[0944] The server's generation means retrieves the necessary information in real time from an external database or API (e.g., a weather information API) based on the user's intentions and emotions. Based on the retrieved information, the generation means generates a natural and relevant response. Since the user's emotional data is also taken into account, the generated response might be something like, "Tomorrow is sunny. It looks like it's going to be a very pleasant day."

[0945] Audio conversion

[0946] The generated text data is sent to the device and converted into voice data by the device's voice synthesis means, using IBM Watson's Text-to-Speech API or similar.

[0947] Providing voice responses

[0948] The generated voice data is provided to the user through the device's speaker, allowing the user to receive an intuitive and easy-to-understand voice response.

[0949] Specific examples

[0950] The following describes the process when a user asks, "What's the weather going to be like tomorrow?" The user's voice input is acquired by the device and converted into text data "What's the weather going to be like tomorrow?" by the voice recognition means. At the same time, the emotion recognition means analyzes the voice tone and recognizes that the user is excited. This text data is sent to the server and interpreted as a "weather information request" by the natural language processing means. Next, the generation means calls a weather information API to acquire tomorrow's weather information and generates a response sentence such as "Tomorrow is sunny. It looks like it's going to be a very pleasant day." Finally, this response sentence is converted into voice data by the voice synthesis means and provided to the user through the device's speaker. This process allows the user to receive an appropriate voice response that takes into account their emotions.

[0951] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0952] Step 1: Getting voice input

[0953] The user speaks a question or command to the device. For example, "Tell me what the weather will be tomorrow." The device captures this voice through its built-in microphone or an external microphone. The captured voice data is stored in a buffer.

[0954] Input: User's voice commands

[0955] Operation: Pick up sound with the device's microphone and save it in a buffer

[0956] Output: Audio data

[0957] Step 2: Voice Recognition

[0958] The acquired voice data is sent to the device's voice recognition means. The device then converts the voice data into text data using a method such as Google's Speech-to-Text API. For example, the generated text data would be, "Tell me what the weather will be like tomorrow."

[0959] Input: Audio data

[0960] Operation: Converts speech to text using the Speech-to-Text API

[0961] Output: Text data "What's the weather going to be like tomorrow?"

[0962] Step 3: Recognize emotions

[0963] The voice data is simultaneously transmitted to the device's emotion recognition means, which analyzes the voice tone and speech pattern to recognize the user's emotion. For example, the device may analyze the voice pitch and speed to determine that the user is "excited."

[0964] Input: Audio data

[0965] Operation: Voice tone and speech pattern analysis

[0966] Output: Emotion data (e.g. "Excited")

[0967] Step 4: Natural Language Processing

[0968] The text data generated by the speech recognition means is sent from the terminal to the server. The server passes the text data to the natural language processing means, which uses a machine learning algorithm to analyze the user's intent. As a result, the text data is interpreted as a "weather information request."

[0969] Input: Text data "What's the weather going to be like tomorrow?"

[0970] Operation: Intention analysis using natural language processing (machine learning algorithms)

[0971] Output: Intent data (e.g., "Request weather information")

[0972] Step 5: Obtaining and generating information

[0973] The server's generation means retrieves information from an external database or API based on the user's intent and emotional data. For example, it retrieves tomorrow's weather information from a weather information API. Based on the retrieved information, the generation means generates a natural and relevant response. Since the user's emotional data is also taken into consideration, a response such as "Tomorrow will be sunny. It looks like it will be a very pleasant day" is generated.

[0974] Input: Intent data (weather information request), emotion data

[0975] Operation: Retrieving information from external databases or APIs and generating appropriate responses

[0976] Output: Response text data (e.g. "Tomorrow is sunny. It looks like it's going to be a very pleasant day.")

[0977] Step 6: Audio conversion

[0978] The generated text data is sent to the device and converted into voice data by the device's voice synthesis means, using IBM Watson's Text-to-Speech API or similar.

[0979] Input: Response text data

[0980] Operation: Convert text data to audio data (using the Text-to-Speech API)

[0981] Output: Speech data (e.g. "Tomorrow is sunny. It looks like it's going to be a very pleasant day.")

[0982] Step 7: Provide a voice response

[0983] The generated voice data is provided to the user through the device's speaker. The device plays back the voice data and conveys information to the user in an intuitive and easy-to-understand format.

[0984] Input: Audio data

[0985] Operation: Play audio data through the speaker

[0986] Output: A spoken response to the user (e.g., "Tomorrow is sunny. It's going to be a very pleasant day.")

[0987] In this way, the system can perform emotion recognition from voice input and provide natural responses according to the user's emotional state.

[0988] (Application example 2)

[0989] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0990] Conventional speech recognition and response systems simply convert text into speech and provide information without considering the user's emotions. This results in a lack of appropriate information provided based on the user's emotions, leading to a decline in user satisfaction, especially in stressful environments or places where frequent guidance is required. Furthermore, improved customer service is required in brick-and-mortar stores, and voice guidance can improve the quality of customer service by providing prompt and accurate guidance. To solve these issues, a system that recognizes the user's emotions and adjusts responses accordingly is needed.

[0991]

[0992] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0993] In this invention, the server includes means for acquiring voice input, speech recognition means for converting the acquired voice input into text data, natural language processing means for analyzing the text data and understanding the user's intention, generation means for acquiring and generating information based on the user's intention, speech synthesis means for receiving the generated information as text data and converting it into voice data, means for providing the voice data to the user, emotion analysis means for recognizing the user's emotion, and emotion adjustment means for adjusting responses based on the user's emotional state. This makes it possible to provide information that is appropriately adjusted according to the user's emotion, thereby improving the quality of customer service, particularly in providing guidance and information in physical stores.

[0994] "Audio input" refers to audio signals emitted by a user to the system through a microphone.

[0995] "Speech recognition means" refers to technology that converts voice signals into text data.

[0996] "Natural language processing means" refers to technology for analyzing text data and understanding the user's intent.

[0997] "Generation means" refers to a technology that acquires information based on the user's intentions and generates text data based on that information.

[0998] "Speech synthesis means" refers to a technology that converts generated text data into voice data.

[0999] "Means for providing audio data to a user" refers to technology for transmitting generated audio data to a user through a speaker or the like.

[1000] "Emotion analysis means" refers to technology that recognizes a user's emotions by analyzing the user's tone of voice and speech patterns.

[1001] "Emotion adjustment means" refers to a technique for adjusting the content of a response based on the recognized emotions of a user.

[1002] A system for implementing the present invention includes means for obtaining speech input, speech recognition means, natural language processing means, generation means, emotion analysis means, emotion modulation means, speech synthesis means, and means for providing a speech response.

[1003] Hardware and software used

[1004] Hardware:

[1005] Smart glasses (with built-in microphone, speaker, and camera)

[1006] software:

[1007] Speech recognition API (e.g., converting voice data into text using a speech recognition API)

[1008] Natural language processing APIs (e.g., using natural language processing APIs to analyze text data)

[1009] Speech synthesis API (e.g., converting text data into speech data using a speech synthesis API)

[1010] Emotion analysis API (analyzes voice tone and speech patterns)

[1011] System processing overview

[1012] Acquiring voice input

[1013] The user speaks a question into the microphone of the smart glasses. An example question is "Where is the bread section?" This voice signal is picked up by the microphone of the smart glasses.

[1014] Voice Recognition

[1015] The acquired voice signal is converted into text data using a speech recognition API, which generates the text data "Where is the bread section?"

[1016] emotion recognition

[1017] At the same time, an emotion analysis API is used to analyze voice tone and speech patterns to recognize the user's emotions, for example, to determine whether the user is excited or calm.

[1018] Natural Language Processing

[1019] The generated text data is sent to a natural language processing API, which analyzes the user's intent. In this case, the intent is understood to be "directions to the bread section."

[1020] Information Acquisition and Generation

[1021] Once the user's intention is analyzed, the generation means obtains related information from the cloud server of the smart glasses and generates a response. For example, specific location information of the bread section is obtained, and a response sentence such as "The bread section is at the end of the aisle on the left" is generated as text data.

[1022] emotional regulation

[1023] The emotion adjustment means adjusts the response sentence based on the recognized emotion of the user, for example, adding calming words if the user is excited.

[1024] Audio conversion

[1025] The generated response is converted into voice data using a speech synthesis API, which generates the voice data, "The bread section is down the aisle on the left, straight ahead."

[1026] Providing voice responses

[1027] Finally, the generated audio data is provided to the user through the speakers of the smart glasses.

[1028] Specific examples

[1029] For example, if a user asks, "What are the current sales?" the same series of processes are performed, providing the following concrete example:

[1030] Question: "What are the current sales?"

[1031] Example prompt: Tell me about this week's sales.

[1032] Response: "All drinks are 10% off this week. Give them a try! It's a great deal."

[1033] This provides a highly convenient shopping experience, allowing users to easily obtain information in-store and find the products they need.

[1034] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1035] Step 1:

[1036] Acquiring voice input

[1037] The user speaks a question into the microphone of the smart glasses. An example question here is "Where is the bread section?" The microphone built into the smart glasses picks up the voice signal and stores it in a buffer as voice data. This voice data is then sent to the next step of speech recognition.

[1038] Input: User speech ("Where is the bread section?")

[1039] Output: Audio data (audio signal)

[1040] Step 2:

[1041] Voice Recognition

[1042] The device's speech recognition API receives the voice data stored in the buffer. The speech recognition API analyzes the voice signal, breaks it down into phonemes, and generates corresponding text data. The generated text data is "Where is the bread section?"

[1043] Input: Audio data

[1044] Output: Text data ("Where is the bread section?")

[1045] Step 3:

[1046] emotion recognition

[1047] The device's emotion analysis API receives the voice data. The emotion analysis API analyzes the voice tone and speech patterns to recognize the user's emotional state. For example, it analyzes the voice tone, speaking speed, and strength to determine whether the user is in a "calm" state.

[1048] Input: Audio data

[1049] Output: Emotion data ("Calm")

[1050] Step 4:

[1051] Natural Language Processing

[1052] The text data is sent to the server's natural language processing API. The natural language processing API analyzes the text data and understands the user's intent. As a result of the analysis, it is determined that the user is asking where the bread section is. This analysis result is passed to the generation means.

[1053] Input: Text data ("Where is the bread section?")

[1054] Output: Intent data ("Where is the bread section?")

[1055] Step 5:

[1056] Information Acquisition and Generation

[1057] The server's generation means retrieves location information for the bread section from an external database in real time. For example, it executes a query to the database and retrieves information such as "The bread section is located at the end of the aisle on the left." A response sentence is generated based on the retrieved information. The generated response sentence is "The bread section is located at the end of the aisle on the left."

[1058] Input: Intent data ("location of the bread section")

[1059] Output: Response ("The bread section is down the left aisle, straight ahead.")

[1060] Step 6:

[1061] emotional regulation

[1062] The emotion adjustment means adjusts the generated response sentence based on the recognized emotion data ("calm"). If the emotion data is "calm", it is determined that no additional adjustment is necessary for the response sentence, and the response sentence is used as is.

[1063] Input: Response ("The bread section is down the left aisle, straight ahead."), Emotion ("Calm")

[1064] Output: Adjusted response ("The bakery is down the left aisle, straight ahead.")

[1065] Step 7:

[1066] Audio conversion

[1067] The generated text data is sent to the device's speech synthesis API and converted into voice data. The speech synthesis API converts the text data into voice signals, generating voice data such as, "The bread section is down the aisle on the left, straight ahead."

[1068] Input: Adjusted response ("The bakery is down the left aisle, straight ahead.")

[1069] Output: Speech data ("The bread section is down the left aisle, straight ahead.")

[1070] Step 8:

[1071] Providing voice responses

[1072] Finally, the generated voice data is provided to the user through the smart glasses speaker, and the user receives the voice data stating, "The bread section is down the left aisle, straight ahead."

[1073] Input: Speech data ("The bread section is down the left aisle, straight ahead.")

[1074] Output: Voice guidance ("The bread section is down the left aisle, straight ahead.")

[1075] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1076] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1077] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1078] [Fourth embodiment]

[1079] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1080] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1081] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1082] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1083] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1084] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1085] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1086] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1087] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1088] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1089] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1090] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1091] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1092] The present invention is a system including means for acquiring a voice input, means for speech recognition, means for natural language processing, means for generating, means for synthesizing speech, and means for providing a voice response. The following describes an embodiment of the system.

[1093] Acquiring voice input

[1094] The user speaks a question or instruction to the terminal. For example, they might say, "Tell me what the weather will be tomorrow." This speech is picked up through the terminal's built-in microphone or a connected external microphone. The means for acquiring speech input stores the speech data in a buffer and sends it to the speech recognition means, which is the next step.

[1095] Voice Recognition

[1096] The acquired voice data is sent to the terminal's voice recognition means. This means converts the acquired voice data into text data. Existing technology (e.g., voice recognition API) can be used for the voice recognition means. In this step, text data such as "Tell me what the weather will be like tomorrow" is generated.

[1097] Natural Language Processing

[1098] The text data generated by the speech recognition means is sent from the terminal to the server. The server analyzes the text data using natural language processing means to understand the user's intent. This natural language processing means can use a machine learning algorithm (e.g., a natural language processing API). In this step, the text data is interpreted as a "weather information request."

[1099] Information Acquisition and Generation

[1100] After the user's intent is interpreted, the server's generator retrieves and generates information. This generator retrieves information in real time from external databases or APIs (e.g., weather information APIs). Based on the retrieved information, the generator generates a natural and relevant response. In this step, a response such as "Tomorrow will be sunny" is generated.

[1101] Audio conversion

[1102] The text data generated by the generation means is sent to the terminal. The terminal converts the text data into voice data using a voice synthesis means. This means can use existing technology (e.g., a voice synthesis API). In this step, voice data saying "Tomorrow will be sunny" is generated.

[1103] Providing voice responses

[1104] Finally, the generated voice data is provided to the user through the speaker of the terminal, and the terminal plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand manner.

[1105] Specific examples

[1106] As an example, the flow when a user asks "What's the weather going to be like tomorrow?" is shown below. The user's voice input is acquired by the terminal and converted into text data "What's the weather going to be like tomorrow?" by the speech recognition means. This text data is sent to the server and interpreted as a "weather information request" by the natural language processing means. Next, the generation means calls the weather information API to acquire tomorrow's weather information, and generates a response such as "It will be sunny tomorrow." Finally, this response is converted into voice data by the voice synthesis means and provided to the user through the terminal's speaker.

[1107] In this way, the system of the present invention utilizes generative AI to provide the user with the necessary information through voice dialogue, providing an environment that is easy to use even for elderly people and children who are unfamiliar with using computers.

[1108] The processing flow will be explained below.

[1109] Step 1:

[1110] The user speaks to the device, saying, "Tell me what the weather will be tomorrow." The device picks up the voice data through its built-in microphone or a connected external microphone.

[1111] Step 2:

[1112] The acquired voice data is sent to the device's voice recognition means. This means converts the voice data into text data. Specifically, it calls a voice recognition API to generate text data such as "Tell me what the weather will be like tomorrow."

[1113] Step 3:

[1114] The text data generated by the speech recognition means is transmitted from the terminal to the server, which then passes the received text data to the natural language processing means.

[1115] Step 4:

[1116] The server's natural language processing means analyzes the text data and understands the user's intent. Specifically, it uses a machine learning algorithm to interpret it as a "weather information request."

[1117] Step 5:

[1118] After the user's intention is interpreted, the server's generating means operates. The server sends a request to an external weather information API to obtain weather information in real time.

[1119] Step 6:

[1120] Based on the data obtained from the weather information API, the server's generation means generates a response such as "It will be sunny tomorrow." The generated response is sent to the terminal as text data.

[1121] Step 7:

[1122] The device passes the received response to the speech synthesis means. This means converts text data into speech data. Specifically, it calls the speech synthesis API to generate speech data saying, "Tomorrow will be sunny."

[1123] Step 8:

[1124] The generated voice data is provided to the user through the speaker of the terminal, and the terminal plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand format.

[1125] By following these steps, users can obtain the necessary information using the generation AI while interacting with the system via voice.

[1126] Example 1

[1127] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1128] Many users today desire intuitive voice-based interfaces, but existing systems have issues with speech recognition accuracy and response speed. It is also difficult to perform the entire process from voice input to data search and response generation in real time, making them difficult to use, especially for users who are unfamiliar with technology, such as the elderly and children. Furthermore, current technologies require complex integration of different APIs and buffer management, making it difficult to achieve stable operation.

[1129] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1130] In this invention, the server includes means for acquiring voice input, speech recognition means for converting the acquired voice input into text data, natural language processing means for analyzing the text data and understanding the user's intention, information generation means for acquiring and generating information based on the user's intention, speech synthesis means for receiving the generated information as text data and converting it into voice data, and means for providing the voice data to the user. This enables the user to intuitively interact with the system using voice and obtain necessary information in real time.

[1131] "Means for acquiring voice input" refers to a device or method for acquiring voice uttered by a user and recording it as digital data.

[1132] "Speech recognition means" refers to technology or devices that convert acquired voice data into text data, and includes speech recognition APIs, etc.

[1133] "Natural language processing means" refers to technology or devices that analyze text data and understand the user's intentions and requests, and includes natural language processing APIs and machine learning algorithms.

[1134] "Information generation means" refers to technology or devices that acquire necessary information from external databases, APIs, etc. based on the user's intentions, and generate a response sentence to convey that information to the user.

[1135] "Speech synthesis means" refers to technology or devices that convert generated text data into voice data for output as voice, and includes speech synthesis APIs, etc.

[1136] The "means for providing audio data to a user" refers to a technique or device for making the generated audio data audible to a user through a speaker or other audio output device.

[1137] A "buffer" is a storage device that temporarily stores data to facilitate transmission and processing.

[1138] An "external database" is a database or API that exists outside the system and provides specific information.

[1139] A "machine learning algorithm" is a technique or method for training a model using large amounts of data to make appropriate predictions and classifications for new data.

[1140] MODE FOR CARRYING OUT THE INVENTION

[1141] The present invention is a system that allows a user to obtain information through a voice interface. The system comprises the following elements: means for obtaining voice input, means for speech recognition, means for natural language processing, means for generating information, means for speech synthesis, and means for providing voice data.

[1142] Acquiring voice input

[1143] The user speaks a question or instruction to the terminal. For example, the user might say, "Tell me what the weather will be like tomorrow." The terminal captures this voice using a built-in microphone or a connected external microphone and stores it in a buffer as digital voice data. The voice input means then transmits this voice data to the next step.

[1144] Voice Recognition

[1145] The device sends the acquired voice data to the voice recognition means. The device converts the voice data into text data using a voice recognition API (for example, Google Cloud Speech-to-Text). At this stage, the voice is converted into text data such as "Tell me what the weather will be tomorrow."

[1146] Natural Language Processing

[1147] The text data is sent from the device to the server. The server analyzes the text data using natural language processing means to understand the user's intent. This means can use a natural language processing API (e.g., OpenAI GPT-3). Through this step, the text data is interpreted as a "weather information request."

[1148] Information Acquisition and Generation

[1149] After the user's intent is interpreted, the server's information generation means retrieves and generates information in real time from external databases or APIs (e.g., weather information APIs). Based on the retrieved information, the generation means creates a natural and relevant response. At this stage, a response such as "Tomorrow will be sunny" is generated.

[1150] Audio conversion

[1151] The generated text data is sent from the server to the terminal. The terminal converts the text data into voice data using a voice synthesis means. This means can use a voice synthesis API (for example, Amazon Polly or Google Text-to-Speech). In this step, voice data saying "Tomorrow will be sunny" is generated.

[1152] Providing voice responses

[1153] Finally, the generated voice data is provided to the user through the device's speaker. The device plays back the voice data and conveys information to the user in an intuitive and easy-to-understand manner.

[1154] Specific examples

[1155] For example, if a user asks "What's the weather going to be like tomorrow?", the processing flow is as follows:

[1156] 1. The user says, "What's the weather going to be like tomorrow?"

[1157] 2. The device captures audio using the built-in microphone and stores it in a buffer.

[1158] 3. The device uses a voice recognition API to convert the voice into text data: "Tell me what the weather will be like tomorrow."

[1159] 4. The device sends this text data to the server.

[1160] 5. The server uses a natural language processing API to analyze the text data and understand the user's intent.

[1161] 6. The server retrieves weather information using an external database or API.

[1162] 7. The server generates a text response saying "Tomorrow will be sunny."

[1163] 8. The server sends the generated text data to the terminal.

[1164] 9. The device uses the speech synthesis API to generate voice data saying, "Tomorrow will be sunny."

[1165] 10. The device speaker plays the audio data and provides it to the user.

[1166] In this way, the system's specific processing flow allows intuitive and efficient processing of a series of processes from voice input to information acquisition and voice response. Examples of prompt sentences include the following:

[1167] "If a user asks about tomorrow's weather, explain how the system processes voice input and responds."

[1168] Using such prompts allows us to effectively leverage generative AI models to get fast and accurate responses.

[1169] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1170] Step 1:

[1171] The user speaks a question or instruction to the device. For example, they might say, "Tell me what the weather will be tomorrow." This voice is picked up as user input through the device's built-in microphone or an external microphone. The device stores this voice input in a buffer as digital voice data. This data becomes the input for the next process.

[1172] Step 2:

[1173] The device retrieves the voice data stored in the buffer and sends it to the voice recognition means. Specifically, it calls a voice recognition API (for example, Google Cloud Speech-to-Text) and converts this voice data into text data. At this stage, the input is digital voice data, and the output is text data such as "What's the weather going to be like tomorrow?"

[1174] Step 3:

[1175] The device sends the generated text data to the server. The server passes this text data to a natural language processing means. The server analyzes the text data using a natural language processing API (e.g., OpenAI GPT-3) to understand the user's intent. In this step, the input is the text data "Tell me what the weather will be tomorrow," and the output is data interpreted as a "weather information request."

[1176] Step 4:

[1177] The server uses information generation means to obtain and generate the necessary information based on the interpreted user intent. Specifically, it obtains weather data in real time from an external database or API (for example, a weather information API). In this process, the input is a "weather information request," and the output is a response such as "It will be sunny tomorrow." Natural language processing is performed based on the obtained weather information to generate the response.

[1178] Step 5:

[1179] The server sends the generated response sentence to the terminal as text data. The terminal receives this text data and converts it into voice data using a voice synthesis means. A voice synthesis API (for example, Amazon Polly or Google Text-to-Speech) is used as the voice synthesis means. In this step, the input is the text data "Tomorrow will be sunny," and the output is voice data.

[1180] Step 6:

[1181] The terminal plays the generated voice data through a speaker and provides it to the user. The user listens to this voice response and accepts it as an answer to their question. At this final stage, the input is voice data and the output is voice information that the user hears.

[1182] In this way, each processing step of the system is executed successively, thereby achieving a series of data processing steps from voice input to voice response.

[1183] (Application example 1)

[1184] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1185] Autonomous vehicles require systems that allow users to give voice instructions and easily access navigation and other information services. Current systems have difficulty accurately understanding the user's intentions and obtaining the necessary information in real time to provide it via voice. Furthermore, improving the accuracy and response speed of the voice interface for users while driving is also a challenge.

[1186] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1187] In this invention, the server includes means for acquiring voice input, speech recognition means for converting the acquired voice input into text data, natural language processing means for analyzing the text data and understanding the user's intention, generation means for acquiring and generating information based on the user's intention, speech synthesis means for receiving the generated information as text data and converting it into voice data, and means for providing the voice data to the user. This enables the user to give voice instructions to operate an autonomous vehicle and use navigation and other information services with high accuracy in real time.

[1188] "Means for acquiring voice input" refers to devices or software that collect the voice uttered by the user as data.

[1189] "Speech recognition means" refers to a device or software that converts acquired voice data into text data.

[1190] "Natural language processing means" refers to devices or software that analyze text data and understand the user's intent.

[1191] "Generation means" refers to a device or software that acquires necessary information based on the user's intentions and generates that information.

[1192] "Speech synthesis means" refers to a device or software that converts the generated information from text data into speech data.

[1193] The "means for providing audio data to a user" refers to a device or software for playing back the generated audio data and providing it to a user.

[1194] "Navigation and information services" refers to functions that provide route guidance and various information for autonomous vehicles.

[1195] "Multiple means for responding to voice-based operational instructions" refers to multiple devices and software that allow the user to give voice instructions and the system to respond accordingly.

[1196] A "machine learning algorithm" is a technology that learns patterns and features from data and automatically analyzes them.

[1197] An "external database" is a collection of data that exists outside the system and is referenced to provide required information.

[1198] "Acquiring information in real time" means instantly acquiring and providing information requested by a user.

[1199] The present invention provides a system for providing navigation and information services to an autonomous vehicle by allowing a user to give voice instructions for operation. The system includes a means for acquiring voice input, a speech recognition means, a natural language processing means, a generation means, a speech synthesis means, and a means for providing voice data to the user.

[1200] Acquiring voice input

[1201] The user speaks a question or instruction into a microphone installed in the vehicle. For example, the user might say, "Find a nearby restaurant." This speech is picked up by the vehicle's built-in microphone. The means for acquiring speech input stores this speech data in a buffer and sends it to the speech recognition means, which is the next step.

[1202] Voice Recognition

[1203] The acquired voice data is sent to the vehicle's voice recognition means. This means converts the acquired voice data into text data. Existing technology (e.g., SpeechRecognition library) can be used for the voice recognition means. In this step, the text data "Find nearby restaurants" is generated.

[1204] Natural Language Processing

[1205] The text data generated by the speech recognition means is sent from the vehicle's computer system to a server. The server uses natural language processing means to analyze the text data and understand the user's intent. This natural language processing means can use machine learning algorithms (e.g., Dialogflow API). In this step, the text data is interpreted as a "request for restaurant location information."

[1206] Information Acquisition and Generation

[1207] After the user's intent is interpreted, the server's generator retrieves and generates relevant information. This generator retrieves information in real time from external databases or APIs (e.g., Google Maps API). Based on the retrieved information, the generator generates a natural and relevant response. In this step, a response such as "The nearest restaurant is ____" is generated.

[1208] Conversion to audio data

[1209] The text data generated by the generation means is sent to the vehicle's computer system. The vehicle converts the text data into voice data using a voice synthesis means. This means can use existing technology (e.g., Google Cloud Text-to-Speech API). In this step, voice data such as "The nearest restaurant is ____" is generated.

[1210] Providing voice responses

[1211] Finally, the generated voice data is provided to the user through the vehicle's speakers, and the vehicle's system plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand manner.

[1212] As a specific example, when a user utters "Tell me where the nearest gas station is," the user's voice input is acquired by the vehicle and converted into text data "Tell me where the nearest gas station is" by speech recognition means. This text data is sent to the server and interpreted as a "request for gas station location information" by natural language processing means. Next, generation means refers to an external database to acquire information about nearby gas stations, and generates a response sentence such as "The nearest gas station is XX." Finally, this response sentence is converted into voice data by speech synthesis means and provided to the user through the vehicle's speaker.

[1213] Prompt Sentence Examples

[1214] "Find a nearby restaurant"

[1215] Increase the temperature inside the car

[1216] "Tell me where the nearest gas station is."

[1217] As a result, the system of the present invention provides an environment in which users can intuitively use various functions of an autonomous vehicle through voice control.

[1218] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1219] Step 1:

[1220] A user speaks a question or command into a microphone in an autonomous vehicle. For example, they say, "Find a nearby restaurant." The input is the user's voice, and the output is the voice data captured by the vehicle's microphone.

[1221] Step 2:

[1222] The terminal stores the acquired voice data in a buffer and sends it to the voice recognition means. The input is the voice data acquired by the microphone, and the output is the voice data sent to the voice recognition means.

[1223] Step 3:

[1224] The device's speech recognition means converts the voice data into text data. Existing speech recognition technology (e.g., the SpeechRecognition library) is used here. The input is voice data, and the output is text data such as "Find nearby restaurants."

[1225] Step 4:

[1226] The terminal sends text data to the server. The input is the text data, and the output is the text data sent to the server.

[1227] Step 5:

[1228] The server's natural language processing means analyzes the text data and understands the user's intent. At this stage, machine learning algorithms (e.g., Dialogflow API) are used. The input is text data, and the output is an interpretation result that includes the user's intent, such as "request restaurant location information."

[1229] Step 6:

[1230] The server's generation means retrieves relevant information based on the user's intent in real time from external databases or APIs (e.g., Google Maps API). The input is the result of interpreting the user's intent, and the output is relevant information, such as "location information of the nearest restaurant."

[1231] Step 7:

[1232] The server's generation means generates a natural and relevant response based on the acquired information. The input is the acquired information, and the output is a response sentence such as "The nearest restaurant is ____."

[1233] Step 8:

[1234] The server sends the generated response to the terminal. The input is the response, and the output is the response sent to the terminal.

[1235] Step 9:

[1236] The device's speech synthesis means converts the response sentence into voice data. At this stage, existing speech synthesis technology (e.g., Google Cloud Text-to-Speech API) is used. The input is text data, and the output is voice data such as "The nearest restaurant is ____."

[1237] Step 10:

[1238] The terminal provides voice data to the user through a speaker, where the input is the voice data and the output is the voice information provided to the user.

[1239] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1240] The present invention is a system including a means for acquiring a voice input, a voice recognition means, a natural language processing means, a generation means, an emotion engine, a voice synthesis means, and a means for providing a voice response. An embodiment of the system is shown below.

[1241] Acquiring voice input

[1242] The user speaks a question or instruction to the device. For example, "Tell me what the weather will be tomorrow." This speech is picked up through the device's built-in microphone or a connected external microphone. The means for acquiring speech input stores the speech data in a buffer and sends it to the speech recognition means and emotion engine, which are the next steps.

[1243] Voice Recognition

[1244] The acquired voice data is sent to the device's voice recognition means. This means converts the voice data into text data. Existing technology (e.g., voice recognition API) can be used for the voice recognition means. In this step, the text data "Tell me what the weather will be tomorrow" is generated.

[1245] Emotion recognition

[1246] The voice data is also sent to the emotion engine. The device's emotion engine analyzes the voice tone and speech patterns to recognize the user's emotions. For example, when a user says, "Tell me the weather tomorrow," the tone, speed, and strength of the voice are analyzed to determine emotions such as "excited" or "calm."

[1247] Natural Language Processing

[1248] The text data generated by the speech recognition means is sent from the terminal to the server. The server passes the received text data to the natural language processing means. The server's natural language processing means analyzes the text data and understands the user's intent. Specifically, it uses a machine learning algorithm to interpret it as a "weather information request." The natural language processing means also uses an emotion engine to analyze the user's emotions.

[1249] Information Acquisition and Generation

[1250] After the user's intent is interpreted, the server's generation means retrieves and generates information. This retrieves information in real time from external databases and APIs (e.g., weather information APIs). Based on the retrieved information, the generation means generates a natural and relevant response. In this process, it uses data from the emotion engine to adjust the response. For example, if the user is excited, it generates a response that includes calming words. In this step, the response generated is, "Tomorrow is sunny. It looks like it's going to be a very pleasant day."

[1251] Audio conversion

[1252] The text data generated by the generation means is sent to the terminal. The terminal converts the text data into voice data using a voice synthesis means. This means can use existing technology (e.g., a voice synthesis API). In this step, voice data is generated that says, "Tomorrow is sunny. It looks like it's going to be a very pleasant day."

[1253] Providing voice responses

[1254] Finally, the generated voice data is provided to the user through the speaker of the terminal, and the terminal plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand manner.

[1255] Specific examples

[1256] As an example, the flow when a user asks, "What's the weather going to be like tomorrow?" is shown below. The user's voice input is acquired by the terminal and converted into text data "What's the weather going to be like tomorrow?" by the speech recognition means. At the same time, the emotion engine analyzes the voice tone and recognizes that the user is excited. This text data is sent to the server and interpreted as a "weather information request" by the natural language processing means. Next, the generation means calls the weather information API to acquire tomorrow's weather information and generates a response such as "It's going to be sunny tomorrow. It looks like it's going to be a very pleasant day." Finally, this response is converted into voice data by the speech synthesis means and provided to the user through the terminal's speaker.

[1257] In this way, the system of the present invention improves the usability of the AI ​​generator by recognizing emotions in response to voice input and providing an appropriate response according to the user's emotional state, thereby providing an environment that can be easily used by elderly people and children who are unfamiliar with using computers.

[1258] The processing flow will be explained below.

[1259] Step 1:

[1260] The user speaks to the device, saying, "Tell me the weather tomorrow." The device acquires voice data through the built-in microphone or a connected external microphone. The means for acquiring voice input stores the voice data in a buffer and sends it to the speech recognition means and emotion engine, which are the next steps.

[1261] Step 2:

[1262] The acquired voice data is sent to the terminal's voice recognition means. This means converts the voice data into text data. Existing technology (e.g., voice recognition API) can be used for the voice recognition means. In this step, the text data "Tell me what the weather will be like tomorrow" is generated.

[1263] Step 3:

[1264] At the same time, the voice data is also sent to the device's emotion engine, which analyzes the voice tone and speech patterns to recognize the user's emotions. For example, when a user says, "Tell me the weather tomorrow," the engine analyzes the tone, speed, and volume of the voice to determine emotions such as "excited" or "calm."

[1265] Step 4:

[1266] The text data generated by the speech recognition means is sent from the terminal to the server. The server passes the received text data to the natural language processing means. The server's natural language processing means analyzes the text data and understands the user's intent. Specifically, it uses a machine learning algorithm to interpret the text data as a "weather information request."

[1267] Step 5:

[1268] At the same time as the results of natural language processing, emotional information analyzed by the device's emotion engine is also sent to the server, allowing the server to simultaneously grasp the user's intentions and emotional state.

[1269] Step 6:

[1270] After the user's intent is interpreted and their emotional state is grasped, the server's generation means acquires and generates information. This means acquires information in real time from external databases or APIs (e.g., weather information APIs). Based on the acquired information, the generation means utilizes data from the emotion engine to adjust the response. For example, if the user is excited, the server generates a response that includes calming words such as, "Tomorrow is sunny. It looks like it's going to be a very pleasant day."

[1271] Step 7:

[1272] The text data generated by the generation means is sent to the terminal. The terminal converts the text data into voice data using a voice synthesis means. This means can use existing technology (e.g., a voice synthesis API). In this step, voice data is generated that says, "Tomorrow will be sunny. It looks like it will be a very pleasant day."

[1273] Step 8:

[1274] The generated voice data is provided to the user through the device's speaker. The device then plays back the generated voice data to convey information to the user in an intuitive and easy-to-understand manner, using the tone and speed corresponding to the information provided by the emotion engine.

[1275] Through these steps, users can use the AI ​​to interact with the system through voice and obtain optimal information according to their emotional state. This system provides an environment that is easy to use, even for elderly people and children who are unfamiliar with using computers.

[1276] Example 2

[1277] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1278] Existing speech recognition systems can only provide simple responses to user voice commands and are unable to provide adaptive responses that take the user's emotional state into account. Intuitive and easy-to-understand responses are particularly required for people who are unfamiliar with PC operation, such as the elderly and children. Given this background, there is a need for a system that can recognize a user's emotions and provide natural responses in response to them.

[1279] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring a voice input, a voice recognition means for converting the acquired voice input into text data, an emotion recognition means for recognizing a user's emotion from the analyzed voice data, a natural language processing means for analyzing the text data and understanding the user's intention, a generation means for acquiring and generating information based on the user's intention and emotion, a voice synthesis means for receiving the generated information as text data and converting it into voice data, and a means for providing the voice data to the user. This makes it possible to provide an appropriate and natural voice response according to the user's emotional state.

[1280] The "means for acquiring voice input" refers to a means for collecting voice data from a user using a device such as a microphone and converting it into a format that can be processed within the system.

[1281] The "voice recognition means" is a means for converting acquired voice input into text data and generating character information from voice information.

[1282] The "emotion recognition means" is a means for analyzing voice data and estimating and recognizing the user's emotions from the tone of the sound and speech patterns.

[1283] "Natural language processing means" refers to means that use machine learning algorithms to analyze text data and understand user intent and requirements.

[1284] The "generation means" is a means for obtaining necessary information from an external database or API based on the user's intentions and emotions, and generating a response based on this information.

[1285] The "voice synthesis means" is a means for converting the generated text data into voice data and outputting it in a reproducible voice format.

[1286] The "means for providing the user with voice data" refers to a means for letting the user hear the generated voice data through a speaker or the like.

[1287] This invention is a system for interacting with a user through voice input and providing an appropriate response based on the user's intentions and emotions. Specifically, it includes a means for acquiring voice input, a voice recognition means, an emotion recognition means, a natural language processing means, a generation means, a voice synthesis means, and a means for providing voice data to the user.

[1288] Acquiring voice input

[1289] A user speaks questions or commands into a device such as a smartphone or tablet that supports voice recognition. For example, they might say, "Tell me what the weather will be tomorrow." The device captures the voice through its built-in or external microphone and stores the voice data in a buffer.

[1290] Voice Recognition

[1291] The captured voice data is sent to the device's voice recognition means, which then converts the voice data into text data using existing technologies such as Google's Speech-to-Text API. For example, the text data generated would be "What's the weather going to be like tomorrow?"

[1292] Emotion recognition

[1293] The voice data is simultaneously sent to the device's emotion recognition means, which analyzes the voice tone and speech patterns to recognize the user's emotions. Specifically, the user's emotional state, such as "excited" or "calm," is determined from the voice pitch and speed.

[1294] Natural Language Processing

[1295] The text data generated by the speech recognition means is sent from the terminal to the server. The server passes the text data to the natural language processing means, which uses a machine learning algorithm to analyze the user's intent. This results in the text data being interpreted as a "weather information request."

[1296] Information Acquisition and Generation

[1297] The server's generation means retrieves the necessary information in real time from an external database or API (e.g., a weather information API) based on the user's intentions and emotions. Based on the retrieved information, the generation means generates a natural and relevant response. Since the user's emotional data is also taken into account, the generated response might be something like, "Tomorrow is sunny. It looks like it's going to be a very pleasant day."

[1298] Audio conversion

[1299] The generated text data is sent to the device and converted into voice data by the device's voice synthesis means, using IBM Watson's Text-to-Speech API or similar.

[1300] Providing voice responses

[1301] The generated voice data is provided to the user through the device's speaker, allowing the user to receive an intuitive and easy-to-understand voice response.

[1302] Specific examples

[1303] The following describes the process when a user asks, "What's the weather going to be like tomorrow?" The user's voice input is acquired by the device and converted into text data "What's the weather going to be like tomorrow?" by the voice recognition means. At the same time, the emotion recognition means analyzes the voice tone and recognizes that the user is excited. This text data is sent to the server and interpreted as a "weather information request" by the natural language processing means. Next, the generation means calls a weather information API to acquire tomorrow's weather information and generates a response sentence such as "Tomorrow is sunny. It looks like it's going to be a very pleasant day." Finally, this response sentence is converted into voice data by the voice synthesis means and provided to the user through the device's speaker. This process allows the user to receive an appropriate voice response that takes into account their emotions.

[1304] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1305] Step 1: Getting voice input

[1306] The user speaks a question or command to the device. For example, "Tell me what the weather will be tomorrow." The device captures this voice through its built-in microphone or an external microphone. The captured voice data is stored in a buffer.

[1307] Input: User's voice commands

[1308] Operation: Pick up sound with the device's microphone and save it in a buffer

[1309] Output: Audio data

[1310] Step 2: Voice Recognition

[1311] The acquired voice data is sent to the device's voice recognition means. The device then converts the voice data into text data using a method such as Google's Speech-to-Text API. For example, the generated text data would be, "Tell me what the weather will be like tomorrow."

[1312] Input: Audio data

[1313] Operation: Converts speech to text using the Speech-to-Text API

[1314] Output: Text data "What's the weather going to be like tomorrow?"

[1315] Step 3: Recognize emotions

[1316] The voice data is simultaneously transmitted to the device's emotion recognition means, which analyzes the voice tone and speech pattern to recognize the user's emotion. For example, the device may analyze the voice pitch and speed to determine that the user is "excited."

[1317] Input: Audio data

[1318] Operation: Voice tone and speech pattern analysis

[1319] Output: Emotion data (e.g. "Excited")

[1320] Step 4: Natural Language Processing

[1321] The text data generated by the speech recognition means is sent from the terminal to the server. The server passes the text data to the natural language processing means, which uses a machine learning algorithm to analyze the user's intent. As a result, the text data is interpreted as a "weather information request."

[1322] Input: Text data "What's the weather going to be like tomorrow?"

[1323] Operation: Intention analysis using natural language processing (machine learning algorithms)

[1324] Output: Intent data (e.g., "Request weather information")

[1325] Step 5: Obtaining and generating information

[1326] The server's generation means retrieves information from an external database or API based on the user's intent and emotional data. For example, it retrieves tomorrow's weather information from a weather information API. Based on the retrieved information, the generation means generates a natural and relevant response. Since the user's emotional data is also taken into consideration, a response such as "Tomorrow will be sunny. It looks like it will be a very pleasant day" is generated.

[1327] Input: Intent data (weather information request), emotion data

[1328] Operation: Retrieving information from external databases or APIs and generating appropriate responses

[1329] Output: Response text data (e.g. "Tomorrow is sunny. It looks like it's going to be a very pleasant day.")

[1330] Step 6: Audio conversion

[1331] The generated text data is sent to the device and converted into voice data by the device's voice synthesis means, using IBM Watson's Text-to-Speech API or similar.

[1332] Input: Response text data

[1333] Operation: Convert text data to audio data (using the Text-to-Speech API)

[1334] Output: Speech data (e.g. "Tomorrow is sunny. It looks like it's going to be a very pleasant day.")

[1335] Step 7: Provide a voice response

[1336] The generated voice data is provided to the user through the device's speaker. The device plays back the voice data and conveys information to the user in an intuitive and easy-to-understand format.

[1337] Input: Audio data

[1338] Operation: Play audio data through the speaker

[1339] Output: A spoken response to the user (e.g., "Tomorrow is sunny. It's going to be a very pleasant day.")

[1340] In this way, the system can perform emotion recognition from voice input and provide natural responses according to the user's emotional state.

[1341] (Application example 2)

[1342] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1343] Conventional speech recognition and response systems simply convert text into speech and provide information without considering the user's emotions. This results in a lack of appropriate information provided based on the user's emotions, leading to a decline in user satisfaction, especially in stressful environments or places where frequent guidance is required. Furthermore, improved customer service is required in brick-and-mortar stores, and voice guidance can improve the quality of customer service by providing prompt and accurate guidance. To solve these issues, a system that recognizes the user's emotions and adjusts responses accordingly is needed.

[1344]

[1345] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1346] In this invention, the server includes means for acquiring voice input, speech recognition means for converting the acquired voice input into text data, natural language processing means for analyzing the text data and understanding the user's intention, generation means for acquiring and generating information based on the user's intention, speech synthesis means for receiving the generated information as text data and converting it into voice data, means for providing the voice data to the user, emotion analysis means for recognizing the user's emotion, and emotion adjustment means for adjusting responses based on the user's emotional state. This makes it possible to provide information that is appropriately adjusted according to the user's emotion, thereby improving the quality of customer service, particularly in providing guidance and information in physical stores.

[1347] "Audio input" refers to audio signals emitted by a user to the system through a microphone.

[1348] "Speech recognition means" refers to technology that converts voice signals into text data.

[1349] "Natural language processing means" refers to technology for analyzing text data and understanding the user's intent.

[1350] "Generation means" refers to a technology that acquires information based on the user's intentions and generates text data based on that information.

[1351] "Speech synthesis means" refers to a technology that converts generated text data into voice data.

[1352] "Means for providing audio data to a user" refers to technology for transmitting generated audio data to a user through a speaker or the like.

[1353] "Emotion analysis means" refers to technology that recognizes a user's emotions by analyzing the user's tone of voice and speech patterns.

[1354] "Emotion adjustment means" refers to a technique for adjusting the content of a response based on the recognized emotions of a user.

[1355] A system for implementing the present invention includes means for obtaining speech input, speech recognition means, natural language processing means, generation means, emotion analysis means, emotion modulation means, speech synthesis means, and means for providing a speech response.

[1356] Hardware and software used

[1357] Hardware:

[1358] Smart glasses (with built-in microphone, speaker, and camera)

[1359] software:

[1360] Speech recognition API (e.g., converting voice data into text using a speech recognition API)

[1361] Natural language processing APIs (e.g., using natural language processing APIs to analyze text data)

[1362] Speech synthesis API (e.g., converting text data into speech data using a speech synthesis API)

[1363] Emotion analysis API (analyzes voice tone and speech patterns)

[1364] System processing overview

[1365] Acquiring voice input

[1366] The user speaks a question into the microphone of the smart glasses. An example question is "Where is the bread section?" This voice signal is picked up by the microphone of the smart glasses.

[1367] Voice Recognition

[1368] The acquired voice signal is converted into text data using a speech recognition API, which generates the text data "Where is the bread section?"

[1369] emotion recognition

[1370] At the same time, an emotion analysis API is used to analyze voice tone and speech patterns to recognize the user's emotions, for example, to determine whether the user is excited or calm.

[1371] Natural Language Processing

[1372] The generated text data is sent to a natural language processing API, which analyzes the user's intent. In this case, the intent is understood to be "directions to the bread section."

[1373] Information Acquisition and Generation

[1374] Once the user's intention is analyzed, the generation means obtains related information from the cloud server of the smart glasses and generates a response. For example, specific location information of the bread section is obtained, and a response sentence such as "The bread section is at the end of the aisle on the left" is generated as text data.

[1375] emotional regulation

[1376] The emotion adjustment means adjusts the response sentence based on the recognized emotion of the user, for example, adding calming words if the user is excited.

[1377] Audio conversion

[1378] The generated response is converted into voice data using a speech synthesis API, which generates the voice data, "The bread section is down the aisle on the left, straight ahead."

[1379] Providing voice responses

[1380] Finally, the generated audio data is provided to the user through the speakers of the smart glasses.

[1381] Specific examples

[1382] For example, if a user asks, "What are the current sales?" the same series of processes are performed, providing the following concrete example:

[1383] Question: "What are the current sales?"

[1384] Example prompt: Tell me about this week's sales.

[1385] Response: "All drinks are 10% off this week. Give them a try! It's a great deal."

[1386] This provides a highly convenient shopping experience, allowing users to easily obtain information in-store and find the products they need.

[1387] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1388] Step 1:

[1389] Acquiring voice input

[1390] The user speaks a question into the microphone of the smart glasses. An example question here is "Where is the bread section?" The microphone built into the smart glasses picks up the voice signal and stores it in a buffer as voice data. This voice data is then sent to the next step of speech recognition.

[1391] Input: User speech ("Where is the bread section?")

[1392] Output: Audio data (audio signal)

[1393] Step 2:

[1394] Voice Recognition

[1395] The device's speech recognition API receives the voice data stored in the buffer. The speech recognition API analyzes the voice signal, breaks it down into phonemes, and generates corresponding text data. The generated text data is "Where is the bread section?"

[1396] Input: Audio data

[1397] Output: Text data ("Where is the bread section?")

[1398] Step 3:

[1399] emotion recognition

[1400] The device's emotion analysis API receives the voice data. The emotion analysis API analyzes the voice tone and speech patterns to recognize the user's emotional state. For example, it analyzes the voice tone, speaking speed, and strength to determine whether the user is in a "calm" state.

[1401] Input: Audio data

[1402] Output: Emotion data ("Calm")

[1403] Step 4:

[1404] Natural Language Processing

[1405] The text data is sent to the server's natural language processing API. The natural language processing API analyzes the text data and understands the user's intent. As a result of the analysis, it is determined that the user is asking where the bread section is. This analysis result is passed to the generation means.

[1406] Input: Text data ("Where is the bread section?")

[1407] Output: Intent data ("Where is the bread section?")

[1408] Step 5:

[1409] Information Acquisition and Generation

[1410] The server's generation means retrieves location information for the bread section from an external database in real time. For example, it executes a query to the database and retrieves information such as "The bread section is located at the end of the aisle on the left." A response sentence is generated based on the retrieved information. The generated response sentence is "The bread section is located at the end of the aisle on the left."

[1411] Input: Intent data ("location of the bread section")

[1412] Output: Response ("The bread section is down the left aisle, straight ahead.")

[1413] Step 6:

[1414] emotional regulation

[1415] The emotion adjustment means adjusts the generated response sentence based on the recognized emotion data ("calm"). If the emotion data is "calm", it is determined that no additional adjustment is necessary for the response sentence, and the response sentence is used as is.

[1416] Input: Response ("The bread section is down the left aisle, straight ahead."), Emotion ("Calm")

[1417] Output: Adjusted response ("The bakery is down the left aisle, straight ahead.")

[1418] Step 7:

[1419] Audio conversion

[1420] The generated text data is sent to the device's speech synthesis API and converted into voice data. The speech synthesis API converts the text data into voice signals, generating voice data such as, "The bread section is down the aisle on the left, straight ahead."

[1421] Input: Adjusted response ("The bakery is down the left aisle, straight ahead.")

[1422] Output: Speech data ("The bread section is down the left aisle, straight ahead.")

[1423] Step 8:

[1424] Providing voice responses

[1425] Finally, the generated voice data is provided to the user through the smart glasses speaker, and the user receives the voice data stating, "The bread section is down the left aisle, straight ahead."

[1426] Input: Speech data ("The bread section is down the left aisle, straight ahead.")

[1427] Output: Voice guidance ("The bread section is down the left aisle, straight ahead.")

[1428] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1429] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1430] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1431] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1432] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1433] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1434] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1435] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1436] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1437] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1438] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1439] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1440] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1441] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1442] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1443] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1444] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1445] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1446] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1447] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1448] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1449] The following is further disclosed regarding the above embodiment.

[1450] (Claim 1)

[1451] a means for obtaining a voice input;

[1452] a speech recognition means for converting the acquired speech input into text data;

[1453] natural language processing means for analyzing text data and understanding user intent;

[1454] A generating means for acquiring and generating information based on a user's intention;

[1455] a voice synthesis means for receiving the generated information as text data and converting it into voice data;

[1456] means for providing audio data to a user;

[1457] A system including:

[1458] (Claim 2)

[1459] 10. The system of claim 1, wherein the natural language processing means uses machine learning algorithms to analyze the user's intent.

[1460] (Claim 3)

[1461] 2. The system of claim 1, wherein the generating means obtains information in real time from an external database.

[1462] "Example 1"

[1463] (Claim 1)

[1464] a means for obtaining a voice input;

[1465] a speech recognition means for converting the acquired speech input into text data;

[1466] natural language processing means for analyzing text data and understanding user intent;

[1467] an information generating means for acquiring and generating information based on a user's intention;

[1468] a voice synthesis means for receiving the generated information as text data and converting it into voice data;

[1469] means for providing audio data to a user;

[1470] A system including:

[1471] (Claim 2)

[1472] 10. The system of claim 1, wherein the natural language processing means uses machine learning algorithms to analyze the user's intent.

[1473] (Claim 3)

[1474] 2. The system according to claim 1, wherein the information generating means obtains information in real time from an external database.

[1475] (Claim 4)

[1476] 2. The system according to claim 1, wherein the speech recognition means, the natural language processing means, and the speech synthesis means utilize a speech recognition API, a natural language processing API, and a speech synthesis API, respectively.

[1477] (Claim 5)

[1478] 2. The system according to claim 1, wherein the voice data acquired by the voice input means is stored in a buffer and is sequentially transmitted to the voice recognition means.

[1479] (Claim 6)

[1480] 2. The system according to claim 1, wherein the speech synthesis means instantly converts the generated text data into speech data, and the providing means reproduces the speech data to the user.

[1481] "Application Example 1"

[1482] (Claim 1)

[1483] a means for obtaining a voice input;

[1484] a speech recognition means for converting the acquired speech input into text data;

[1485] natural language processing means for analyzing text data and understanding user intent;

[1486] A generating means for acquiring and generating information based on a user's intention;

[1487] a voice synthesis means for receiving the generated information as text data and converting it into voice data;

[1488] means for providing audio data to a user;

[1489] A system that provides navigation and information services in a vehicle and includes multiple means for responding to voice commands.

[1490] (Claim 2)

[1491] 10. The system of claim 1, wherein the natural language processing means uses machine learning algorithms to analyze the user's intent.

[1492] (Claim 3)

[1493] 2. The system of claim 1, wherein the generating means obtains information in real time from an external database.

[1494] "Example 2: Combining Emotion Engines"

[1495] (Claim 1)

[1496] a means for obtaining a voice input;

[1497] a speech recognition means for converting the acquired speech input into text data;

[1498] emotion recognition means for recognizing the emotion of a user from the analyzed voice data;

[1499] natural language processing means for analyzing text data and understanding user intent;

[1500] A generating means for acquiring and generating information based on the user's intentions and emotions;

[1501] a voice synthesis means for receiving the generated information as text data and converting it into voice data;

[1502] means for providing audio data to a user;

[1503] A system including:

[1504] (Claim 2)

[1505] 10. The system of claim 1, wherein the natural language processing means uses machine learning algorithms to analyze the user's intent.

[1506] (Claim 3)

[1507] 2. The system of claim 1, wherein the generating means obtains information in real time from an external database.

[1508] "Application example 2 when combining emotion engines"

[1509] (Claim 1)

[1510] a means for obtaining a voice input;

[1511] a speech recognition means for converting the acquired speech input into text data;

[1512] natural language processing means for analyzing text data and understanding user intent;

[1513] A generating means for acquiring and generating information based on a user's intention;

[1514] a voice synthesis means for receiving the generated information as text data and converting it into voice data;

[1515] means for providing audio data to a user;

[1516] emotion analysis means for recognizing the emotion of a user;

[1517] emotion adjustment means for adjusting a response based on the user's emotional state;

[1518] A system including:

[1519] (Claim 2)

[1520] 10. The system of claim 1, wherein the natural language processing means uses machine learning algorithms to analyze the user's intent.

[1521] (Claim 3)

[1522] 2. The system of claim 1, wherein the generating means obtains information in real time from an external database. [Explanation of symbols]

[1523] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for obtaining a voice input; a speech recognition means for converting the acquired speech input into text data; natural language processing means for analyzing text data and understanding user intent; A generating means for acquiring and generating information based on a user's intention; a voice synthesis means for receiving the generated information as text data and converting it into voice data; means for providing audio data to a user; A system including:

2. 10. The system of claim 1, wherein the natural language processing means uses machine learning algorithms to analyze the user's intent.

3. 2. The system of claim 1, wherein the generating means obtains information in real time from an external database.

Images