System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving an input of a user and converting the input into text data; means for analyzing the text data and grasping an intention of the user; means for generating an appropriate response based on an analysis result; means for acquiring necessary information and reflecting the information on a response in cooperation with an external service; and means for providing the response to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, balancing work and family life is a major challenge for many individuals. Office workers raising children in particular face various pressures regarding time and health management. In these circumstances, there is a need for methods that provide appropriate support and enable users to live their daily lives efficiently and comfortably. Conventional systems have had difficulty accurately grasping users' emotions and situations and connecting with necessary external services. Therefore, it is necessary to develop a system that can quickly and accurately provide the support users require. [Means for solving the problem]

[0005] The present invention relates to a system that includes a means for receiving user input and converting the input into text data, a means for analyzing the text data to understand the user's intent, a means for generating an appropriate response based on the analysis results, a means for acquiring necessary information through collaboration with an external service and incorporating it into the response, and a means for providing the response to the user. Furthermore, the response generation means includes a means for analyzing the emotions in the text data and generating a response based on those emotions, thereby realizing support that takes into account the user's emotions. Furthermore, the means for collaborating with external services accesses a reservation service via an API, acquires reservation status, and provides it to the user, allowing the user to obtain the information they need without any effort. In this way, the present invention aims to provide optimal support tailored to the user's needs and solve problems such as time and health management.

[0006] "User" means any individual or legal entity that uses the System.

[0007] "Input" means voice, text, or other data provided by a user to a system.

[0008] "Text data" refers to textual information converted by voice recognition or other methods.

[0009] "Analysis" is the process of extracting meaning and intent from text data.

[0010] "Intention" refers to the goal or desire that the user is trying to achieve through the system.

[0011] A "response" is any information or instruction that a system provides to a user.

[0012] "External services" refer to other online services or APIs that the system connects to.

[0013] "API" stands for Application Programming Interface, an interface that allows different software components to communicate with each other.

[0014] "Sentiment analysis" is a technique for inferring user emotions from text data.

[0015] A "reservation service" is an online system for reserving specific times or resources in advance.

[0016] "Get information" refers to the process of gathering the necessary data from external services.

[0017] "Provision" means that the system communicates analysis results and information obtained from external services to the user. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention relates to a system that receives user input, converts the input into text data, analyzes it to generate an appropriate response, and provides the response to the user in cooperation with an external service.

[0040] System configuration

[0041] The system mainly consists of the following components:

[0042] 1. User terminal: A device used by a user that has the ability to receive voice input and send it to a server.

[0043] 2. Server: A central device that manages the speech recognition engine, natural language processing engine, generative AI model, and collaboration with external services.

[0044] 3. External services: Online services that connect to the system via external APIs, such as reservation services.

[0045] System Operation Overview

[0046] The user inputs a request such as a refresh or reservation into the user terminal by voice. The terminal sends this voice data to the server. The server performs the following process and provides the final response to the user.

[0047] Program processing explanation

[0048] 1. Accept user input:

[0049] The user speaks to the device, saying, "I'm tired and want to refresh myself."

[0050] The device records this audio data and sends it to the server.

[0051] 2. Speech to text conversion:

[0052] The server uses a voice recognition engine to convert the received voice data into text data.

[0053] This conversion generates the text "I'm tired so I want to refresh myself."

[0054] 3. Text Analysis:

[0055] The server uses a natural language processing (NLP) engine to analyze the text data.

[0056] The NLP engine understands the meaning and structure of the text and identifies the user's intent.

[0057] For example, understand that the user wants refresh suggestions.

[0058] 4. Sentiment analysis:

[0059] The server uses a sentiment analysis engine to analyze the sentiment of the text data.

[0060] Here, the user's emotions, such as "tired," are identified.

[0061] 5. Generating appropriate responses:

[0062] The server uses a generative AI model to generate a response based on the user's emotions and intent.

[0063] In response generation, suggestions such as "How about some yoga to relax?"

[0064] 6. Decision on external service integration:

[0065] Based on the response generated by the server, it determines whether to link with an external service.

[0066] If it is determined that a yoga studio reservation is necessary, an external service API is called.

[0067] 7. Calling external service APIs:

[0068] The server checks the reservation status through an external service API.

[0069] For example, get information about available slots at a yoga studio.

[0070] 8. Generate the final response:

[0071] The server integrates the responses of the generated AI model with information obtained from external services.

[0072] Generate the final response "Yoga studio reservations available. Available tomorrow at 5:00 PM."

[0073] 9. Providing Responses to Users:

[0074] The device displays the final response received from the server to the user.

[0075] This allows users to receive specific support, such as booking yoga studios.

[0076] In this way, the system provides optimal support according to the user's needs and effectively resolves issues related to time and health management. For example, for a user who is tired from work and wants to refresh themselves, the system provides accurate yoga suggestions and reservation information. This system allows users to live their daily lives efficiently and comfortably.

[0077] The processing flow will be explained below.

[0078] Step 1:

[0079] The user speaks to the device, saying, "I'm tired, so I want to refresh myself." The device records this speech and sends it to the server as audio data.

[0080] Step 2:

[0081] The server uses a speech recognition engine to convert the received voice data into text data. The speech recognition engine analyzes the voice data and generates the text "I'm tired, so I want to refresh myself."

[0082] Step 3:

[0083] The server uses a natural language processing (NLP) engine to analyze the generated text data. The NLP engine parses the text and understands the user's intent as "I'm looking for refresh suggestions."

[0084] Step 4:

[0085] The server uses a sentiment analysis engine to determine the sentiment of the text data, identifying emotions such as "tired" and passing the results on to the next step of processing.

[0086] Step 5:

[0087] The server uses the generative AI model to generate a response appropriate to the user's emotions and needs. Based on the instruction "Suggest yoga to refresh yourself," the generative AI model creates a response message saying, "How about yoga to relax?"

[0088] Step 6:

[0089] The server analyzes the generated response message and determines whether it is necessary to connect to an external service. From the generated message, it determines that "a reservation at a yoga studio is required."

[0090] Step 7:

[0091] The server calls an external service API and sends a request to get reservation information for the yoga studio. The API checks for available slots.

[0092] Step 8:

[0093] The external service receives the request and returns available slot information to the server, such as "Tomorrow is available at 5:00 PM."

[0094] Step 9:

[0095] The server combines the response of the generated AI model with the reservation information received from the external service, and creates a final response message saying, "Yoga studio reservations are available. We have availability tomorrow at 5:00 PM."

[0096] Step 10:

[0097] The server sends a final response message to the device, which receives it and displays it to the user. The user can then take specific action, such as booking a yoga studio, based on the displayed information.

[0098] This process allows users to receive efficient and personalized support.

[0099] Example 1

[0100] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0101] Current dialogue systems must integrate multiple technologies, such as speech recognition, natural language processing, and sentiment analysis, to respond appropriately to diverse user requests. However, existing systems do not fully integrate these functions, making it difficult to accurately grasp the user's intentions and emotions and provide appropriate responses. Furthermore, they are unable to efficiently connect with external services, making it difficult to quickly provide the information and services users desire.

[0102] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0103] In this invention, the server includes means for receiving voice input and converting it into voice data, means for converting the voice data into text data, means for analyzing the text data to understand the user's intention, means for analyzing the sentiment of the text data, means for generating an appropriate response based on the analysis results and the sentiment analysis results, means for acquiring necessary information in cooperation with an external service and reflecting it in the response, and means for providing the response to the user, thereby enabling accurate and prompt responses to a variety of user requests.

[0104] The "means for receiving and converting voice input into voice data" is a component that has the function of converting the contents of a user's voice input into digital voice data.

[0105] The "means for converting the voice data into text data" is a component that has the function of analyzing digital voice data and converting it into corresponding text format information.

[0106] The "means for analyzing the text data to understand the user's intention" is a component that has the function of analyzing the text data using natural language processing technology and identifying the intention implied by the user's utterance.

[0107] The "means for analyzing the emotion of the text data" is a component that has a function of identifying the emotion of the user based on the analyzed text data.

[0108] The "means for generating an appropriate response based on the analysis results and emotion analysis results" is a component that has the function of generating the optimal response to be provided to the user based on the user's intentions and emotions.

[0109] "Means of interacting with external services to obtain necessary information and reflect it in the response" refers to a component that has the function of accessing the APIs of other external systems and services, obtaining necessary data and information, and incorporating it into the response to the user.

[0110] The "means for providing the response to the user" is a component having an interface or function for transmitting the generated response to the user.

[0111] A "generative AI model" is a machine learning model that learns from large amounts of data in advance and performs natural language generation and text analysis.

[0112] A "prompt" is a textual instruction or question input to a generative AI model that is used to elicit a specific response or generated result.

[0113] This invention relates to a system that converts voice input into text, analyzes it to generate appropriate responses, and links with external services. How the invention is realized using specific hardware and software is described below.

[0114] Hardware and software used

[0115] The system is implemented primarily using the following hardware and software:

[0116] User terminal: A device that allows users to input voice, and is equipped with a microphone and network connection capabilities.

[0117] Server: A central device that manages the speech recognition engine, natural language processing engine, generative AI model, and collaboration with external services. Here, we use Google® Cloud Speech-to-Text API, OpenAI® GPT-4®, and IBM Watson® Tone Analyzer.

[0118] Accepting and converting voice input

[0119] The user speaks, "I'm tired, so I want to refresh myself." The device records this voice data and sends it over the network to a server. The server then converts the received voice data into text data using the Google Cloud Speech-to-Text API.

[0120] Text data analysis

[0121] The server uses OpenAI's GPT-4 to analyze the converted text data and understand the user's intent. At the same time, it uses IBM Watson Tone Analyzer to analyze the sentiment of the text data. This identifies that the user's intent is "I want to refresh myself" and that the sentiment is "I'm tired."

[0122] Generating a response

[0123] The server uses OpenAI's ChatGPT (registered trademark), a generative AI model, to generate responses based on the user's intent and emotions. Examples of prompts are as follows:

[0124] User: I'm tired from work and want to refresh myself.

[0125] System: How about some yoga to relax? There's a class available at John Smith Studio at 5:00 PM. Would you like to book it?

[0126] Thus, the suggestion: "How about some yoga to relax?"

[0127] Integration with external services

[0128] The server then decides whether to connect to an external service based on the generated response. For example, if it determines that a yoga studio reservation is required, it checks the reservation status through the external service's API. Specifically, it accesses the reservation system's API and obtains information about available slots at the yoga studio.

[0129] Providing a final response

[0130] The server combines the generated response with information obtained from external services to generate a final response: "You can make a reservation at the yoga studio. It's available tomorrow at 5:00 PM." This specific response is provided to the user via the device.

[0131] In this way, the system receives and analyzes the user's voice input, generates appropriate responses, and provides them to the user. For example, if a user is tired from work, the system can provide yoga suggestions and reservation information, allowing the user to live their daily life efficiently and comfortably.

[0132] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0133] Step 1:

[0134] The user speaks, "I'm tired, so I want to refresh myself." The device records this voice data. The input is voice data, which is then converted into a recording. Specifically, the device uses a microphone to capture the voice signal and saves it as a digital audio file.

[0135] Step 2:

[0136] The device sends the recorded audio data to the server. The input is a digital audio file, which is then sent to the server over the network. Specifically, the audio file is uploaded using an HTTP request over the internet connection.

[0137] Step 3:

[0138] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text data. The input is a digital audio file, and the output is text data. Specifically, the server passes the audio file to the API and obtains the text, "I'm tired, so I want to refresh myself."

[0139] Step 4:

[0140] The server uses OpenAI's GPT-4 to analyze the text data and understand the user's intent. The input is the converted text data, and the output is the analysis result that identifies the user's intent. Specifically, the text is passed to GPT-4, which understands that the user is requesting a refresh suggestion.

[0141] Step 5:

[0142] The server uses IBM Watson Tone Analyzer to analyze the sentiment of the text data. The input is the text data, and the output is the sentiment analysis result. Specifically, the text is passed to the analysis engine to identify the sentiment of "tired."

[0143] Step 6:

[0144] The server uses OpenAI's ChatGPT, a generative AI model, to generate a response based on the user's intent and emotions. The input is the user's intent and the results of the emotion analysis, and the output is the generated response. Specifically, the following prompt is passed to ChatGPT:

[0145] User: I'm tired and want to refresh myself.

[0146] System: How about yoga to relax?

[0147] I get the response:

[0148] Step 7:

[0149] The server determines whether to collaborate with an external service based on the generated response. The input is the generated response, and the output is a decision on whether to collaborate with an external service. Specifically, since the response contains "yoga suggestions," the server determines that a reservation at a yoga studio is necessary.

[0150] Step 8:

[0151] The server obtains available slot information through the reservation system's API. The input is a reservation request, and the output is available slot information. Specifically, the server obtains information such as "Available tomorrow at 5:00 PM" through the API.

[0152] Step 9:

[0153] The server integrates the generated response with information from the external service to generate a final response. The input is the generated response and information obtained from the external service, and the output is the final response. Specifically, it generates a response that says, "You can make a reservation at the yoga studio. There is an opening tomorrow at 5:00 PM."

[0154] Step 10:

[0155] The terminal displays the final response received from the server to the user. The input is the final response, and the output is the response displayed on the user's terminal screen. Specifically, the display shows "Yoga studio reservations are available. Available tomorrow at 5:00 PM."

[0156] (Application example 1)

[0157] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0158] Conventional voice input systems lack the ability to convert voice data into text and perform text analysis in the process of generating appropriate responses from user speech. Furthermore, integration with external services is limited, making it difficult to provide real-time information, such as inventory inquiries and reservation services. Furthermore, the lack of visual responses provided via smart devices results in a suboptimal user experience. To solve these problems, a system that utilizes more advanced natural language processing, external service integration, and a visual interface on smart devices is required.

[0159] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0160] In this invention, the server includes means for receiving user input and converting the input into text data, means for analyzing the text data to understand the user's intent, means for generating an appropriate response based on the analysis results, means for acquiring necessary information in cooperation with an external service and reflecting the information in the response, means for generating a detailed response based on the user's intent using a generative AI model, and means for providing the response through a visual interface of a smart device. This enables real-time and accurate information to be provided in response to a variety of user requests.

[0161] "User input" refers to information provided by a user to a system in the form of speech or text.

[0162] "Text data" is data in the form of a character string converted from voice data via a voice recognition engine.

[0163] "Analysis" refers to the process of using a natural language processing engine to understand the meaning and intent of text data.

[0164] An "appropriate response" is one that generates information or instructions that are in line with the user's intentions based on the analysis results.

[0165] "External services" refers to external online services and databases that are linked via an application program interface (API).

[0166] A "generative AI model" is an artificial intelligence model that uses deep learning and other techniques to generate text data based on the user's intentions and emotions.

[0167] "Smart devices" refers to information terminals with internet connectivity, such as smartphones, smart glasses, and head-mounted displays.

[0168] "Visual interface" refers to a means of interacting with users through information and messages displayed on the screen of a smart device.

[0169] We will explain the system to realize this application example. This system mainly consists of the following components:

[0170] User terminal: A smart device such as smart glasses that has the ability to receive voice input and send it to a server.

[0171] Server: A central device that manages the voice recognition engine, natural language processing engine, generative AI model, and collaboration with external services.

[0172] External services: Online services that connect to the system via external APIs, such as transaction information or inventory status.

[0173] System Operation Overview

[0174] Accepting user input

[0175] The user speaks to the smart glasses, saying, "Please show me the cosmetics section." The smart glasses receive this voice input and send it to the server. The user's device records the voice data and sends it to the server.

[0176] Speech-to-text conversion

[0177] The server uses a speech recognition engine (for example, Google Cloud Speech-to-Text API) to convert the received voice data into text data. This conversion generates the text "Please show me the cosmetics section."

[0178] Text Analysis

[0179] The server uses a natural language processing engine (e.g., IBM Watson NLP API) to analyze the text data and identify the user's intent. The NLP engine understands the meaning and structure of the text and understands that the user is seeking guidance.

[0180] Generating an appropriate response

[0181] The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate a response based on the user's intent, such as "The cosmetics section is on the second floor. Please take the escalator."

[0182] Deciding whether to link with external services

[0183] The server determines whether or not it needs to connect to an external service based on the generated response. For example, if a user inquires about the stock of a specific product, it determines that it needs to check the stock information.

[0184] Calling external service APIs

[0185] The server connects to the store's inventory management system via an external service API to check the stock status of a specific product. For example, it performs a process such as "calling an API to obtain stock information and obtaining the stock status of lipstick."

[0186] Generate the final response

[0187] The server combines the generated AI model's response with information obtained from external services to generate a final response, such as "The cosmetics section is on the second floor. Please take the escalator. The lipstick you are looking for is in stock."

[0188] Providing a response to the user

[0189] The user device (smart glasses) provides the final response received from the server in a display format, allowing the user to visually confirm specific guidance information in response to voice input.

[0190] Examples of specific examples and prompts

[0191] As a concrete example, consider a case where a user asks "What is the stock status of lipstick?" The server converts the speech to text and then inputs the following prompt into the generative AI model to get a response:

[0192] Example prompt sentence:

[0193] User Question: What is the availability of lipstick?

[0194] Generate an appropriate response.

[0195] Example response:

[0196] "Lipstick is in stock. The cosmetics section is on the second floor. Please take the escalator."

[0197] In this way, a system can be built that generates appropriate responses in real time based on the user's voice input and delivers them through smart devices.

[0198] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0199] Step 1:

[0200] The user inputs the information by voice. The user speaks to the smart glasses, saying, "Please show me the cosmetics section." The voice input (voice data) is sent to the device.

[0201] Step 2:

[0202] The device records the audio data and sends it to the server. Input: Audio data. Output: Audio data sent to the server.

[0203] Step 3:

[0204] The server uses a speech recognition engine to convert the voice data into text data. Input: Voice data. Output: Text data saying "Please show me to the cosmetics section." Specific operation: Uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API).

[0205] Step 4:

[0206] The server uses a natural language processing engine to analyze the text data and identify the user's intent. Input: Text data. Output: User intent (e.g., "I need guidance"). Specific behavior: Uses a natural language processing engine (e.g., IBM Watson NLP API).

[0207] Step 5:

[0208] The server uses a generative AI model to generate a response based on the user's intent. Input: User's intent. Output: Response text (e.g., "The cosmetics section is on the second floor. Please use the escalator."). Specific behavior: Uses a generative AI model (e.g., OpenAI GPT-4).

[0209] Step 6:

[0210] The server determines whether or not integration with an external service is necessary based on the generated response. Input: Response text. Output: Necessity of integration with an external service (e.g., "Stock information needs to be confirmed").

[0211] Step 7:

[0212] If necessary, the server calls an external service API to check the stock status of a specific product. Input: Product name. Output: Stock status data. Specific operation: Calls an external service API (e.g., inventory management system API) to obtain stock information.

[0213] Step 8:

[0214] The server combines the generated AI model's response with information obtained from external services. Input: Response text, stock status data. Output: Final response text (e.g., "The cosmetics section is on the second floor. Please take the escalator. The lipstick you are looking for is in stock.").

[0215] Step 9:

[0216] Provides the final response received by the user device from the server in a display format. Input: Final response text. Output: Response displayed in the user's field of view.

[0217] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0218] This invention relates to a system that converts user input into text data, analyzes emotions and intentions, and generates appropriate responses. It also includes a means for linking with external services to obtain necessary information and reflect it in the response. The invention particularly aims to provide responses based on the user's emotions by combining an emotion engine, thereby optimizing support for the user.

[0219] System configuration

[0220] The system consists of the following main components:

[0221] 1. User terminal: A device used by the user (smartphone, tablet, PC, etc.) that has the function of receiving voice input and sending it to the server.

[0222] 2. Server: A central device that manages the speech recognition engine, natural language processing engine, generative AI model, emotion engine, and collaboration with external services.

[0223] 3. External services: These are online services that connect to the system through APIs, such as reservation services.

[0224] System Operation Overview

[0225] The user inputs a request such as a refresh or reservation into the user terminal by voice. The terminal sends this voice data to the server. The server performs the following process and provides the final response to the user.

[0226] Program processing explanation

[0227] 1. Accept user input:

[0228] The user speaks to the device, saying, "I'm tired and want to refresh myself."

[0229] The device records this audio data and sends it to the server.

[0230] 2. Speech to text conversion:

[0231] The server converts the received voice data into text data using a voice recognition engine.

[0232] The voice recognition engine analyzes the voice data and generates the text "I'm tired, so I want to refresh myself."

[0233] 3. Text Analysis:

[0234] The server uses a natural language processing (NLP) engine to analyze the generated text data.

[0235] The NLP engine parses the text and understands the user's intent as "I'm looking for refresh suggestions."

[0236] 4. Sentiment analysis:

[0237] The server uses an emotion engine to determine the emotion of the text data.

[0238] The emotion engine identifies emotions such as "tired" and passes the results on to the next process.

[0239] 5. Generating appropriate responses:

[0240] The server uses generative AI models to generate responses that are appropriate to the user's emotions and needs.

[0241] Based on the instruction "Suggest yoga to refresh," the generative AI model creates a response message saying, "How about yoga to relax?"

[0242] 6. Decision on external service integration:

[0243] The server analyzes the generated response message and determines whether it is necessary to link with an external service.

[0244] From the generated message, it is determined that "a reservation at a yoga studio is required."

[0245] 7. Calling external service APIs:

[0246] The server calls an external service API and sends a request to get reservation information for a yoga studio.

[0247] Check available slots via API.

[0248] 8. External service response processing:

[0249] The external service receives the request and returns available slot information to the server, such as "Tomorrow is available at 5:00 PM."

[0250] 9. Generate the final response:

[0251] The server integrates the response of the generated AI model with the reservation information received from the external service.

[0252] As a final response, create a message that reads, "Yoga studio reservations are available. We're available tomorrow at 5:00 PM."

[0253] 10. Providing Responses to Users:

[0254] The server sends a final response message to the terminal, which receives it and displays it to the user.

[0255] Based on the information displayed, users can take specific actions, such as booking a yoga studio.

[0256] Through this series of processes, users can receive efficient and individualized support. One feature of this system is that it generates responses taking into account the user's emotions, thereby improving the user experience. For example, if a user feels like "refreshing" after returning home tired from work, the system can not only provide information but also make appropriate suggestions based on that emotion.

[0257] The processing flow will be explained below.

[0258] Step 1:

[0259] The user speaks to the device, saying, "I'm tired, so I want to refresh myself." The device records this speech and sends it to the server as audio data.

[0260] Step 2:

[0261] The server uses a speech recognition engine to convert the received voice data into text data. The speech recognition engine analyzes the voice data and generates the text "I'm tired, so I want to refresh myself."

[0262] Step 3:

[0263] The server uses a natural language processing (NLP) engine to analyze the generated text data. The NLP engine parses the text and understands the user's intent as "I'm looking for refresh suggestions."

[0264] Step 4:

[0265] The server uses an emotion engine to determine the emotion of the text data. The emotion engine identifies an emotion such as "tired" and passes the result on to the next process.

[0266] Step 5:

[0267] The server uses the generative AI model to generate a response appropriate to the user's emotions and needs. Based on the instruction "Suggest yoga to refresh yourself," the generative AI model creates a response message saying, "How about yoga to relax?"

[0268] Step 6:

[0269] The server analyzes the generated response message and determines whether it is necessary to connect to an external service. From the generated message, it determines that "a reservation at a yoga studio is required."

[0270] Step 7:

[0271] The server calls an external service API and sends a request to get reservation information for the yoga studio. The API checks for available slots.

[0272] Step 8:

[0273] The external service receives the request and returns available slot information to the server, such as "Tomorrow is available at 5:00 PM."

[0274] Step 9:

[0275] The server combines the response of the generated AI model with the reservation information received from the external service, and creates a final response message saying, "Yoga studio reservations are available. We have availability tomorrow at 5:00 PM."

[0276] Step 10:

[0277] The server sends a final response message to the device, which receives it and displays it to the user. The user can then take specific action, such as booking a yoga studio, based on the displayed information.

[0278] This process allows users to receive efficient and personalized support.

[0279] Example 2

[0280] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0281] Many current systems can receive user voice input, but they have limitations in their ability to fully understand the user's emotions and intentions and generate appropriate responses. They also have difficulty linking with external services to obtain necessary information and provide consistent responses to users. This results in a poor user experience and lowers user satisfaction. The objective of this invention is to provide a system that generates responses based on the user's emotions and intentions and links with external services to provide high-quality support.

[0282] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0283] In this invention, the server includes means for receiving a user's voice input and converting the voice into text data, means for analyzing the text data to understand the user's intention, means for analyzing the emotion in the text data and generating a response according to the emotion, means for analyzing the generated response and coordinating with an external service to obtain necessary information and reflect this in the response, and means for providing the final response to the user. This allows the user to receive an appropriate response based on their own emotions and intentions. Furthermore, by coordinating with an external service to obtain necessary information, the server can receive consistent, high-quality support.

[0284] "Voice input" refers to instructions or requests that a user gives to a system through voice.

[0285] "Text data" refers to textual information converted from voice input, and is digital information in a format that can be understood by humans.

[0286] "Sentiment analysis" is the process of detecting emotional information contained in text data and identifying that emotion.

[0287] A "natural language processing engine" is software or algorithms that analyze text data and perform syntactic analysis and extract intent.

[0288] "Response generation" is the process of creating an appropriate response message based on the analyzed intentions and emotions.

[0289] "External Services" refers to online services that connect to the system through APIs and provide additional information or functionality.

[0290] "Reservation information" refers to detailed information such as available reservation dates, times, and locations obtained from external services.

[0291] "Final Response" refers to the final form of the response message generated for delivery to the user.

[0292] "User experience" is a general term for the ease of use and satisfaction that users feel when using a system.

[0293] This invention relates to a system that receives a user's voice input, converts the voice into text data, analyzes the text data to understand the user's intentions and emotions, generates an appropriate response, and works with external services to obtain necessary information and provide it to the user.

[0294] The system mainly consists of the following components:

[0295] 1. User device: A device (smartphone, tablet, PC, etc.) that allows the user to input voice data and has the function of sending voice data to a server.

[0296] 2. Server: A central device that manages the speech recognition engine, natural language processing engine, generative AI model, emotion engine, and collaboration with external services.

[0297] 3. External services: These are online services that connect to the system through APIs, such as reservation services.

[0298] Hardware and software used

[0299] Speech recognition engine: Converts user speech into text data using the Google Speech-to-Text API.

[0300] Natural language processing engine: Uses the Google Cloud Natural Language API to analyze text data and understand user intent.

[0301] Sentiment Engine: Uses IBM Watson Tone Analyzer to determine the sentiment of text data.

[0302] Generative AI model: OpenAI's GPT-3® is used to generate responses appropriate to the user's emotions and needs.

[0303] External Service API: Use the ClassPass API or other booking service APIs to get the information you need.

[0304] Actual operation example

[0305] When a user comes home tired from work, they say to their smartphone, "I'm tired, so I want to refresh myself." The smartphone records the voice data and sends it to the server. The server then performs the following processes:

[0306] 1. Speech recognition: The server uses the Google Speech-to-Text API to convert the received voice data into text data. The resulting text is "I'm tired, so I want to refresh myself."

[0307] 2. Natural Language Processing: The server uses the Google Cloud Natural Language API to parse the text and understand the user's intent as "I'm looking for refresh suggestions."

[0308] 3. Sentiment analysis: The server uses IBM Watson Tone Analyzer to identify the emotion "tired."

[0309] 4. Response Generation: The server uses OpenAI's GPT-3 to create a response message: "How about yoga to relax?"

[0310] 5. External Service Integration: The server analyzes the generated message and determines that a yoga studio reservation is required. It calls the ClassPass API to obtain the reservation information.

[0311] 6. Generate final response: Based on the obtained reservation information, create a message saying, "Yoga studio reservations are available. Available tomorrow at 5:00 PM."

[0312] 7. Provide to user: Send the final response message to the terminal and display it to the user.

[0313] Example prompt sentence:

[0314] User's voice input: "I'm tired and want to refresh myself."

[0315] The system responds: "How about some yoga to relax? We have a yoga studio available for booking. I'm available tomorrow at 5:00 PM."

[0316] This allows users to consistently receive relevant suggestions based on their emotions and intentions, as well as the booking information they need.

[0317] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0318] Step 1:

[0319] Accepting user voice input

[0320] The user speaks into their smartphone, saying, "I'm tired and want to refresh myself."

[0321] This voice becomes the input data.

[0322] The device uses a built-in microphone to record the user's voice and convert it into digital audio data.

[0323] The output is audio data.

[0324] Step 2:

[0325] Sending audio data

[0326] The device sends the recorded audio data to the server.

[0327] Specifically, the terminal sends the audio data to the server as an HTTP request.

[0328] The input is the voice data from the terminal, and the output is the voice data sent to the server.

[0329] Step 3:

[0330] Speech-to-text conversion

[0331] The server uses a speech recognition engine to convert the voice data into text data.

[0332] For example, it calls the Google Speech-to-Text API to analyze voice data.

[0333] The voice waveform is converted into text data, and the text "I'm tired, so I want to refresh myself" is generated.

[0334] The input is audio data and the output is text data.

[0335] Step 4:

[0336] Text Analysis

[0337] The server uses a natural language processing engine to analyze the generated text data.

[0338] For example, it uses the Google Cloud Natural Language API to parse the text and understand the user's intent as "I'm looking for refresh suggestions."

[0339] The input is text data and the output is the parsed intent.

[0340] Step 5:

[0341] sentiment analysis

[0342] The server uses an emotion engine to determine the emotion of the text data.

[0343] For example, IBM Watson Tone Analyzer is used to identify the emotion meaning "tiredness" from the analyzed intent.

[0344] The input is text data and the output is identified emotion data.

[0345] Step 6:

[0346] Generating an appropriate response

[0347] The server uses generative AI models to generate responses that are appropriate to the user's emotions and needs.

[0348] For example, OpenAI's GPT-3 is used to create a response message based on the instruction "Suggest yoga for refreshment."

[0349] The input is the user's intent and emotional data, and the output is the generated response message.

[0350] Step 7:

[0351] Deciding whether to link with external services

[0352] The server analyzes the generated response message and determines whether it is necessary to link with an external service.

[0353] From the generated AI's output message, it is determined that "a reservation at a yoga studio is required."

[0354] The input is the generated response message, and the output is the decision on the need for external service integration.

[0355] Step 8:

[0356] Calling external service APIs

[0357] The server calls an external service API and sends a request to get reservation information for a yoga studio.

[0358] For example, use the ClassPass API to check available slots for reservation.

[0359] The input is information that requires collaboration with an external service, and the output is reservation information obtained from the external service.

[0360] Step 9:

[0361] Response processing for external services

[0362] The external service receives the request and returns available slot information to the server.

[0363] For example, provide information such as "I'm free tomorrow at 5:00 p.m."

[0364] The server receives this information.

[0365] The input is the response from the external service API, and the output is the available slot information.

[0366] Step 10:

[0367] Generate the final response

[0368] The server integrates the response of the generated AI model with the reservation information received from the external service.

[0369] As a final response, create a message that reads, "Yoga studio reservations are available. We're available tomorrow at 5:00 PM."

[0370] The input is the response of the generative AI model and reservation information obtained from an external service, and the output is the final response message.

[0371] Step 11:

[0372] Providing a response to the user

[0373] The server sends a final response message to the terminal.

[0374] The terminal receives this message and displays it to the user.

[0375] Based on the information displayed, users can take specific actions, such as booking a yoga studio.

[0376] The input is the final response message, and the output is the user's action.

[0377] (Application example 2)

[0378] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0379] Conventional systems have difficulty generating appropriate responses based on user emotions, making it difficult to quickly and efficiently provide the products and services that users specifically want, especially in physical stores.In addition, there was insufficient integration with inventory information, and the information provided to users was often out of date, limiting the ability to improve the user experience.

[0380] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0381] In this invention, the server includes means for receiving user input and converting the input into text data, means for analyzing the text data to understand the user's intention, means for generating an appropriate response based on the analysis results, means for acquiring necessary information in cooperation with an external service and reflecting the acquired information in the response, means for providing the response to the user, means for acquiring inventory information in cooperation with the inventory system of each store, and means for reflecting the acquired inventory information in the response. This enables an appropriate response based on the user's emotions and reflecting the latest inventory information.

[0382] "User" refers to a person who uses the system or receives information or services from the system.

[0383] "Input" refers to the act of a user providing data or information to a system, whether by voice, text, touch, or other means.

[0384] "Text data" refers to character string data converted from voice input, and is the information that forms the basis for the system's analysis.

[0385] "Analysis" refers to the process of analyzing input text data to understand the user's intentions and emotions.

[0386] "Response generation" refers to the process of creating appropriate messages and actions based on the user's intent and emotions.

[0387] "External services" are external online services or databases that the system connects to and that are accessed through APIs.

[0388] "Information" means any data or knowledge to be provided to the User, including stock availability, available slots, recommended products, etc.

[0389] "Providing" refers to the act of displaying or notifying the user of a response generated by the system or information obtained.

[0390] An "inventory system" is a system that manages the inventory information held by each store, and it is possible to link with the system via an API.

[0391] "Acquisition" refers to the act of the system obtaining necessary information from an external service or inventory system.

[0392] The system for implementing this invention comprises a user terminal, a server, and an external service. The user terminal is a device such as a smartphone or tablet that receives voice input and transmits it to the server. The server is a central device that drives a speech recognition engine, a natural language processing engine, a generative AI model, and an emotion engine, and links with external services.

[0393] Hardware and Software Used

[0394] 1. Speech recognition engine: Using the Google Speech-to-Text API, it converts the voice data entered by the user into text data, analyzes the voice data, and generates accurate text.

[0395] 2. Natural Language Processing Engine (NLP): Using the Google Natural Language API, the generated text data is analyzed to understand the user's intent and extract the information needed for further processing.

[0396] 3. Emotion Engine: Uses IBM Watson Tone Analyzer to analyze the emotions in text data and provide information based on those emotions.

[0397] 4. Generative AI model: OpenAI GPT-3 is used to generate responses appropriate to the user's emotions and needs.

[0398] 5. Integration with external services: Connect with inventory systems, reservation services, etc. via API to obtain necessary information.

[0399] Example of a system

[0400] When a user speaks into a smartphone in a physical store saying, "I'm tired and I want a product that will help me relax," this voice input is recorded and sent to a server. The server uses the Google Speech-to-Text API to convert the voice data into text data. This text data is analyzed using the Google Natural Language API to understand the user's intent of "I want to relax." Sentiment analysis is then performed using IBM Watson Tone Analyzer, and the result is determined to be "I'm tired and want to relax."

[0401] Based on the results of this analysis, OpenAI GPT-3 generates the optimal response, creating a message such as, "Would you like some aromatic oils or herbal tea to help you relax?" Next, it connects with the store's inventory system via API to check stock information. For example, it may obtain information such as, "Aroma oils are in stock." The final response, including this result, is then provided to the user.

[0402] Prompt Sentence Examples

[0403] "The user said, 'I'm tired and I want something to relax me.' Sentiment analysis tells you that the user feels tired and wants to relax. Based on this, make suggestions for relaxation products that you can offer in-store."

[0404] Such systems can improve user experience by providing real-time, specific responses based on user sentiment, and also improve store operations by quickly checking inventory and suggesting appropriate products.

[0405] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0406] Program processing flow

[0407] Step 1:

[0408] The user inputs voice data into their smartphone. Specifically, they say, "I'm tired, so I want a product that will help me relax" at a physical store. This voice data is recorded and sent to the server by the user's device.

[0409] Input: User's voice data

[0410] Output: Sending audio data to the server

[0411] Step 2:

[0412] The server converts the received voice data into text data using the Google Speech-to-Text API. This API analyzes the voice data and generates text such as "I'm tired, so I'd like a product that helps me relax."

[0413] Input: Audio data

[0414] Output: Text data

[0415] Step 3:

[0416] The server uses the Google Natural Language API to analyze the generated text data. The NLP engine parses the text and understands the user's intent as "I want to relax."

[0417] Input: Text data

[0418] Output: Analysis of user intent

[0419] Step 4:

[0420] The server uses IBM Watson Tone Analyzer to analyze the sentiment of the text data, and the sentiment engine identifies emotions such as "I'm tired and want to relax" and passes the results on to the next step in the process.

[0421] Input: Text data

[0422] Output: Emotion analysis results

[0423] Step 5:

[0424] The server uses OpenAI GPT-3 to generate a response appropriate to the user's emotions and needs. Based on the instruction to "suggest products for relaxation," the generative AI model creates a response message such as "How about some aromatic oils or herbal teas to help you relax?"

[0425] Input: Emotion analysis results, user intent

[0426] Output: Response message

[0427] Step 6:

[0428] The server analyzes the generated response message and determines whether it needs to link with the store's inventory system, for example, to check whether a specific product is in stock.

[0429] Input: Response message

[0430] Output: A request for inventory

[0431] Step 7:

[0432] The server calls the store's inventory system API and sends a request to obtain inventory information. The API returns information such as "Aroma oil is in stock."

[0433] Input: Request for inventory check

[0434] Output: Inventory information

[0435] Step 8:

[0436] The server generates the final response based on the inventory information it retrieves. It combines the output of OpenAI GPT-3 with information from the inventory system to create a message that says, "We have aromatic oils and herbal teas in stock. You can see them on the shelves right now."

[0437] Input: Inventory information, response message from the generative AI model

[0438] Output: Final response message

[0439] Step 9:

[0440] The server sends a final response message to the user terminal, which receives the message and displays it to the user.

[0441] Input: Final response message

[0442] Output: Send message to user terminal

[0443] Step 10:

[0444] The user checks the display on the device and searches for the suggested product in the appropriate location in the store. For example, the user may take an action such as "I'm going to look for aroma oils and herbal teas."

[0445] Input: Display last response message

[0446] Output: User actions

[0447] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0448] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0449] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0450] [Second embodiment]

[0451] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0452] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0453] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0454] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0455] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0456] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0457] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0458] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0459] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0460] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0461] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0462] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0463] This invention relates to a system that receives user input, converts the input into text data, analyzes it to generate an appropriate response, and provides the response to the user in cooperation with an external service.

[0464] System configuration

[0465] The system mainly consists of the following components:

[0466] 1. User terminal: A device used by a user that has the ability to receive voice input and send it to a server.

[0467] 2. Server: A central device that manages the speech recognition engine, natural language processing engine, generative AI model, and collaboration with external services.

[0468] 3. External services: Online services that connect to the system via external APIs, such as reservation services.

[0469] System Operation Overview

[0470] The user inputs a request such as a refresh or reservation into the user terminal by voice. The terminal sends this voice data to the server. The server performs the following process and provides the final response to the user.

[0471] Program processing explanation

[0472] 1. Accept user input:

[0473] The user speaks to the device, saying, "I'm tired and want to refresh myself."

[0474] The device records this audio data and sends it to the server.

[0475] 2. Speech to text conversion:

[0476] The server uses a voice recognition engine to convert the received voice data into text data.

[0477] This conversion generates the text "I'm tired so I want to refresh myself."

[0478] 3. Text Analysis:

[0479] The server uses a natural language processing (NLP) engine to analyze the text data.

[0480] The NLP engine understands the meaning and structure of the text and identifies the user's intent.

[0481] For example, understand that the user wants refresh suggestions.

[0482] 4. Sentiment analysis:

[0483] The server uses a sentiment analysis engine to analyze the sentiment of the text data.

[0484] Here, the user's emotions, such as "tired," are identified.

[0485] 5. Generating appropriate responses:

[0486] The server uses a generative AI model to generate a response based on the user's emotions and intent.

[0487] In response generation, suggestions such as "How about some yoga to relax?"

[0488] 6. Decision on external service integration:

[0489] Based on the response generated by the server, it determines whether to link with an external service.

[0490] If it is determined that a yoga studio reservation is necessary, an external service API is called.

[0491] 7. Calling external service APIs:

[0492] The server checks the reservation status through an external service API.

[0493] For example, get information about available slots at a yoga studio.

[0494] 8. Generate the final response:

[0495] The server integrates the responses of the generated AI model with information obtained from external services.

[0496] Generate the final response "Yoga studio reservations available. Available tomorrow at 5:00 PM."

[0497] 9. Providing Responses to Users:

[0498] The device displays the final response received from the server to the user.

[0499] This allows users to receive specific support, such as booking yoga studios.

[0500] In this way, the system provides optimal support according to the user's needs and effectively resolves issues related to time and health management. For example, for a user who is tired from work and wants to refresh themselves, the system provides accurate yoga suggestions and reservation information. This system allows users to live their daily lives efficiently and comfortably.

[0501] The processing flow will be explained below.

[0502] Step 1:

[0503] The user speaks to the device, saying, "I'm tired, so I want to refresh myself." The device records this speech and sends it to the server as audio data.

[0504] Step 2:

[0505] The server uses a speech recognition engine to convert the received voice data into text data. The speech recognition engine analyzes the voice data and generates the text "I'm tired, so I want to refresh myself."

[0506] Step 3:

[0507] The server uses a natural language processing (NLP) engine to analyze the generated text data. The NLP engine parses the text and understands the user's intent as "I'm looking for refresh suggestions."

[0508] Step 4:

[0509] The server uses a sentiment analysis engine to determine the sentiment of the text data, identifying emotions such as "tired" and passing the results on to the next step of processing.

[0510] Step 5:

[0511] The server uses the generative AI model to generate a response appropriate to the user's emotions and needs. Based on the instruction "Suggest yoga to refresh yourself," the generative AI model creates a response message saying, "How about yoga to relax?"

[0512] Step 6:

[0513] The server analyzes the generated response message and determines whether it is necessary to connect to an external service. From the generated message, it determines that "a reservation at a yoga studio is required."

[0514] Step 7:

[0515] The server calls an external service API and sends a request to get reservation information for the yoga studio. The API checks for available slots.

[0516] Step 8:

[0517] The external service receives the request and returns available slot information to the server, such as "Tomorrow is available at 5:00 PM."

[0518] Step 9:

[0519] The server combines the response of the generated AI model with the reservation information received from the external service, and creates a final response message saying, "Yoga studio reservations are available. We have availability tomorrow at 5:00 PM."

[0520] Step 10:

[0521] The server sends a final response message to the device, which receives it and displays it to the user. The user can then take specific action, such as booking a yoga studio, based on the displayed information.

[0522] This process allows users to receive efficient and personalized support.

[0523] Example 1

[0524] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0525] Current dialogue systems must integrate multiple technologies, such as speech recognition, natural language processing, and sentiment analysis, to respond appropriately to diverse user requests. However, existing systems do not fully integrate these functions, making it difficult to accurately grasp the user's intentions and emotions and provide appropriate responses. Furthermore, they are unable to efficiently connect with external services, making it difficult to quickly provide the information and services users desire.

[0526] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0527] In this invention, the server includes means for receiving voice input and converting it into voice data, means for converting the voice data into text data, means for analyzing the text data to understand the user's intention, means for analyzing the sentiment of the text data, means for generating an appropriate response based on the analysis results and the sentiment analysis results, means for acquiring necessary information in cooperation with an external service and reflecting it in the response, and means for providing the response to the user, thereby enabling accurate and prompt responses to a variety of user requests.

[0528] The "means for receiving and converting voice input into voice data" is a component that has the function of converting the contents of a user's voice input into digital voice data.

[0529] The "means for converting the voice data into text data" is a component that has the function of analyzing digital voice data and converting it into corresponding text format information.

[0530] The "means for analyzing the text data to understand the user's intention" is a component that has the function of analyzing the text data using natural language processing technology and identifying the intention implied by the user's utterance.

[0531] The "means for analyzing the emotion of the text data" is a component that has a function of identifying the emotion of the user based on the analyzed text data.

[0532] The "means for generating an appropriate response based on the analysis results and emotion analysis results" is a component that has the function of generating the optimal response to be provided to the user based on the user's intentions and emotions.

[0533] "Means of interacting with external services to obtain necessary information and reflect it in the response" refers to a component that has the function of accessing the APIs of other external systems and services, obtaining necessary data and information, and incorporating it into the response to the user.

[0534] The "means for providing the response to the user" is a component having an interface or function for transmitting the generated response to the user.

[0535] A "generative AI model" is a machine learning model that learns from large amounts of data in advance and performs natural language generation and text analysis.

[0536] A "prompt" is a textual instruction or question input to a generative AI model that is used to elicit a specific response or generated result.

[0537] This invention relates to a system that converts voice input into text, analyzes it to generate appropriate responses, and links with external services. How the invention is realized using specific hardware and software is described below.

[0538] Hardware and software used

[0539] The system is implemented primarily using the following hardware and software:

[0540] User terminal: A device that allows users to input voice, and is equipped with a microphone and network connection capabilities.

[0541] Server: This is the central device that manages the speech recognition engine, natural language processing engine, generative AI model, and collaboration with external services. Here, we use Google Cloud Speech-to-Text API, OpenAI's GPT-4, and IBM Watson Tone Analyzer.

[0542] Accepting and converting voice input

[0543] The user speaks, "I'm tired, so I want to refresh myself." The device records this voice data and sends it over the network to a server. The server then converts the received voice data into text data using the Google Cloud Speech-to-Text API.

[0544] Text data analysis

[0545] The server uses OpenAI's GPT-4 to analyze the converted text data and understand the user's intent. At the same time, it uses IBM Watson Tone Analyzer to analyze the sentiment of the text data. This identifies that the user's intent is "I want to refresh myself" and that the sentiment is "I'm tired."

[0546] Generating a response

[0547] The server uses OpenAI's ChatGPT, a generative AI model, to generate responses based on the user's intent and emotions. Here are some example prompts:

[0548] User: I'm tired from work and want to refresh myself.

[0549] System: How about some yoga to relax? There's a class available at John Smith Studio at 5:00 PM. Would you like to book it?

[0550] Thus, the suggestion: "How about some yoga to relax?"

[0551] Integration with external services

[0552] The server then decides whether to connect to an external service based on the generated response. For example, if it determines that a yoga studio reservation is required, it checks the reservation status through the external service's API. Specifically, it accesses the reservation system's API and obtains information about available slots at the yoga studio.

[0553] Providing a final response

[0554] The server combines the generated response with information obtained from external services to generate a final response: "You can make a reservation at the yoga studio. It's available tomorrow at 5:00 PM." This specific response is provided to the user via the device.

[0555] In this way, the system receives and analyzes the user's voice input, generates appropriate responses, and provides them to the user. For example, if a user is tired from work, the system can provide yoga suggestions and reservation information, allowing the user to live their daily life efficiently and comfortably.

[0556] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0557] Step 1:

[0558] The user speaks, "I'm tired, so I want to refresh myself." The device records this voice data. The input is voice data, which is then converted into a recording. Specifically, the device uses a microphone to capture the voice signal and saves it as a digital audio file.

[0559] Step 2:

[0560] The device sends the recorded audio data to the server. The input is a digital audio file, which is then sent to the server over the network. Specifically, the audio file is uploaded using an HTTP request over the internet connection.

[0561] Step 3:

[0562] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text data. The input is a digital audio file, and the output is text data. Specifically, the server passes the audio file to the API and obtains the text, "I'm tired, so I want to refresh myself."

[0563] Step 4:

[0564] The server uses OpenAI's GPT-4 to analyze the text data and understand the user's intent. The input is the converted text data, and the output is the analysis result that identifies the user's intent. Specifically, the text is passed to GPT-4, which understands that the user is requesting a refresh suggestion.

[0565] Step 5:

[0566] The server uses IBM Watson Tone Analyzer to analyze the sentiment of the text data. The input is the text data, and the output is the sentiment analysis result. Specifically, the text is passed to the analysis engine to identify the sentiment of "tired."

[0567] Step 6:

[0568] The server uses OpenAI's ChatGPT, a generative AI model, to generate a response based on the user's intent and emotions. The input is the user's intent and the results of the emotion analysis, and the output is the generated response. Specifically, the following prompt is passed to ChatGPT:

[0569] User: I'm tired and want to refresh myself.

[0570] System: How about yoga to relax?

[0571] I get the response:

[0572] Step 7:

[0573] The server determines whether to collaborate with an external service based on the generated response. The input is the generated response, and the output is a decision on whether to collaborate with an external service. Specifically, since the response contains "yoga suggestions," the server determines that a reservation at a yoga studio is necessary.

[0574] Step 8:

[0575] The server obtains available slot information through the reservation system's API. The input is a reservation request, and the output is available slot information. Specifically, the server obtains information such as "Available tomorrow at 5:00 PM" through the API.

[0576] Step 9:

[0577] The server integrates the generated response with information from the external service to generate a final response. The input is the generated response and information obtained from the external service, and the output is the final response. Specifically, it generates a response that says, "You can make a reservation at the yoga studio. There is an opening tomorrow at 5:00 PM."

[0578] Step 10:

[0579] The terminal displays the final response received from the server to the user. The input is the final response, and the output is the response displayed on the user's terminal screen. Specifically, the display shows "Yoga studio reservations are available. Available tomorrow at 5:00 PM."

[0580] (Application example 1)

[0581] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0582] Conventional voice input systems lack the ability to convert voice data into text and perform text analysis in the process of generating appropriate responses from user speech. Furthermore, integration with external services is limited, making it difficult to provide real-time information, such as inventory inquiries and reservation services. Furthermore, the lack of visual responses provided via smart devices results in a suboptimal user experience. To solve these problems, a system that utilizes more advanced natural language processing, external service integration, and a visual interface on smart devices is required.

[0583] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0584] In this invention, the server includes means for receiving user input and converting the input into text data, means for analyzing the text data to understand the user's intent, means for generating an appropriate response based on the analysis results, means for acquiring necessary information in cooperation with an external service and reflecting the information in the response, means for generating a detailed response based on the user's intent using a generative AI model, and means for providing the response through a visual interface of a smart device. This enables real-time and accurate information to be provided in response to a variety of user requests.

[0585] "User input" refers to information provided by a user to a system in the form of speech or text.

[0586] "Text data" is data in the form of a character string converted from voice data via a voice recognition engine.

[0587] "Analysis" refers to the process of using a natural language processing engine to understand the meaning and intent of text data.

[0588] An "appropriate response" is one that generates information or instructions that are in line with the user's intentions based on the analysis results.

[0589] "External services" refers to external online services and databases that are linked via an application program interface (API).

[0590] A "generative AI model" is an artificial intelligence model that uses deep learning and other techniques to generate text data based on the user's intentions and emotions.

[0591] "Smart devices" refers to information terminals with internet connectivity, such as smartphones, smart glasses, and head-mounted displays.

[0592] "Visual interface" refers to a means of interacting with users through information and messages displayed on the screen of a smart device.

[0593] We will explain the system to realize this application example. This system mainly consists of the following components:

[0594] User terminal: A smart device such as smart glasses that has the ability to receive voice input and send it to a server.

[0595] Server: A central device that manages the voice recognition engine, natural language processing engine, generative AI model, and collaboration with external services.

[0596] External services: Online services that connect to the system via external APIs, such as transaction information or inventory status.

[0597] System Operation Overview

[0598] Accepting user input

[0599] The user speaks to the smart glasses, saying, "Please show me the cosmetics section." The smart glasses receive this voice input and send it to the server. The user's device records the voice data and sends it to the server.

[0600] Speech-to-text conversion

[0601] The server uses a speech recognition engine (for example, Google Cloud Speech-to-Text API) to convert the received voice data into text data. This conversion generates the text "Please show me the cosmetics section."

[0602] Text Analysis

[0603] The server uses a natural language processing engine (e.g., IBM Watson NLP API) to analyze the text data and identify the user's intent. The NLP engine understands the meaning and structure of the text and understands that the user is seeking guidance.

[0604] Generating an appropriate response

[0605] The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate a response based on the user's intent, such as "The cosmetics section is on the second floor. Please take the escalator."

[0606] Deciding whether to link with external services

[0607] The server determines whether or not it needs to connect to an external service based on the generated response. For example, if a user inquires about the stock of a specific product, it determines that it needs to check the stock information.

[0608] Calling external service APIs

[0609] The server connects to the store's inventory management system via an external service API to check the stock status of a specific product. For example, it performs a process such as "calling an API to obtain stock information and obtaining the stock status of lipstick."

[0610] Generate the final response

[0611] The server combines the generated AI model's response with information obtained from external services to generate a final response, such as "The cosmetics section is on the second floor. Please take the escalator. The lipstick you are looking for is in stock."

[0612] Providing a response to the user

[0613] The user device (smart glasses) provides the final response received from the server in a display format, allowing the user to visually confirm specific guidance information in response to voice input.

[0614] Examples of specific examples and prompts

[0615] As a concrete example, consider a case where a user asks "What is the stock status of lipstick?" The server converts the speech to text and then inputs the following prompt into the generative AI model to get a response:

[0616] Example prompt sentence:

[0617] User Question: What is the availability of lipstick?

[0618] Generate an appropriate response.

[0619] Example response:

[0620] "Lipstick is in stock. The cosmetics section is on the second floor. Please take the escalator."

[0621] In this way, a system can be built that generates appropriate responses in real time based on the user's voice input and delivers them through smart devices.

[0622] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0623] Step 1:

[0624] The user inputs the information by voice. The user speaks to the smart glasses, saying, "Please show me the cosmetics section." The voice input (voice data) is sent to the device.

[0625] Step 2:

[0626] The device records the audio data and sends it to the server. Input: Audio data. Output: Audio data sent to the server.

[0627] Step 3:

[0628] The server uses a speech recognition engine to convert the voice data into text data. Input: Voice data. Output: Text data saying "Please show me to the cosmetics section." Specific operation: Uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API).

[0629] Step 4:

[0630] The server uses a natural language processing engine to analyze the text data and identify the user's intent. Input: Text data. Output: User intent (e.g., "I need guidance"). Specific behavior: Uses a natural language processing engine (e.g., IBM Watson NLP API).

[0631] Step 5:

[0632] The server uses a generative AI model to generate a response based on the user's intent. Input: User's intent. Output: Response text (e.g., "The cosmetics section is on the second floor. Please use the escalator."). Specific behavior: Uses a generative AI model (e.g., OpenAI GPT-4).

[0633] Step 6:

[0634] The server determines whether or not integration with an external service is necessary based on the generated response. Input: Response text. Output: Necessity of integration with an external service (e.g., "Stock information needs to be confirmed").

[0635] Step 7:

[0636] If necessary, the server calls an external service API to check the stock status of a specific product. Input: Product name. Output: Stock status data. Specific operation: Calls an external service API (e.g., inventory management system API) to obtain stock information.

[0637] Step 8:

[0638] The server combines the generated AI model's response with information obtained from external services. Input: Response text, stock status data. Output: Final response text (e.g., "The cosmetics section is on the second floor. Please take the escalator. The lipstick you are looking for is in stock.").

[0639] Step 9:

[0640] Provides the final response received by the user device from the server in a display format. Input: Final response text. Output: Response displayed in the user's field of view.

[0641] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0642] This invention relates to a system that converts user input into text data, analyzes emotions and intentions, and generates appropriate responses. It also includes a means for linking with external services to obtain necessary information and reflect it in the response. The invention particularly aims to provide responses based on the user's emotions by combining an emotion engine, thereby optimizing support for the user.

[0643] System configuration

[0644] The system consists of the following main components:

[0645] 1. User terminal: A device used by the user (smartphone, tablet, PC, etc.) that has the function of receiving voice input and sending it to the server.

[0646] 2. Server: A central device that manages the speech recognition engine, natural language processing engine, generative AI model, emotion engine, and collaboration with external services.

[0647] 3. External services: These are online services that connect to the system through APIs, such as reservation services.

[0648] System Operation Overview

[0649] The user inputs a request such as a refresh or reservation into the user terminal by voice. The terminal sends this voice data to the server. The server performs the following process and provides the final response to the user.

[0650] Program processing explanation

[0651] 1. Accept user input:

[0652] The user speaks to the device, saying, "I'm tired and want to refresh myself."

[0653] The device records this audio data and sends it to the server.

[0654] 2. Speech to text conversion:

[0655] The server converts the received voice data into text data using a voice recognition engine.

[0656] The voice recognition engine analyzes the voice data and generates the text "I'm tired, so I want to refresh myself."

[0657] 3. Text Analysis:

[0658] The server uses a natural language processing (NLP) engine to analyze the generated text data.

[0659] The NLP engine parses the text and understands the user's intent as "I'm looking for refresh suggestions."

[0660] 4. Sentiment analysis:

[0661] The server uses an emotion engine to determine the emotion of the text data.

[0662] The emotion engine identifies emotions such as "tired" and passes the results on to the next process.

[0663] 5. Generating appropriate responses:

[0664] The server uses generative AI models to generate responses that are appropriate to the user's emotions and needs.

[0665] Based on the instruction "Suggest yoga to refresh," the generative AI model creates a response message saying, "How about yoga to relax?"

[0666] 6. Decision on external service integration:

[0667] The server analyzes the generated response message and determines whether it is necessary to link with an external service.

[0668] From the generated message, it is determined that "a reservation at a yoga studio is required."

[0669] 7. Calling external service APIs:

[0670] The server calls an external service API and sends a request to get reservation information for a yoga studio.

[0671] Check available slots via API.

[0672] 8. External service response processing:

[0673] The external service receives the request and returns available slot information to the server, such as "Tomorrow is available at 5:00 PM."

[0674] 9. Generate the final response:

[0675] The server integrates the response of the generated AI model with the reservation information received from the external service.

[0676] As a final response, create a message that reads, "Yoga studio reservations are available. We're available tomorrow at 5:00 PM."

[0677] 10. Providing Responses to Users:

[0678] The server sends a final response message to the terminal, which receives it and displays it to the user.

[0679] Based on the information displayed, users can take specific actions, such as booking a yoga studio.

[0680] Through this series of processes, users can receive efficient and individualized support. One feature of this system is that it generates responses taking into account the user's emotions, thereby improving the user experience. For example, if a user feels like "refreshing" after returning home tired from work, the system can not only provide information but also make appropriate suggestions based on that emotion.

[0681] The processing flow will be explained below.

[0682] Step 1:

[0683] The user speaks to the device, saying, "I'm tired, so I want to refresh myself." The device records this speech and sends it to the server as audio data.

[0684] Step 2:

[0685] The server uses a speech recognition engine to convert the received voice data into text data. The speech recognition engine analyzes the voice data and generates the text "I'm tired, so I want to refresh myself."

[0686] Step 3:

[0687] The server uses a natural language processing (NLP) engine to analyze the generated text data. The NLP engine parses the text and understands the user's intent as "I'm looking for refresh suggestions."

[0688] Step 4:

[0689] The server uses an emotion engine to determine the emotion of the text data. The emotion engine identifies an emotion such as "tired" and passes the result on to the next process.

[0690] Step 5:

[0691] The server uses the generative AI model to generate a response appropriate to the user's emotions and needs. Based on the instruction "Suggest yoga to refresh yourself," the generative AI model creates a response message saying, "How about yoga to relax?"

[0692] Step 6:

[0693] The server analyzes the generated response message and determines whether it is necessary to connect to an external service. From the generated message, it determines that "a reservation at a yoga studio is required."

[0694] Step 7:

[0695] The server calls an external service API and sends a request to get reservation information for the yoga studio. The API checks for available slots.

[0696] Step 8:

[0697] The external service receives the request and returns available slot information to the server, such as "Tomorrow is available at 5:00 PM."

[0698] Step 9:

[0699] The server combines the response of the generated AI model with the reservation information received from the external service, and creates a final response message saying, "Yoga studio reservations are available. We have availability tomorrow at 5:00 PM."

[0700] Step 10:

[0701] The server sends a final response message to the device, which receives it and displays it to the user. The user can then take specific action, such as booking a yoga studio, based on the displayed information.

[0702] This process allows users to receive efficient and personalized support.

[0703] Example 2

[0704] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0705] Many current systems can receive user voice input, but they have limitations in their ability to fully understand the user's emotions and intentions and generate appropriate responses. They also have difficulty linking with external services to obtain necessary information and provide consistent responses to users. This results in a poor user experience and lowers user satisfaction. The objective of this invention is to provide a system that generates responses based on the user's emotions and intentions and links with external services to provide high-quality support.

[0706] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0707] In this invention, the server includes means for receiving a user's voice input and converting the voice into text data, means for analyzing the text data to understand the user's intention, means for analyzing the emotion in the text data and generating a response according to the emotion, means for analyzing the generated response and coordinating with an external service to obtain necessary information and reflect this in the response, and means for providing the final response to the user. This allows the user to receive an appropriate response based on their own emotions and intentions. Furthermore, by coordinating with an external service to obtain necessary information, the server can receive consistent, high-quality support.

[0708] "Voice input" refers to instructions or requests that a user gives to a system through voice.

[0709] "Text data" refers to textual information converted from voice input, and is digital information in a format that can be understood by humans.

[0710] "Sentiment analysis" is the process of detecting emotional information contained in text data and identifying that emotion.

[0711] A "natural language processing engine" is software or algorithms that analyze text data and perform syntactic analysis and extract intent.

[0712] "Response generation" is the process of creating an appropriate response message based on the analyzed intentions and emotions.

[0713] "External Services" refers to online services that connect to the system through APIs and provide additional information or functionality.

[0714] "Reservation information" refers to detailed information such as available reservation dates, times, and locations obtained from external services.

[0715] "Final Response" refers to the final form of the response message generated for delivery to the user.

[0716] "User experience" is a general term for the ease of use and satisfaction that users feel when using a system.

[0717] This invention relates to a system that receives a user's voice input, converts the voice into text data, analyzes the text data to understand the user's intentions and emotions, generates an appropriate response, and works with external services to obtain necessary information and provide it to the user.

[0718] The system mainly consists of the following components:

[0719] 1. User device: A device (smartphone, tablet, PC, etc.) that allows the user to input voice data and has the function of sending voice data to a server.

[0720] 2. Server: A central device that manages the speech recognition engine, natural language processing engine, generative AI model, emotion engine, and collaboration with external services.

[0721] 3. External services: These are online services that connect to the system through APIs, such as reservation services.

[0722] Hardware and software used

[0723] Speech recognition engine: Converts user speech into text data using the Google Speech-to-Text API.

[0724] Natural language processing engine: Uses the Google Cloud Natural Language API to analyze text data and understand user intent.

[0725] Sentiment Engine: Uses IBM Watson Tone Analyzer to determine the sentiment of text data.

[0726] Generative AI model: OpenAI's GPT-3 is used to generate responses appropriate to the user's emotions and needs.

[0727] External Service API: Use the ClassPass API or other booking service APIs to get the information you need.

[0728] Actual operation example

[0729] When a user comes home tired from work, they say to their smartphone, "I'm tired, so I want to refresh myself." The smartphone records the voice data and sends it to the server. The server then performs the following processes:

[0730] 1. Speech recognition: The server uses the Google Speech-to-Text API to convert the received voice data into text data. The resulting text is "I'm tired, so I want to refresh myself."

[0731] 2. Natural Language Processing: The server uses the Google Cloud Natural Language API to parse the text and understand the user's intent as "I'm looking for refresh suggestions."

[0732] 3. Sentiment analysis: The server uses IBM Watson Tone Analyzer to identify the emotion "tired."

[0733] 4. Response Generation: The server uses OpenAI's GPT-3 to create a response message: "How about yoga to relax?"

[0734] 5. External Service Integration: The server analyzes the generated message and determines that a yoga studio reservation is required. It calls the ClassPass API to obtain the reservation information.

[0735] 6. Generate final response: Based on the obtained reservation information, create a message saying, "Yoga studio reservations are available. Available tomorrow at 5:00 PM."

[0736] 7. Provide to user: Send the final response message to the terminal and display it to the user.

[0737] Example prompt sentence:

[0738] User's voice input: "I'm tired and want to refresh myself."

[0739] The system responds: "How about some yoga to relax? We have a yoga studio available for booking. I'm available tomorrow at 5:00 PM."

[0740] This allows users to consistently receive relevant suggestions based on their emotions and intentions, as well as the booking information they need.

[0741] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0742] Step 1:

[0743] Accepting user voice input

[0744] The user speaks into their smartphone, saying, "I'm tired and want to refresh myself."

[0745] This voice becomes the input data.

[0746] The device uses a built-in microphone to record the user's voice and convert it into digital audio data.

[0747] The output is audio data.

[0748] Step 2:

[0749] Sending audio data

[0750] The device sends the recorded audio data to the server.

[0751] Specifically, the terminal sends the audio data to the server as an HTTP request.

[0752] The input is the voice data from the terminal, and the output is the voice data sent to the server.

[0753] Step 3:

[0754] Speech-to-text conversion

[0755] The server uses a speech recognition engine to convert the voice data into text data.

[0756] For example, it calls the Google Speech-to-Text API to analyze voice data.

[0757] The voice waveform is converted into text data, and the text "I'm tired, so I want to refresh myself" is generated.

[0758] The input is audio data and the output is text data.

[0759] Step 4:

[0760] Text Analysis

[0761] The server uses a natural language processing engine to analyze the generated text data.

[0762] For example, it uses the Google Cloud Natural Language API to parse the text and understand the user's intent as "I'm looking for refresh suggestions."

[0763] The input is text data and the output is the parsed intent.

[0764] Step 5:

[0765] sentiment analysis

[0766] The server uses an emotion engine to determine the emotion of the text data.

[0767] For example, IBM Watson Tone Analyzer is used to identify the emotion meaning "tiredness" from the analyzed intent.

[0768] The input is text data and the output is identified emotion data.

[0769] Step 6:

[0770] Generating an appropriate response

[0771] The server uses generative AI models to generate responses that are appropriate to the user's emotions and needs.

[0772] For example, OpenAI's GPT-3 is used to create a response message based on the instruction "Suggest yoga for refreshment."

[0773] The input is the user's intent and emotional data, and the output is the generated response message.

[0774] Step 7:

[0775] Deciding whether to link with external services

[0776] The server analyzes the generated response message and determines whether it is necessary to link with an external service.

[0777] From the generated AI's output message, it is determined that "a reservation at a yoga studio is required."

[0778] The input is the generated response message, and the output is the decision on the need for external service integration.

[0779] Step 8:

[0780] Calling external service APIs

[0781] The server calls an external service API and sends a request to get reservation information for a yoga studio.

[0782] For example, use the ClassPass API to check available slots for reservation.

[0783] The input is information that requires collaboration with an external service, and the output is reservation information obtained from the external service.

[0784] Step 9:

[0785] Response processing for external services

[0786] The external service receives the request and returns available slot information to the server.

[0787] For example, provide information such as "I'm free tomorrow at 5:00 p.m."

[0788] The server receives this information.

[0789] The input is the response from the external service API, and the output is the available slot information.

[0790] Step 10:

[0791] Generate the final response

[0792] The server integrates the response of the generated AI model with the reservation information received from the external service.

[0793] As a final response, create a message that reads, "Yoga studio reservations are available. We're available tomorrow at 5:00 PM."

[0794] The input is the response of the generative AI model and reservation information obtained from an external service, and the output is the final response message.

[0795] Step 11:

[0796] Providing a response to the user

[0797] The server sends a final response message to the terminal.

[0798] The terminal receives this message and displays it to the user.

[0799] Based on the information displayed, users can take specific actions, such as booking a yoga studio.

[0800] The input is the final response message, and the output is the user's action.

[0801] (Application example 2)

[0802] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0803] Conventional systems have difficulty generating appropriate responses based on user emotions, making it difficult to quickly and efficiently provide the products and services that users specifically want, especially in physical stores.In addition, there was insufficient integration with inventory information, and the information provided to users was often out of date, limiting the ability to improve the user experience.

[0804] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0805] In this invention, the server includes means for receiving user input and converting the input into text data, means for analyzing the text data to understand the user's intention, means for generating an appropriate response based on the analysis results, means for acquiring necessary information in cooperation with an external service and reflecting the acquired information in the response, means for providing the response to the user, means for acquiring inventory information in cooperation with the inventory system of each store, and means for reflecting the acquired inventory information in the response. This enables an appropriate response based on the user's emotions and reflecting the latest inventory information.

[0806] "User" refers to a person who uses the system or receives information or services from the system.

[0807] "Input" refers to the act of a user providing data or information to a system, whether by voice, text, touch, or other means.

[0808] "Text data" refers to character string data converted from voice input, and is the information that forms the basis for the system's analysis.

[0809] "Analysis" refers to the process of analyzing input text data to understand the user's intentions and emotions.

[0810] "Response generation" refers to the process of creating appropriate messages and actions based on the user's intent and emotions.

[0811] "External services" are external online services or databases that the system connects to and that are accessed through APIs.

[0812] "Information" means any data or knowledge to be provided to the User, including stock availability, available slots, recommended products, etc.

[0813] "Providing" refers to the act of displaying or notifying the user of a response generated by the system or information obtained.

[0814] An "inventory system" is a system that manages the inventory information held by each store, and it is possible to link with the system via an API.

[0815] "Acquisition" refers to the act of the system obtaining necessary information from an external service or inventory system.

[0816] The system for implementing this invention comprises a user terminal, a server, and an external service. The user terminal is a device such as a smartphone or tablet that receives voice input and transmits it to the server. The server is a central device that drives a speech recognition engine, a natural language processing engine, a generative AI model, and an emotion engine, and links with external services.

[0817] Hardware and Software Used

[0818] 1. Speech recognition engine: Using the Google Speech-to-Text API, it converts the voice data entered by the user into text data, analyzes the voice data, and generates accurate text.

[0819] 2. Natural Language Processing Engine (NLP): Using the Google Natural Language API, the generated text data is analyzed to understand the user's intent and extract the information needed for further processing.

[0820] 3. Emotion Engine: Uses IBM Watson Tone Analyzer to analyze the emotions in text data and provide information based on those emotions.

[0821] 4. Generative AI model: OpenAI GPT-3 is used to generate responses appropriate to the user's emotions and needs.

[0822] 5. Integration with external services: Connect with inventory systems, reservation services, etc. via API to obtain necessary information.

[0823] Example of a system

[0824] When a user speaks into a smartphone in a physical store saying, "I'm tired and I want a product that will help me relax," this voice input is recorded and sent to a server. The server uses the Google Speech-to-Text API to convert the voice data into text data. This text data is analyzed using the Google Natural Language API to understand the user's intent of "I want to relax." Sentiment analysis is then performed using IBM Watson Tone Analyzer, and the result is determined to be "I'm tired and want to relax."

[0825] Based on the results of this analysis, OpenAI GPT-3 generates the optimal response, creating a message such as, "Would you like some aromatic oils or herbal tea to help you relax?" Next, it connects with the store's inventory system via API to check stock information. For example, it may obtain information such as, "Aroma oils are in stock." The final response, including this result, is then provided to the user.

[0826] Prompt Sentence Examples

[0827] "The user said, 'I'm tired and I want something to relax me.' Sentiment analysis tells you that the user feels tired and wants to relax. Based on this, make suggestions for relaxation products that you can offer in-store."

[0828] Such systems can improve user experience by providing real-time, specific responses based on user sentiment, and also improve store operations by quickly checking inventory and suggesting appropriate products.

[0829] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0830] Program processing flow

[0831] Step 1:

[0832] The user inputs voice data into their smartphone. Specifically, they say, "I'm tired, so I want a product that will help me relax" at a physical store. This voice data is recorded and sent to the server by the user's device.

[0833] Input: User's voice data

[0834] Output: Sending audio data to the server

[0835] Step 2:

[0836] The server converts the received voice data into text data using the Google Speech-to-Text API. This API analyzes the voice data and generates text such as "I'm tired, so I'd like a product that helps me relax."

[0837] Input: Audio data

[0838] Output: Text data

[0839] Step 3:

[0840] The server uses the Google Natural Language API to analyze the generated text data. The NLP engine parses the text and understands the user's intent as "I want to relax."

[0841] Input: Text data

[0842] Output: Analysis of user intent

[0843] Step 4:

[0844] The server uses IBM Watson Tone Analyzer to analyze the sentiment of the text data, and the sentiment engine identifies emotions such as "I'm tired and want to relax" and passes the results on to the next step in the process.

[0845] Input: Text data

[0846] Output: Emotion analysis results

[0847] Step 5:

[0848] The server uses OpenAI GPT-3 to generate a response appropriate to the user's emotions and needs. Based on the instruction to "suggest products for relaxation," the generative AI model creates a response message such as "How about some aromatic oils or herbal teas to help you relax?"

[0849] Input: Emotion analysis results, user intent

[0850] Output: Response message

[0851] Step 6:

[0852] The server analyzes the generated response message and determines whether it needs to link with the store's inventory system, for example, to check whether a specific product is in stock.

[0853] Input: Response message

[0854] Output: A request for inventory

[0855] Step 7:

[0856] The server calls the store's inventory system API and sends a request to obtain inventory information. The API returns information such as "Aroma oil is in stock."

[0857] Input: Request for inventory check

[0858] Output: Inventory information

[0859] Step 8:

[0860] The server generates the final response based on the inventory information it retrieves. It combines the output of OpenAI GPT-3 with information from the inventory system to create a message that says, "We have aromatic oils and herbal teas in stock. You can see them on the shelves right now."

[0861] Input: Inventory information, response message from the generative AI model

[0862] Output: Final response message

[0863] Step 9:

[0864] The server sends a final response message to the user terminal, which receives the message and displays it to the user.

[0865] Input: Final response message

[0866] Output: Send message to user terminal

[0867] Step 10:

[0868] The user checks the display on the device and searches for the suggested product in the appropriate location in the store. For example, the user may take an action such as "I'm going to look for aroma oils and herbal teas."

[0869] Input: Display last response message

[0870] Output: User actions

[0871] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0872] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0873] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0874] [Third embodiment]

[0875] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0876] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0877] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0878] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0879] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0880] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0881] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0882] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0883] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0884] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0885] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0886] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0887] This invention relates to a system that receives user input, converts the input into text data, analyzes it to generate an appropriate response, and provides the response to the user in cooperation with an external service.

[0888] System configuration

[0889] The system mainly consists of the following components:

[0890] 1. User terminal: A device used by a user that has the ability to receive voice input and send it to a server.

[0891] 2. Server: A central device that manages the speech recognition engine, natural language processing engine, generative AI model, and collaboration with external services.

[0892] 3. External services: Online services that connect to the system via external APIs, such as reservation services.

[0893] System Operation Overview

[0894] The user inputs a request such as a refresh or reservation into the user terminal by voice. The terminal sends this voice data to the server. The server performs the following process and provides the final response to the user.

[0895] Program processing explanation

[0896] 1. Accept user input:

[0897] The user speaks to the device, saying, "I'm tired and want to refresh myself."

[0898] The device records this audio data and sends it to the server.

[0899] 2. Speech to text conversion:

[0900] The server uses a voice recognition engine to convert the received voice data into text data.

[0901] This conversion generates the text "I'm tired so I want to refresh myself."

[0902] 3. Text Analysis:

[0903] The server uses a natural language processing (NLP) engine to analyze the text data.

[0904] The NLP engine understands the meaning and structure of the text and identifies the user's intent.

[0905] For example, understand that the user wants refresh suggestions.

[0906] 4. Sentiment analysis:

[0907] The server uses a sentiment analysis engine to analyze the sentiment of the text data.

[0908] Here, the user's emotions, such as "tired," are identified.

[0909] 5. Generating appropriate responses:

[0910] The server uses a generative AI model to generate a response based on the user's emotions and intent.

[0911] In response generation, suggestions such as "How about some yoga to relax?"

[0912] 6. Decision on external service integration:

[0913] Based on the response generated by the server, it determines whether to link with an external service.

[0914] If it is determined that a yoga studio reservation is necessary, an external service API is called.

[0915] 7. Calling external service APIs:

[0916] The server checks the reservation status through an external service API.

[0917] For example, get information about available slots at a yoga studio.

[0918] 8. Generate the final response:

[0919] The server integrates the responses of the generated AI model with information obtained from external services.

[0920] Generate the final response "Yoga studio reservations available. Available tomorrow at 5:00 PM."

[0921] 9. Providing Responses to Users:

[0922] The device displays the final response received from the server to the user.

[0923] This allows users to receive specific support, such as booking yoga studios.

[0924] In this way, the system provides optimal support according to the user's needs and effectively resolves issues related to time and health management. For example, for a user who is tired from work and wants to refresh themselves, the system provides accurate yoga suggestions and reservation information. This system allows users to live their daily lives efficiently and comfortably.

[0925] The processing flow will be explained below.

[0926] Step 1:

[0927] The user speaks to the device, saying, "I'm tired, so I want to refresh myself." The device records this speech and sends it to the server as audio data.

[0928] Step 2:

[0929] The server uses a speech recognition engine to convert the received voice data into text data. The speech recognition engine analyzes the voice data and generates the text "I'm tired, so I want to refresh myself."

[0930] Step 3:

[0931] The server uses a natural language processing (NLP) engine to analyze the generated text data. The NLP engine parses the text and understands the user's intent as "I'm looking for refresh suggestions."

[0932] Step 4:

[0933] The server uses a sentiment analysis engine to determine the sentiment of the text data, identifying emotions such as "tired" and passing the results on to the next step of processing.

[0934] Step 5:

[0935] The server uses the generative AI model to generate a response appropriate to the user's emotions and needs. Based on the instruction "Suggest yoga to refresh yourself," the generative AI model creates a response message saying, "How about yoga to relax?"

[0936] Step 6:

[0937] The server analyzes the generated response message and determines whether it is necessary to connect to an external service. From the generated message, it determines that "a reservation at a yoga studio is required."

[0938] Step 7:

[0939] The server calls an external service API and sends a request to get reservation information for the yoga studio. The API checks for available slots.

[0940] Step 8:

[0941] The external service receives the request and returns available slot information to the server, such as "Tomorrow is available at 5:00 PM."

[0942] Step 9:

[0943] The server combines the response of the generated AI model with the reservation information received from the external service, and creates a final response message saying, "Yoga studio reservations are available. We have availability tomorrow at 5:00 PM."

[0944] Step 10:

[0945] The server sends a final response message to the device, which receives it and displays it to the user. The user can then take specific action, such as booking a yoga studio, based on the displayed information.

[0946] This process allows users to receive efficient and personalized support.

[0947] Example 1

[0948] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0949] Current dialogue systems must integrate multiple technologies, such as speech recognition, natural language processing, and sentiment analysis, to respond appropriately to diverse user requests. However, existing systems do not fully integrate these functions, making it difficult to accurately grasp the user's intentions and emotions and provide appropriate responses. Furthermore, they are unable to efficiently connect with external services, making it difficult to quickly provide the information and services users desire.

[0950] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0951] In this invention, the server includes means for receiving voice input and converting it into voice data, means for converting the voice data into text data, means for analyzing the text data to understand the user's intention, means for analyzing the sentiment of the text data, means for generating an appropriate response based on the analysis results and the sentiment analysis results, means for acquiring necessary information in cooperation with an external service and reflecting it in the response, and means for providing the response to the user, thereby enabling accurate and prompt responses to a variety of user requests.

[0952] The "means for receiving and converting voice input into voice data" is a component that has the function of converting the contents of a user's voice input into digital voice data.

[0953] The "means for converting the voice data into text data" is a component that has the function of analyzing digital voice data and converting it into corresponding text format information.

[0954] The "means for analyzing the text data to understand the user's intention" is a component that has the function of analyzing the text data using natural language processing technology and identifying the intention implied by the user's utterance.

[0955] The "means for analyzing the emotion of the text data" is a component that has a function of identifying the emotion of the user based on the analyzed text data.

[0956] The "means for generating an appropriate response based on the analysis results and emotion analysis results" is a component that has the function of generating the optimal response to be provided to the user based on the user's intentions and emotions.

[0957] "Means of interacting with external services to obtain necessary information and reflect it in the response" refers to a component that has the function of accessing the APIs of other external systems and services, obtaining necessary data and information, and incorporating it into the response to the user.

[0958] The "means for providing the response to the user" is a component having an interface or function for transmitting the generated response to the user.

[0959] A "generative AI model" is a machine learning model that learns from large amounts of data in advance and performs natural language generation and text analysis.

[0960] A "prompt" is a textual instruction or question input to a generative AI model that is used to elicit a specific response or generated result.

[0961] This invention relates to a system that converts voice input into text, analyzes it to generate appropriate responses, and links with external services. How the invention is realized using specific hardware and software is described below.

[0962] Hardware and software used

[0963] The system is implemented primarily using the following hardware and software:

[0964] User terminal: A device that allows users to input voice, and is equipped with a microphone and network connection capabilities.

[0965] Server: This is the central device that manages the speech recognition engine, natural language processing engine, generative AI model, and collaboration with external services. Here, we use Google Cloud Speech-to-Text API, OpenAI's GPT-4, and IBM Watson Tone Analyzer.

[0966] Accepting and converting voice input

[0967] The user speaks, "I'm tired, so I want to refresh myself." The device records this voice data and sends it over the network to a server. The server then converts the received voice data into text data using the Google Cloud Speech-to-Text API.

[0968] Text data analysis

[0969] The server uses OpenAI's GPT-4 to analyze the converted text data and understand the user's intent. At the same time, it uses IBM Watson Tone Analyzer to analyze the sentiment of the text data. This identifies that the user's intent is "I want to refresh myself" and that the sentiment is "I'm tired."

[0970] Generating a response

[0971] The server uses OpenAI's ChatGPT, a generative AI model, to generate responses based on the user's intent and emotions. Here are some example prompts:

[0972] User: I'm tired from work and want to refresh myself.

[0973] System: How about some yoga to relax? There's a class available at John Smith Studio at 5:00 PM. Would you like to book it?

[0974] Thus, the suggestion: "How about some yoga to relax?"

[0975] Integration with external services

[0976] The server then decides whether to connect to an external service based on the generated response. For example, if it determines that a yoga studio reservation is required, it checks the reservation status through the external service's API. Specifically, it accesses the reservation system's API and obtains information about available slots at the yoga studio.

[0977] Providing a final response

[0978] The server combines the generated response with information obtained from external services to generate a final response: "You can make a reservation at the yoga studio. It's available tomorrow at 5:00 PM." This specific response is provided to the user via the device.

[0979] In this way, the system receives and analyzes the user's voice input, generates appropriate responses, and provides them to the user. For example, if a user is tired from work, the system can provide yoga suggestions and reservation information, allowing the user to live their daily life efficiently and comfortably.

[0980] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0981] Step 1:

[0982] The user speaks, "I'm tired, so I want to refresh myself." The device records this voice data. The input is voice data, which is then converted into a recording. Specifically, the device uses a microphone to capture the voice signal and saves it as a digital audio file.

[0983] Step 2:

[0984] The device sends the recorded audio data to the server. The input is a digital audio file, which is then sent to the server over the network. Specifically, the audio file is uploaded using an HTTP request over the internet connection.

[0985] Step 3:

[0986] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text data. The input is a digital audio file, and the output is text data. Specifically, the server passes the audio file to the API and obtains the text, "I'm tired, so I want to refresh myself."

[0987] Step 4:

[0988] The server uses OpenAI's GPT-4 to analyze the text data and understand the user's intent. The input is the converted text data, and the output is the analysis result that identifies the user's intent. Specifically, the text is passed to GPT-4, which understands that the user is requesting a refresh suggestion.

[0989] Step 5:

[0990] The server uses IBM Watson Tone Analyzer to analyze the sentiment of the text data. The input is the text data, and the output is the sentiment analysis result. Specifically, the text is passed to the analysis engine to identify the sentiment of "tired."

[0991] Step 6:

[0992] The server uses OpenAI's ChatGPT, a generative AI model, to generate a response based on the user's intent and emotions. The input is the user's intent and the results of the emotion analysis, and the output is the generated response. Specifically, the following prompt is passed to ChatGPT:

[0993] User: I'm tired and want to refresh myself.

[0994] System: How about yoga to relax?

[0995] I get the response:

[0996] Step 7:

[0997] The server determines whether to collaborate with an external service based on the generated response. The input is the generated response, and the output is a decision on whether to collaborate with an external service. Specifically, since the response contains "yoga suggestions," the server determines that a reservation at a yoga studio is necessary.

[0998] Step 8:

[0999] The server obtains available slot information through the reservation system's API. The input is a reservation request, and the output is available slot information. Specifically, the server obtains information such as "Available tomorrow at 5:00 PM" through the API.

[1000] Step 9:

[1001] The server integrates the generated response with information from the external service to generate a final response. The input is the generated response and information obtained from the external service, and the output is the final response. Specifically, it generates a response that says, "You can make a reservation at the yoga studio. There is an opening tomorrow at 5:00 PM."

[1002] Step 10:

[1003] The terminal displays the final response received from the server to the user. The input is the final response, and the output is the response displayed on the user's terminal screen. Specifically, the display shows "Yoga studio reservations are available. Available tomorrow at 5:00 PM."

[1004] (Application example 1)

[1005] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1006] Conventional voice input systems lack the ability to convert voice data into text and perform text analysis in the process of generating appropriate responses from user speech. Furthermore, integration with external services is limited, making it difficult to provide real-time information, such as inventory inquiries and reservation services. Furthermore, the lack of visual responses provided via smart devices results in a suboptimal user experience. To solve these problems, a system that utilizes more advanced natural language processing, external service integration, and a visual interface on smart devices is required.

[1007] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1008] In this invention, the server includes means for receiving user input and converting the input into text data, means for analyzing the text data to understand the user's intent, means for generating an appropriate response based on the analysis results, means for acquiring necessary information in cooperation with an external service and reflecting the information in the response, means for generating a detailed response based on the user's intent using a generative AI model, and means for providing the response through a visual interface of a smart device. This enables real-time and accurate information to be provided in response to a variety of user requests.

[1009] "User input" refers to information provided by a user to a system in the form of speech or text.

[1010] "Text data" is data in the form of a character string converted from voice data via a voice recognition engine.

[1011] "Analysis" refers to the process of using a natural language processing engine to understand the meaning and intent of text data.

[1012] An "appropriate response" is one that generates information or instructions that are in line with the user's intentions based on the analysis results.

[1013] "External services" refers to external online services and databases that are linked via an application program interface (API).

[1014] A "generative AI model" is an artificial intelligence model that uses deep learning and other techniques to generate text data based on the user's intentions and emotions.

[1015] "Smart devices" refers to information terminals with internet connectivity, such as smartphones, smart glasses, and head-mounted displays.

[1016] "Visual interface" refers to a means of interacting with users through information and messages displayed on the screen of a smart device.

[1017] We will explain the system to realize this application example. This system mainly consists of the following components:

[1018] User terminal: A smart device such as smart glasses that has the ability to receive voice input and send it to a server.

[1019] Server: A central device that manages the voice recognition engine, natural language processing engine, generative AI model, and collaboration with external services.

[1020] External services: Online services that connect to the system via external APIs, such as transaction information or inventory status.

[1021] System Operation Overview

[1022] Accepting user input

[1023] The user speaks to the smart glasses, saying, "Please show me the cosmetics section." The smart glasses receive this voice input and send it to the server. The user's device records the voice data and sends it to the server.

[1024] Speech-to-text conversion

[1025] The server uses a speech recognition engine (for example, Google Cloud Speech-to-Text API) to convert the received voice data into text data. This conversion generates the text "Please show me the cosmetics section."

[1026] Text Analysis

[1027] The server uses a natural language processing engine (e.g., IBM Watson NLP API) to analyze the text data and identify the user's intent. The NLP engine understands the meaning and structure of the text and understands that the user is seeking guidance.

[1028] Generating an appropriate response

[1029] The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate a response based on the user's intent, such as "The cosmetics section is on the second floor. Please take the escalator."

[1030] Deciding whether to link with external services

[1031] The server determines whether or not it needs to connect to an external service based on the generated response. For example, if a user inquires about the stock of a specific product, it determines that it needs to check the stock information.

[1032] Calling external service APIs

[1033] The server connects to the store's inventory management system via an external service API to check the stock status of a specific product. For example, it performs a process such as "calling an API to obtain stock information and obtaining the stock status of lipstick."

[1034] Generate the final response

[1035] The server combines the generated AI model's response with information obtained from external services to generate a final response, such as "The cosmetics section is on the second floor. Please take the escalator. The lipstick you are looking for is in stock."

[1036] Providing a response to the user

[1037] The user device (smart glasses) provides the final response received from the server in a display format, allowing the user to visually confirm specific guidance information in response to voice input.

[1038] Examples of specific examples and prompts

[1039] As a concrete example, consider a case where a user asks "What is the stock status of lipstick?" The server converts the speech to text and then inputs the following prompt into the generative AI model to get a response:

[1040] Example prompt sentence:

[1041] User Question: What is the availability of lipstick?

[1042] Generate an appropriate response.

[1043] Example response:

[1044] "Lipstick is in stock. The cosmetics section is on the second floor. Please take the escalator."

[1045] In this way, a system can be built that generates appropriate responses in real time based on the user's voice input and delivers them through smart devices.

[1046] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1047] Step 1:

[1048] The user inputs the information by voice. The user speaks to the smart glasses, saying, "Please show me the cosmetics section." The voice input (voice data) is sent to the device.

[1049] Step 2:

[1050] The device records the audio data and sends it to the server. Input: Audio data. Output: Audio data sent to the server.

[1051] Step 3:

[1052] The server uses a speech recognition engine to convert the voice data into text data. Input: Voice data. Output: Text data saying "Please show me to the cosmetics section." Specific operation: Uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API).

[1053] Step 4:

[1054] The server uses a natural language processing engine to analyze the text data and identify the user's intent. Input: Text data. Output: User intent (e.g., "I need guidance"). Specific behavior: Uses a natural language processing engine (e.g., IBM Watson NLP API).

[1055] Step 5:

[1056] The server uses a generative AI model to generate a response based on the user's intent. Input: User's intent. Output: Response text (e.g., "The cosmetics section is on the second floor. Please use the escalator."). Specific behavior: Uses a generative AI model (e.g., OpenAI GPT-4).

[1057] Step 6:

[1058] The server determines whether or not integration with an external service is necessary based on the generated response. Input: Response text. Output: Necessity of integration with an external service (e.g., "Stock information needs to be confirmed").

[1059] Step 7:

[1060] If necessary, the server calls an external service API to check the stock status of a specific product. Input: Product name. Output: Stock status data. Specific operation: Calls an external service API (e.g., inventory management system API) to obtain stock information.

[1061] Step 8:

[1062] The server combines the generated AI model's response with information obtained from external services. Input: Response text, stock status data. Output: Final response text (e.g., "The cosmetics section is on the second floor. Please take the escalator. The lipstick you are looking for is in stock.").

[1063] Step 9:

[1064] Provides the final response received by the user device from the server in a display format. Input: Final response text. Output: Response displayed in the user's field of view.

[1065] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1066] This invention relates to a system that converts user input into text data, analyzes emotions and intentions, and generates appropriate responses. It also includes a means for linking with external services to obtain necessary information and reflect it in the response. The invention particularly aims to provide responses based on the user's emotions by combining an emotion engine, thereby optimizing support for the user.

[1067] System configuration

[1068] The system consists of the following main components:

[1069] 1. User terminal: A device used by the user (smartphone, tablet, PC, etc.) that has the function of receiving voice input and sending it to the server.

[1070] 2. Server: A central device that manages the speech recognition engine, natural language processing engine, generative AI model, emotion engine, and collaboration with external services.

[1071] 3. External services: These are online services that connect to the system through APIs, such as reservation services.

[1072] System Operation Overview

[1073] The user inputs a request such as a refresh or reservation into the user terminal by voice. The terminal sends this voice data to the server. The server performs the following process and provides the final response to the user.

[1074] Program processing explanation

[1075] 1. Accept user input:

[1076] The user speaks to the device, saying, "I'm tired and want to refresh myself."

[1077] The device records this audio data and sends it to the server.

[1078] 2. Speech to text conversion:

[1079] The server converts the received voice data into text data using a voice recognition engine.

[1080] The voice recognition engine analyzes the voice data and generates the text "I'm tired, so I want to refresh myself."

[1081] 3. Text Analysis:

[1082] The server uses a natural language processing (NLP) engine to analyze the generated text data.

[1083] The NLP engine parses the text and understands the user's intent as "I'm looking for refresh suggestions."

[1084] 4. Sentiment analysis:

[1085] The server uses an emotion engine to determine the emotion of the text data.

[1086] The emotion engine identifies emotions such as "tired" and passes the results on to the next process.

[1087] 5. Generating appropriate responses:

[1088] The server uses generative AI models to generate responses that are appropriate to the user's emotions and needs.

[1089] Based on the instruction "Suggest yoga to refresh," the generative AI model creates a response message saying, "How about yoga to relax?"

[1090] 6. Decision on external service integration:

[1091] The server analyzes the generated response message and determines whether it is necessary to link with an external service.

[1092] From the generated message, it is determined that "a reservation at a yoga studio is required."

[1093] 7. Calling external service APIs:

[1094] The server calls an external service API and sends a request to get reservation information for a yoga studio.

[1095] Check available slots via API.

[1096] 8. External service response processing:

[1097] The external service receives the request and returns available slot information to the server, such as "Tomorrow is available at 5:00 PM."

[1098] 9. Generate the final response:

[1099] The server integrates the response of the generated AI model with the reservation information received from the external service.

[1100] As a final response, create a message that reads, "Yoga studio reservations are available. We're available tomorrow at 5:00 PM."

[1101] 10. Providing Responses to Users:

[1102] The server sends a final response message to the terminal, which receives it and displays it to the user.

[1103] Based on the information displayed, users can take specific actions, such as booking a yoga studio.

[1104] Through this series of processes, users can receive efficient and individualized support. One feature of this system is that it generates responses taking into account the user's emotions, thereby improving the user experience. For example, if a user feels like "refreshing" after returning home tired from work, the system can not only provide information but also make appropriate suggestions based on that emotion.

[1105] The processing flow will be explained below.

[1106] Step 1:

[1107] The user speaks to the device, saying, "I'm tired, so I want to refresh myself." The device records this speech and sends it to the server as audio data.

[1108] Step 2:

[1109] The server uses a speech recognition engine to convert the received voice data into text data. The speech recognition engine analyzes the voice data and generates the text "I'm tired, so I want to refresh myself."

[1110] Step 3:

[1111] The server uses a natural language processing (NLP) engine to analyze the generated text data. The NLP engine parses the text and understands the user's intent as "I'm looking for refresh suggestions."

[1112] Step 4:

[1113] The server uses an emotion engine to determine the emotion of the text data. The emotion engine identifies an emotion such as "tired" and passes the result on to the next process.

[1114] Step 5:

[1115] The server uses the generative AI model to generate a response appropriate to the user's emotions and needs. Based on the instruction "Suggest yoga to refresh yourself," the generative AI model creates a response message saying, "How about yoga to relax?"

[1116] Step 6:

[1117] The server analyzes the generated response message and determines whether it is necessary to connect to an external service. From the generated message, it determines that "a reservation at a yoga studio is required."

[1118] Step 7:

[1119] The server calls an external service API and sends a request to get reservation information for the yoga studio. The API checks for available slots.

[1120] Step 8:

[1121] The external service receives the request and returns available slot information to the server, such as "Tomorrow is available at 5:00 PM."

[1122] Step 9:

[1123] The server combines the response of the generated AI model with the reservation information received from the external service, and creates a final response message saying, "Yoga studio reservations are available. We have availability tomorrow at 5:00 PM."

[1124] Step 10:

[1125] The server sends a final response message to the device, which receives it and displays it to the user. The user can then take specific action, such as booking a yoga studio, based on the displayed information.

[1126] This process allows users to receive efficient and personalized support.

[1127] Example 2

[1128] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1129] Many current systems can receive user voice input, but they have limitations in their ability to fully understand the user's emotions and intentions and generate appropriate responses. They also have difficulty linking with external services to obtain necessary information and provide consistent responses to users. This results in a poor user experience and lowers user satisfaction. The objective of this invention is to provide a system that generates responses based on the user's emotions and intentions and links with external services to provide high-quality support.

[1130] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1131] In this invention, the server includes means for receiving a user's voice input and converting the voice into text data, means for analyzing the text data to understand the user's intention, means for analyzing the emotion in the text data and generating a response according to the emotion, means for analyzing the generated response and coordinating with an external service to obtain necessary information and reflect this in the response, and means for providing the final response to the user. This allows the user to receive an appropriate response based on their own emotions and intentions. Furthermore, by coordinating with an external service to obtain necessary information, the server can receive consistent, high-quality support.

[1132] "Voice input" refers to instructions or requests that a user gives to a system through voice.

[1133] "Text data" refers to textual information converted from voice input, and is digital information in a format that can be understood by humans.

[1134] "Sentiment analysis" is the process of detecting emotional information contained in text data and identifying that emotion.

[1135] A "natural language processing engine" is software or algorithms that analyze text data and perform syntactic analysis and extract intent.

[1136] "Response generation" is the process of creating an appropriate response message based on the analyzed intentions and emotions.

[1137] "External Services" refers to online services that connect to the system through APIs and provide additional information or functionality.

[1138] "Reservation information" refers to detailed information such as available reservation dates, times, and locations obtained from external services.

[1139] "Final Response" refers to the final form of the response message generated for delivery to the user.

[1140] "User experience" is a general term for the ease of use and satisfaction that users feel when using a system.

[1141] This invention relates to a system that receives a user's voice input, converts the voice into text data, analyzes the text data to understand the user's intentions and emotions, generates an appropriate response, and works with external services to obtain necessary information and provide it to the user.

[1142] The system mainly consists of the following components:

[1143] 1. User device: A device (smartphone, tablet, PC, etc.) that allows the user to input voice data and has the function of sending voice data to a server.

[1144] 2. Server: A central device that manages the speech recognition engine, natural language processing engine, generative AI model, emotion engine, and collaboration with external services.

[1145] 3. External services: These are online services that connect to the system through APIs, such as reservation services.

[1146] Hardware and software used

[1147] Speech recognition engine: Converts user speech into text data using the Google Speech-to-Text API.

[1148] Natural language processing engine: Uses the Google Cloud Natural Language API to analyze text data and understand user intent.

[1149] Sentiment Engine: Uses IBM Watson Tone Analyzer to determine the sentiment of text data.

[1150] Generative AI model: OpenAI's GPT-3 is used to generate responses appropriate to the user's emotions and needs.

[1151] External Service API: Use the ClassPass API or other booking service APIs to get the information you need.

[1152] Actual operation example

[1153] When a user comes home tired from work, they say to their smartphone, "I'm tired, so I want to refresh myself." The smartphone records the voice data and sends it to the server. The server then performs the following processes:

[1154] 1. Speech recognition: The server uses the Google Speech-to-Text API to convert the received voice data into text data. The resulting text is "I'm tired, so I want to refresh myself."

[1155] 2. Natural Language Processing: The server uses the Google Cloud Natural Language API to parse the text and understand the user's intent as "I'm looking for refresh suggestions."

[1156] 3. Sentiment analysis: The server uses IBM Watson Tone Analyzer to identify the emotion "tired."

[1157] 4. Response Generation: The server uses OpenAI's GPT-3 to create a response message: "How about yoga to relax?"

[1158] 5. External Service Integration: The server analyzes the generated message and determines that a yoga studio reservation is required. It calls the ClassPass API to obtain the reservation information.

[1159] 6. Generate final response: Based on the obtained reservation information, create a message saying, "Yoga studio reservations are available. Available tomorrow at 5:00 PM."

[1160] 7. Provide to user: Send the final response message to the terminal and display it to the user.

[1161] Example prompt sentence:

[1162] User's voice input: "I'm tired and want to refresh myself."

[1163] The system responds: "How about some yoga to relax? We have a yoga studio available for booking. I'm available tomorrow at 5:00 PM."

[1164] This allows users to consistently receive relevant suggestions based on their emotions and intentions, as well as the booking information they need.

[1165] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1166] Step 1:

[1167] Accepting user voice input

[1168] The user speaks into their smartphone, saying, "I'm tired and want to refresh myself."

[1169] This voice becomes the input data.

[1170] The device uses a built-in microphone to record the user's voice and convert it into digital audio data.

[1171] The output is audio data.

[1172] Step 2:

[1173] Sending audio data

[1174] The device sends the recorded audio data to the server.

[1175] Specifically, the terminal sends the audio data to the server as an HTTP request.

[1176] The input is the voice data from the terminal, and the output is the voice data sent to the server.

[1177] Step 3:

[1178] Speech-to-text conversion

[1179] The server uses a speech recognition engine to convert the voice data into text data.

[1180] For example, it calls the Google Speech-to-Text API to analyze voice data.

[1181] The voice waveform is converted into text data, and the text "I'm tired, so I want to refresh myself" is generated.

[1182] The input is audio data and the output is text data.

[1183] Step 4:

[1184] Text Analysis

[1185] The server uses a natural language processing engine to analyze the generated text data.

[1186] For example, it uses the Google Cloud Natural Language API to parse the text and understand the user's intent as "I'm looking for refresh suggestions."

[1187] The input is text data and the output is the parsed intent.

[1188] Step 5:

[1189] sentiment analysis

[1190] The server uses an emotion engine to determine the emotion of the text data.

[1191] For example, IBM Watson Tone Analyzer is used to identify the emotion meaning "tiredness" from the analyzed intent.

[1192] The input is text data and the output is identified emotion data.

[1193] Step 6:

[1194] Generating an appropriate response

[1195] The server uses generative AI models to generate responses that are appropriate to the user's emotions and needs.

[1196] For example, OpenAI's GPT-3 is used to create a response message based on the instruction "Suggest yoga for refreshment."

[1197] The input is the user's intent and emotional data, and the output is the generated response message.

[1198] Step 7:

[1199] Deciding whether to link with external services

[1200] The server analyzes the generated response message and determines whether it is necessary to link with an external service.

[1201] From the generated AI's output message, it is determined that "a reservation at a yoga studio is required."

[1202] The input is the generated response message, and the output is the decision on the need for external service integration.

[1203] Step 8:

[1204] Calling external service APIs

[1205] The server calls an external service API and sends a request to get reservation information for a yoga studio.

[1206] For example, use the ClassPass API to check available slots for reservation.

[1207] The input is information that requires collaboration with an external service, and the output is reservation information obtained from the external service.

[1208] Step 9:

[1209] Response processing for external services

[1210] The external service receives the request and returns available slot information to the server.

[1211] For example, provide information such as "I'm free tomorrow at 5:00 p.m."

[1212] The server receives this information.

[1213] The input is the response from the external service API, and the output is the available slot information.

[1214] Step 10:

[1215] Generate the final response

[1216] The server integrates the response of the generated AI model with the reservation information received from the external service.

[1217] As a final response, create a message that reads, "Yoga studio reservations are available. We're available tomorrow at 5:00 PM."

[1218] The input is the response of the generative AI model and reservation information obtained from an external service, and the output is the final response message.

[1219] Step 11:

[1220] Providing a response to the user

[1221] The server sends a final response message to the terminal.

[1222] The terminal receives this message and displays it to the user.

[1223] Based on the information displayed, users can take specific actions, such as booking a yoga studio.

[1224] The input is the final response message, and the output is the user's action.

[1225] (Application example 2)

[1226] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1227] Conventional systems have difficulty generating appropriate responses based on user emotions, making it difficult to quickly and efficiently provide the products and services that users specifically want, especially in physical stores.In addition, there was insufficient integration with inventory information, and the information provided to users was often out of date, limiting the ability to improve the user experience.

[1228] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1229] In this invention, the server includes means for receiving user input and converting the input into text data, means for analyzing the text data to understand the user's intention, means for generating an appropriate response based on the analysis results, means for acquiring necessary information in cooperation with an external service and reflecting the acquired information in the response, means for providing the response to the user, means for acquiring inventory information in cooperation with the inventory system of each store, and means for reflecting the acquired inventory information in the response. This enables an appropriate response based on the user's emotions and reflecting the latest inventory information.

[1230] "User" refers to a person who uses the system or receives information or services from the system.

[1231] "Input" refers to the act of a user providing data or information to a system, whether by voice, text, touch, or other means.

[1232] "Text data" refers to character string data converted from voice input, and is the information that forms the basis for the system's analysis.

[1233] "Analysis" refers to the process of analyzing input text data to understand the user's intentions and emotions.

[1234] "Response generation" refers to the process of creating appropriate messages and actions based on the user's intent and emotions.

[1235] "External services" are external online services or databases that the system connects to and that are accessed through APIs.

[1236] "Information" means any data or knowledge to be provided to the User, including stock availability, available slots, recommended products, etc.

[1237] "Providing" refers to the act of displaying or notifying the user of a response generated by the system or information obtained.

[1238] An "inventory system" is a system that manages the inventory information held by each store, and it is possible to link with the system via an API.

[1239] "Acquisition" refers to the act of the system obtaining necessary information from an external service or inventory system.

[1240] The system for implementing this invention comprises a user terminal, a server, and an external service. The user terminal is a device such as a smartphone or tablet that receives voice input and transmits it to the server. The server is a central device that drives a speech recognition engine, a natural language processing engine, a generative AI model, and an emotion engine, and links with external services.

[1241] Hardware and Software Used

[1242] 1. Speech recognition engine: Using the Google Speech-to-Text API, it converts the voice data entered by the user into text data, analyzes the voice data, and generates accurate text.

[1243] 2. Natural Language Processing Engine (NLP): Using the Google Natural Language API, the generated text data is analyzed to understand the user's intent and extract the information needed for further processing.

[1244] 3. Emotion Engine: Uses IBM Watson Tone Analyzer to analyze the emotions in text data and provide information based on those emotions.

[1245] 4. Generative AI model: OpenAI GPT-3 is used to generate responses appropriate to the user's emotions and needs.

[1246] 5. Integration with external services: Connect with inventory systems, reservation services, etc. via API to obtain necessary information.

[1247] Example of a system

[1248] When a user speaks into a smartphone in a physical store saying, "I'm tired and I want a product that will help me relax," this voice input is recorded and sent to a server. The server uses the Google Speech-to-Text API to convert the voice data into text data. This text data is analyzed using the Google Natural Language API to understand the user's intent of "I want to relax." Sentiment analysis is then performed using IBM Watson Tone Analyzer, and the result is determined to be "I'm tired and want to relax."

[1249] Based on the results of this analysis, OpenAI GPT-3 generates the optimal response, creating a message such as, "Would you like some aromatic oils or herbal tea to help you relax?" Next, it connects with the store's inventory system via API to check stock information. For example, it may obtain information such as, "Aroma oils are in stock." The final response, including this result, is then provided to the user.

[1250] Prompt Sentence Examples

[1251] "The user said, 'I'm tired and I want something to relax me.' Sentiment analysis tells you that the user feels tired and wants to relax. Based on this, make suggestions for relaxation products that you can offer in-store."

[1252] Such systems can improve user experience by providing real-time, specific responses based on user sentiment, and also improve store operations by quickly checking inventory and suggesting appropriate products.

[1253] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1254] Program processing flow

[1255] Step 1:

[1256] The user inputs voice data into their smartphone. Specifically, they say, "I'm tired, so I want a product that will help me relax" at a physical store. This voice data is recorded and sent to the server by the user's device.

[1257] Input: User's voice data

[1258] Output: Sending audio data to the server

[1259] Step 2:

[1260] The server converts the received voice data into text data using the Google Speech-to-Text API. This API analyzes the voice data and generates text such as "I'm tired, so I'd like a product that helps me relax."

[1261] Input: Audio data

[1262] Output: Text data

[1263] Step 3:

[1264] The server uses the Google Natural Language API to analyze the generated text data. The NLP engine parses the text and understands the user's intent as "I want to relax."

[1265] Input: Text data

[1266] Output: Analysis of user intent

[1267] Step 4:

[1268] The server uses IBM Watson Tone Analyzer to analyze the sentiment of the text data, and the sentiment engine identifies emotions such as "I'm tired and want to relax" and passes the results on to the next step in the process.

[1269] Input: Text data

[1270] Output: Emotion analysis results

[1271] Step 5:

[1272] The server uses OpenAI GPT-3 to generate a response appropriate to the user's emotions and needs. Based on the instruction to "suggest products for relaxation," the generative AI model creates a response message such as "How about some aromatic oils or herbal teas to help you relax?"

[1273] Input: Emotion analysis results, user intent

[1274] Output: Response message

[1275] Step 6:

[1276] The server analyzes the generated response message and determines whether it needs to link with the store's inventory system, for example, to check whether a specific product is in stock.

[1277] Input: Response message

[1278] Output: A request for inventory

[1279] Step 7:

[1280] The server calls the store's inventory system API and sends a request to obtain inventory information. The API returns information such as "Aroma oil is in stock."

[1281] Input: Request for inventory check

[1282] Output: Inventory information

[1283] Step 8:

[1284] The server generates the final response based on the inventory information it retrieves. It combines the output of OpenAI GPT-3 with information from the inventory system to create a message that says, "We have aromatic oils and herbal teas in stock. You can see them on the shelves right now."

[1285] Input: Inventory information, response message from the generative AI model

[1286] Output: Final response message

[1287] Step 9:

[1288] The server sends a final response message to the user terminal, which receives the message and displays it to the user.

[1289] Input: Final response message

[1290] Output: Send message to user terminal

[1291] Step 10:

[1292] The user checks the display on the device and searches for the suggested product in the appropriate location in the store. For example, the user may take an action such as "I'm going to look for aroma oils and herbal teas."

[1293] Input: Display last response message

[1294] Output: User actions

[1295] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1296] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1297] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1298] [Fourth embodiment]

[1299] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1300] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1301] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1302] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1303] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1304] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1305] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1306] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1307] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1308] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1309] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1310] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1311] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1312] This invention relates to a system that receives user input, converts the input into text data, analyzes it to generate an appropriate response, and provides the response to the user in cooperation with an external service.

[1313] System configuration

[1314] The system mainly consists of the following components:

[1315] 1. User terminal: A device used by a user that has the ability to receive voice input and send it to a server.

[1316] 2. Server: A central device that manages the speech recognition engine, natural language processing engine, generative AI model, and collaboration with external services.

[1317] 3. External services: Online services that connect to the system via external APIs, such as reservation services.

[1318] System Operation Overview

[1319] The user inputs a request such as a refresh or reservation into the user terminal by voice. The terminal sends this voice data to the server. The server performs the following process and provides the final response to the user.

[1320] Program processing explanation

[1321] 1. Accept user input:

[1322] The user speaks to the device, saying, "I'm tired and want to refresh myself."

[1323] The device records this audio data and sends it to the server.

[1324] 2. Speech to text conversion:

[1325] The server uses a voice recognition engine to convert the received voice data into text data.

[1326] This conversion generates the text "I'm tired so I want to refresh myself."

[1327] 3. Text Analysis:

[1328] The server uses a natural language processing (NLP) engine to analyze the text data.

[1329] The NLP engine understands the meaning and structure of the text and identifies the user's intent.

[1330] For example, understand that the user wants refresh suggestions.

[1331] 4. Sentiment analysis:

[1332] The server uses a sentiment analysis engine to analyze the sentiment of the text data.

[1333] Here, the user's emotions, such as "tired," are identified.

[1334] 5. Generating appropriate responses:

[1335] The server uses a generative AI model to generate a response based on the user's emotions and intent.

[1336] In response generation, suggestions such as "How about some yoga to relax?"

[1337] 6. Decision on external service integration:

[1338] Based on the response generated by the server, it determines whether to link with an external service.

[1339] If it is determined that a yoga studio reservation is necessary, an external service API is called.

[1340] 7. Calling external service APIs:

[1341] The server checks the reservation status through an external service API.

[1342] For example, get information about available slots at a yoga studio.

[1343] 8. Generate the final response:

[1344] The server integrates the responses of the generated AI model with information obtained from external services.

[1345] Generate the final response "Yoga studio reservations available. Available tomorrow at 5:00 PM."

[1346] 9. Providing Responses to Users:

[1347] The device displays the final response received from the server to the user.

[1348] This allows users to receive specific support, such as booking yoga studios.

[1349] In this way, the system provides optimal support according to the user's needs and effectively resolves issues related to time and health management. For example, for a user who is tired from work and wants to refresh themselves, the system provides accurate yoga suggestions and reservation information. This system allows users to live their daily lives efficiently and comfortably.

[1350] The processing flow will be explained below.

[1351] Step 1:

[1352] The user speaks to the device, saying, "I'm tired, so I want to refresh myself." The device records this speech and sends it to the server as audio data.

[1353] Step 2:

[1354] The server uses a speech recognition engine to convert the received voice data into text data. The speech recognition engine analyzes the voice data and generates the text "I'm tired, so I want to refresh myself."

[1355] Step 3:

[1356] The server uses a natural language processing (NLP) engine to analyze the generated text data. The NLP engine parses the text and understands the user's intent as "I'm looking for refresh suggestions."

[1357] Step 4:

[1358] The server uses a sentiment analysis engine to determine the sentiment of the text data, identifying emotions such as "tired" and passing the results on to the next step of processing.

[1359] Step 5:

[1360] The server uses the generative AI model to generate a response appropriate to the user's emotions and needs. Based on the instruction "Suggest yoga to refresh yourself," the generative AI model creates a response message saying, "How about yoga to relax?"

[1361] Step 6:

[1362] The server analyzes the generated response message and determines whether it is necessary to connect to an external service. From the generated message, it determines that "a reservation at a yoga studio is required."

[1363] Step 7:

[1364] The server calls an external service API and sends a request to get reservation information for the yoga studio. The API checks for available slots.

[1365] Step 8:

[1366] The external service receives the request and returns available slot information to the server, such as "Tomorrow is available at 5:00 PM."

[1367] Step 9:

[1368] The server combines the response of the generated AI model with the reservation information received from the external service, and creates a final response message saying, "Yoga studio reservations are available. We have availability tomorrow at 5:00 PM."

[1369] Step 10:

[1370] The server sends a final response message to the device, which receives it and displays it to the user. The user can then take specific action, such as booking a yoga studio, based on the displayed information.

[1371] This process allows users to receive efficient and personalized support.

[1372] Example 1

[1373] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1374] Current dialogue systems must integrate multiple technologies, such as speech recognition, natural language processing, and sentiment analysis, to respond appropriately to diverse user requests. However, existing systems do not fully integrate these functions, making it difficult to accurately grasp the user's intentions and emotions and provide appropriate responses. Furthermore, they are unable to efficiently connect with external services, making it difficult to quickly provide the information and services users desire.

[1375] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1376] In this invention, the server includes means for receiving voice input and converting it into voice data, means for converting the voice data into text data, means for analyzing the text data to understand the user's intention, means for analyzing the sentiment of the text data, means for generating an appropriate response based on the analysis results and the sentiment analysis results, means for acquiring necessary information in cooperation with an external service and reflecting it in the response, and means for providing the response to the user, thereby enabling accurate and prompt responses to a variety of user requests.

[1377] The "means for receiving and converting voice input into voice data" is a component that has the function of converting the contents of a user's voice input into digital voice data.

[1378] The "means for converting the voice data into text data" is a component that has the function of analyzing digital voice data and converting it into corresponding text format information.

[1379] The "means for analyzing the text data to understand the user's intention" is a component that has the function of analyzing the text data using natural language processing technology and identifying the intention implied by the user's utterance.

[1380] The "means for analyzing the emotion of the text data" is a component that has a function of identifying the emotion of the user based on the analyzed text data.

[1381] The "means for generating an appropriate response based on the analysis results and emotion analysis results" is a component that has the function of generating the optimal response to be provided to the user based on the user's intentions and emotions.

[1382] "Means of interacting with external services to obtain necessary information and reflect it in the response" refers to a component that has the function of accessing the APIs of other external systems and services, obtaining necessary data and information, and incorporating it into the response to the user.

[1383] The "means for providing the response to the user" is a component having an interface or function for transmitting the generated response to the user.

[1384] A "generative AI model" is a machine learning model that learns from large amounts of data in advance and performs natural language generation and text analysis.

[1385] A "prompt" is a textual instruction or question input to a generative AI model that is used to elicit a specific response or generated result.

[1386] This invention relates to a system that converts voice input into text, analyzes it to generate appropriate responses, and links with external services. How the invention is realized using specific hardware and software is described below.

[1387] Hardware and software used

[1388] The system is implemented primarily using the following hardware and software:

[1389] User terminal: A device that allows users to input voice, and is equipped with a microphone and network connection capabilities.

[1390] Server: This is the central device that manages the speech recognition engine, natural language processing engine, generative AI model, and collaboration with external services. Here, we use Google Cloud Speech-to-Text API, OpenAI's GPT-4, and IBM Watson Tone Analyzer.

[1391] Accepting and converting voice input

[1392] The user speaks, "I'm tired, so I want to refresh myself." The device records this voice data and sends it over the network to a server. The server then converts the received voice data into text data using the Google Cloud Speech-to-Text API.

[1393] Text data analysis

[1394] The server uses OpenAI's GPT-4 to analyze the converted text data and understand the user's intent. At the same time, it uses IBM Watson Tone Analyzer to analyze the sentiment of the text data. This identifies that the user's intent is "I want to refresh myself" and that the sentiment is "I'm tired."

[1395] Generating a response

[1396] The server uses OpenAI's ChatGPT, a generative AI model, to generate responses based on the user's intent and emotions. Here are some example prompts:

[1397] User: I'm tired from work and want to refresh myself.

[1398] System: How about some yoga to relax? There's a class available at John Smith Studio at 5:00 PM. Would you like to book it?

[1399] Thus, the suggestion: "How about some yoga to relax?"

[1400] Integration with external services

[1401] The server then decides whether to connect to an external service based on the generated response. For example, if it determines that a yoga studio reservation is required, it checks the reservation status through the external service's API. Specifically, it accesses the reservation system's API and obtains information about available slots at the yoga studio.

[1402] Providing a final response

[1403] The server combines the generated response with information obtained from external services to generate a final response: "You can make a reservation at the yoga studio. It's available tomorrow at 5:00 PM." This specific response is provided to the user via the device.

[1404] In this way, the system receives and analyzes the user's voice input, generates appropriate responses, and provides them to the user. For example, if a user is tired from work, the system can provide yoga suggestions and reservation information, allowing the user to live their daily life efficiently and comfortably.

[1405] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1406] Step 1:

[1407] The user speaks, "I'm tired, so I want to refresh myself." The device records this voice data. The input is voice data, which is then converted into a recording. Specifically, the device uses a microphone to capture the voice signal and saves it as a digital audio file.

[1408] Step 2:

[1409] The device sends the recorded audio data to the server. The input is a digital audio file, which is then sent to the server over the network. Specifically, the audio file is uploaded using an HTTP request over the internet connection.

[1410] Step 3:

[1411] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text data. The input is a digital audio file, and the output is text data. Specifically, the server passes the audio file to the API and obtains the text, "I'm tired, so I want to refresh myself."

[1412] Step 4:

[1413] The server uses OpenAI's GPT-4 to analyze the text data and understand the user's intent. The input is the converted text data, and the output is the analysis result that identifies the user's intent. Specifically, the text is passed to GPT-4, which understands that the user is requesting a refresh suggestion.

[1414] Step 5:

[1415] The server uses IBM Watson Tone Analyzer to analyze the sentiment of the text data. The input is the text data, and the output is the sentiment analysis result. Specifically, the text is passed to the analysis engine to identify the sentiment of "tired."

[1416] Step 6:

[1417] The server uses OpenAI's ChatGPT, a generative AI model, to generate a response based on the user's intent and emotions. The input is the user's intent and the results of the emotion analysis, and the output is the generated response. Specifically, the following prompt is passed to ChatGPT:

[1418] User: I'm tired and want to refresh myself.

[1419] System: How about yoga to relax?

[1420] I get the response:

[1421] Step 7:

[1422] The server determines whether to collaborate with an external service based on the generated response. The input is the generated response, and the output is a decision on whether to collaborate with an external service. Specifically, since the response contains "yoga suggestions," the server determines that a reservation at a yoga studio is necessary.

[1423] Step 8:

[1424] The server obtains available slot information through the reservation system's API. The input is a reservation request, and the output is available slot information. Specifically, the server obtains information such as "Available tomorrow at 5:00 PM" through the API.

[1425] Step 9:

[1426] The server integrates the generated response with information from the external service to generate a final response. The input is the generated response and information obtained from the external service, and the output is the final response. Specifically, it generates a response that says, "You can make a reservation at the yoga studio. There is an opening tomorrow at 5:00 PM."

[1427] Step 10:

[1428] The terminal displays the final response received from the server to the user. The input is the final response, and the output is the response displayed on the user's terminal screen. Specifically, the display shows "Yoga studio reservations are available. Available tomorrow at 5:00 PM."

[1429] (Application example 1)

[1430] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1431] Conventional voice input systems lack the ability to convert voice data into text and perform text analysis in the process of generating appropriate responses from user speech. Furthermore, integration with external services is limited, making it difficult to provide real-time information, such as inventory inquiries and reservation services. Furthermore, the lack of visual responses provided via smart devices results in a suboptimal user experience. To solve these problems, a system that utilizes more advanced natural language processing, external service integration, and a visual interface on smart devices is required.

[1432] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1433] In this invention, the server includes means for receiving user input and converting the input into text data, means for analyzing the text data to understand the user's intent, means for generating an appropriate response based on the analysis results, means for acquiring necessary information in cooperation with an external service and reflecting the information in the response, means for generating a detailed response based on the user's intent using a generative AI model, and means for providing the response through a visual interface of a smart device. This enables real-time and accurate information to be provided in response to a variety of user requests.

[1434] "User input" refers to information provided by a user to a system in the form of speech or text.

[1435] "Text data" is data in the form of a character string converted from voice data via a voice recognition engine.

[1436] "Analysis" refers to the process of using a natural language processing engine to understand the meaning and intent of text data.

[1437] An "appropriate response" is one that generates information or instructions that are in line with the user's intentions based on the analysis results.

[1438] "External services" refers to external online services and databases that are linked via an application program interface (API).

[1439] A "generative AI model" is an artificial intelligence model that uses deep learning and other techniques to generate text data based on the user's intentions and emotions.

[1440] "Smart devices" refers to information terminals with internet connectivity, such as smartphones, smart glasses, and head-mounted displays.

[1441] "Visual interface" refers to a means of interacting with users through information and messages displayed on the screen of a smart device.

[1442] We will explain the system to realize this application example. This system mainly consists of the following components:

[1443] User terminal: A smart device such as smart glasses that has the ability to receive voice input and send it to a server.

[1444] Server: A central device that manages the voice recognition engine, natural language processing engine, generative AI model, and collaboration with external services.

[1445] External services: Online services that connect to the system via external APIs, such as transaction information or inventory status.

[1446] System Operation Overview

[1447] Accepting user input

[1448] The user speaks to the smart glasses, saying, "Please show me the cosmetics section." The smart glasses receive this voice input and send it to the server. The user's device records the voice data and sends it to the server.

[1449] Speech-to-text conversion

[1450] The server uses a speech recognition engine (for example, Google Cloud Speech-to-Text API) to convert the received voice data into text data. This conversion generates the text "Please show me the cosmetics section."

[1451] Text Analysis

[1452] The server uses a natural language processing engine (e.g., IBM Watson NLP API) to analyze the text data and identify the user's intent. The NLP engine understands the meaning and structure of the text and understands that the user is seeking guidance.

[1453] Generating an appropriate response

[1454] The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate a response based on the user's intent, such as "The cosmetics section is on the second floor. Please take the escalator."

[1455] Deciding whether to link with external services

[1456] The server determines whether or not it needs to connect to an external service based on the generated response. For example, if a user inquires about the stock of a specific product, it determines that it needs to check the stock information.

[1457] Calling external service APIs

[1458] The server connects to the store's inventory management system via an external service API to check the stock status of a specific product. For example, it performs a process such as "calling an API to obtain stock information and obtaining the stock status of lipstick."

[1459] Generate the final response

[1460] The server combines the generated AI model's response with information obtained from external services to generate a final response, such as "The cosmetics section is on the second floor. Please take the escalator. The lipstick you are looking for is in stock."

[1461] Providing a response to the user

[1462] The user device (smart glasses) provides the final response received from the server in a display format, allowing the user to visually confirm specific guidance information in response to voice input.

[1463] Examples of specific examples and prompts

[1464] As a concrete example, consider a case where a user asks "What is the stock status of lipstick?" The server converts the speech to text and then inputs the following prompt into the generative AI model to get a response:

[1465] Example prompt sentence:

[1466] User Question: What is the availability of lipstick?

[1467] Generate an appropriate response.

[1468] Example response:

[1469] "Lipstick is in stock. The cosmetics section is on the second floor. Please take the escalator."

[1470] In this way, a system can be built that generates appropriate responses in real time based on the user's voice input and delivers them through smart devices.

[1471] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1472] Step 1:

[1473] The user inputs the information by voice. The user speaks to the smart glasses, saying, "Please show me the cosmetics section." The voice input (voice data) is sent to the device.

[1474] Step 2:

[1475] The device records the audio data and sends it to the server. Input: Audio data. Output: Audio data sent to the server.

[1476] Step 3:

[1477] The server uses a speech recognition engine to convert the voice data into text data. Input: Voice data. Output: Text data saying "Please show me to the cosmetics section." Specific operation: Uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API).

[1478] Step 4:

[1479] The server uses a natural language processing engine to analyze the text data and identify the user's intent. Input: Text data. Output: User intent (e.g., "I need guidance"). Specific behavior: Uses a natural language processing engine (e.g., IBM Watson NLP API).

[1480] Step 5:

[1481] The server uses a generative AI model to generate a response based on the user's intent. Input: User's intent. Output: Response text (e.g., "The cosmetics section is on the second floor. Please use the escalator."). Specific behavior: Uses a generative AI model (e.g., OpenAI GPT-4).

[1482] Step 6:

[1483] The server determines whether or not integration with an external service is necessary based on the generated response. Input: Response text. Output: Necessity of integration with an external service (e.g., "Stock information needs to be confirmed").

[1484] Step 7:

[1485] If necessary, the server calls an external service API to check the stock status of a specific product. Input: Product name. Output: Stock status data. Specific operation: Calls an external service API (e.g., inventory management system API) to obtain stock information.

[1486] Step 8:

[1487] The server combines the generated AI model's response with information obtained from external services. Input: Response text, stock status data. Output: Final response text (e.g., "The cosmetics section is on the second floor. Please take the escalator. The lipstick you are looking for is in stock.").

[1488] Step 9:

[1489] Provides the final response received by the user device from the server in a display format. Input: Final response text. Output: Response displayed in the user's field of view.

[1490] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1491] This invention relates to a system that converts user input into text data, analyzes emotions and intentions, and generates appropriate responses. It also includes a means for linking with external services to obtain necessary information and reflect it in the response. The invention particularly aims to provide responses based on the user's emotions by combining an emotion engine, thereby optimizing support for the user.

[1492] System configuration

[1493] The system consists of the following main components:

[1494] 1. User terminal: A device used by the user (smartphone, tablet, PC, etc.) that has the function of receiving voice input and sending it to the server.

[1495] 2. Server: A central device that manages the speech recognition engine, natural language processing engine, generative AI model, emotion engine, and collaboration with external services.

[1496] 3. External services: These are online services that connect to the system through APIs, such as reservation services.

[1497] System Operation Overview

[1498] The user inputs a request such as a refresh or reservation into the user terminal by voice. The terminal sends this voice data to the server. The server performs the following process and provides the final response to the user.

[1499] Program processing explanation

[1500] 1. Accept user input:

[1501] The user speaks to the device, saying, "I'm tired and want to refresh myself."

[1502] The device records this audio data and sends it to the server.

[1503] 2. Speech to text conversion:

[1504] The server converts the received voice data into text data using a voice recognition engine.

[1505] The voice recognition engine analyzes the voice data and generates the text "I'm tired, so I want to refresh myself."

[1506] 3. Text Analysis:

[1507] The server uses a natural language processing (NLP) engine to analyze the generated text data.

[1508] The NLP engine parses the text and understands the user's intent as "I'm looking for refresh suggestions."

[1509] 4. Sentiment analysis:

[1510] The server uses an emotion engine to determine the emotion of the text data.

[1511] The emotion engine identifies emotions such as "tired" and passes the results on to the next process.

[1512] 5. Generating appropriate responses:

[1513] The server uses generative AI models to generate responses that are appropriate to the user's emotions and needs.

[1514] Based on the instruction "Suggest yoga to refresh," the generative AI model creates a response message saying, "How about yoga to relax?"

[1515] 6. Decision on external service integration:

[1516] The server analyzes the generated response message and determines whether it is necessary to link with an external service.

[1517] From the generated message, it is determined that "a reservation at a yoga studio is required."

[1518] 7. Calling external service APIs:

[1519] The server calls an external service API and sends a request to get reservation information for a yoga studio.

[1520] Check available slots via API.

[1521] 8. External service response processing:

[1522] The external service receives the request and returns available slot information to the server, such as "Tomorrow is available at 5:00 PM."

[1523] 9. Generate the final response:

[1524] The server integrates the response of the generated AI model with the reservation information received from the external service.

[1525] As a final response, create a message that reads, "Yoga studio reservations are available. We're available tomorrow at 5:00 PM."

[1526] 10. Providing Responses to Users:

[1527] The server sends a final response message to the terminal, which receives it and displays it to the user.

[1528] Based on the information displayed, users can take specific actions, such as booking a yoga studio.

[1529] Through this series of processes, users can receive efficient and individualized support. One feature of this system is that it generates responses taking into account the user's emotions, thereby improving the user experience. For example, if a user feels like "refreshing" after returning home tired from work, the system can not only provide information but also make appropriate suggestions based on that emotion.

[1530] The processing flow will be explained below.

[1531] Step 1:

[1532] The user speaks to the device, saying, "I'm tired, so I want to refresh myself." The device records this speech and sends it to the server as audio data.

[1533] Step 2:

[1534] The server uses a speech recognition engine to convert the received voice data into text data. The speech recognition engine analyzes the voice data and generates the text "I'm tired, so I want to refresh myself."

[1535] Step 3:

[1536] The server uses a natural language processing (NLP) engine to analyze the generated text data. The NLP engine parses the text and understands the user's intent as "I'm looking for refresh suggestions."

[1537] Step 4:

[1538] The server uses an emotion engine to determine the emotion of the text data. The emotion engine identifies an emotion such as "tired" and passes the result on to the next process.

[1539] Step 5:

[1540] The server uses the generative AI model to generate a response appropriate to the user's emotions and needs. Based on the instruction "Suggest yoga to refresh yourself," the generative AI model creates a response message saying, "How about yoga to relax?"

[1541] Step 6:

[1542] The server analyzes the generated response message and determines whether it is necessary to connect to an external service. From the generated message, it determines that "a reservation at a yoga studio is required."

[1543] Step 7:

[1544] The server calls an external service API and sends a request to get reservation information for the yoga studio. The API checks for available slots.

[1545] Step 8:

[1546] The external service receives the request and returns available slot information to the server, such as "Tomorrow is available at 5:00 PM."

[1547] Step 9:

[1548] The server combines the response of the generated AI model with the reservation information received from the external service, and creates a final response message saying, "Yoga studio reservations are available. We have availability tomorrow at 5:00 PM."

[1549] Step 10:

[1550] The server sends a final response message to the device, which receives it and displays it to the user. The user can then take specific action, such as booking a yoga studio, based on the displayed information.

[1551] This process allows users to receive efficient and personalized support.

[1552] Example 2

[1553] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1554] Many current systems can receive user voice input, but they have limitations in their ability to fully understand the user's emotions and intentions and generate appropriate responses. They also have difficulty linking with external services to obtain necessary information and provide consistent responses to users. This results in a poor user experience and lowers user satisfaction. The objective of this invention is to provide a system that generates responses based on the user's emotions and intentions and links with external services to provide high-quality support.

[1555] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1556] In this invention, the server includes means for receiving a user's voice input and converting the voice into text data, means for analyzing the text data to understand the user's intention, means for analyzing the emotion in the text data and generating a response according to the emotion, means for analyzing the generated response and coordinating with an external service to obtain necessary information and reflect this in the response, and means for providing the final response to the user. This allows the user to receive an appropriate response based on their own emotions and intentions. Furthermore, by coordinating with an external service to obtain necessary information, the server can receive consistent, high-quality support.

[1557] "Voice input" refers to instructions or requests that a user gives to a system through voice.

[1558] "Text data" refers to textual information converted from voice input, and is digital information in a format that can be understood by humans.

[1559] "Sentiment analysis" is the process of detecting emotional information contained in text data and identifying that emotion.

[1560] A "natural language processing engine" is software or algorithms that analyze text data and perform syntactic analysis and extract intent.

[1561] "Response generation" is the process of creating an appropriate response message based on the analyzed intentions and emotions.

[1562] "External Services" refers to online services that connect to the system through APIs and provide additional information or functionality.

[1563] "Reservation information" refers to detailed information such as available reservation dates, times, and locations obtained from external services.

[1564] "Final Response" refers to the final form of the response message generated for delivery to the user.

[1565] "User experience" is a general term for the ease of use and satisfaction that users feel when using a system.

[1566] This invention relates to a system that receives a user's voice input, converts the voice into text data, analyzes the text data to understand the user's intentions and emotions, generates an appropriate response, and works with external services to obtain necessary information and provide it to the user.

[1567] The system mainly consists of the following components:

[1568] 1. User device: A device (smartphone, tablet, PC, etc.) that allows the user to input voice data and has the function of sending voice data to a server.

[1569] 2. Server: A central device that manages the speech recognition engine, natural language processing engine, generative AI model, emotion engine, and collaboration with external services.

[1570] 3. External services: These are online services that connect to the system through APIs, such as reservation services.

[1571] Hardware and software used

[1572] Speech recognition engine: Converts user speech into text data using the Google Speech-to-Text API.

[1573] Natural language processing engine: Uses the Google Cloud Natural Language API to analyze text data and understand user intent.

[1574] Sentiment Engine: Uses IBM Watson Tone Analyzer to determine the sentiment of text data.

[1575] Generative AI model: OpenAI's GPT-3 is used to generate responses appropriate to the user's emotions and needs.

[1576] External Service API: Use the ClassPass API or other booking service APIs to get the information you need.

[1577] Actual operation example

[1578] When a user comes home tired from work, they say to their smartphone, "I'm tired, so I want to refresh myself." The smartphone records the voice data and sends it to the server. The server then performs the following processes:

[1579] 1. Speech recognition: The server uses the Google Speech-to-Text API to convert the received voice data into text data. The resulting text is "I'm tired, so I want to refresh myself."

[1580] 2. Natural Language Processing: The server uses the Google Cloud Natural Language API to parse the text and understand the user's intent as "I'm looking for refresh suggestions."

[1581] 3. Sentiment analysis: The server uses IBM Watson Tone Analyzer to identify the emotion "tired."

[1582] 4. Response Generation: The server uses OpenAI's GPT-3 to create a response message: "How about yoga to relax?"

[1583] 5. External Service Integration: The server analyzes the generated message and determines that a yoga studio reservation is required. It calls the ClassPass API to obtain the reservation information.

[1584] 6. Generate final response: Based on the obtained reservation information, create a message saying, "Yoga studio reservations are available. Available tomorrow at 5:00 PM."

[1585] 7. Provide to user: Send the final response message to the terminal and display it to the user.

[1586] Example prompt sentence:

[1587] User's voice input: "I'm tired and want to refresh myself."

[1588] The system responds: "How about some yoga to relax? We have a yoga studio available for booking. I'm available tomorrow at 5:00 PM."

[1589] This allows users to consistently receive relevant suggestions based on their emotions and intentions, as well as the booking information they need.

[1590] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1591] Step 1:

[1592] Accepting user voice input

[1593] The user speaks into their smartphone, saying, "I'm tired and want to refresh myself."

[1594] This voice becomes the input data.

[1595] The device uses a built-in microphone to record the user's voice and convert it into digital audio data.

[1596] The output is audio data.

[1597] Step 2:

[1598] Sending audio data

[1599] The device sends the recorded audio data to the server.

[1600] Specifically, the terminal sends the audio data to the server as an HTTP request.

[1601] The input is the voice data from the terminal, and the output is the voice data sent to the server.

[1602] Step 3:

[1603] Speech-to-text conversion

[1604] The server uses a speech recognition engine to convert the voice data into text data.

[1605] For example, it calls the Google Speech-to-Text API to analyze voice data.

[1606] The voice waveform is converted into text data, and the text "I'm tired, so I want to refresh myself" is generated.

[1607] The input is audio data and the output is text data.

[1608] Step 4:

[1609] Text Analysis

[1610] The server uses a natural language processing engine to analyze the generated text data.

[1611] For example, it uses the Google Cloud Natural Language API to parse the text and understand the user's intent as "I'm looking for refresh suggestions."

[1612] The input is text data and the output is the parsed intent.

[1613] Step 5:

[1614] sentiment analysis

[1615] The server uses an emotion engine to determine the emotion of the text data.

[1616] For example, IBM Watson Tone Analyzer is used to identify the emotion meaning "tiredness" from the analyzed intent.

[1617] The input is text data and the output is identified emotion data.

[1618] Step 6:

[1619] Generating an appropriate response

[1620] The server uses generative AI models to generate responses that are appropriate to the user's emotions and needs.

[1621] For example, OpenAI's GPT-3 is used to create a response message based on the instruction "Suggest yoga for refreshment."

[1622] The input is the user's intent and emotional data, and the output is the generated response message.

[1623] Step 7:

[1624] Deciding whether to link with external services

[1625] The server analyzes the generated response message and determines whether it is necessary to link with an external service.

[1626] From the generated AI's output message, it is determined that "a reservation at a yoga studio is required."

[1627] The input is the generated response message, and the output is the decision on the need for external service integration.

[1628] Step 8:

[1629] Calling external service APIs

[1630] The server calls an external service API and sends a request to get reservation information for a yoga studio.

[1631] For example, use the ClassPass API to check available slots for reservation.

[1632] The input is information that requires collaboration with an external service, and the output is reservation information obtained from the external service.

[1633] Step 9:

[1634] Response processing for external services

[1635] The external service receives the request and returns available slot information to the server.

[1636] For example, provide information such as "I'm free tomorrow at 5:00 p.m."

[1637] The server receives this information.

[1638] The input is the response from the external service API, and the output is the available slot information.

[1639] Step 10:

[1640] Generate the final response

[1641] The server integrates the response of the generated AI model with the reservation information received from the external service.

[1642] As a final response, create a message that reads, "Yoga studio reservations are available. We're available tomorrow at 5:00 PM."

[1643] The input is the response of the generative AI model and reservation information obtained from an external service, and the output is the final response message.

[1644] Step 11:

[1645] Providing a response to the user

[1646] The server sends a final response message to the terminal.

[1647] The terminal receives this message and displays it to the user.

[1648] Based on the information displayed, users can take specific actions, such as booking a yoga studio.

[1649] The input is the final response message, and the output is the user's action.

[1650] (Application example 2)

[1651] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1652] Conventional systems have difficulty generating appropriate responses based on user emotions, making it difficult to quickly and efficiently provide the products and services that users specifically want, especially in physical stores.In addition, there was insufficient integration with inventory information, and the information provided to users was often out of date, limiting the ability to improve the user experience.

[1653] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1654] In this invention, the server includes means for receiving user input and converting the input into text data, means for analyzing the text data to understand the user's intention, means for generating an appropriate response based on the analysis results, means for acquiring necessary information in cooperation with an external service and reflecting the acquired information in the response, means for providing the response to the user, means for acquiring inventory information in cooperation with the inventory system of each store, and means for reflecting the acquired inventory information in the response. This enables an appropriate response based on the user's emotions and reflecting the latest inventory information.

[1655] "User" refers to a person who uses the system or receives information or services from the system.

[1656] "Input" refers to the act of a user providing data or information to a system, whether by voice, text, touch, or other means.

[1657] "Text data" refers to character string data converted from voice input, and is the information that forms the basis for the system's analysis.

[1658] "Analysis" refers to the process of analyzing input text data to understand the user's intentions and emotions.

[1659] "Response generation" refers to the process of creating appropriate messages and actions based on the user's intent and emotions.

[1660] "External services" are external online services or databases that the system connects to and that are accessed through APIs.

[1661] "Information" means any data or knowledge to be provided to the User, including stock availability, available slots, recommended products, etc.

[1662] "Providing" refers to the act of displaying or notifying the user of a response generated by the system or information obtained.

[1663] An "inventory system" is a system that manages the inventory information held by each store, and it is possible to link with the system via an API.

[1664] "Acquisition" refers to the act of the system obtaining necessary information from an external service or inventory system.

[1665] The system for implementing this invention comprises a user terminal, a server, and an external service. The user terminal is a device such as a smartphone or tablet that receives voice input and transmits it to the server. The server is a central device that drives a speech recognition engine, a natural language processing engine, a generative AI model, and an emotion engine, and links with external services.

[1666] Hardware and Software Used

[1667] 1. Speech recognition engine: Using the Google Speech-to-Text API, it converts the voice data entered by the user into text data, analyzes the voice data, and generates accurate text.

[1668] 2. Natural Language Processing Engine (NLP): Using the Google Natural Language API, the generated text data is analyzed to understand the user's intent and extract the information needed for further processing.

[1669] 3. Emotion Engine: Uses IBM Watson Tone Analyzer to analyze the emotions in text data and provide information based on those emotions.

[1670] 4. Generative AI model: OpenAI GPT-3 is used to generate responses appropriate to the user's emotions and needs.

[1671] 5. Integration with external services: Connect with inventory systems, reservation services, etc. via API to obtain necessary information.

[1672] Example of a system

[1673] When a user speaks into a smartphone in a physical store saying, "I'm tired and I want a product that will help me relax," this voice input is recorded and sent to a server. The server uses the Google Speech-to-Text API to convert the voice data into text data. This text data is analyzed using the Google Natural Language API to understand the user's intent of "I want to relax." Sentiment analysis is then performed using IBM Watson Tone Analyzer, and the result is determined to be "I'm tired and want to relax."

[1674] Based on the results of this analysis, OpenAI GPT-3 generates the optimal response, creating a message such as, "Would you like some aromatic oils or herbal tea to help you relax?" Next, it connects with the store's inventory system via API to check stock information. For example, it may obtain information such as, "Aroma oils are in stock." The final response, including this result, is then provided to the user.

[1675] Prompt Sentence Examples

[1676] "The user said, 'I'm tired and I want something to relax me.' Sentiment analysis tells you that the user feels tired and wants to relax. Based on this, make suggestions for relaxation products that you can offer in-store."

[1677] Such systems can improve user experience by providing real-time, specific responses based on user sentiment, and also improve store operations by quickly checking inventory and suggesting appropriate products.

[1678] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1679] Program processing flow

[1680] Step 1:

[1681] The user inputs voice data into their smartphone. Specifically, they say, "I'm tired, so I want a product that will help me relax" at a physical store. This voice data is recorded and sent to the server by the user's device.

[1682] Input: User's voice data

[1683] Output: Sending audio data to the server

[1684] Step 2:

[1685] The server converts the received voice data into text data using the Google Speech-to-Text API. This API analyzes the voice data and generates text such as "I'm tired, so I'd like a product that helps me relax."

[1686] Input: Audio data

[1687] Output: Text data

[1688] Step 3:

[1689] The server uses the Google Natural Language API to analyze the generated text data. The NLP engine parses the text and understands the user's intent as "I want to relax."

[1690] Input: Text data

[1691] Output: Analysis of user intent

[1692] Step 4:

[1693] The server uses IBM Watson Tone Analyzer to analyze the sentiment of the text data, and the sentiment engine identifies emotions such as "I'm tired and want to relax" and passes the results on to the next step in the process.

[1694] Input: Text data

[1695] Output: Emotion analysis results

[1696] Step 5:

[1697] The server uses OpenAI GPT-3 to generate a response appropriate to the user's emotions and needs. Based on the instruction to "suggest products for relaxation," the generative AI model creates a response message such as "How about some aromatic oils or herbal teas to help you relax?"

[1698] Input: Emotion analysis results, user intent

[1699] Output: Response message

[1700] Step 6:

[1701] The server analyzes the generated response message and determines whether it needs to link with the store's inventory system, for example, to check whether a specific product is in stock.

[1702] Input: Response message

[1703] Output: A request for inventory

[1704] Step 7:

[1705] The server calls the store's inventory system API and sends a request to obtain inventory information. The API returns information such as "Aroma oil is in stock."

[1706] Input: Request for inventory check

[1707] Output: Inventory information

[1708] Step 8:

[1709] The server generates the final response based on the inventory information it retrieves. It combines the output of OpenAI GPT-3 with information from the inventory system to create a message that says, "We have aromatic oils and herbal teas in stock. You can see them on the shelves right now."

[1710] Input: Inventory information, response message from the generative AI model

[1711] Output: Final response message

[1712] Step 9:

[1713] The server sends a final response message to the user terminal, which receives the message and displays it to the user.

[1714] Input: Final response message

[1715] Output: Send message to user terminal

[1716] Step 10:

[1717] The user checks the display on the device and searches for the suggested product in the appropriate location in the store. For example, the user may take an action such as "I'm going to look for aroma oils and herbal teas."

[1718] Input: Display last response message

[1719] Output: User actions

[1720] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1721] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1722] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1723] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1724] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1725] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1726] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1727] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1728] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1729] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1730] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1731] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1732] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1733] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1734] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1735] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1736] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1737] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1738] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1739] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1740] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1741] The following is further disclosed regarding the above embodiment.

[1742] (Claim 1)

[1743] a means for receiving user input and converting the input into text data;

[1744] means for analyzing the text data to understand the user's intent;

[1745] means for generating an appropriate response based on the analysis results;

[1746] A means of linking with external services to obtain necessary information and reflect it in responses,

[1747] means for providing said response to a user;

[1748] A system including:

[1749] (Claim 2)

[1750] 2. The system according to claim 1, wherein the response generating means further comprises means for analyzing emotions in the text data and generating a response according to the emotions.

[1751] (Claim 3)

[1752] 2. The system according to claim 1, wherein the means for linking with the external service includes means for accessing a reservation service via an API, obtaining reservation status, and providing it to the user.

[1753] "Example 1"

[1754] (Claim 1)

[1755] means for receiving and converting audio input into audio data;

[1756] means for converting the voice data into text data;

[1757] means for analyzing the text data to understand the user's intent;

[1758] means for analyzing sentiment in the text data;

[1759] means for generating an appropriate response based on the analysis results and sentiment analysis results;

[1760] A means of linking with external services to obtain necessary information and reflect it in responses,

[1761] means for providing said response to a user;

[1762] A system including:

[1763] (Claim 2)

[1764] 2. The system of claim 1, wherein the response generation means further comprises means for generating a response based on a prompt sentence using a generative AI model.

[1765] (Claim 3)

[1766] 2. The system according to claim 1, wherein the means for linking with the external service includes means for accessing a reservation service via an API, obtaining reservation status, and providing it to the user.

[1767] "Application Example 1"

[1768] (Claim 1)

[1769] a means for receiving user input and converting the input into text data;

[1770] means for analyzing the text data to understand the user's intent;

[1771] means for generating an appropriate response based on the analysis results;

[1772] A means of linking with external services to obtain necessary information and reflect it in responses,

[1773] A means for generating detailed responses based on user intent using a generative AI model; and

[1774] a means for providing a response through a visual interface on the smart device;

[1775] A system including:

[1776] (Claim 2)

[1777] 2. The system according to claim 1, wherein the response generating means further comprises means for analyzing emotions in the text data and generating a response according to the emotions.

[1778] (Claim 3)

[1779] The system according to claim 1, wherein the means for linking with the external service includes means for accessing commercial transactions and inventory status through an API and providing the same to the user.

[1780] "Example 2: Combining Emotion Engines"

[1781] (Claim 1)

[1782] means for receiving a user's voice input and converting the voice into text data;

[1783] means for analyzing the text data to understand the user's intent;

[1784] means for analyzing emotions in the text data and generating a response according to the emotions;

[1785] A means to analyze the generated response, obtain necessary information by linking with external services, and reflect it in the response;

[1786] means for providing said final response to a user;

[1787] A system including:

[1788] (Claim 2)

[1789] 2. The system according to claim 1, wherein the response generating means further comprises means for analyzing text data using a natural language processing engine to understand the user's intent.

[1790] (Claim 3)

[1791] 2. The system according to claim 1, wherein the means for obtaining information in cooperation with the external service includes means for accessing an external reservation service via an API to obtain reservation status and providing it to the user.

[1792] "Application example 2 when combining emotion engines"

[1793] (Claim 1)

[1794] a means for receiving user input and converting the input into text data;

[1795] means for analyzing the text data to understand the user's intent;

[1796] means for generating an appropriate response based on the analysis results;

[1797] A means of linking with external services to obtain necessary information and reflect it in responses,

[1798] means for providing said response to a user;

[1799] A means of obtaining inventory information by linking with each store's inventory system, and

[1800] A means for reflecting the acquired inventory information in a response;

[1801] A system including:

[1802] (Claim 2)

[1803] 2. The system according to claim 1, wherein the response generating means further comprises means for analyzing emotions in the text data and generating a response according to the emotions.

[1804] (Claim 3)

[1805] 2. The system according to claim 1, wherein the means for linking with the external service includes means for accessing a reservation service via an API, obtaining reservation status, and providing it to the user. [Explanation of symbols]

[1806] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for receiving user input and converting the input into text data; means for analyzing the text data to understand the user's intent; means for generating an appropriate response based on the analysis results; A means of linking with external services to obtain necessary information and reflect it in responses, means for providing said response to a user; A system including:

2. 2. The system according to claim 1, wherein the response generating means further comprises means for analyzing emotions in the text data and generating a response according to the emotions.

3. The system according to claim 1 , wherein the means for linking with the external service includes means for accessing a reservation service through an API, obtaining reservation status, and providing it to the user.

Images