system

Abstract

We provide the system. [Solution] A terminal means for receiving user requests, A server means for analyzing the aforementioned request and retrieving related information, A transmission means for generating the retrieved information in natural language and transmitting it to the terminal means, A system that includes this.

Description

Technical Field

[0001] The technology of this disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern information society, a system that can quickly and accurately answer various questions and requests of users has become an essential service. However, existing digital assistant systems have problems in efficiency and accuracy in context understanding of requests, extraction of relevant information, and generation of appropriate answers. Therefore, it is necessary to solve the problem that it is difficult for users to obtain quick and appropriate information.

Means for Solving the Problems

[0005] This invention provides a system comprising a terminal means for receiving user requests and a server means that uses a natural language processing engine for request analysis. Specifically, the server means searches for information from multiple databases based on the request and generates a response that is relevant to the user's context. Furthermore, by generating this information in natural language and transmitting it to the terminal means, it is possible to provide a response that is easy for the user to understand and highly relevant. This method makes it possible to efficiently and accurately satisfy user requests.

[0006] "Terminal means" refers to a device or interface used to receive requests from users and transmit them to a server.

[0007] "Server-side means" refers to the entire system used to analyze user requests, search and extract necessary information, and generate appropriate responses.

[0008] "Requirements analysis" refers to the process of understanding the requests received from users and clarifying their intentions.

[0009] A "natural language processing engine" refers to algorithms and tools that enable computers to understand and analyze human language.

[0010] "Search methods" refer to mechanisms and processes for finding relevant information from databases and other sources.

[0011] "Filtering methods" refer to the process of selecting necessary information from retrieved data and eliminating unnecessary information.

[0012] "Transmission method" refers to the means of sending responses generated on the server to a terminal and providing information to the user. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2]It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

MODE FOR CARRYING OUT THE INVENTION

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention relates to a digital assistant system designed to enable users to quickly obtain various types of information. The system implements a process of receiving user questions and requests, and generating and providing appropriate answers. The terminal in the system is responsible for receiving user input, and information can be transmitted via voice or text using a microphone or keyboard.

[0035] The terminal transmits the received information to the server via the internet or other communication methods. The server uses a natural language processing engine to analyze the incoming user information. This analysis process understands the user's request in context and identifies relevant keywords and phrases.

[0036] After analysis, the server searches multiple databases for the necessary information. This includes FAQ databases, product and service information, and operating and procedural instructions. The server filters the information to extract what is relevant to the user's request and selects the most appropriate information for the user's request.

[0037] The server then generates a response in natural language based on the information it has gathered. The generated response is sent to the terminal in text or audio format. The terminal then presents this information to the user, providing them with the knowledge and support they need immediately.

[0038] As a concrete example, suppose a user asks the device, "Please tell me the specifications of the latest product." The device sends the question to the server, which uses natural language processing to analyze the keywords "latest product" and "specifications." The server searches the relevant database for the specifications of the latest product and constructs an appropriate answer. Finally, the device presents the user with an answer such as, "The specifications of the latest product ABC are as follows..." This process makes it possible to respond to requests quickly and appropriately.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The user inputs questions or requests via voice or text through the terminal. The terminal converts this input into digital data. This data is then prepared to be sent to the server for analysis.

[0042] Step 2:

[0043] The terminal sends user input data to the server. Care is taken to minimize delays and data loss during this data transfer using a communication protocol.

[0044] Step 3:

[0045] The server passes the received data to a natural language processing (NLP) engine. The server tokenizes the input data and performs morphological analysis to understand the structure and meaning of the request. In this process, it extracts information necessary to accurately grasp the user's intent.

[0046] Step 4:

[0047] Based on the analysis results, the server searches relevant databases. These databases include product specifications, service procedure information, FAQs, and more. The server then applies a filtering algorithm to extract the most relevant information.

[0048] Step 5:

[0049] The server constructs the optimal response based on the extracted information. Using natural language generation (NLG) technology, the information is rearranged in a user-friendly format. Links to subsequent actions and recommendations are also incorporated as needed.

[0050] Step 6:

[0051] The server sends the generated response to the terminal. The response is provided in audio format via a speech synthesis tool or in text format via screen display.

[0052] Step 7:

[0053] The device displays the response sent from the server to the user. Based on the feedback from the device, the user confirms the necessary information and decides on their next action.

[0054] (Example 1)

[0055] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0056] Conventional information retrieval systems struggle to properly understand user requests and provide relevant information quickly and accurately. As a result, users may have to spend a considerable amount of time and effort to obtain the information they need, thus compromising convenience.

[0057] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0058] In this invention, the server includes information receiving means for receiving information as audio or text, analysis means for analyzing the received information using natural language processing technology and extracting keywords based on context, and search means for searching for target information from multiple data storage systems based on the analyzed information. This makes it possible to provide optimal information based on user requests quickly and accurately.

[0059] "Information receiving means" refers to a mechanism that plays a role in obtaining user requests in voice or text format.

[0060] "Analysis means" refers to a device or program that has the function of analyzing received information using natural language processing technology and identifying keywords based on context.

[0061] A "search method" is a mechanism that performs a process of tracking relevant information from multiple data storage systems based on keywords identified by an analysis method.

[0062] "Natural language processing technology" refers to the technology that enables computers to understand, interpret, and generate human language, and is used to grasp the meaning of text and voice input.

[0063] A "data storage system" is a system for efficiently managing and storing information, and includes databases and cloud storage that store data in various formats.

[0064] This invention begins with a terminal that allows the user to provide information in voice or text format. The terminal is responsible for processing the input data and transmitting it to a server. Here, the terminal can use speech recognition software to convert voice input into text. General speech recognition technology is used for this conversion.

[0065] The server analyzes the received information using natural language processing (NLP) techniques. This analysis utilizes NLP libraries available in programming languages ​​such as Python. This process involves contextual analysis and keyword extraction. The server then uses the analysis results to access the data storage system and search for the relevant information. SQL databases and cloud storage are used as the data storage system.

[0066] The server generates answers using a natural language generation AI model based on the information obtained from the search. The generated answers are formatted in a human-readable style and sent to the device in text or audio format. The device then presents this information to the user either on the screen or via audio.

[0067] As a concrete example, consider a scenario where a user requests the device to "tell me about the camera performance of the latest smartphones." The device sends this request to a server, which uses natural language processing technology to analyze the keywords "smartphone" and "camera performance." The server searches its data storage system for relevant information and extracts detailed specifications and comparison information. Finally, the device provides the user with details such as, "The camera performance of the latest smartphone ABC is 12 megapixels..."

[0068] A concrete example of a prompt message would be something like, "Please analyze the details of my smartphone and provide me with the relevant information." This system allows users to easily obtain the information they want.

[0069] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0070] Step 1:

[0071] The user enters information.

[0072] Users input questions and requests via voice or text through their device. For example, they might use the microphone to say, "Tell me about the camera performance of the latest smartphones." The input is then saved as digital data on the device.

[0073] Step 2:

[0074] Processing and transmitting input data.

[0075] The device uses speech recognition technology to convert voice input into text. After the voice data is converted into text data, it is sent to a server via the internet. The input data is sent in a state prepared for further detailed analysis on the server.

[0076] Step 3:

[0077] Data analysis.

[0078] The server analyzes the received text data using natural language processing technology. Specifically, the server grasps the context and extracts keywords such as "smartphone" and "camera performance." This helps understand the essence of the user's request and lays the foundation for searching for relevant information.

[0079] Step 4:

[0080] Information retrieval.

[0081] The server searches for relevant information from the data storage system based on the extracted keywords. It executes SQL queries against databases and cloud storage to retrieve the corresponding product information and specifications. As a result of this process, the relevant information is imported into the server as search results.

[0082] Step 5:

[0083] Generating the answer.

[0084] Based on the acquired information, the server uses a natural language generation AI model to generate easy-to-understand responses. In this process, the information is organized in a way that is easy for the user to understand and structured into a human-readable document.

[0085] Step 6:

[0086] Providing the answer.

[0087] The generated response is sent from the server to the terminal. The terminal either displays the information on its screen or uses speech synthesis technology to explain it to the user verbally. As a result, the user can obtain the necessary information from the terminal and receive a satisfactory response.

[0088] (Application Example 1)

[0089] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0090] Users have difficulty obtaining information quickly and efficiently, particularly regarding electronic payments, where it is time-consuming to access campaign information, discounts, and transaction history. Providing this information quickly is necessary to improve the user experience and enhance convenience.

[0091] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0092] In this invention, the server includes communication equipment means for receiving user requests, computing means for analyzing the requests and retrieving relevant data, and transmitting device means for generating the retrieved data in natural language and transmitting it to the communication equipment means. This makes it possible for users to request information in natural language and quickly obtain relevant information.

[0093] "Communication equipment means" refers to a device that receives requests from users, transmits that information to a server, and returns the received information to the user.

[0094] A "computer means" is a device that has the function of analyzing user requests and identifying and specifying related data.

[0095] A "transmission device means" is a device for transmitting information generated in natural language on the server to the user's communication device.

[0096] "Voice recognition means" refers to a function that performs the process of converting the user's voice input into text data.

[0097] "Natural language processing tools" are functions that extract important keywords from text and understand the intent and context of a request.

[0098] A "natural language processing program" is a software technology that handles the process of analyzing user requests and generating specific information.

[0099] An "extraction method" is a function that filters necessary data from multiple information sources and selects the data most relevant to the user's request.

[0100] As an embodiment of this invention, the digital assistant system is configured as follows: First, the user inputs information by voice or text using a smartphone or similar communication device. The communication device receives the input request and transmits the data to a server. The server converts the voice into text using speech recognition means and further extracts important keywords from the text using natural language processing means.

[0101] The server performs a search based on extracted keywords using computing means. During this process, data is filtered from multiple databases using extraction means to select the information most relevant to the user's request. The selected information is then generated in a user-friendly format by a natural language processing program and returned to the user's communication device via a transmission device.

[0102] For example, if a user asks by voice, "Please tell me about this month's credit card benefits," the server will respond to the user's request by generating and sending information such as, "This month's credit card benefits include a 20% discount at XYZ stores." An example of a prompt would be, "Please tell me about this month's credit card benefits." In this way, users can quickly obtain information that answers their questions and requests.

[0103] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0104] Step 1:

[0105] The terminal receives information from the user via voice or text data. The input voice data is converted into text data using speech recognition technology on the terminal. This prepares the device for converting the voice into text format.

[0106] Step 2:

[0107] The terminal sends the converted text data to the server. The server analyzes this text data using natural language processing tools and extracts key keywords. In this process, the intent of the prompt sentence is clarified.

[0108] Step 3:

[0109] The server uses computing power to search for relevant information based on the extracted keywords. Based on the input keywords, it selects the necessary data from multiple databases and filters that information to obtain the most suitable information for the user's request.

[0110] Step 4:

[0111] The server uses a natural language processing program to generate information selected by the computer into a user-friendly natural language format. During this process, the information is formatted as text data.

[0112] Step 5:

[0113] The server sends the generated response data back to the terminal using a transmission device. The terminal presents the received response to the user and, if necessary, can convert it back into audio and play it. This allows the user to receive information that appropriately addresses their request.

[0114] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0115] This invention provides a digital assistant system that can analyze user requests and recognize the emotions associated with those requests. The terminal device is responsible for receiving user input and transmitting it to the server. This input is collected in either voice or text format. In the case of voice input, factors such as voice tone and speed are also analyzed.

[0116] The server uses a natural language processing engine and an emotion engine to analyze the received data and identify the user's emotions along with their requests. The natural language processing engine clarifies the intent of the request, and the emotion engine recognizes the user's emotions based on the linguistic expressions and vocal characteristics used.

[0117] The server searches for relevant information based on the analyzed data. Here, it's crucial to extract necessary information from multiple databases. It's also possible to adjust the presentation method and content of information according to the user's emotional state, as recognized by the emotion engine. For example, if the user expresses dissatisfaction, the response will be generated in a way that emphasizes more detailed information and solutions.

[0118] The generated response is sent to the device using natural language generation technology. This response is presented in a tone that matches the user's emotional state. The device then delivers the response either as text on the screen or as audio using a speech synthesis engine.

[0119] As a concrete example, suppose a user asks the terminal, "Please tell me about recent malfunctions in electrical appliances." The server's emotion engine detects that this question is expressed with some anxiety or confusion. Based on this information, the server uses a problem-solving approach to generate and provide the user with an answer that emphasizes detailed procedures and contact information for support. In this way, it becomes possible to provide information in a manner that is appropriate to the user's emotions.

[0120] The following describes the processing flow.

[0121] Step 1:

[0122] The user inputs questions or requests via voice or text through the terminal. The terminal receives this input as digital data and prepares to send it to the server.

[0123] Step 2:

[0124] The terminal sends user input data to the server. This communication uses a highly reliable protocol to maintain data accuracy.

[0125] Step 3:

[0126] The server passes the received data to a natural language processing (NLP) engine. The server uses this engine to analyze the intent of the request and determine what the user wants.

[0127] Step 4:

[0128] The server uses an emotion engine to analyze the user's emotional state from their input. Specifically, it analyzes linguistic expressions and speech features (in the case of speech) to identify emotions.

[0129] Step 5:

[0130] The server searches multiple databases to retrieve relevant information based on the analysis results. This information not only satisfies the user's requests but is also tailored to their emotions.

[0131] Step 6:

[0132] The server constructs an appropriate response based on the acquired information. It adjusts the tone and content of the response to match the user's emotions, taking into account the results of the emotion engine's analysis.

[0133] Step 7:

[0134] The server sends the generated response to the device. The device then presents the response to the user using text display or speech synthesis. This allows the user to receive the necessary information in a way that resonates with their emotions.

[0135] (Example 2)

[0136] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0137] Traditional digital assistant systems can provide information in response to user requests, but they struggle to respond flexibly while considering the user's emotional state. As a result, the nuances and methods of providing the information the user desires are not optimized according to their emotions, limiting the improvement of the user experience.

[0138] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0139] In this invention, the server includes means for analyzing requests using a natural language processing engine and an emotion engine, means for retrieving information from multiple information sources based on the analysis results, and means for generating information in a manner that corresponds to the user's emotions. This makes it possible to provide information tailored to the user's emotional state.

[0140] A "terminal device" is a device that receives requests and emotional states from a user, and has an interface function for collecting information in voice or text format and transmitting it to a server.

[0141] A "server system" is a system that analyzes the received user's requests and emotional state and provides computing resources to generate appropriate information, incorporating a natural language processing engine and an emotion engine.

[0142] A "natural language processing engine" is a software component that analyzes natural language contained in user requests and understands their intent.

[0143] An "emotion engine" is a software component that evaluates the text and audio characteristics contained in the received data and has the function of recognizing the user's emotional state.

[0144] A "generation method" is a mechanism that generates and provides information in a tone and content appropriate to the user's emotions based on the analysis results, and utilizes natural language generation technology.

[0145] A "transmission means" is a system that has a communication function for sending the generated information back to the terminal means in an appropriate format.

[0146] This invention relates to a digital assistant system that analyzes a user's requests and emotional state and provides information based on them. The system consists of terminal means, server means, and communication means for linking them together.

[0147] The terminal receives voice or text input from the user. At this stage, interface devices such as a microphone or keyboard are used. In the case of voice input, a speech recognition engine is used to convert the voice data into text. During this process, voice characteristics such as tone and speed are also collected.

[0148] The server receives the collected data and performs analysis using a natural language processing engine and an emotion engine. The natural language processing engine analyzes the structure of the language to clarify the intent of the user's request. Meanwhile, the emotion engine recognizes the user's emotional state from text and voice characteristics. This analysis prepares the server to provide appropriate information not only for the user's needs but also for their emotions.

[0149] Based on the analysis results, the server searches multiple databases for the necessary information and selects it. During this process, the tone and content of the information provided are adjusted according to the user's emotional state. Natural language generation technology is used to generate the requested information in a format suitable for the user.

[0150] Finally, the server sends the generated information to the terminal. The terminal either displays the information as text on the screen or transmits it as voice using a speech synthesis engine. This ultimately allows the user to receive a more personalized response to their request.

[0151] As a concrete example, suppose a user asks, "Please tell me about recent malfunctions in electrical appliances." In this case, the server's emotion engine detects that the user is feeling anxious and generates a problem-solving response that emphasizes detailed procedures and information about support contacts, which it then provides to the user.

[0152] An example prompt would be, "A user has a question about a malfunction in an electrical appliance and is expressing anxiety. In this case, how should we generate an answer that emphasizes detailed steps?" The AI ​​model then suggests an answer that reflects the user's emotions.

[0153] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0154] Step 1:

[0155] The device receives voice or text input from the user. Specifically, if the user says to the device, "Tell me about recent malfunctions in my electronic devices," the device uses its microphone to capture voice data. This input is then fed into a speech recognition engine and converted into text data. In the case of voice input, the tone and speed of the voice are also analyzed to provide data for emotion recognition.

[0156] Step 2:

[0157] The terminal sends the converted text data and analysis data, including speech characteristics, to the server. A secure communication protocol is used for transmission, and the data is transferred to the server. The input here is the text data and speech characteristics, and the output is the analysis data received by the server.

[0158] Step 3:

[0159] The server uses the received parsed data to analyze the user's request using a natural language processing engine. The text data is parsed syntactically to clarify its sentence structure and determine the intent of the request. In this step, the input is the parsed data, and the output is a clarified intent of the request.

[0160] Step 4:

[0161] The server uses an emotion engine to analyze the user's emotional state. It recognizes emotions such as confusion or anxiety from vocal characteristics and textual expressions. The input is analysis data including vocal characteristics, and the output is the user's determined emotional state.

[0162] Step 5:

[0163] The server searches for relevant information from multiple sources based on the analysis results. In this process, it filters the necessary information from the database and extracts the information best suited to the user's request and emotional state. The input is the user's intention and emotional state, and the output is the selected information.

[0164] Step 6:

[0165] The server generates information in a format suitable for the user using natural language generation technology. This process determines the tone and content based on the user's emotions, adjusting them if kindness or encouragement is needed. The input here is the selected information, and the output is the generated response.

[0166] Step 7:

[0167] The server sends the generated response to the terminal. The terminal receives this response and presents it to the user in a format suitable for them, either displayed or spoken aloud. A speech synthesis engine is used to deliver the response to the user in a natural voice. Here, the input is the generated response, and the output is the information presented to the user.

[0168] Step 8:

[0169] Users receive information provided through their devices, and their concerns and questions are addressed. This allows users to obtain satisfactory answers to their requests.

[0170] (Application Example 2)

[0171] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0172] In recent years, there has been a growing demand for digital assistants that can provide information more appropriately and quickly in response to user requests. However, conventional systems have struggled to provide information while fully considering the user's emotions, limiting the improvement of user satisfaction. In particular, consumer devices such as home robots require smooth conversation and operation while taking the user's emotions into consideration.

[0173] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0174] In this invention, the server includes a computing means for analyzing the user's requests and emotions and retrieving relevant information; a data processing means for adjusting the method of presenting information according to the user's emotions; and a transmission means for generating the retrieved information in natural language and outputting the information as voice or text using a user interface. This makes it possible to provide appropriate information that takes the user's emotions into consideration.

[0175] "User requests" refer to the instructions, questions, and other forms of communication that users express to the system.

[0176] "Emotion" refers to the psychological or emotional state expressed by a user when making a request.

[0177] A "terminal device" refers to equipment or devices used by users to input requests, and can accept both voice and text input.

[0178] A "processing unit" is a hardware or software system that analyzes user requests and emotions and processes the necessary information.

[0179] "Natural language" refers to the language that humans use in everyday life, and which is used by computers to understand and process.

[0180] A "transmission device" is a device or system used to deliver analyzed information to the user, and the information is expressed in either audio or text.

[0181] A "data processing device" is a device used to adjust the method of presenting information according to the user's emotional state.

[0182] A "knowledge base" is a general term for databases and information sources used to store information.

[0183] A "selection device" is a device or mechanism used to select specific information from among multiple options.

[0184] This invention is a digital assistant system that accurately processes user requests and emotions and provides appropriate information. To implement it, a terminal device, a computing device, a transmitting device, and a data processing device are used in combination. Specifically, it operates in the following manner.

[0185] The terminal device receives user requests and emotions in voice and text format. The received data is sent to the computing unit via the network. The computing unit uses SpaCy as its natural language processing engine to analyze user requests. It also uses Google® Cloud Natural Language API as its emotion engine to analyze user emotions.

[0186] Based on analyzed requests and emotions, the computing unit searches for the necessary information from the knowledge base. By using ElasticSearch® as the database management system, it is possible to quickly retrieve information.

[0187] Furthermore, the data processing device adjusts how information is presented according to the user's emotions. For example, if a user shows anxiety, the information they receive can be presented in a warmer tone.

[0188] The retrieved and refined information is returned to the terminal device via the transmission device. The terminal device has a synthesized speech function and can provide information to the user by voice. It can also display and provide visually improved information to the user on the screen.

[0189] As a concrete example, if a household robot receives the question "What's the weather like today?", its computing unit first analyzes the "anxiety" and "curiosity" contained in the question and then searches for weather information. If it determines that the user is in a hurry, it quickly provides concise and important information and responds via voice through the terminal device, "The weather today is sunny. There is nothing to be careful about."

[0190] An example of a prompt message is, "As a user request, analyze voice messages indicating hunger and provide appropriate recommendations." In this way, the system can provide a customized experience tailored to the user's diverse emotional states.

[0191] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0192] Step 1:

[0193] The terminal device receives requests from the user in voice or text format. This input also includes the user's emotions. The terminal device converts this raw data into a digital signal and transmits it to the processing unit via the network.

[0194] Step 2:

[0195] The computing unit analyzes the received data. First, it uses SpaCy, a natural language processing engine, to interpret the request content in text format. During this process, the input data is parsed to clarify the intent of the user's request.

[0196] Step 3:

[0197] The computing unit uses the Google Cloud Natural Language API to evaluate the user's emotions. In this step, it extracts emotional features from the input text and outputs them as an emotion score. This score is used to adjust how the information is presented in subsequent processing.

[0198] Step 4:

[0199] The computing unit searches for relevant information from the knowledge base based on the analyzed request content and sentiment data. Elasticsearch is used to quickly and efficiently retrieve information that matches the request. The input is the analyzed request content, and the output is a list of the corresponding information.

[0200] Step 5:

[0201] The data processing device optimizes the acquired information based on the user's sentiment score. This process generates information adjusted to a tone that reflects the user's emotions. The input is a list of information and the sentiment score, and the output is the final response.

[0202] Step 6:

[0203] The transmitting device sends optimized information back to the terminal device. The terminal device either conveys the information to the user via voice using synthesized speech or provides text information through a screen display. In doing so, the information is output in the most easily understandable format for the user.

[0204] Step 7:

[0205] The system receives the information provided by the user and, if necessary, inputs additional requests or feedback into the terminal device. This repeats the cycle, resulting in continuous interaction.

[0206] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0207] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0208] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0209] [Second Embodiment]

[0210] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0211] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0212] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0213] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0214] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0215] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0216] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0217] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0218] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0219] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0220] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0221] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0222] This invention relates to a digital assistant system designed to enable users to quickly obtain various types of information. The system implements a process of receiving user questions and requests, and generating and providing appropriate answers. The terminal in the system is responsible for receiving user input, and information can be transmitted via voice or text using a microphone or keyboard.

[0223] The terminal transmits the received information to the server via the internet or other communication methods. The server uses a natural language processing engine to analyze the incoming user information. This analysis process understands the user's request in context and identifies relevant keywords and phrases.

[0224] After analysis, the server searches multiple databases for the necessary information. This includes FAQ databases, product and service information, and operating and procedural instructions. The server filters the information to extract what is relevant to the user's request and selects the most appropriate information for the user's request.

[0225] The server then generates a response in natural language based on the information it has gathered. The generated response is sent to the terminal in text or audio format. The terminal then presents this information to the user, providing them with the knowledge and support they need immediately.

[0226] As a concrete example, suppose a user asks the device, "Please tell me the specifications of the latest product." The device sends the question to the server, which uses natural language processing to analyze the keywords "latest product" and "specifications." The server searches the relevant database for the specifications of the latest product and constructs an appropriate answer. Finally, the device presents the user with an answer such as, "The specifications of the latest product ABC are as follows..." This process makes it possible to respond to requests quickly and appropriately.

[0227] The following describes the processing flow.

[0228] Step 1:

[0229] The user inputs questions or requests via voice or text through the terminal. The terminal converts this input into digital data. This data is then prepared to be sent to the server for analysis.

[0230] Step 2:

[0231] The terminal sends user input data to the server. Care is taken to minimize delays and data loss during this data transfer using a communication protocol.

[0232] Step 3:

[0233] The server passes the received data to a natural language processing (NLP) engine. The server tokenizes the input data and performs morphological analysis to understand the structure and meaning of the request. In this process, it extracts information necessary to accurately grasp the user's intent.

[0234] Step 4:

[0235] Based on the analysis results, the server searches relevant databases. These databases include product specifications, service procedure information, FAQs, and more. The server then applies a filtering algorithm to extract the most relevant information.

[0236] Step 5:

[0237] The server constructs the optimal response based on the extracted information. Using natural language generation (NLG) technology, the information is rearranged in a user-friendly format. Links to subsequent actions and recommendations are also incorporated as needed.

[0238] Step 6:

[0239] The server sends the generated response to the terminal. The response is provided in audio format via a speech synthesis tool or in text format via screen display.

[0240] Step 7:

[0241] The device displays the response sent from the server to the user. Based on the feedback from the device, the user confirms the necessary information and decides on their next action.

[0242] (Example 1)

[0243] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0244] Conventional information retrieval systems struggle to properly understand user requests and provide relevant information quickly and accurately. As a result, users may have to spend a considerable amount of time and effort to obtain the information they need, thus compromising convenience.

[0245] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0246] In this invention, the server includes information receiving means for receiving information as audio or text, analysis means for analyzing the received information using natural language processing technology and extracting keywords based on context, and search means for searching for target information from multiple data storage systems based on the analyzed information. This makes it possible to provide optimal information based on user requests quickly and accurately.

[0247] "Information receiving means" refers to a mechanism that plays a role in obtaining user requests in voice or text format.

[0248] "Analysis means" refers to a device or program that has the function of analyzing received information using natural language processing technology and identifying keywords based on context.

[0249] A "search method" is a mechanism that performs a process of tracking relevant information from multiple data storage systems based on keywords identified by an analysis method.

[0250] "Natural language processing technology" refers to the technology that enables computers to understand, interpret, and generate human language, and is used to grasp the meaning of text and voice input.

[0251] A "data storage system" is a system for efficiently managing and storing information, and includes databases and cloud storage that store data in various formats.

[0252] This invention begins with a terminal that allows the user to provide information in voice or text format. The terminal is responsible for processing the input data and transmitting it to a server. Here, the terminal can use speech recognition software to convert voice input into text. General speech recognition technology is used for this conversion.

[0253] The server analyzes the received information using natural language processing (NLP) techniques. This analysis utilizes NLP libraries available in programming languages ​​such as Python. This process involves contextual analysis and keyword extraction. The server then uses the analysis results to access the data storage system and search for the relevant information. SQL databases and cloud storage are used as the data storage system.

[0254] The server generates answers using a natural language generation AI model based on the information obtained from the search. The generated answers are formatted in a human-readable style and sent to the device in text or audio format. The device then presents this information to the user either on the screen or via audio.

[0255] As a concrete example, consider a scenario where a user requests the device to "tell me about the camera performance of the latest smartphones." The device sends this request to a server, which uses natural language processing technology to analyze the keywords "smartphone" and "camera performance." The server searches its data storage system for relevant information and extracts detailed specifications and comparison information. Finally, the device provides the user with details such as, "The camera performance of the latest smartphone ABC is 12 megapixels..."

[0256] A concrete example of a prompt message would be something like, "Please analyze the details of my smartphone and provide me with the relevant information." This system allows users to easily obtain the information they want.

[0257] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0258] Step 1:

[0259] The user enters information.

[0260] Users input questions and requests via voice or text through their device. For example, they might use the microphone to say, "Tell me about the camera performance of the latest smartphones." The input is then saved as digital data on the device.

[0261] Step 2:

[0262] Processing and transmitting input data.

[0263] The device uses speech recognition technology to convert voice input into text. After the voice data is converted into text data, it is sent to a server via the internet. The input data is sent in a state prepared for further detailed analysis on the server.

[0264] Step 3:

[0265] Data analysis.

[0266] The server analyzes the received text data using natural language processing technology. Specifically, the server grasps the context and extracts keywords such as "smartphone" and "camera performance." This helps understand the essence of the user's request and lays the foundation for searching for relevant information.

[0267] Step 4:

[0268] Information retrieval.

[0269] The server searches for relevant information from the data storage system based on the extracted keywords. It executes SQL queries against databases and cloud storage to retrieve the corresponding product information and specifications. As a result of this process, the relevant information is imported into the server as search results.

[0270] Step 5:

[0271] Generating the answer.

[0272] Based on the acquired information, the server uses a natural language generation AI model to generate easy-to-understand responses. In this process, the information is organized in a way that is easy for the user to understand and structured into a human-readable document.

[0273] Step 6:

[0274] Providing the answer.

[0275] The generated response is sent from the server to the terminal. The terminal either displays the information on its screen or uses speech synthesis technology to explain it to the user verbally. As a result, the user can obtain the necessary information from the terminal and receive a satisfactory response.

[0276] (Application Example 1)

[0277] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0278] Users have difficulty obtaining information quickly and efficiently, particularly regarding electronic payments, where it is time-consuming to access campaign information, discounts, and transaction history. Providing this information quickly is necessary to improve the user experience and enhance convenience.

[0279] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0280] In this invention, the server includes communication device means for receiving a user's request, computer means for analyzing the request and searching for relevant data, and transmission device means for generating the searched data in natural language and transmitting it to the communication device means. Thereby, the user can request information in natural language and quickly obtain relevant information.

[0281] The "communication device means" is a device for receiving a request from a user, transmitting the information to the server, and returning the received information to the user.

[0282] The "computer means" is a device having a function of analyzing a user's request and identifying and specifying relevant data.

[0283] The "transmission device means" is a device for transmitting information in natural language generated by the server to the user's communication device.

[0284] The "voice recognition means" is a function for executing a process of converting a user's voice input into text data.

[0285] The "natural language analysis means" is a function for extracting important keywords from text and understanding the intention and context of a request.

[0286] The "natural language processing program" is a software technology responsible for analyzing a user's request and generating specific information.

[0287] The "extraction means" is a function for filtering necessary data from a plurality of information sources and selecting the data most relevant to a user's request.

[0288] As an embodiment of this invention, the digital assistant system is configured as follows: First, the user inputs information by voice or text using a smartphone or similar communication device. The communication device receives the input request and transmits the data to a server. The server converts the voice into text using speech recognition means and further extracts important keywords from the text using natural language processing means.

[0289] The server performs a search based on extracted keywords using computing means. During this process, data is filtered from multiple databases using extraction means to select the information most relevant to the user's request. The selected information is then generated in a user-friendly format by a natural language processing program and returned to the user's communication device via a transmission device.

[0290] For example, if a user asks by voice, "Please tell me about this month's credit card benefits," the server will respond to the user's request by generating and sending information such as, "This month's credit card benefits include a 20% discount at XYZ stores." An example of a prompt would be, "Please tell me about this month's credit card benefits." In this way, users can quickly obtain information that answers their questions and requests.

[0291] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0292] Step 1:

[0293] The terminal receives information from the user via voice or text data. The input voice data is converted into text data using speech recognition technology on the terminal. This prepares the device for converting the voice into text format.

[0294] Step 2:

[0295] The terminal sends the converted text data to the server. The server analyzes this text data using natural language processing tools and extracts key keywords. In this process, the intent of the prompt sentence is clarified.

[0296] Step 3:

[0297] The server uses computing power to search for relevant information based on the extracted keywords. Based on the input keywords, it selects the necessary data from multiple databases and filters that information to obtain the most suitable information for the user's request.

[0298] Step 4:

[0299] The server uses a natural language processing program to generate information selected by the computer into a user-friendly natural language format. During this process, the information is formatted as text data.

[0300] Step 5:

[0301] The server sends the generated response data back to the terminal using a transmission device. The terminal presents the received response to the user and, if necessary, can convert it back into audio and play it. This allows the user to receive information that appropriately addresses their request.

[0302] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0303] This invention provides a digital assistant system that can analyze user requests and recognize the emotions associated with those requests. The terminal device is responsible for receiving user input and transmitting it to the server. This input is collected in either voice or text format. In the case of voice input, factors such as voice tone and speed are also analyzed.

[0304] The server uses a natural language processing engine and an emotion engine to analyze the received data and identify the emotion along with the user's request. The natural language processing engine clarifies the intention of the request, and the emotion engine recognizes the user's emotion based on the language expressions and voice characteristics used.

[0305] The server conducts a search for relevant information based on the analyzed data. Here, it is important to extract the necessary information from multiple databases. It is also possible to adjust the presentation method and content of the information according to the emotional state of the user recognized by the emotion engine. For example, when the user shows dissatisfaction, an answer is generated in a form that emphasizes more detailed information or solutions.

[0306] The generated answer is sent to the terminal using natural language generation technology. This answer is presented in a tone that suits the user's emotional state. The terminal displays the text on the screen or conveys the answer in voice using a speech synthesis engine.

[0307] As a specific example, suppose the user makes an inquiry to the terminal asking "Please tell me about the problems with recent electrical appliances." The server detects with the emotion engine that this question is expressed with an emotion of somewhat anxiety and confusion. Based on this information, the server generates an answer in a problem-solving approach, emphasizing detailed procedures and information on support channels, and provides it to the user. In this way, it becomes possible to provide information in a way suitable for the user's emotion.

[0308] The following describes the processing flow.

[0309] Step 1:

[0310] The user inputs a question or request in voice or text through the terminal. The terminal receives this input as digital data and prepares to send it to the server.

[0311] Step 2:

[0312] The terminal sends user input data to the server. This communication uses a highly reliable protocol to maintain data accuracy.

[0313] Step 3:

[0314] The server passes the received data to a natural language processing (NLP) engine. The server uses this engine to analyze the intent of the request and determine what the user wants.

[0315] Step 4:

[0316] The server uses an emotion engine to analyze the user's emotional state from their input. Specifically, it analyzes linguistic expressions and speech features (in the case of speech) to identify emotions.

[0317] Step 5:

[0318] The server searches multiple databases to retrieve relevant information based on the analysis results. This information not only satisfies the user's requests but is also tailored to their emotions.

[0319] Step 6:

[0320] The server constructs an appropriate response based on the acquired information. It adjusts the tone and content of the response to match the user's emotions, taking into account the results of the emotion engine's analysis.

[0321] Step 7:

[0322] The server sends the generated response to the device. The device then presents the response to the user using text display or speech synthesis. This allows the user to receive the necessary information in a way that resonates with their emotions.

[0323] (Example 2)

[0324] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0325] Traditional digital assistant systems can provide information in response to user requests, but they struggle to respond flexibly while considering the user's emotional state. As a result, the nuances and methods of providing the information the user desires are not optimized according to their emotions, limiting the improvement of the user experience.

[0326] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0327] In this invention, the server includes means for analyzing requests using a natural language processing engine and an emotion engine, means for retrieving information from multiple information sources based on the analysis results, and means for generating information in a manner that corresponds to the user's emotions. This makes it possible to provide information tailored to the user's emotional state.

[0328] A "terminal device" is a device that receives requests and emotional states from a user, and has an interface function for collecting information in voice or text format and transmitting it to a server.

[0329] A "server system" is a system that analyzes the received user's requests and emotional state and provides computing resources to generate appropriate information, incorporating a natural language processing engine and an emotion engine.

[0330] A "natural language processing engine" is a software component that analyzes natural language contained in user requests and understands their intent.

[0331] An "emotion engine" is a software component that evaluates the text and audio characteristics contained in the received data and has the function of recognizing the user's emotional state.

[0332] A "generation method" is a mechanism that generates and provides information in a tone and content appropriate to the user's emotions based on the analysis results, and utilizes natural language generation technology.

[0333] A "transmission means" is a system that has a communication function for sending the generated information back to the terminal means in an appropriate format.

[0334] This invention relates to a digital assistant system that analyzes a user's requests and emotional state and provides information based on them. The system consists of terminal means, server means, and communication means for linking them together.

[0335] The terminal receives voice or text input from the user. At this stage, interface devices such as a microphone or keyboard are used. In the case of voice input, a speech recognition engine is used to convert the voice data into text. During this process, voice characteristics such as tone and speed are also collected.

[0336] The server receives the collected data and performs analysis using a natural language processing engine and an emotion engine. The natural language processing engine analyzes the structure of the language to clarify the intent of the user's request. Meanwhile, the emotion engine recognizes the user's emotional state from text and voice characteristics. This analysis prepares the server to provide appropriate information not only for the user's needs but also for their emotions.

[0337] Based on the analysis results, the server searches multiple databases for the necessary information and selects it. During this process, the tone and content of the information provided are adjusted according to the user's emotional state. Natural language generation technology is used to generate the requested information in a format suitable for the user.

[0338] Finally, the server sends the generated information to the terminal. The terminal either displays the information as text on the screen or transmits it as voice using a speech synthesis engine. This ultimately allows the user to receive a more personalized response to their request.

[0339] As a concrete example, suppose a user asks, "Please tell me about recent malfunctions in electrical appliances." In this case, the server's emotion engine detects that the user is feeling anxious and generates a problem-solving response that emphasizes detailed procedures and information about support contacts, which it then provides to the user.

[0340] An example prompt would be, "A user has a question about a malfunction in an electrical appliance and is expressing anxiety. In this case, how should we generate an answer that emphasizes detailed steps?" The AI ​​model then suggests an answer that reflects the user's emotions.

[0341] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0342] Step 1:

[0343] The device receives voice or text input from the user. Specifically, if the user says to the device, "Tell me about recent malfunctions in my electronic devices," the device uses its microphone to capture voice data. This input is then fed into a speech recognition engine and converted into text data. In the case of voice input, the tone and speed of the voice are also analyzed to provide data for emotion recognition.

[0344] Step 2:

[0345] The terminal sends the converted text data and analysis data, including speech characteristics, to the server. A secure communication protocol is used for transmission, and the data is transferred to the server. The input here is the text data and speech characteristics, and the output is the analysis data received by the server.

[0346] Step 3:

[0347] The server uses the received parsed data to analyze the user's request using a natural language processing engine. The text data is parsed syntactically to clarify its sentence structure and determine the intent of the request. In this step, the input is the parsed data, and the output is a clarified intent of the request.

[0348] Step 4:

[0349] The server uses an emotion engine to analyze the user's emotional state. It recognizes emotions such as confusion or anxiety from vocal characteristics and textual expressions. The input is analysis data including vocal characteristics, and the output is the user's determined emotional state.

[0350] Step 5:

[0351] The server searches for relevant information from multiple sources based on the analysis results. In this process, it filters the necessary information from the database and extracts the information best suited to the user's request and emotional state. The input is the user's intention and emotional state, and the output is the selected information.

[0352] Step 6:

[0353] The server generates information in a format suitable for the user using natural language generation technology. This process determines the tone and content based on the user's emotions, adjusting them if kindness or encouragement is needed. The input here is the selected information, and the output is the generated response.

[0354] Step 7:

[0355] The server sends the generated response to the terminal. The terminal receives this response and presents it to the user in a format suitable for them, either displayed or spoken aloud. A speech synthesis engine is used to deliver the response to the user in a natural voice. Here, the input is the generated response, and the output is the information presented to the user.

[0356] Step 8:

[0357] Users receive information provided through their devices, and their concerns and questions are addressed. This allows users to obtain satisfactory answers to their requests.

[0358] (Application Example 2)

[0359] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0360] In recent years, there has been a growing demand for digital assistants that can provide information more appropriately and quickly in response to user requests. However, conventional systems have struggled to provide information while fully considering the user's emotions, limiting the improvement of user satisfaction. In particular, consumer devices such as home robots require smooth conversation and operation while taking the user's emotions into consideration.

[0361] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0362] In this invention, the server includes a computing means for analyzing the user's requests and emotions and retrieving relevant information; a data processing means for adjusting the method of presenting information according to the user's emotions; and a transmission means for generating the retrieved information in natural language and outputting the information as voice or text using a user interface. This makes it possible to provide appropriate information that takes the user's emotions into consideration.

[0363] "User requests" refer to the instructions, questions, and other forms of communication that users express to the system.

[0364] "Emotion" refers to the psychological or emotional state expressed by a user when making a request.

[0365] A "terminal device" refers to equipment or devices used by users to input requests, and can accept both voice and text input.

[0366] A "processing unit" is a hardware or software system that analyzes user requests and emotions and processes the necessary information.

[0367] "Natural language" refers to the language that humans use in everyday life, and which is used by computers to understand and process.

[0368] A "transmission device" is a device or system used to deliver analyzed information to the user, and the information is expressed in either audio or text.

[0369] A "data processing device" is a device used to adjust the method of presenting information according to the user's emotional state.

[0370] A "knowledge base" is a general term for databases and information sources used to store information.

[0371] A "selection device" is a device or mechanism used to select specific information from among multiple options.

[0372] This invention is a digital assistant system that accurately processes user requests and emotions and provides appropriate information. To implement it, a terminal device, a computing device, a transmitting device, and a data processing device are used in combination. Specifically, it operates in the following manner.

[0373] The terminal device receives user requests and emotions in voice and text format. The received data is sent to the computing unit via the network. The computing unit uses SpaCy as its natural language processing engine to analyze user requests. It also uses the Google Cloud Natural Language API as its emotion engine to analyze user emotions.

[0374] Based on analyzed requirements and emotions, the computing unit searches the knowledge base for the necessary information. Using Elasticsearch as the database management system allows for rapid information retrieval.

[0375] Furthermore, the data processing device adjusts how information is presented according to the user's emotions. For example, if a user shows anxiety, the information they receive can be presented in a warmer tone.

[0376] The retrieved and refined information is returned to the terminal device via the transmission device. The terminal device has a synthesized speech function and can provide information to the user by voice. It can also display and provide visually improved information to the user on the screen.

[0377] As a concrete example, if a household robot receives the question "What's the weather like today?", its computing unit first analyzes the "anxiety" and "curiosity" contained in the question and then searches for weather information. If it determines that the user is in a hurry, it quickly provides concise and important information and responds via voice through the terminal device, "The weather today is sunny. There is nothing to be careful about."

[0378] An example of a prompt message is, "As a user request, analyze voice messages indicating hunger and provide appropriate recommendations." In this way, the system can provide a customized experience tailored to the user's diverse emotional states.

[0379] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0380] Step 1:

[0381] The terminal device receives requests from the user in voice or text format. This input also includes the user's emotions. The terminal device converts this raw data into a digital signal and transmits it to the processing unit via the network.

[0382] Step 2:

[0383] The computing unit analyzes the received data. First, it uses SpaCy, a natural language processing engine, to interpret the request content in text format. During this process, the input data is parsed to clarify the intent of the user's request.

[0384] Step 3:

[0385] The computing unit uses the Google Cloud Natural Language API to evaluate the user's emotions. In this step, it extracts emotional features from the input text and outputs them as an emotion score. This score is used to adjust how the information is presented in subsequent processing.

[0386] Step 4:

[0387] The computing unit searches for relevant information from the knowledge base based on the analyzed request content and sentiment data. Elasticsearch is used to quickly and efficiently retrieve information that matches the request. The input is the analyzed request content, and the output is a list of the corresponding information.

[0388] Step 5:

[0389] The data processing device optimizes the acquired information based on the user's sentiment score. This process generates information adjusted to a tone that reflects the user's emotions. The input is a list of information and the sentiment score, and the output is the final response.

[0390] Step 6:

[0391] The transmitting device sends optimized information back to the terminal device. The terminal device either conveys the information to the user via voice using synthesized speech or provides text information through a screen display. In doing so, the information is output in the most easily understandable format for the user.

[0392] Step 7:

[0393] The system receives the information provided by the user and, if necessary, inputs additional requests or feedback into the terminal device. This repeats the cycle, resulting in continuous interaction.

[0394] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0395] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0396] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0397] [Third Embodiment]

[0398] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0399] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0400] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0401] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0402] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0403] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0404] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0405] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0406] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0407] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0408] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0409] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0410] This invention relates to a digital assistant system designed to enable users to quickly obtain various types of information. The system implements a process of receiving user questions and requests, and generating and providing appropriate answers. The terminal in the system is responsible for receiving user input, and information can be transmitted via voice or text using a microphone or keyboard.

[0411] The terminal transmits the received information to the server via the internet or other communication methods. The server uses a natural language processing engine to analyze the incoming user information. This analysis process understands the user's request in context and identifies relevant keywords and phrases.

[0412] After analysis, the server searches multiple databases for the necessary information. This includes FAQ databases, product and service information, and operating and procedural instructions. The server filters the information to extract what is relevant to the user's request and selects the most appropriate information for the user's request.

[0413] The server then generates a response in natural language based on the information it has gathered. The generated response is sent to the terminal in text or audio format. The terminal then presents this information to the user, providing them with the knowledge and support they need immediately.

[0414] As a concrete example, suppose a user asks the device, "Please tell me the specifications of the latest product." The device sends the question to the server, which uses natural language processing to analyze the keywords "latest product" and "specifications." The server searches the relevant database for the specifications of the latest product and constructs an appropriate answer. Finally, the device presents the user with an answer such as, "The specifications of the latest product ABC are as follows..." This process makes it possible to respond to requests quickly and appropriately.

[0415] The following describes the processing flow.

[0416] Step 1:

[0417] The user inputs questions or requests via voice or text through the terminal. The terminal converts this input into digital data. This data is then prepared to be sent to the server for analysis.

[0418] Step 2:

[0419] The terminal sends user input data to the server. Care is taken to minimize delays and data loss during this data transfer using a communication protocol.

[0420] Step 3:

[0421] The server passes the received data to a natural language processing (NLP) engine. The server tokenizes the input data and performs morphological analysis to understand the structure and meaning of the request. In this process, it extracts information necessary to accurately grasp the user's intent.

[0422] Step 4:

[0423] Based on the analysis results, the server searches relevant databases. These databases include product specifications, service procedure information, FAQs, and more. The server then applies a filtering algorithm to extract the most relevant information.

[0424] Step 5:

[0425] The server constructs the optimal response based on the extracted information. Using natural language generation (NLG) technology, the information is rearranged in a user-friendly format. Links to subsequent actions and recommendations are also incorporated as needed.

[0426] Step 6:

[0427] The server sends the generated response to the terminal. The response is provided in audio format via a speech synthesis tool or in text format via screen display.

[0428] Step 7:

[0429] The device displays the response sent from the server to the user. Based on the feedback from the device, the user confirms the necessary information and decides on their next action.

[0430] (Example 1)

[0431] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0432] Conventional information retrieval systems struggle to properly understand user requests and provide relevant information quickly and accurately. As a result, users may have to spend a considerable amount of time and effort to obtain the information they need, thus compromising convenience.

[0433] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0434] In this invention, the server includes information receiving means for receiving information as audio or text, analysis means for analyzing the received information using natural language processing technology and extracting keywords based on context, and search means for searching for target information from multiple data storage systems based on the analyzed information. This makes it possible to provide optimal information based on user requests quickly and accurately.

[0435] "Information receiving means" refers to a mechanism that plays a role in obtaining user requests in voice or text format.

[0436] "Analysis means" refers to a device or program that has the function of analyzing received information using natural language processing technology and identifying keywords based on context.

[0437] A "search method" is a mechanism that performs a process of tracking relevant information from multiple data storage systems based on keywords identified by an analysis method.

[0438] "Natural language processing technology" refers to the technology that enables computers to understand, interpret, and generate human language, and is used to grasp the meaning of text and voice input.

[0439] A "data storage system" is a system for efficiently managing and storing information, and includes databases and cloud storage that store data in various formats.

[0440] This invention begins with a terminal that allows the user to provide information in voice or text format. The terminal is responsible for processing the input data and transmitting it to a server. Here, the terminal can use speech recognition software to convert voice input into text. General speech recognition technology is used for this conversion.

[0441] The server analyzes the received information using natural language processing (NLP) techniques. This analysis utilizes NLP libraries available in programming languages ​​such as Python. This process involves contextual analysis and keyword extraction. The server then uses the analysis results to access the data storage system and search for the relevant information. SQL databases and cloud storage are used as the data storage system.

[0442] The server generates answers using a natural language generation AI model based on the information obtained from the search. The generated answers are formatted in a human-readable style and sent to the device in text or audio format. The device then presents this information to the user either on the screen or via audio.

[0443] As a concrete example, consider a scenario where a user requests the device to "tell me about the camera performance of the latest smartphones." The device sends this request to a server, which uses natural language processing technology to analyze the keywords "smartphone" and "camera performance." The server searches its data storage system for relevant information and extracts detailed specifications and comparison information. Finally, the device provides the user with details such as, "The camera performance of the latest smartphone ABC is 12 megapixels..."

[0444] A concrete example of a prompt message would be something like, "Please analyze the details of my smartphone and provide me with the relevant information." This system allows users to easily obtain the information they want.

[0445] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0446] Step 1:

[0447] The user enters information.

[0448] Users input questions and requests via voice or text through their device. For example, they might use the microphone to say, "Tell me about the camera performance of the latest smartphones." The input is then saved as digital data on the device.

[0449] Step 2:

[0450] Processing and transmitting input data.

[0451] The device uses speech recognition technology to convert voice input into text. After the voice data is converted into text data, it is sent to a server via the internet. The input data is sent in a state prepared for further detailed analysis on the server.

[0452] Step 3:

[0453] Data analysis.

[0454] The server analyzes the received text data using natural language processing technology. Specifically, the server grasps the context and extracts keywords such as "smartphone" and "camera performance." This helps understand the essence of the user's request and lays the foundation for searching for relevant information.

[0455] Step 4:

[0456] Information retrieval.

[0457] The server searches for relevant information from the data storage system based on the extracted keywords. It executes SQL queries against databases and cloud storage to retrieve the corresponding product information and specifications. As a result of this process, the relevant information is imported into the server as search results.

[0458] Step 5:

[0459] Generating the answer.

[0460] Based on the acquired information, the server uses a natural language generation AI model to generate easy-to-understand responses. In this process, the information is organized in a way that is easy for the user to understand and structured into a human-readable document.

[0461] Step 6:

[0462] Providing the answer.

[0463] The generated response is sent from the server to the terminal. The terminal either displays the information on its screen or uses speech synthesis technology to explain it to the user verbally. As a result, the user can obtain the necessary information from the terminal and receive a satisfactory response.

[0464] (Application Example 1)

[0465] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0466] Users have difficulty obtaining information quickly and efficiently, particularly regarding electronic payments, where it is time-consuming to access campaign information, discounts, and transaction history. Providing this information quickly is necessary to improve the user experience and enhance convenience.

[0467] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0468] In this invention, the server includes communication equipment means for receiving user requests, computing means for analyzing the requests and retrieving relevant data, and transmitting device means for generating the retrieved data in natural language and transmitting it to the communication equipment means. This makes it possible for users to request information in natural language and quickly obtain relevant information.

[0469] "Communication equipment means" refers to a device that receives requests from users, transmits that information to a server, and returns the received information to the user.

[0470] A "computer means" is a device that has the function of analyzing user requests and identifying and specifying related data.

[0471] A "transmission device means" is a device for transmitting information generated in natural language on the server to the user's communication device.

[0472] "Voice recognition means" refers to a function that performs the process of converting the user's voice input into text data.

[0473] "Natural language processing tools" are functions that extract important keywords from text and understand the intent and context of a request.

[0474] A "natural language processing program" is a software technology that handles the process of analyzing user requests and generating specific information.

[0475] An "extraction method" is a function that filters necessary data from multiple information sources and selects the data most relevant to the user's request.

[0476] As an embodiment of this invention, the digital assistant system is configured as follows: First, the user inputs information by voice or text using a smartphone or similar communication device. The communication device receives the input request and transmits the data to a server. The server converts the voice into text using speech recognition means and further extracts important keywords from the text using natural language processing means.

[0477] The server performs a search based on extracted keywords using computing means. During this process, data is filtered from multiple databases using extraction means to select the information most relevant to the user's request. The selected information is then generated in a user-friendly format by a natural language processing program and returned to the user's communication device via a transmission device.

[0478] For example, if a user asks by voice, "Please tell me about this month's credit card benefits," the server will respond to the user's request by generating and sending information such as, "This month's credit card benefits include a 20% discount at XYZ stores." An example of a prompt would be, "Please tell me about this month's credit card benefits." In this way, users can quickly obtain information that answers their questions and requests.

[0479] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0480] Step 1:

[0481] The terminal receives information from the user via voice or text data. The input voice data is converted into text data using speech recognition technology on the terminal. This prepares the device for converting the voice into text format.

[0482] Step 2:

[0483] The terminal sends the converted text data to the server. The server analyzes this text data using natural language processing tools and extracts key keywords. In this process, the intent of the prompt sentence is clarified.

[0484] Step 3:

[0485] The server uses computing power to search for relevant information based on the extracted keywords. Based on the input keywords, it selects the necessary data from multiple databases and filters that information to obtain the most suitable information for the user's request.

[0486] Step 4:

[0487] The server uses a natural language processing program to generate information selected by the computer into a user-friendly natural language format. During this process, the information is formatted as text data.

[0488] Step 5:

[0489] The server sends the generated response data back to the terminal using a transmission device. The terminal presents the received response to the user and, if necessary, can convert it back into audio and play it. This allows the user to receive information that appropriately addresses their request.

[0490] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0491] This invention provides a digital assistant system that can analyze user requests and recognize the emotions associated with those requests. The terminal device is responsible for receiving user input and transmitting it to the server. This input is collected in either voice or text format. In the case of voice input, factors such as voice tone and speed are also analyzed.

[0492] The server uses a natural language processing engine and an emotion engine to analyze the received data and identify the user's emotions along with their requests. The natural language processing engine clarifies the intent of the request, and the emotion engine recognizes the user's emotions based on the linguistic expressions and vocal characteristics used.

[0493] The server searches for relevant information based on the analyzed data. Here, it's crucial to extract necessary information from multiple databases. It's also possible to adjust the presentation method and content of information according to the user's emotional state, as recognized by the emotion engine. For example, if the user expresses dissatisfaction, the response will be generated in a way that emphasizes more detailed information and solutions.

[0494] The generated response is sent to the device using natural language generation technology. This response is presented in a tone that matches the user's emotional state. The device then delivers the response either as text on the screen or as audio using a speech synthesis engine.

[0495] As a concrete example, suppose a user asks the terminal, "Please tell me about recent malfunctions in electrical appliances." The server's emotion engine detects that this question is expressed with some anxiety or confusion. Based on this information, the server uses a problem-solving approach to generate and provide the user with an answer that emphasizes detailed procedures and contact information for support. In this way, it becomes possible to provide information in a manner that is appropriate to the user's emotions.

[0496] The following describes the processing flow.

[0497] Step 1:

[0498] The user inputs questions or requests via voice or text through the terminal. The terminal receives this input as digital data and prepares to send it to the server.

[0499] Step 2:

[0500] The terminal sends user input data to the server. This communication uses a highly reliable protocol to maintain data accuracy.

[0501] Step 3:

[0502] The server passes the received data to a natural language processing (NLP) engine. The server uses this engine to analyze the intent of the request and determine what the user wants.

[0503] Step 4:

[0504] The server uses an emotion engine to analyze the user's emotional state from their input. Specifically, it analyzes linguistic expressions and speech features (in the case of speech) to identify emotions.

[0505] Step 5:

[0506] The server searches multiple databases to retrieve relevant information based on the analysis results. This information not only satisfies the user's requests but is also tailored to their emotions.

[0507] Step 6:

[0508] The server constructs an appropriate response based on the acquired information. It adjusts the tone and content of the response to match the user's emotions, taking into account the results of the emotion engine's analysis.

[0509] Step 7:

[0510] The server sends the generated response to the device. The device then presents the response to the user using text display or speech synthesis. This allows the user to receive the necessary information in a way that resonates with their emotions.

[0511] (Example 2)

[0512] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0513] Traditional digital assistant systems can provide information in response to user requests, but they struggle to respond flexibly while considering the user's emotional state. As a result, the nuances and methods of providing the information the user desires are not optimized according to their emotions, limiting the improvement of the user experience.

[0514] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0515] In this invention, the server includes means for analyzing requests using a natural language processing engine and an emotion engine, means for retrieving information from multiple information sources based on the analysis results, and means for generating information in a manner that corresponds to the user's emotions. This makes it possible to provide information tailored to the user's emotional state.

[0516] A "terminal device" is a device that receives requests and emotional states from a user, and has an interface function for collecting information in voice or text format and transmitting it to a server.

[0517] A "server system" is a system that analyzes the received user's requests and emotional state and provides computing resources to generate appropriate information, incorporating a natural language processing engine and an emotion engine.

[0518] A "natural language processing engine" is a software component that analyzes natural language contained in user requests and understands their intent.

[0519] An "emotion engine" is a software component that evaluates the text and audio characteristics contained in the received data and has the function of recognizing the user's emotional state.

[0520] A "generation method" is a mechanism that generates and provides information in a tone and content appropriate to the user's emotions based on the analysis results, and utilizes natural language generation technology.

[0521] A "transmission means" is a system that has a communication function for sending the generated information back to the terminal means in an appropriate format.

[0522] This invention relates to a digital assistant system that analyzes a user's requests and emotional state and provides information based on them. The system consists of terminal means, server means, and communication means for linking them together.

[0523] The terminal receives voice or text input from the user. At this stage, interface devices such as a microphone or keyboard are used. In the case of voice input, a speech recognition engine is used to convert the voice data into text. During this process, voice characteristics such as tone and speed are also collected.

[0524] The server receives the collected data and performs analysis using a natural language processing engine and an emotion engine. The natural language processing engine analyzes the structure of the language to clarify the intent of the user's request. Meanwhile, the emotion engine recognizes the user's emotional state from text and voice characteristics. This analysis prepares the server to provide appropriate information not only for the user's needs but also for their emotions.

[0525] Based on the analysis results, the server searches multiple databases for the necessary information and selects it. During this process, the tone and content of the information provided are adjusted according to the user's emotional state. Natural language generation technology is used to generate the requested information in a format suitable for the user.

[0526] Finally, the server sends the generated information to the terminal. The terminal either displays the information as text on the screen or transmits it as voice using a speech synthesis engine. This ultimately allows the user to receive a more personalized response to their request.

[0527] As a concrete example, suppose a user asks, "Please tell me about recent malfunctions in electrical appliances." In this case, the server's emotion engine detects that the user is feeling anxious and generates a problem-solving response that emphasizes detailed procedures and information about support contacts, which it then provides to the user.

[0528] An example prompt would be, "A user has a question about a malfunction in an electrical appliance and is expressing anxiety. In this case, how should we generate an answer that emphasizes detailed steps?" The AI ​​model then suggests an answer that reflects the user's emotions.

[0529] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0530] Step 1:

[0531] The device receives voice or text input from the user. Specifically, if the user says to the device, "Tell me about recent malfunctions in my electronic devices," the device uses its microphone to capture voice data. This input is then fed into a speech recognition engine and converted into text data. In the case of voice input, the tone and speed of the voice are also analyzed to provide data for emotion recognition.

[0532] Step 2:

[0533] The terminal sends the converted text data and analysis data, including speech characteristics, to the server. A secure communication protocol is used for transmission, and the data is transferred to the server. The input here is the text data and speech characteristics, and the output is the analysis data received by the server.

[0534] Step 3:

[0535] The server uses the received parsed data to analyze the user's request using a natural language processing engine. The text data is parsed syntactically to clarify its sentence structure and determine the intent of the request. In this step, the input is the parsed data, and the output is a clarified intent of the request.

[0536] Step 4:

[0537] The server uses an emotion engine to analyze the user's emotional state. It recognizes emotions such as confusion or anxiety from vocal characteristics and textual expressions. The input is analysis data including vocal characteristics, and the output is the user's determined emotional state.

[0538] Step 5:

[0539] The server searches for relevant information from multiple sources based on the analysis results. In this process, it filters the necessary information from the database and extracts the information best suited to the user's request and emotional state. The input is the user's intention and emotional state, and the output is the selected information.

[0540] Step 6:

[0541] The server generates information in a format suitable for the user using natural language generation technology. This process determines the tone and content based on the user's emotions, adjusting them if kindness or encouragement is needed. The input here is the selected information, and the output is the generated response.

[0542] Step 7:

[0543] The server sends the generated response to the terminal. The terminal receives this response and presents it to the user in a format suitable for them, either displayed or spoken aloud. A speech synthesis engine is used to deliver the response to the user in a natural voice. Here, the input is the generated response, and the output is the information presented to the user.

[0544] Step 8:

[0545] Users receive information provided through their devices, and their concerns and questions are addressed. This allows users to obtain satisfactory answers to their requests.

[0546] (Application Example 2)

[0547] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0548] In recent years, there has been a growing demand for digital assistants that can provide information more appropriately and quickly in response to user requests. However, conventional systems have struggled to provide information while fully considering the user's emotions, limiting the improvement of user satisfaction. In particular, consumer devices such as home robots require smooth conversation and operation while taking the user's emotions into consideration.

[0549] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0550] In this invention, the server includes a computing means for analyzing the user's requests and emotions and retrieving relevant information; a data processing means for adjusting the method of presenting information according to the user's emotions; and a transmission means for generating the retrieved information in natural language and outputting the information as voice or text using a user interface. This makes it possible to provide appropriate information that takes the user's emotions into consideration.

[0551] "User requests" refer to the instructions, questions, and other forms of communication that users express to the system.

[0552] "Emotion" refers to the psychological or emotional state expressed by a user when making a request.

[0553] A "terminal device" refers to equipment or devices used by users to input requests, and can accept both voice and text input.

[0554] A "processing unit" is a hardware or software system that analyzes user requests and emotions and processes the necessary information.

[0555] "Natural language" refers to the language that humans use in everyday life, and which is used by computers to understand and process.

[0556] A "transmission device" is a device or system used to deliver analyzed information to the user, and the information is expressed in either audio or text.

[0557] A "data processing device" is a device used to adjust the method of presenting information according to the user's emotional state.

[0558] A "knowledge base" is a general term for databases and information sources used to store information.

[0559] A "selection device" is a device or mechanism used to select specific information from among multiple options.

[0560] This invention is a digital assistant system that accurately processes user requests and emotions and provides appropriate information. To implement it, a terminal device, a computing device, a transmitting device, and a data processing device are used in combination. Specifically, it operates in the following manner.

[0561] The terminal device receives user requests and emotions in voice and text format. The received data is sent to the computing unit via the network. The computing unit uses SpaCy as its natural language processing engine to analyze user requests. It also uses the Google Cloud Natural Language API as its emotion engine to analyze user emotions.

[0562] Based on analyzed requirements and emotions, the computing unit searches the knowledge base for the necessary information. Using Elasticsearch as the database management system allows for rapid information retrieval.

[0563] Furthermore, the data processing device adjusts how information is presented according to the user's emotions. For example, if a user shows anxiety, the information they receive can be presented in a warmer tone.

[0564] The retrieved and refined information is returned to the terminal device via the transmission device. The terminal device has a synthesized speech function and can provide information to the user by voice. It can also display and provide visually improved information to the user on the screen.

[0565] As a concrete example, if a household robot receives the question "What's the weather like today?", its computing unit first analyzes the "anxiety" and "curiosity" contained in the question and then searches for weather information. If it determines that the user is in a hurry, it quickly provides concise and important information and responds via voice through the terminal device, "The weather today is sunny. There is nothing to be careful about."

[0566] An example of a prompt message is, "As a user request, analyze voice messages indicating hunger and provide appropriate recommendations." In this way, the system can provide a customized experience tailored to the user's diverse emotional states.

[0567] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0568] Step 1:

[0569] The terminal device receives requests from the user in voice or text format. This input also includes the user's emotions. The terminal device converts this raw data into a digital signal and transmits it to the processing unit via the network.

[0570] Step 2:

[0571] The computing unit analyzes the received data. First, it uses SpaCy, a natural language processing engine, to interpret the request content in text format. During this process, the input data is parsed to clarify the intent of the user's request.

[0572] Step 3:

[0573] The computing unit uses the Google Cloud Natural Language API to evaluate the user's emotions. In this step, it extracts emotional features from the input text and outputs them as an emotion score. This score is used to adjust how the information is presented in subsequent processing.

[0574] Step 4:

[0575] The computing unit searches for relevant information from the knowledge base based on the analyzed request content and sentiment data. Elasticsearch is used to quickly and efficiently retrieve information that matches the request. The input is the analyzed request content, and the output is a list of the corresponding information.

[0576] Step 5:

[0577] The data processing device optimizes the acquired information based on the user's sentiment score. This process generates information adjusted to a tone that reflects the user's emotions. The input is a list of information and the sentiment score, and the output is the final response.

[0578] Step 6:

[0579] The transmitting device sends optimized information back to the terminal device. The terminal device either conveys the information to the user via voice using synthesized speech or provides text information through a screen display. In doing so, the information is output in the most easily understandable format for the user.

[0580] Step 7:

[0581] The system receives the information provided by the user and, if necessary, inputs additional requests or feedback into the terminal device. This repeats the cycle, resulting in continuous interaction.

[0582] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0583] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0584] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0585] [Fourth Embodiment]

[0586] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0587] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0588] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0589] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0590] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0591] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0592] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0593] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0594] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0595] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0596] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0597] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0598] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0599] This invention relates to a digital assistant system designed to enable users to quickly obtain various types of information. The system implements a process of receiving user questions and requests, and generating and providing appropriate answers. The terminal in the system is responsible for receiving user input, and information can be transmitted via voice or text using a microphone or keyboard.

[0600] The terminal transmits the received information to the server via the internet or other communication methods. The server uses a natural language processing engine to analyze the incoming user information. This analysis process understands the user's request in context and identifies relevant keywords and phrases.

[0601] After analysis, the server searches multiple databases for the necessary information. This includes FAQ databases, product and service information, and operating and procedural instructions. The server filters the information to extract what is relevant to the user's request and selects the most appropriate information for the user's request.

[0602] The server then generates a response in natural language based on the information it has gathered. The generated response is sent to the terminal in text or audio format. The terminal then presents this information to the user, providing them with the knowledge and support they need immediately.

[0603] As a concrete example, suppose a user asks the device, "Please tell me the specifications of the latest product." The device sends the question to the server, which uses natural language processing to analyze the keywords "latest product" and "specifications." The server searches the relevant database for the specifications of the latest product and constructs an appropriate answer. Finally, the device presents the user with an answer such as, "The specifications of the latest product ABC are as follows..." This process makes it possible to respond to requests quickly and appropriately.

[0604] The following describes the processing flow.

[0605] Step 1:

[0606] The user inputs questions or requests via voice or text through the terminal. The terminal converts this input into digital data. This data is then prepared to be sent to the server for analysis.

[0607] Step 2:

[0608] The terminal sends user input data to the server. Care is taken to minimize delays and data loss during this data transfer using a communication protocol.

[0609] Step 3:

[0610] The server passes the received data to a natural language processing (NLP) engine. The server tokenizes the input data and performs morphological analysis to understand the structure and meaning of the request. In this process, it extracts information necessary to accurately grasp the user's intent.

[0611] Step 4:

[0612] Based on the analysis results, the server searches relevant databases. These databases include product specifications, service procedure information, FAQs, and more. The server then applies a filtering algorithm to extract the most relevant information.

[0613] Step 5:

[0614] The server constructs the optimal response based on the extracted information. Using natural language generation (NLG) technology, the information is rearranged in a user-friendly format. Links to subsequent actions and recommendations are also incorporated as needed.

[0615] Step 6:

[0616] The server sends the generated response to the terminal. The response is provided in audio format via a speech synthesis tool or in text format via screen display.

[0617] Step 7:

[0618] The device displays the response sent from the server to the user. Based on the feedback from the device, the user confirms the necessary information and decides on their next action.

[0619] (Example 1)

[0620] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0621] Conventional information retrieval systems struggle to properly understand user requests and provide relevant information quickly and accurately. As a result, users may have to spend a considerable amount of time and effort to obtain the information they need, thus compromising convenience.

[0622] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0623] In this invention, the server includes information receiving means for receiving information as audio or text, analysis means for analyzing the received information using natural language processing technology and extracting keywords based on context, and search means for searching for target information from multiple data storage systems based on the analyzed information. This makes it possible to provide optimal information based on user requests quickly and accurately.

[0624] "Information receiving means" refers to a mechanism that plays a role in obtaining user requests in voice or text format.

[0625] "Analysis means" refers to a device or program that has the function of analyzing received information using natural language processing technology and identifying keywords based on context.

[0626] A "search method" is a mechanism that performs a process of tracking relevant information from multiple data storage systems based on keywords identified by an analysis method.

[0627] "Natural language processing technology" refers to the technology that enables computers to understand, interpret, and generate human language, and is used to grasp the meaning of text and voice input.

[0628] A "data storage system" is a system for efficiently managing and storing information, and includes databases and cloud storage that store data in various formats.

[0629] This invention begins with a terminal that allows the user to provide information in voice or text format. The terminal is responsible for processing the input data and transmitting it to a server. Here, the terminal can use speech recognition software to convert voice input into text. General speech recognition technology is used for this conversion.

[0630] The server analyzes the received information using natural language processing (NLP) techniques. This analysis utilizes NLP libraries available in programming languages ​​such as Python. This process involves contextual analysis and keyword extraction. The server then uses the analysis results to access the data storage system and search for the relevant information. SQL databases and cloud storage are used as the data storage system.

[0631] The server generates answers using a natural language generation AI model based on the information obtained from the search. The generated answers are formatted in a human-readable style and sent to the device in text or audio format. The device then presents this information to the user either on the screen or via audio.

[0632] As a concrete example, consider a scenario where a user requests the device to "tell me about the camera performance of the latest smartphones." The device sends this request to a server, which uses natural language processing technology to analyze the keywords "smartphone" and "camera performance." The server searches its data storage system for relevant information and extracts detailed specifications and comparison information. Finally, the device provides the user with details such as, "The camera performance of the latest smartphone ABC is 12 megapixels..."

[0633] A concrete example of a prompt message would be something like, "Please analyze the details of my smartphone and provide me with the relevant information." This system allows users to easily obtain the information they want.

[0634] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0635] Step 1:

[0636] The user enters information.

[0637] Users input questions and requests via voice or text through their device. For example, they might use the microphone to say, "Tell me about the camera performance of the latest smartphones." The input is then saved as digital data on the device.

[0638] Step 2:

[0639] Processing and transmitting input data.

[0640] The device uses speech recognition technology to convert voice input into text. After the voice data is converted into text data, it is sent to a server via the internet. The input data is sent in a state prepared for further detailed analysis on the server.

[0641] Step 3:

[0642] Data analysis.

[0643] The server analyzes the received text data using natural language processing technology. Specifically, the server grasps the context and extracts keywords such as "smartphone" and "camera performance." This helps understand the essence of the user's request and lays the foundation for searching for relevant information.

[0644] Step 4:

[0645] Information retrieval.

[0646] The server searches for relevant information from the data storage system based on the extracted keywords. It executes SQL queries against databases and cloud storage to retrieve the corresponding product information and specifications. As a result of this process, the relevant information is imported into the server as search results.

[0647] Step 5:

[0648] Generating the answer.

[0649] Based on the acquired information, the server uses a natural language generation AI model to generate easy-to-understand responses. In this process, the information is organized in a way that is easy for the user to understand and structured into a human-readable document.

[0650] Step 6:

[0651] Providing the answer.

[0652] The generated response is sent from the server to the terminal. The terminal either displays the information on its screen or uses speech synthesis technology to explain it to the user verbally. As a result, the user can obtain the necessary information from the terminal and receive a satisfactory response.

[0653] (Application Example 1)

[0654] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0655] Users have difficulty obtaining information quickly and efficiently, particularly regarding electronic payments, where it is time-consuming to access campaign information, discounts, and transaction history. Providing this information quickly is necessary to improve the user experience and enhance convenience.

[0656] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0657] In this invention, the server includes communication equipment means for receiving user requests, computing means for analyzing the requests and retrieving relevant data, and transmitting device means for generating the retrieved data in natural language and transmitting it to the communication equipment means. This makes it possible for users to request information in natural language and quickly obtain relevant information.

[0658] "Communication equipment means" refers to a device that receives requests from users, transmits that information to a server, and returns the received information to the user.

[0659] A "computer means" is a device that has the function of analyzing user requests and identifying and specifying related data.

[0660] A "transmission device means" is a device for transmitting information generated in natural language on the server to the user's communication device.

[0661] "Voice recognition means" refers to a function that performs the process of converting the user's voice input into text data.

[0662] "Natural language processing tools" are functions that extract important keywords from text and understand the intent and context of a request.

[0663] A "natural language processing program" is a software technology that handles the process of analyzing user requests and generating specific information.

[0664] An "extraction method" is a function that filters necessary data from multiple information sources and selects the data most relevant to the user's request.

[0665] As an embodiment of this invention, the digital assistant system is configured as follows: First, the user inputs information by voice or text using a smartphone or similar communication device. The communication device receives the input request and transmits the data to a server. The server converts the voice into text using speech recognition means and further extracts important keywords from the text using natural language processing means.

[0666] The server performs a search based on extracted keywords using computing means. During this process, data is filtered from multiple databases using extraction means to select the information most relevant to the user's request. The selected information is then generated in a user-friendly format by a natural language processing program and returned to the user's communication device via a transmission device.

[0667] For example, if a user asks by voice, "Please tell me about this month's credit card benefits," the server will respond to the user's request by generating and sending information such as, "This month's credit card benefits include a 20% discount at XYZ stores." An example of a prompt would be, "Please tell me about this month's credit card benefits." In this way, users can quickly obtain information that answers their questions and requests.

[0668] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0669] Step 1:

[0670] The terminal receives information from the user via voice or text data. The input voice data is converted into text data using speech recognition technology on the terminal. This prepares the device for converting the voice into text format.

[0671] Step 2:

[0672] The terminal sends the converted text data to the server. The server analyzes this text data using natural language processing tools and extracts key keywords. In this process, the intent of the prompt sentence is clarified.

[0673] Step 3:

[0674] The server uses computing power to search for relevant information based on the extracted keywords. Based on the input keywords, it selects the necessary data from multiple databases and filters that information to obtain the most suitable information for the user's request.

[0675] Step 4:

[0676] The server uses a natural language processing program to generate information selected by the computer into a user-friendly natural language format. During this process, the information is formatted as text data.

[0677] Step 5:

[0678] The server sends the generated response data back to the terminal using a transmission device. The terminal presents the received response to the user and, if necessary, can convert it back into audio and play it. This allows the user to receive information that appropriately addresses their request.

[0679] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0680] This invention provides a digital assistant system that can analyze user requests and recognize the emotions associated with those requests. The terminal device is responsible for receiving user input and transmitting it to the server. This input is collected in either voice or text format. In the case of voice input, factors such as voice tone and speed are also analyzed.

[0681] The server uses a natural language processing engine and an emotion engine to analyze the received data and identify the user's emotions along with their requests. The natural language processing engine clarifies the intent of the request, and the emotion engine recognizes the user's emotions based on the linguistic expressions and vocal characteristics used.

[0682] The server searches for relevant information based on the analyzed data. Here, it's crucial to extract necessary information from multiple databases. It's also possible to adjust the presentation method and content of information according to the user's emotional state, as recognized by the emotion engine. For example, if the user expresses dissatisfaction, the response will be generated in a way that emphasizes more detailed information and solutions.

[0683] The generated response is sent to the device using natural language generation technology. This response is presented in a tone that matches the user's emotional state. The device then delivers the response either as text on the screen or as audio using a speech synthesis engine.

[0684] As a concrete example, suppose a user asks the terminal, "Please tell me about recent malfunctions in electrical appliances." The server's emotion engine detects that this question is expressed with some anxiety or confusion. Based on this information, the server uses a problem-solving approach to generate and provide the user with an answer that emphasizes detailed procedures and contact information for support. In this way, it becomes possible to provide information in a manner that is appropriate to the user's emotions.

[0685] The following describes the processing flow.

[0686] Step 1:

[0687] The user inputs questions or requests via voice or text through the terminal. The terminal receives this input as digital data and prepares to send it to the server.

[0688] Step 2:

[0689] The terminal sends user input data to the server. This communication uses a highly reliable protocol to maintain data accuracy.

[0690] Step 3:

[0691] The server passes the received data to a natural language processing (NLP) engine. The server uses this engine to analyze the intent of the request and determine what the user wants.

[0692] Step 4:

[0693] The server uses an emotion engine to analyze the user's emotional state from their input. Specifically, it analyzes linguistic expressions and speech features (in the case of speech) to identify emotions.

[0694] Step 5:

[0695] The server searches multiple databases to retrieve relevant information based on the analysis results. This information not only satisfies the user's requests but is also tailored to their emotions.

[0696] Step 6:

[0697] The server constructs an appropriate response based on the acquired information. It adjusts the tone and content of the response to match the user's emotions, taking into account the results of the emotion engine's analysis.

[0698] Step 7:

[0699] The server sends the generated response to the device. The device then presents the response to the user using text display or speech synthesis. This allows the user to receive the necessary information in a way that resonates with their emotions.

[0700] (Example 2)

[0701] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0702] Traditional digital assistant systems can provide information in response to user requests, but they struggle to respond flexibly while considering the user's emotional state. As a result, the nuances and methods of providing the information the user desires are not optimized according to their emotions, limiting the improvement of the user experience.

[0703] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0704] In this invention, the server includes means for analyzing requests using a natural language processing engine and an emotion engine, means for retrieving information from multiple information sources based on the analysis results, and means for generating information in a manner that corresponds to the user's emotions. This makes it possible to provide information tailored to the user's emotional state.

[0705] A "terminal device" is a device that receives requests and emotional states from a user, and has an interface function for collecting information in voice or text format and transmitting it to a server.

[0706] A "server system" is a system that analyzes the received user's requests and emotional state and provides computing resources to generate appropriate information, incorporating a natural language processing engine and an emotion engine.

[0707] A "natural language processing engine" is a software component that analyzes natural language contained in user requests and understands their intent.

[0708] An "emotion engine" is a software component that evaluates the text and audio characteristics contained in the received data and has the function of recognizing the user's emotional state.

[0709] A "generation method" is a mechanism that generates and provides information in a tone and content appropriate to the user's emotions based on the analysis results, and utilizes natural language generation technology.

[0710] A "transmission means" is a system that has a communication function for sending the generated information back to the terminal means in an appropriate format.

[0711] This invention relates to a digital assistant system that analyzes a user's requests and emotional state and provides information based on them. The system consists of terminal means, server means, and communication means for linking them together.

[0712] The terminal receives voice or text input from the user. At this stage, interface devices such as a microphone or keyboard are used. In the case of voice input, a speech recognition engine is used to convert the voice data into text. During this process, voice characteristics such as tone and speed are also collected.

[0713] The server receives the collected data and performs analysis using a natural language processing engine and an emotion engine. The natural language processing engine analyzes the structure of the language to clarify the intent of the user's request. Meanwhile, the emotion engine recognizes the user's emotional state from text and voice characteristics. This analysis prepares the server to provide appropriate information not only for the user's needs but also for their emotions.

[0714] Based on the analysis results, the server searches multiple databases for the necessary information and selects it. During this process, the tone and content of the information provided are adjusted according to the user's emotional state. Natural language generation technology is used to generate the requested information in a format suitable for the user.

[0715] Finally, the server sends the generated information to the terminal. The terminal either displays the information as text on the screen or transmits it as voice using a speech synthesis engine. This ultimately allows the user to receive a more personalized response to their request.

[0716] As a concrete example, suppose a user asks, "Please tell me about recent malfunctions in electrical appliances." In this case, the server's emotion engine detects that the user is feeling anxious and generates a problem-solving response that emphasizes detailed procedures and information about support contacts, which it then provides to the user.

[0717] An example prompt would be, "A user has a question about a malfunction in an electrical appliance and is expressing anxiety. In this case, how should we generate an answer that emphasizes detailed steps?" The AI ​​model then suggests an answer that reflects the user's emotions.

[0718] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0719] Step 1:

[0720] The device receives voice or text input from the user. Specifically, if the user says to the device, "Tell me about recent malfunctions in my electronic devices," the device uses its microphone to capture voice data. This input is then fed into a speech recognition engine and converted into text data. In the case of voice input, the tone and speed of the voice are also analyzed to provide data for emotion recognition.

[0721] Step 2:

[0722] The terminal sends the converted text data and analysis data, including speech characteristics, to the server. A secure communication protocol is used for transmission, and the data is transferred to the server. The input here is the text data and speech characteristics, and the output is the analysis data received by the server.

[0723] Step 3:

[0724] The server uses the received parsed data to analyze the user's request using a natural language processing engine. The text data is parsed syntactically to clarify its sentence structure and determine the intent of the request. In this step, the input is the parsed data, and the output is a clarified intent of the request.

[0725] Step 4:

[0726] The server uses an emotion engine to analyze the user's emotional state. It recognizes emotions such as confusion or anxiety from vocal characteristics and textual expressions. The input is analysis data including vocal characteristics, and the output is the user's determined emotional state.

[0727] Step 5:

[0728] The server searches for relevant information from multiple sources based on the analysis results. In this process, it filters the necessary information from the database and extracts the information best suited to the user's request and emotional state. The input is the user's intention and emotional state, and the output is the selected information.

[0729] Step 6:

[0730] The server generates information in a format suitable for the user using natural language generation technology. This process determines the tone and content based on the user's emotions, adjusting them if kindness or encouragement is needed. The input here is the selected information, and the output is the generated response.

[0731] Step 7:

[0732] The server sends the generated response to the terminal. The terminal receives this response and presents it to the user in a format suitable for them, either displayed or spoken aloud. A speech synthesis engine is used to deliver the response to the user in a natural voice. Here, the input is the generated response, and the output is the information presented to the user.

[0733] Step 8:

[0734] Users receive information provided through their devices, and their concerns and questions are addressed. This allows users to obtain satisfactory answers to their requests.

[0735] (Application Example 2)

[0736] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0737] In recent years, there has been a growing demand for digital assistants that can provide information more appropriately and quickly in response to user requests. However, conventional systems have struggled to provide information while fully considering the user's emotions, limiting the improvement of user satisfaction. In particular, consumer devices such as home robots require smooth conversation and operation while taking the user's emotions into consideration.

[0738] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0739] In this invention, the server includes a computing means for analyzing the user's requests and emotions and retrieving relevant information; a data processing means for adjusting the method of presenting information according to the user's emotions; and a transmission means for generating the retrieved information in natural language and outputting the information as voice or text using a user interface. This makes it possible to provide appropriate information that takes the user's emotions into consideration.

[0740] "User requests" refer to the instructions, questions, and other forms of communication that users express to the system.

[0741] "Emotion" refers to the psychological or emotional state expressed by a user when making a request.

[0742] A "terminal device" refers to equipment or devices used by users to input requests, and can accept both voice and text input.

[0743] A "processing unit" is a hardware or software system that analyzes user requests and emotions and processes the necessary information.

[0744] "Natural language" refers to the language that humans use in everyday life, and which is used by computers to understand and process.

[0745] A "transmission device" is a device or system used to deliver analyzed information to the user, and the information is expressed in either audio or text.

[0746] A "data processing device" is a device used to adjust the method of presenting information according to the user's emotional state.

[0747] A "knowledge base" is a general term for databases and information sources used to store information.

[0748] A "selection device" is a device or mechanism used to select specific information from among multiple options.

[0749] This invention is a digital assistant system that accurately processes user requests and emotions and provides appropriate information. To implement it, a terminal device, a computing device, a transmitting device, and a data processing device are used in combination. Specifically, it operates in the following manner.

[0750] The terminal device receives user requests and emotions in voice and text format. The received data is sent to the computing unit via the network. The computing unit uses SpaCy as its natural language processing engine to analyze user requests. It also uses the Google Cloud Natural Language API as its emotion engine to analyze user emotions.

[0751] Based on analyzed requirements and emotions, the computing unit searches the knowledge base for the necessary information. Using Elasticsearch as the database management system allows for rapid information retrieval.

[0752] Furthermore, the data processing device adjusts how information is presented according to the user's emotions. For example, if a user shows anxiety, the information they receive can be presented in a warmer tone.

[0753] The retrieved and refined information is returned to the terminal device via the transmission device. The terminal device has a synthesized speech function and can provide information to the user by voice. It can also display and provide visually improved information to the user on the screen.

[0754] As a concrete example, if a household robot receives the question "What's the weather like today?", its computing unit first analyzes the "anxiety" and "curiosity" contained in the question and then searches for weather information. If it determines that the user is in a hurry, it quickly provides concise and important information and responds via voice through the terminal device, "The weather today is sunny. There is nothing to be careful about."

[0755] An example of a prompt message is, "As a user request, analyze voice messages indicating hunger and provide appropriate recommendations." In this way, the system can provide a customized experience tailored to the user's diverse emotional states.

[0756] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0757] Step 1:

[0758] The terminal device receives requests from the user in voice or text format. This input also includes the user's emotions. The terminal device converts this raw data into a digital signal and transmits it to the processing unit via the network.

[0759] Step 2:

[0760] The computing unit analyzes the received data. First, it uses SpaCy, a natural language processing engine, to interpret the request content in text format. During this process, the input data is parsed to clarify the intent of the user's request.

[0761] Step 3:

[0762] The computing unit uses the Google Cloud Natural Language API to evaluate the user's emotions. In this step, it extracts emotional features from the input text and outputs them as an emotion score. This score is used to adjust how the information is presented in subsequent processing.

[0763] Step 4:

[0764] The computing unit searches for relevant information from the knowledge base based on the analyzed request content and sentiment data. Elasticsearch is used to quickly and efficiently retrieve information that matches the request. The input is the analyzed request content, and the output is a list of the corresponding information.

[0765] Step 5:

[0766] The data processing device optimizes the acquired information based on the user's sentiment score. This process generates information adjusted to a tone that reflects the user's emotions. The input is a list of information and the sentiment score, and the output is the final response.

[0767] Step 6:

[0768] The transmitting device sends optimized information back to the terminal device. The terminal device either conveys the information to the user via voice using synthesized speech or provides text information through a screen display. In doing so, the information is output in the most easily understandable format for the user.

[0769] Step 7:

[0770] The system receives the information provided by the user and, if necessary, inputs additional requests or feedback into the terminal device. This repeats the cycle, resulting in continuous interaction.

[0771] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0772] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0773] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0774] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0775] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0776] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0777] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0778] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0779] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0780] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0781] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0782] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0783] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0784] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0785] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0786] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0787] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0788] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0789] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0790] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0791] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0792] The following is further disclosed regarding the embodiments described above.

[0793] (Claim 1)

[0794] A terminal means for receiving user requests,

[0795] A server means for analyzing the aforementioned request and retrieving related information,

[0796] A transmission means for generating the retrieved information in natural language and transmitting it to the terminal means,

[0797] A system that includes this.

[0798] (Claim 2)

[0799] The system according to claim 1, wherein the server means further comprises means for analyzing the request using a natural language processing engine.

[0800] (Claim 3)

[0801] The system according to claim 1, wherein the server means further comprises a filtering means for retrieving information from multiple databases and selecting information that corresponds to a user's request.

[0802] "Example 1"

[0803] (Claim 1)

[0804] Information receiving means for receiving information as audio or text,

[0805] The aforementioned received information is analyzed using natural language processing technology, and analysis means extracts keywords based on context.

[0806] A search method for retrieving target information from multiple data storage systems based on the analyzed information,

[0807] A generation method for generating selected information in natural language and presenting it to the user,

[0808] A system that includes this.

[0809] (Claim 2)

[0810] The system according to claim 1, further comprising an analysis method that uses a natural language processing engine to understand user requests based on extracted keywords.

[0811] (Claim 3)

[0812] The system according to claim 1, wherein the search means further comprises a filtering process that filters the acquired information and selects and provides the most suitable information.

[0813] "Application Example 1"

[0814] (Claim 1)

[0815] Communication equipment means for receiving user requests,

[0816] A computer means for analyzing the aforementioned requirements and searching for related data,

[0817] A transmitting device means for generating the retrieved data in natural language and transmitting it to the communication device means,

[0818] A speech recognition means for converting voice input into text,

[0819] A natural language processing method for analyzing text and extracting important keywords,

[0820] A system that includes this.

[0821] (Claim 2)

[0822] The system according to claim 1, wherein the computing means further comprises means for analyzing the request using a natural language processing program.

[0823] (Claim 3)

[0824] The system according to claim 1, wherein the computing means further comprises extraction means for searching for data from multiple information sources and selecting data that corresponds to the user's request.

[0825] "Example 2 of combining an emotion engine"

[0826] (Claim 1)

[0827] A terminal means for receiving user requests and emotional states,

[0828] A server means for analyzing the aforementioned request using a natural language processing engine and an emotion engine,

[0829] A generation means for retrieving information from multiple sources based on the aforementioned analysis results and providing the information in a manner that responds to the user's emotions,

[0830] A transmission means for generating the aforementioned information in natural language and transmitting it to the terminal means,

[0831] A system that includes this.

[0832] (Claim 2)

[0833] The system according to claim 1, wherein the server means further comprises means for recognizing the user's emotional state and adjusting the generation of information accordingly.

[0834] (Claim 3)

[0835] The system according to claim 1, further comprising a server means for analyzing the user's emotions using the collected voice characteristics and reflecting the results in the method of presenting information.

[0836] "Application example 2 when combining with an emotional engine"

[0837] (Claim 1)

[0838] A terminal device for receiving user requests and emotions,

[0839] A computing device for analyzing the aforementioned requests and emotions and retrieving related information,

[0840] A transmitting device for generating the retrieved information in natural language and transmitting it to the terminal device,

[0841] A data processing device that adjusts the method of presenting information according to the user's emotions,

[0842] A device that outputs information in voice or text using a user interface,

[0843] A system that includes this.

[0844] (Claim 2)

[0845] The system according to claim 1, further comprising means for the computing device to analyze the request using a natural language processing engine and to recognize the emotion using an emotion engine.

[0846] (Claim 3)

[0847] The system according to claim 1, further comprising a selection device that retrieves information from multiple knowledge bases and selects information that corresponds to the user's requests and emotions. [Explanation of symbols]

[0848] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A terminal means for receiving user requests, A server means for analyzing the aforementioned request and retrieving related information, A transmission means for generating the retrieved information in natural language and transmitting it to the terminal means, A system that includes this.

2. The system according to claim 1, wherein the server means further comprises means for analyzing the request using a natural language processing engine.

3. The system according to claim 1, wherein the server means further comprises a filtering means for searching for information from multiple databases and selecting information that corresponds to a user's request.

Images