system

Abstract

Provide a system. 【Solution means】 Means for acquiring voice input, Means for converting the voice input into text data, Means for analyzing the text data and recognizing the user's intention, Means for acquiring an image of the desktop screen, Means for analyzing the acquired image of the desktop screen and extracting information, Means for generating a response based on the extracted information and the user's intention, Means for providing the response to the user as voice or text, A system including the above.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a conventional sales support system, a user has to open a chat interface and manually input information, which is a factor reducing productivity. The present invention aims to solve this problem by providing a means to reduce the operation burden of the user by using voice input and obtain information more quickly and efficiently.

Means for Solving the Problems

[0005] To solve this problem, the present invention provides means for acquiring voice input and means for converting said voice input into text data. Then, by introducing means for analyzing the converted text data and recognizing the user's intent, the information the user is seeking is clearly identified. Furthermore, by means for capturing the desktop screen and analyzing the acquired image, necessary information is efficiently extracted. By integrating these and providing the generated response to the user in voice or text format, smoother and more intuitive operation is achieved.

[0006] "Voice input" is a method of transmitting information to a computer system by having the user speak.

[0007] "Text data" refers to data in a format that represents voice input or other information as a string of characters.

[0008] A "natural language processing engine" is software or algorithms designed to analyze text data and understand its grammatical structure and meaning.

[0009] "User intent" refers to what the user wants from the system, and what kind of information or actions they expect.

[0010] A "screenshot" is the act of capturing the current display content of the desktop screen as image data.

[0011] "Image analysis" is the process of analyzing image data and extracting the information contained within it.

[0012] "Generated response" refers to the answers or information that the system provides to the user based on the analyzed information. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described according to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] To implement this invention, a system is constructed in which a user, a terminal, and a server work together to perform a series of voice-based information processing tasks. This system begins with the user providing information in the form of voice input. When the user gives instructions or asks questions verbally, the terminal captures the audio in real time and converts it into text data using a speech recognition engine. This digitizes the user's voice information, enabling further processing.

[0035] Subsequently, the device uses a natural language processing engine to analyze the transcribed audio information and understand the user's intent. This analysis process involves analyzing the grammatical structure and key elements of the text data to identify the type of information the user is seeking from the system and the necessary actions.

[0036] Next, the device takes a screenshot of the desktop to understand the current context. The acquired image is sent to the server, which extracts information using image analysis algorithms. In this process, specific data points or document sections within the image are identified, and data that aligns with the user's request is extracted.

[0037] The server integrates the extracted information with the results of speech-to-text analysis to generate information useful to the user. This response is formatted in text and, if necessary, converted to speech using speech synthesis technology. The user receives the information output from the terminal as either audio or text, enabling them to smoothly perform necessary desk work.

[0038] For example, if a user asks, "What are the meetings scheduled for today?", the voice input is converted into text data, and subsequent analysis identifies keywords such as "meeting" and "schedule." The server analyzes the display screen of the schedule management application from a screenshot, extracts the relevant schedule information, and responds to the user. In this way, the present invention realizes a rapid and automated information provision process based on voice input.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The device receives the user's voice input. When the user speaks into the voice input device, the device captures voice data in real time through the microphone.

[0042] Step 2:

[0043] The device uses a speech recognition engine to convert the captured audio into text data. The converted text data is temporarily stored in memory for subsequent analysis.

[0044] Step 3:

[0045] The device uses a natural language processing engine to analyze text data. Through this analysis, it examines the user's intent and the information they need, and identifies specific information requests.

[0046] Step 4:

[0047] The device takes a screenshot of the user's desktop screen. This screenshot is necessary to understand the current context.

[0048] Step 5:

[0049] The device sends a screenshot it has taken to the server. The server receives this image data and prepares it for processing.

[0050] Step 6:

[0051] The server analyzes the screenshot to extract the specific information the user is looking for. For example, it might identify numerical data or parts of documents within a particular application.

[0052] Step 7:

[0053] The server integrates the analysis results with the user's voice request and generates an appropriate response. This response includes detailed information tailored to the user's request.

[0054] Step 8:

[0055] The server sends the generated response to the terminal. The terminal receives this response and prepares for output.

[0056] Step 9:

[0057] The device uses a speech synthesis engine to convert text responses into speech. It provides information to the user in either voice or text and answers the user's questions.

[0058] (Example 1)

[0059] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0060] Conventional information acquisition systems have struggled to quickly and accurately retrieve necessary information from voice data and provide it to users. This challenge has led to inefficiencies, particularly in information integration and analysis, and has negatively impacted the user experience.

[0061] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0062] In this invention, the server includes means for acquiring voice data, means for converting the voice data into encoded information, and means for analyzing the encoded information and recognizing the user's request. This enables rapid and accurate processing of voice data and the provision of information based on the user's request.

[0063] "Audio data" refers to information that represents sound vibrations in digital format, and is the information collected from sound.

[0064] "Encoded information" refers to information obtained by analyzing audio or image data and converting it into a digitally understandable format.

[0065] A "display device" refers to a hardware device connected to a computer or electronic device that presents visual information to the user.

[0066] "User" refers to a person who operates or uses the system.

[0067] "Means of conversion" refers to the technical process of replacing data in one format with another.

[0068] "Means of analysis" refers to methods or techniques used to break down data and understand its structure and meaning.

[0069] "Means of acquisition" refers to the technical processes and equipment used to obtain data and information.

[0070] "Means of identification" refers to the process of identifying and extracting necessary data from the analyzed information.

[0071] This invention specifically demonstrates how to implement a system for acquiring and providing information to a user through a voice-based interface. First, the user gives instructions or questions by voice, and voice data is acquired through a microphone installed in the terminal. This terminal converts the voice data into encoded information using speech recognition technology. Google® Cloud Speech-to-Text or other speech recognition technologies may be used in this process.

[0072] Next, the terminal analyzes the generated encoded information using natural language processing technology, such as a generative AI model from OpenAI®, to interpret the user's request. The analysis identifies the type of information the user is seeking. To obtain this information, the terminal captures the current screen state and sends it to the server.

[0073] The server analyzes this screenshot using image analysis techniques, such as OpenCV. The server identifies specific information and data from the screenshot and extracts information that aligns with the user's request. This information is integrated with the analyzed encoded information to generate useful information for the user.

[0074] The server can then format the generated information into text and, if necessary, generate it as speech using speech synthesis technology. As a result, users can receive concise responses from their devices in either voice or text, allowing them to efficiently continue with the necessary tasks.

[0075] For example, if a user prompts their device with "Check the preparation status for next week's meeting," this invention can analyze the voice input, retrieve information from the calendar and related applications, and provide the user with accurate information. In this way, the present invention highly streamlines voice-based information processing and user feedback.

[0076] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0077] Step 1:

[0078] The user gives instructions or asks questions to the device using voice commands.

[0079] Input: User's voice

[0080] Specific action: The user speaks into the microphone and says, "Please check the preparation status for next week's meeting."

[0081] Output: Audio signal input to the terminal

[0082] Step 2:

[0083] The device acquires audio data, sends that audio to a speech recognition engine, and converts it into text data.

[0084] Input: Audio signal

[0085] Data processing: The device uses a speech recognition engine to convert the speech signal into text information. Google Cloud Speech-to-Text, for example, is used.

[0086] Specific operation: The recorded audio is passed to the speech recognition software in real time.

[0087] Output: Text data "Please check the preparation status for next week's meeting."

[0088] Step 3:

[0089] The device uses natural language processing to analyze text data and understand user requests.

[0090] Input: Text data

[0091] Data processing: Using generative AI models such as GPT-3 (registered trademark), we interpret user requests and intentions.

[0092] Specific actions: Extract keywords such as "meeting," "next week," and "preparation status" to identify the type of information the user is looking for.

[0093] Output: Interpreted request information

[0094] Step 4:

[0095] The device takes a screenshot to obtain an image of the current screen.

[0096] Input: Current screen state

[0097] Specific operation: The device uses the screen capture function to save the screen on the monitor as an image.

[0098] Output: Screenshot

[0099] Step 5:

[0100] The system sends screenshots to the server and extracts the necessary information through image analysis.

[0101] Input: Screenshot

[0102] Data processing: Extract necessary data from screenshots using image analysis techniques such as OpenCV.

[0103] Specific action: Retrieve schedule information for "Next week's meeting" from the calendar app screen.

[0104] Output: Extracted meeting information

[0105] Step 6:

[0106] The server integrates the request information and extracted information to generate a response for the user.

[0107] Input: Request information, Extraction information

[0108] Data processing: Generate appropriate response messages based on integrated information.

[0109] Specific actions: Create a detailed description including the date, time, and location of the meeting.

[0110] Output: Response message

[0111] Step 7:

[0112] The server converts the generated response into text or speech and sends it to the terminal.

[0113] Input: Response message

[0114] Data processing: It is also possible to convert the data into speech using speech synthesis technologies such as Amazon Polly.

[0115] Specific action: Generate the text "The next meeting will be held next Monday at 10:00 in Conference Room A."

[0116] Output: Text or voice response

[0117] Step 8:

[0118] The user receives responses through the terminal and makes decisions to carry out their tasks.

[0119] Input: Text or voice response

[0120] Specific action: The user reviews the information received via audio or text and begins preparing for the meeting.

[0121] Output: User guidelines and plans

[0122] (Application Example 1)

[0123] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0124] Voice-based interactions allow users to obtain a wide range of information simply by using voice input, but the process is typically limited to voice recognition and simple text responses. Such systems lack the ability to provide detailed information about the user's surroundings and specific context. Furthermore, in the home, there is a need for systems that allow users to quickly and easily obtain information based on actual objects and situations.

[0125] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0126] In this invention, the server includes means for obtaining voice data, means for converting the voice data into text form, means for analyzing the text data to understand the user's intent, means for acquiring image data, means for analyzing the acquired image data to detect information, means for creating a response based on the detected information and the user's intent, and means for communicating the response to the user in voice or text format. This enables the user to use voice commands to check their surroundings and obtain specific information.

[0127] "Audio data" refers to information that represents audio acquired from a user in digital format.

[0128] "Text format" refers to string-based information obtained as a result of recognizing and converting audio data.

[0129] "User intent" refers to the content that indicates the user's requests or objectives included in the voice input.

[0130] "Image data" refers to visual information acquired using devices such as cameras, represented in digital format.

[0131] "Means of detecting information" refers to the processing and methods used to analyze and extract useful information from acquired image data.

[0132] "Means of generating responses" refers to the process of generating answers or instructions for the user based on detected information and the user's intent.

[0133] "Means of communicating with the user in audio or text format" refers to methods of audio playback or text display that deliver the generated response content to the user in an easily understandable manner.

[0134] This system allows users to quickly obtain everyday information through voice input. The user provides voice data to the terminal, which then uses voice recognition software to convert the voice data into text. The terminal then analyzes the resulting text data using natural language processing software to understand the user's intent.

[0135] The system includes cameras mounted on terminals and robots, which acquire image data. The acquired image data is processed on a server using image analysis algorithms to detect specific information. This makes it possible to quickly obtain information such as the expiration date of food items in a home refrigerator, if the user asks about it by voice.

[0136] The server generates a response based on the detected information and the user's intent, and this response is communicated to the user through voice playback software or a display. This system allows users to obtain practical information on the spot simply by asking specific questions.

[0137] For example, a user might give a voice command such as, "Tell me the expiration date of the milk in the refrigerator." The system accurately recognizes and processes this command and provides relevant information. By using prompts for the generative AI model, such as, "Check the expiration date of the milk in the refrigerator using image analysis and provide the information via voice," it is possible to improve the accuracy of information retrieval and the user experience.

[0138] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0139] Step 1:

[0140] The user provides audio data to the device. What the user says is captured as digital audio data via the microphone. At this point, the audio data becomes the system's input.

[0141] Step 2:

[0142] The terminal uses speech recognition software to convert the audio data into text. The converted text data is output and used as input for the next processing step. This process yields the audio information as a parseable string.

[0143] Step 3:

[0144] The device uses natural language processing software to analyze text data and understand the user's intent. It analyzes the grammatical structure and keywords of the text data, and outputs the analysis results. This analysis clarifies the user's instructions and requests.

[0145] Step 4:

[0146] The device uses its built-in camera to acquire image data of a specified object or area. This image data is output and sent to a server. One example is obtaining an image of a food item specified by the user from inside a refrigerator.

[0147] Step 5:

[0148] The server receives image data and detects specific information by applying image analysis algorithms. During this process, useful data points are extracted, and the analysis results are output. For example, the expiration date can be extracted.

[0149] Step 6:

[0150] The server integrates information extracted from the acquired images with the analyzed user intent to generate a response. The generated response is output in text format, and audio playback software is also provided. The response content is provided in a format useful to the user as specific information in response to the user's question.

[0151] Step 7:

[0152] The device communicates the generated response to the user in either voice or text format. Text-to-speech software converts the text to speech, which is then played through the speaker. This final step allows the user to quickly receive actionable information.

[0153] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0154] To implement the present invention, it is necessary to configure a series of systems that include voice input, text conversion, intent recognition, desktop screen snapshots, image analysis, and an emotion engine that grasps the user's emotions. This system grasps the user's intent by analyzing text data obtained from the user's voice input, and further recognizes the emotions contained in that voice using the emotion engine.

[0155] This process begins with the device receiving the user's voice. A speech recognition engine converts the voice into text data, and then a natural language processing engine analyzes the text to identify the user's intent. Furthermore, an emotion engine infers the user's emotions from elements such as tone and tempo of the voice. This allows the system to not only provide information but also respond flexibly in accordance with the user's emotions.

[0156] The captured desktop screen snapshot is sent to the server, where image analysis extracts the information requested by the user. The server, along with the analyzed information, generates a response that takes into account the user's intent and emotions. This response is provided as text or audio in an emotionally appropriate tone, allowing the user to have a more personalized experience.

[0157] For example, if a user asks "How were my grades today?" in a slightly anxious tone, the emotion engine will detect this anxiety. The system will then retrieve the grade data and provide the user with a response that incorporates an encouraging tone. This allows the user not only to obtain information but also to receive mental support, enabling them to move on to the next action with confidence. In this way, the addition of the emotion engine enables a unique implementation that improves the user experience.

[0158] The following describes the processing flow.

[0159] Step 1:

[0160] The device receives voice input from the user. The user speaks into the voice input device, and the device records it in real time via the microphone.

[0161] Step 2:

[0162] The device uses a speech recognition engine to convert the recorded audio into text data. The converted text data is temporarily stored on the device for subsequent analysis.

[0163] Step 3:

[0164] The device uses a natural language processing engine to analyze text data and understand the intent behind the user's questions and requests. This analysis identifies exactly what the user wants to know.

[0165] Step 4:

[0166] The device uses an emotion engine to analyze elements such as voice tone and speed to recognize the user's emotions. This is an important step in taking the user's psychological state into account.

[0167] Step 5:

[0168] The device captures the user's desktop screen and saves the image as a snapshot. This snapshot is necessary to understand the user's current context.

[0169] Step 6:

[0170] The device sends the captured snapshot to the server, preparing it for image data analysis.

[0171] Step 7:

[0172] The server analyzes the snapshot and extracts the information the user is looking for from the image. This includes reading document content and application data.

[0173] Step 8:

[0174] The server combines the analyzed information with the user's intent and emotional data from the terminal to generate an appropriate response. This response incorporates both detailed content and an emotionally sensitive tone.

[0175] Step 9:

[0176] The server sends the generated response to the terminal. The terminal receives this response and prepares the output for the user.

[0177] Step 10:

[0178] The device uses a speech synthesis engine to convert the generated text responses into speech. Furthermore, it addresses questions by presenting information to the user in a way that reflects appropriate emotions.

[0179] (Example 2)

[0180] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0181] Conventional speech recognition systems could only recognize the user's intent, and struggled to generate responses that took into account emotional changes or the context on the desktop. As a result, users could not receive personalized responses that matched their emotions, leading to a decline in the quality of communication.

[0182] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0183] In this invention, the server includes means for converting voice input into text data, means for recognizing the user's intent and emotions, and means for analyzing images on the desktop screen to extract information. This makes it possible to provide flexible and personalized responses based on the user's intent and emotions.

[0184] "Voice input" is a data format for acquiring information through a user's voice.

[0185] "Text data" refers to data in string format that is converted from voice input.

[0186] "User intent" refers to the purpose or request for which the user is seeking an action or information through voice input.

[0187] "Emotion" refers to the emotional state inferred from the user's voice characteristics.

[0188] A "desktop image" is a snapshot of the visual information displayed on a computer screen.

[0189] A "response" is a message or action provided to the user, generated based on the analyzed information.

[0190] A "speech recognition engine" is a system that provides technology to convert speech input into text data.

[0191] A "natural language processing engine" is a system that provides technology to analyze text data and identify the user's intent.

[0192] One embodiment of this invention is a system that uses speech recognition and natural language processing technologies to precisely analyze user input and generate personalized responses. A specific example of this system is shown below.

[0193] The user provides information to the system via voice input. The terminal receives this voice input and converts the speech into text data using a speech recognition engine. In this process, open-source speech recognition software may be used.

[0194] The device further analyzes the converted text data using a natural language processing engine to identify the user's intent. Commonly available natural language processing models are utilized here. In addition, an emotion engine is used to analyze the user's emotions from the tone and tempo of their voice, preparing to generate an appropriate response.

[0195] The captured desktop screen snapshots are sent by the terminal to the server, where computer vision technology is used to analyze the images. This makes it possible to understand the information the user is requesting and the context of the actions they are performing on the desktop.

[0196] Based on the analyzed user intent, sentiment information, and desktop context, the server generates a response using a generative AI model. This response is delivered as voice or text, customized to the user's emotions and intent.

[0197] For example, if a user asks "What's on my schedule this week?" in an anxious tone, the emotion engine detects that anxiety. The system retrieves the week's calendar information and provides a response that is empathetic to the user's feelings, such as "Your schedule is full this week, but I'm sure you'll have a fulfilling week." An example of a prompt might be something like, "Respond to an anxious user with their schedule for the week in an encouraging tone." Through this process, the user feels better understood and can move on to the next action with confidence.

[0198] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0199] Step 1:

[0200] The user inputs a request to the system by voice. The terminal receives this voice input via its microphone. The input is raw voice data. The terminal activates its speech recognition engine and converts the voice data into digital text data. The output of this step is the string data converted from the voice.

[0201] Step 2:

[0202] The device sends text data to a natural language processing engine to analyze the user's intent. This input is the text data acquired by the device. The natural language processing engine uses parsing algorithms to analyze the context and grammatical structure of the text data and identify the information and actions the user is seeking. The output of this step is data indicating the user's intent.

[0203] Step 3:

[0204] Simultaneously, the device uses an emotion engine to analyze the user's emotions from the audio data. The input is the received audio data, and the emotion engine evaluates elements such as tone, pitch, and tempo of the voice to estimate the user's emotions. The output of this step is information about the identified user emotions.

[0205] Step 4:

[0206] The terminal takes a snapshot of the desktop screen. The input is the currently displayed desktop image. The snapshot is transferred to the server as image data. In this step, the information on the screen is captured for later analysis.

[0207] Step 5:

[0208] The server analyzes the received image data using computer vision technology and extracts useful information from the desktop screen. The input is snapshot image data, and the output is specific information relevant to the user. For example, the server identifies schedule information provided by task management software.

[0209] Step 6:

[0210] The server constructs a response using a generative AI model based on the analyzed intent, sentiment, and screen information. The input is data from steps 2, 3, and 5, which the generative AI model processes as prompts to generate a personalized response. The output of this step is a text or audio message to be presented to the user.

[0211] Step 7:

[0212] The terminal receives responses from the server and provides them to the user as voice or text. The input is the response data generated by the server, and the output is the result directly experienced by the user. By obtaining information and receiving emotionally resonant responses, the user develops a greater affinity with the system.

[0213] (Application Example 2)

[0214] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0215] Conventional voice dialogue systems have the drawback of being monotonous and uniform in their responses, as they do not take into account the user's emotions. As a result, users are often dissatisfied with their interactions with the system, and the quality of the dialogue experience deteriorates. In particular, for personal robots and applications, it is important to appropriately recognize the user's emotions and provide responses accordingly.

[0216] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0217] In this invention, the server includes means for acquiring voice information, means for converting it into text information, means for analyzing the text information to recognize the user's intent and emotions, means for analyzing acquired images to extract information, and means for generating a response based on the extracted information and the user's intent and emotions. This makes it possible to provide flexible and personalized responses that respond to the user's emotions.

[0218] "Audio information" refers to data obtained from the user's spoken words and sounds.

[0219] "Text information" refers to digital data that converts audio information into written form.

[0220] "User intent" refers to the purpose or request that the user wants to convey to the system through voice information.

[0221] "Emotional analysis" is the process of inferring a user's emotions and emotional state from their voice and text information.

[0222] A "display device" is a device that visually displays information from computers and electronic devices.

[0223] A "response" refers to a reply or reaction generated by a system based on the user's intentions and emotions.

[0224] A "speech recognition device" refers to hardware or software technology for converting speech information into text information.

[0225] A "natural language processing system" refers to algorithms and software technologies used to analyze text information and recognize intentions and emotions.

[0226] The system that realizes this application consists of a speech recognition device, a natural language processing device, an emotion analysis device, an image analysis device, and a response generation device. By coordinating these devices, the server analyzes the user's intent and emotions based on the voice information obtained from the user and generates an appropriate response.

[0227] The server first uses a speech recognition device to acquire the user's voice information and converts it into text. This converted text information is then analyzed by a natural language processing device to identify the user's intent. Next, an emotion analysis device infers the user's emotions from the text information. This process utilizes emotion analysis technologies such as IBM Watson® Tone Analyzer.

[0228] The acquired intent and emotion information, along with image information from the display device of the user's terminal, is sent to an image analysis device. Necessary information is extracted from the display device's image. For example, the content displayed on the screen is analyzed using the Google Cloud Vision API, and data corresponding to the user's request is extracted.

[0229] Ultimately, the server uses a response generator to produce a response based on the user's intent and emotions. This response is delivered as voice or text, in a tone that matches the user's emotions. This allows the user to have a personalized experience.

[0230] As a concrete example, consider a scenario where a user asks, "What's the latest news?" If sentiment analysis reveals that the user is excited, the system will provide positive and interesting news in an upbeat tone. An example of a prompt using a generative AI model would be, "Considering the user's excitement, please provide positive news."

[0231] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0232] Step 1:

[0233] The user provides voice input. The terminal acquires this voice and inputs it into the speech recognition device. At this point, the voice information is the input data. The server uses the speech recognition device to convert this voice information into text information and outputs it.

[0234] Step 2:

[0235] The server inputs the acquired text information into a natural language processing unit. At this stage, the input is text information. The server performs natural language processing, analyzing the text information to identify the user's intent. The analyzed user intent is then output.

[0236] Step 3:

[0237] The server inputs text information into the sentiment analysis device and performs sentiment analysis. The input is the user's text information, and the result of the sentiment analysis is the user's emotional state. The server outputs this emotional state.

[0238] Step 4:

[0239] The terminal captures a snapshot of the display device's current screen and sends the image information to the server. The server uses an image analysis device to analyze this image information. The input is the image from the display device, and the server extracts the necessary information from it and outputs it.

[0240] Step 5:

[0241] The server receives the output of the natural language processing unit (user intent), the emotion analysis unit (user emotion), and the image analysis unit (necessary information), and inputs them into the response generation unit. Based on this data, the server generates an appropriate response and outputs it. The generated response is sent to the terminal as audio or text and provided to the user.

[0242] Step 6:

[0243] When providing a response to the user, the server inputs prompts into the generating AI model, which then completes and adjusts the response. These prompts are typically in the format of, "Considering the user's excitement, please provide positive news." This process ensures the response aligns with the user's emotions.

[0244] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0245] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0246] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0247] [Second Embodiment]

[0248] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0249] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0250] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0251] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0252] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0253] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0254] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0255] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0256] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0257] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0258] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0259] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0260] To implement this invention, a system is constructed in which a user, a terminal, and a server work together to perform a series of voice-based information processing tasks. This system begins with the user providing information in the form of voice input. When the user gives instructions or asks questions verbally, the terminal captures the audio in real time and converts it into text data using a speech recognition engine. This digitizes the user's voice information, enabling further processing.

[0261] Subsequently, the device uses a natural language processing engine to analyze the transcribed audio information and understand the user's intent. This analysis process involves analyzing the grammatical structure and key elements of the text data to identify the type of information the user is seeking from the system and the necessary actions.

[0262] Next, the device takes a screenshot of the desktop to understand the current context. The acquired image is sent to the server, which extracts information using image analysis algorithms. In this process, specific data points or document sections within the image are identified, and data that aligns with the user's request is extracted.

[0263] The server integrates the extracted information with the results of speech-to-text analysis to generate information useful to the user. This response is formatted in text and, if necessary, converted to speech using speech synthesis technology. The user receives the information output from the terminal as either audio or text, enabling them to smoothly perform necessary desk work.

[0264] For example, if a user asks, "What are the meetings scheduled for today?", the voice input is converted into text data, and subsequent analysis identifies keywords such as "meeting" and "schedule." The server analyzes the display screen of the schedule management application from a screenshot, extracts the relevant schedule information, and responds to the user. In this way, the present invention realizes a rapid and automated information provision process based on voice input.

[0265] The following describes the processing flow.

[0266] Step 1:

[0267] The device receives the user's voice input. When the user speaks into the voice input device, the device captures voice data in real time through the microphone.

[0268] Step 2:

[0269] The device uses a speech recognition engine to convert the captured audio into text data. The converted text data is temporarily stored in memory for subsequent analysis.

[0270] Step 3:

[0271] The device uses a natural language processing engine to analyze text data. Through this analysis, it examines the user's intent and the information they need, and identifies specific information requests.

[0272] Step 4:

[0273] The device takes a screenshot of the user's desktop screen. This screenshot is necessary to understand the current context.

[0274] Step 5:

[0275] The device sends a screenshot it has taken to the server. The server receives this image data and prepares it for processing.

[0276] Step 6:

[0277] The server analyzes the screenshot to extract the specific information the user is looking for. For example, it might identify numerical data or parts of documents within a particular application.

[0278] Step 7:

[0279] The server integrates the analysis results with the user's voice request and generates an appropriate response. This response includes detailed information tailored to the user's request.

[0280] Step 8:

[0281] The server sends the generated response to the terminal. The terminal receives this response and prepares for output.

[0282] Step 9:

[0283] The terminal uses the speech synthesis engine to convert the text response into speech, provides information to the user in speech or text, and answers the user's questions.

[0284] (Example 1)

[0285] Next, Example 1 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0286] In the conventional information acquisition system, it was difficult to quickly and accurately acquire the necessary information from speech and provide it to the user. This problem was particularly lacking in efficiency in the integration and analysis of information, causing damage to the user experience.

[0287] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0288] In this invention, the server includes means for acquiring speech data, means for converting the speech data into encoded information, and means for analyzing the encoded information to recognize the user's request. Thereby, the speech data can be processed quickly and accurately, and information can be provided based on the user's request.

[0289] "Speech data" refers to information that represents the vibration of sound in digital form and refers to information collected from speech.

[0290] "Encoded information" refers to information obtained by analyzing speech or image data and converting it into a form that can be understood in digital form.

[0291] "Display device" refers to a hardware device that is connected to a computer or electronic device and presents visual information to the user.

[0292] "User" refers to a person who operates or uses the system.

[0293] "Means of conversion" refers to the technical process of replacing data in one format with another.

[0294] "Means of analysis" refers to methods or techniques used to break down data and understand its structure and meaning.

[0295] "Means of acquisition" refers to the technical processes and equipment used to obtain data and information.

[0296] "Means of identification" refers to the process of identifying and extracting necessary data from the analyzed information.

[0297] This invention specifically demonstrates how to implement a system for acquiring and providing information to a user through a voice-based interface. First, the user gives instructions or questions by voice, and voice data is acquired through a microphone installed in the terminal. This terminal converts the voice data into encoded information using speech recognition technology. Google Cloud Speech-to-Text or other speech recognition technologies may be used in this process.

[0298] Next, the terminal analyzes the generated encoded information using natural language processing techniques, such as OpenAI's generative AI model, to interpret the user's request. The analysis identifies the type of information the user is seeking. To obtain this information, the terminal captures the current screen state and sends it to the server.

[0299] The server analyzes this screenshot using image analysis techniques, such as OpenCV. The server identifies specific information and data from the screenshot and extracts information that aligns with the user's request. This information is integrated with the analyzed encoded information to generate useful information for the user.

[0300] Finally, the server can format the generated information into text format and also generate it as audio using speech synthesis technology if necessary. As a result, the user can receive a concise response from the terminal in audio or text and can efficiently continue with the necessary tasks.

[0301] As a specific example, when the user prompts the terminal with "Check the preparation status of next week's meeting", this invention can analyze the voice input, obtain information from the calendar and related applications, and provide accurate information to the user. Thus, this invention highly improves the efficiency of voice-based information processing and feedback to the user.

[0302] The flow of the specific process in Example 1 will be described using FIG. 11.

[0303] Step 1:

[0304] The user gives an instruction or asks a question to the terminal by voice.

[0305] Input: User's voice

[0306] Specific operation: The user speaks towards the microphone "Check the preparation status of next week's meeting".

[0307] Output: The voice signal input to the terminal

[0308] Step 2:

[0309] The terminal acquires the voice data, sends the voice to a voice recognition engine, and converts it into text data.

[0310] Input: Voice signal

[0311] Data processing: The terminal uses a voice recognition engine to convert the voice signal into text information. For example, Google Cloud Speech-to-Text is used.

[0312] Specific operation: The recorded audio is passed to the speech recognition software in real time.

[0313] Output: Text data "Please check the preparation status for next week's meeting."

[0314] Step 3:

[0315] The device uses natural language processing to analyze text data and understand user requests.

[0316] Input: Text data

[0317] Data processing: Using generative AI models such as GPT-3, we interpret user requests and intentions.

[0318] Specific actions: Extract keywords such as "meeting," "next week," and "preparation status" to identify the type of information the user is looking for.

[0319] Output: Interpreted request information

[0320] Step 4:

[0321] The device takes a screenshot to obtain an image of the current screen.

[0322] Input: Current screen state

[0323] Specific operation: The device uses the screen capture function to save the screen on the monitor as an image.

[0324] Output: Screenshot

[0325] Step 5:

[0326] The system sends screenshots to the server and extracts the necessary information through image analysis.

[0327] Input: Screenshot

[0328] Data processing: Extract necessary data from screenshots using image analysis techniques such as OpenCV.

[0329] Specific action: Retrieve schedule information for "Next week's meeting" from the calendar app screen.

[0330] Output: Extracted meeting information

[0331] Step 6:

[0332] The server integrates the request information and extracted information to generate a response for the user.

[0333] Input: Request information, Extraction information

[0334] Data processing: Generate appropriate response messages based on integrated information.

[0335] Specific actions: Create a detailed description including the date, time, and location of the meeting.

[0336] Output: Response message

[0337] Step 7:

[0338] The server converts the generated response into text or speech and sends it to the terminal.

[0339] Input: Response message

[0340] Data processing: It is also possible to convert the data into speech using speech synthesis technologies such as Amazon Polly.

[0341] Specific action: Generate the text "The next meeting will be held next Monday at 10:00 in Conference Room A."

[0342] Output: Text or voice response

[0343] Step 8:

[0344] The user receives responses through the terminal and makes decisions to carry out their tasks.

[0345] Input: Text or voice response

[0346] Specific action: The user reviews the information received via audio or text and begins preparing for the meeting.

[0347] Output: User guidelines and plans

[0348] (Application Example 1)

[0349] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0350] Voice-based interactions allow users to obtain a wide range of information simply by using voice input, but the process is typically limited to voice recognition and simple text responses. Such systems lack the ability to provide detailed information about the user's surroundings and specific context. Furthermore, in the home, there is a need for systems that allow users to quickly and easily obtain information based on actual objects and situations.

[0351] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0352] In this invention, the server includes means for obtaining voice data, means for converting the voice data into text form, means for analyzing the text data to understand the user's intent, means for acquiring image data, means for analyzing the acquired image data to detect information, means for creating a response based on the detected information and the user's intent, and means for communicating the response to the user in voice or text format. This enables the user to use voice commands to check their surroundings and obtain specific information.

[0353] "Audio data" refers to information that represents audio acquired from a user in digital format.

[0354] "Text format" refers to string-based information obtained as a result of recognizing and converting audio data.

[0355] "User intent" refers to the content that indicates the user's requests or objectives included in the voice input.

[0356] "Image data" refers to visual information acquired using devices such as cameras, represented in digital format.

[0357] "Means of detecting information" refers to the processing and methods used to analyze and extract useful information from acquired image data.

[0358] "Means of generating responses" refers to the process of generating answers or instructions for the user based on detected information and the user's intent.

[0359] "Means of communicating with the user in audio or text format" refers to methods of audio playback or text display that deliver the generated response content to the user in an easily understandable manner.

[0360] This system allows users to quickly obtain everyday information through voice input. The user provides voice data to the terminal, which then uses voice recognition software to convert the voice data into text. The terminal then analyzes the resulting text data using natural language processing software to understand the user's intent.

[0361] The system includes cameras mounted on terminals and robots, which acquire image data. The acquired image data is processed on a server using image analysis algorithms to detect specific information. This makes it possible to quickly obtain information such as the expiration date of food items in a home refrigerator, if the user asks about it by voice.

[0362] The server generates a response based on the detected information and the user's intent, and this response is communicated to the user through voice playback software or a display. This system allows users to obtain practical information on the spot simply by asking specific questions.

[0363] For example, a user might give a voice command such as, "Tell me the expiration date of the milk in the refrigerator." The system accurately recognizes and processes this command and provides relevant information. By using prompts for the generative AI model, such as, "Check the expiration date of the milk in the refrigerator using image analysis and provide the information via voice," it is possible to improve the accuracy of information retrieval and the user experience.

[0364] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0365] Step 1:

[0366] The user provides audio data to the device. What the user says is captured as digital audio data via the microphone. At this point, the audio data becomes the system's input.

[0367] Step 2:

[0368] The terminal uses speech recognition software to convert the audio data into text. The converted text data is output and used as input for the next processing step. This process yields the audio information as a parseable string.

[0369] Step 3:

[0370] The device uses natural language processing software to analyze text data and understand the user's intent. It analyzes the grammatical structure and keywords of the text data, and outputs the analysis results. This analysis clarifies the user's instructions and requests.

[0371] Step 4:

[0372] The device uses its built-in camera to acquire image data of a specified object or area. This image data is output and sent to a server. One example is obtaining an image of a food item specified by the user from inside a refrigerator.

[0373] Step 5:

[0374] The server receives image data and detects specific information by applying image analysis algorithms. During this process, useful data points are extracted, and the analysis results are output. For example, the expiration date can be extracted.

[0375] Step 6:

[0376] The server integrates information extracted from the acquired images with the analyzed user intent to generate a response. The generated response is output in text format, and audio playback software is also provided. The response content is provided in a format useful to the user as specific information in response to the user's question.

[0377] Step 7:

[0378] The device communicates the generated response to the user in either voice or text format. Text-to-speech software converts the text to speech, which is then played through the speaker. This final step allows the user to quickly receive actionable information.

[0379] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0380] To implement the present invention, it is necessary to configure a series of systems that include voice input, text conversion, intent recognition, desktop screen snapshots, image analysis, and an emotion engine that grasps the user's emotions. This system grasps the user's intent by analyzing text data obtained from the user's voice input, and further recognizes the emotions contained in that voice using the emotion engine.

[0381] This process begins with the device receiving the user's voice. A speech recognition engine converts the voice into text data, and then a natural language processing engine analyzes the text to identify the user's intent. Furthermore, an emotion engine infers the user's emotions from elements such as tone and tempo of the voice. This allows the system to not only provide information but also respond flexibly in accordance with the user's emotions.

[0382] The captured desktop screen snapshot is sent to the server, where image analysis extracts the information requested by the user. The server, along with the analyzed information, generates a response that takes into account the user's intent and emotions. This response is provided as text or audio in an emotionally appropriate tone, allowing the user to have a more personalized experience.

[0383] For example, if a user asks "How were my grades today?" in a slightly anxious tone, the emotion engine will detect this anxiety. The system will then retrieve the grade data and provide the user with a response that incorporates an encouraging tone. This allows the user not only to obtain information but also to receive mental support, enabling them to move on to the next action with confidence. In this way, the addition of the emotion engine enables a unique implementation that improves the user experience.

[0384] The following describes the processing flow.

[0385] Step 1:

[0386] The device receives voice input from the user. The user speaks into the voice input device, and the device records it in real time via the microphone.

[0387] Step 2:

[0388] The device uses a speech recognition engine to convert the recorded audio into text data. The converted text data is temporarily stored on the device for subsequent analysis.

[0389] Step 3:

[0390] The device uses a natural language processing engine to analyze text data and understand the intent behind the user's questions and requests. This analysis identifies exactly what the user wants to know.

[0391] Step 4:

[0392] The device uses an emotion engine to analyze elements such as voice tone and speed to recognize the user's emotions. This is an important step in taking the user's psychological state into account.

[0393] Step 5:

[0394] The device captures the user's desktop screen and saves the image as a snapshot. This snapshot is necessary to understand the user's current context.

[0395] Step 6:

[0396] The device sends the captured snapshot to the server, preparing it for image data analysis.

[0397] Step 7:

[0398] The server analyzes the snapshot and extracts the information the user is looking for from the image. This includes reading document content and application data.

[0399] Step 8:

[0400] The server combines the analyzed information with the user's intent and emotional data from the terminal to generate an appropriate response. This response incorporates both detailed content and an emotionally sensitive tone.

[0401] Step 9:

[0402] The server sends the generated response to the terminal. The terminal receives this response and prepares the output for the user.

[0403] Step 10:

[0404] The device uses a speech synthesis engine to convert the generated text responses into speech. Furthermore, it addresses questions by presenting information to the user in a way that reflects appropriate emotions.

[0405] (Example 2)

[0406] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0407] Conventional speech recognition systems could only recognize the user's intent, and struggled to generate responses that took into account emotional changes or the context on the desktop. As a result, users could not receive personalized responses that matched their emotions, leading to a decline in the quality of communication.

[0408] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0409] In this invention, the server includes means for converting voice input into text data, means for recognizing the user's intent and emotions, and means for analyzing images on the desktop screen to extract information. This makes it possible to provide flexible and personalized responses based on the user's intent and emotions.

[0410] "Voice input" is a data format for acquiring information through a user's voice.

[0411] "Text data" refers to data in string format that is converted from voice input.

[0412] "User intent" refers to the purpose or request for which the user is seeking an action or information through voice input.

[0413] "Emotion" refers to the emotional state inferred from the user's voice characteristics.

[0414] A "desktop image" is a snapshot of the visual information displayed on a computer screen.

[0415] A "response" is a message or action provided to the user, generated based on the analyzed information.

[0416] A "speech recognition engine" is a system that provides technology to convert speech input into text data.

[0417] A "natural language processing engine" is a system that provides technology to analyze text data and identify the user's intent.

[0418] One embodiment of this invention is a system that uses speech recognition and natural language processing technologies to precisely analyze user input and generate personalized responses. A specific example of this system is shown below.

[0419] The user provides information to the system via voice input. The terminal receives this voice input and converts the speech into text data using a speech recognition engine. In this process, open-source speech recognition software may be used.

[0420] The device further analyzes the converted text data using a natural language processing engine to identify the user's intent. Commonly available natural language processing models are utilized here. In addition, an emotion engine is used to analyze the user's emotions from the tone and tempo of their voice, preparing to generate an appropriate response.

[0421] The captured desktop screen snapshots are sent by the terminal to the server, where computer vision technology is used to analyze the images. This makes it possible to understand the information the user is requesting and the context of the actions they are performing on the desktop.

[0422] Based on the analyzed user intent, sentiment information, and desktop context, the server generates a response using a generative AI model. This response is delivered as voice or text, customized to the user's emotions and intent.

[0423] For example, if a user asks "What's on my schedule this week?" in an anxious tone, the emotion engine detects that anxiety. The system retrieves the week's calendar information and provides a response that is empathetic to the user's feelings, such as "Your schedule is full this week, but I'm sure you'll have a fulfilling week." An example of a prompt might be something like, "Respond to an anxious user with their schedule for the week in an encouraging tone." Through this process, the user feels better understood and can move on to the next action with confidence.

[0424] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0425] Step 1:

[0426] The user inputs a request to the system by voice. The terminal receives this voice input via its microphone. The input is raw voice data. The terminal activates its speech recognition engine and converts the voice data into digital text data. The output of this step is the string data converted from the voice.

[0427] Step 2:

[0428] The device sends text data to a natural language processing engine to analyze the user's intent. This input is the text data acquired by the device. The natural language processing engine uses parsing algorithms to analyze the context and grammatical structure of the text data and identify the information and actions the user is seeking. The output of this step is data indicating the user's intent.

[0429] Step 3:

[0430] Simultaneously, the device uses an emotion engine to analyze the user's emotions from the audio data. The input is the received audio data, and the emotion engine evaluates elements such as tone, pitch, and tempo of the voice to estimate the user's emotions. The output of this step is information about the identified user emotions.

[0431] Step 4:

[0432] The terminal takes a snapshot of the desktop screen. The input is the currently displayed desktop image. The snapshot is transferred to the server as image data. In this step, the information on the screen is captured for later analysis.

[0433] Step 5:

[0434] The server analyzes the received image data using computer vision technology and extracts useful information from the desktop screen. The input is snapshot image data, and the output is specific information relevant to the user. For example, the server identifies schedule information provided by task management software.

[0435] Step 6:

[0436] The server constructs a response using a generative AI model based on the analyzed intent, sentiment, and screen information. The input is data from steps 2, 3, and 5, which the generative AI model processes as prompts to generate a personalized response. The output of this step is a text or audio message to be presented to the user.

[0437] Step 7:

[0438] The terminal receives responses from the server and provides them to the user as voice or text. The input is the response data generated by the server, and the output is the result directly experienced by the user. By obtaining information and receiving emotionally resonant responses, the user develops a greater affinity with the system.

[0439] (Application Example 2)

[0440] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0441] Conventional voice dialogue systems have the drawback of being monotonous and uniform in their responses, as they do not take into account the user's emotions. As a result, users are often dissatisfied with their interactions with the system, and the quality of the dialogue experience deteriorates. In particular, for personal robots and applications, it is important to appropriately recognize the user's emotions and provide responses accordingly.

[0442] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0443] In this invention, the server includes means for acquiring voice information, means for converting it into text information, means for analyzing the text information to recognize the user's intent and emotions, means for analyzing acquired images to extract information, and means for generating a response based on the extracted information and the user's intent and emotions. This makes it possible to provide flexible and personalized responses that respond to the user's emotions.

[0444] "Audio information" refers to data obtained from the user's spoken words and sounds.

[0445] "Text information" refers to digital data that converts audio information into written form.

[0446] "User intent" refers to the purpose or request that the user wants to convey to the system through voice information.

[0447] "Emotional analysis" is the process of inferring a user's emotions and emotional state from their voice and text information.

[0448] A "display device" is a device that visually displays information from computers and electronic devices.

[0449] A "response" refers to a reply or reaction generated by a system based on the user's intentions and emotions.

[0450] A "speech recognition device" refers to hardware or software technology for converting speech information into text information.

[0451] A "natural language processing system" refers to algorithms and software technologies used to analyze text information and recognize intentions and emotions.

[0452] The system that realizes this application consists of a speech recognition device, a natural language processing device, an emotion analysis device, an image analysis device, and a response generation device. By coordinating these devices, the server analyzes the user's intent and emotions based on the voice information obtained from the user and generates an appropriate response.

[0453] The server first uses a speech recognition device to acquire the user's voice information and converts it into text. This converted text information is then analyzed by a natural language processing device to identify the user's intent. Next, an emotion analyzer infers the user's emotions from the text information. This process utilizes emotion analysis technologies such as IBM Watson Tone Analyzer.

[0454] The acquired intent and emotion information, along with image information from the display device of the user's terminal, is sent to an image analysis device. Necessary information is extracted from the display device's image. For example, the content displayed on the screen is analyzed using the Google Cloud Vision API, and data corresponding to the user's request is extracted.

[0455] Ultimately, the server uses a response generator to produce a response based on the user's intent and emotions. This response is delivered as voice or text, in a tone that matches the user's emotions. This allows the user to have a personalized experience.

[0456] As a concrete example, consider a scenario where a user asks, "What's the latest news?" If sentiment analysis reveals that the user is excited, the system will provide positive and interesting news in an upbeat tone. An example of a prompt using a generative AI model would be, "Considering the user's excitement, please provide positive news."

[0457] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0458] Step 1:

[0459] The user provides voice input. The terminal acquires this voice and inputs it into the speech recognition device. At this point, the voice information is the input data. The server uses the speech recognition device to convert this voice information into text information and outputs it.

[0460] Step 2:

[0461] The server inputs the acquired text information into a natural language processing unit. At this stage, the input is text information. The server performs natural language processing, analyzing the text information to identify the user's intent. The analyzed user intent is then output.

[0462] Step 3:

[0463] The server inputs text information into the sentiment analysis device and performs sentiment analysis. The input is the user's text information, and the result of the sentiment analysis is the user's emotional state. The server outputs this emotional state.

[0464] Step 4:

[0465] The terminal captures a snapshot of the display device's current screen and sends the image information to the server. The server uses an image analysis device to analyze this image information. The input is the image from the display device, and the server extracts the necessary information from it and outputs it.

[0466] Step 5:

[0467] The server receives the output of the natural language processing unit (user intent), the emotion analysis unit (user emotion), and the image analysis unit (necessary information), and inputs them into the response generation unit. Based on this data, the server generates an appropriate response and outputs it. The generated response is sent to the terminal as audio or text and provided to the user.

[0468] Step 6:

[0469] When providing a response to the user, the server inputs prompts into the generating AI model, which then completes and adjusts the response. These prompts are typically in the format of, "Considering the user's excitement, please provide positive news." This process ensures the response aligns with the user's emotions.

[0470] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0471] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0472] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0473] [Third Embodiment]

[0474] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0475] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0476] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0477] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0478] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0479] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0480] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0481] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0482] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0483] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0484] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0485] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0486] To implement this invention, a system is constructed in which a user, a terminal, and a server work together to perform a series of voice-based information processing tasks. This system begins with the user providing information in the form of voice input. When the user gives instructions or asks questions verbally, the terminal captures the audio in real time and converts it into text data using a speech recognition engine. This digitizes the user's voice information, enabling further processing.

[0487] Subsequently, the device uses a natural language processing engine to analyze the transcribed audio information and understand the user's intent. This analysis process involves analyzing the grammatical structure and key elements of the text data to identify the type of information the user is seeking from the system and the necessary actions.

[0488] Next, the device takes a screenshot of the desktop to understand the current context. The acquired image is sent to the server, which extracts information using image analysis algorithms. In this process, specific data points or document sections within the image are identified, and data that aligns with the user's request is extracted.

[0489] The server integrates the extracted information with the results of speech-to-text analysis to generate information useful to the user. This response is formatted in text and, if necessary, converted to speech using speech synthesis technology. The user receives the information output from the terminal as either audio or text, enabling them to smoothly perform necessary desk work.

[0490] For example, if a user asks, "What are the meetings scheduled for today?", the voice input is converted into text data, and subsequent analysis identifies keywords such as "meeting" and "schedule." The server analyzes the display screen of the schedule management application from a screenshot, extracts the relevant schedule information, and responds to the user. In this way, the present invention realizes a rapid and automated information provision process based on voice input.

[0491] The following describes the processing flow.

[0492] Step 1:

[0493] The device receives the user's voice input. When the user speaks into the voice input device, the device captures voice data in real time through the microphone.

[0494] Step 2:

[0495] The device uses a speech recognition engine to convert the captured audio into text data. The converted text data is temporarily stored in memory for subsequent analysis.

[0496] Step 3:

[0497] The device uses a natural language processing engine to analyze text data. Through this analysis, it examines the user's intent and the information they need, and identifies specific information requests.

[0498] Step 4:

[0499] The device takes a screenshot of the user's desktop screen. This screenshot is necessary to understand the current context.

[0500] Step 5:

[0501] The device sends a screenshot it has taken to the server. The server receives this image data and prepares it for processing.

[0502] Step 6:

[0503] The server analyzes the screenshot to extract the specific information the user is looking for. For example, it might identify numerical data or parts of documents within a particular application.

[0504] Step 7:

[0505] The server integrates the analysis results with the user's voice request and generates an appropriate response. This response includes detailed information tailored to the user's request.

[0506] Step 8:

[0507] The server sends the generated response to the terminal. The terminal receives this response and prepares for output.

[0508] Step 9:

[0509] The device uses a speech synthesis engine to convert text responses into speech. It provides information to the user in either voice or text and answers the user's questions.

[0510] (Example 1)

[0511] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0512] Conventional information acquisition systems have struggled to quickly and accurately retrieve necessary information from voice data and provide it to users. This challenge has led to inefficiencies, particularly in information integration and analysis, and has negatively impacted the user experience.

[0513] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0514] In this invention, the server includes means for acquiring voice data, means for converting the voice data into encoded information, and means for analyzing the encoded information and recognizing the user's request. This enables rapid and accurate processing of voice data and the provision of information based on the user's request.

[0515] "Audio data" refers to information that represents sound vibrations in digital format, and is the information collected from sound.

[0516] "Encoded information" refers to information obtained by analyzing audio or image data and converting it into a digitally understandable format.

[0517] A "display device" refers to a hardware device connected to a computer or electronic device that presents visual information to the user.

[0518] "User" refers to a person who operates or uses the system.

[0519] "Means of conversion" refers to the technical process of replacing data in one format with another.

[0520] "Means of analysis" refers to methods or techniques used to break down data and understand its structure and meaning.

[0521] "Means of acquisition" refers to the technical processes and equipment used to obtain data and information.

[0522] "Means of identification" refers to the process of identifying and extracting necessary data from the analyzed information.

[0523] This invention specifically demonstrates how to implement a system for acquiring and providing information to a user through a voice-based interface. First, the user gives instructions or questions by voice, and voice data is acquired through a microphone installed in the terminal. This terminal converts the voice data into encoded information using speech recognition technology. Google Cloud Speech-to-Text or other speech recognition technologies may be used in this process.

[0524] Next, the terminal analyzes the generated encoded information using natural language processing techniques, such as OpenAI's generative AI model, to interpret the user's request. The analysis identifies the type of information the user is seeking. To obtain this information, the terminal captures the current screen state and sends it to the server.

[0525] The server analyzes this screenshot using image analysis techniques, such as OpenCV. The server identifies specific information and data from the screenshot and extracts information that aligns with the user's request. This information is integrated with the analyzed encoded information to generate useful information for the user.

[0526] The server can then format the generated information into text and, if necessary, generate it as speech using speech synthesis technology. As a result, users can receive concise responses from their terminal in either voice or text, allowing them to efficiently continue with the required tasks.

[0527] For example, if a user prompts their device with "Check the preparation status for next week's meeting," this invention can analyze the voice input, retrieve information from the calendar and related applications, and provide the user with accurate information. In this way, the present invention highly streamlines voice-based information processing and user feedback.

[0528] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0529] Step 1:

[0530] The user gives instructions or asks questions to the device using voice commands.

[0531] Input: User's voice

[0532] Specific action: The user speaks into the microphone and says, "Please check the preparation status for next week's meeting."

[0533] Output: Audio signal input to the terminal

[0534] Step 2:

[0535] The device acquires audio data, sends that audio to a speech recognition engine, and converts it into text data.

[0536] Input: Audio signal

[0537] Data processing: The device uses a speech recognition engine to convert the speech signal into text information. Google Cloud Speech-to-Text, for example, is used.

[0538] Specific operation: The recorded audio is passed to the speech recognition software in real time.

[0539] Output: Text data "Please check the preparation status for next week's meeting."

[0540] Step 3:

[0541] The device uses natural language processing to analyze text data and understand user requests.

[0542] Input: Text data

[0543] Data processing: Using generative AI models such as GPT-3, we interpret user requests and intentions.

[0544] Specific actions: Extract keywords such as "meeting," "next week," and "preparation status" to identify the type of information the user is looking for.

[0545] Output: Interpreted request information

[0546] Step 4:

[0547] The device takes a screenshot to obtain an image of the current screen.

[0548] Input: Current screen state

[0549] Specific operation: The device uses the screen capture function to save the screen on the monitor as an image.

[0550] Output: Screenshot

[0551] Step 5:

[0552] The system sends screenshots to the server and extracts the necessary information through image analysis.

[0553] Input: Screenshot

[0554] Data processing: Extract necessary data from screenshots using image analysis techniques such as OpenCV.

[0555] Specific action: Retrieve schedule information for "Next week's meeting" from the calendar app screen.

[0556] Output: Extracted meeting information

[0557] Step 6:

[0558] The server integrates the request information and extracted information to generate a response for the user.

[0559] Input: Request information, Extraction information

[0560] Data processing: Generate appropriate response messages based on integrated information.

[0561] Specific actions: Create a detailed description including the date, time, and location of the meeting.

[0562] Output: Response message

[0563] Step 7:

[0564] The server converts the generated response into text or speech and sends it to the terminal.

[0565] Input: Response message

[0566] Data processing: It is also possible to convert the data into speech using speech synthesis technologies such as Amazon Polly.

[0567] Specific action: Generate the text "The next meeting will be held next Monday at 10:00 in Conference Room A."

[0568] Output: Text or voice response

[0569] Step 8:

[0570] The user receives responses through the terminal and makes decisions to carry out their tasks.

[0571] Input: Text or voice response

[0572] Specific action: The user reviews the information received via audio or text and begins preparing for the meeting.

[0573] Output: User guidelines and plans

[0574] (Application Example 1)

[0575] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0576] Voice-based interactions allow users to obtain a wide range of information simply by using voice input, but the process is typically limited to voice recognition and simple text responses. Such systems lack the ability to provide detailed information about the user's surroundings and specific context. Furthermore, in the home, there is a need for systems that allow users to quickly and easily obtain information based on actual objects and situations.

[0577] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0578] In this invention, the server includes means for obtaining voice data, means for converting the voice data into text form, means for analyzing the text data to understand the user's intent, means for acquiring image data, means for analyzing the acquired image data to detect information, means for creating a response based on the detected information and the user's intent, and means for communicating the response to the user in voice or text format. This enables the user to use voice commands to check their surroundings and obtain specific information.

[0579] "Audio data" refers to information that represents audio acquired from a user in digital format.

[0580] "Text format" refers to string-based information obtained as a result of recognizing and converting audio data.

[0581] "User intent" refers to the content that indicates the user's requests or objectives included in the voice input.

[0582] "Image data" refers to visual information acquired using devices such as cameras, represented in digital format.

[0583] "Means of detecting information" refers to the processing and methods used to analyze and extract useful information from acquired image data.

[0584] "Means of generating responses" refers to the process of generating answers or instructions for the user based on detected information and the user's intent.

[0585] "Means of communicating with the user in audio or text format" refers to methods of audio playback or text display that deliver the generated response content to the user in an easily understandable manner.

[0586] This system allows users to quickly obtain everyday information through voice input. The user provides voice data to the terminal, which then uses voice recognition software to convert the voice data into text. The terminal then analyzes the resulting text data using natural language processing software to understand the user's intent.

[0587] The system includes cameras mounted on terminals and robots, which acquire image data. The acquired image data is processed on a server using image analysis algorithms to detect specific information. This makes it possible to quickly obtain information such as the expiration date of food items in a home refrigerator, if the user asks about it by voice.

[0588] The server generates a response based on the detected information and the user's intent, and this response is communicated to the user through voice playback software or a display. This system allows users to obtain practical information on the spot simply by asking specific questions.

[0589] For example, a user might give a voice command such as, "Tell me the expiration date of the milk in the refrigerator." The system accurately recognizes and processes this command and provides relevant information. By using prompts for the generative AI model, such as, "Check the expiration date of the milk in the refrigerator using image analysis and provide the information via voice," it is possible to improve the accuracy of information retrieval and the user experience.

[0590] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0591] Step 1:

[0592] The user provides audio data to the device. What the user says is captured as digital audio data via the microphone. At this point, the audio data becomes the system's input.

[0593] Step 2:

[0594] The terminal uses speech recognition software to convert the audio data into text. The converted text data is output and used as input for the next processing step. This process yields the audio information as a parseable string.

[0595] Step 3:

[0596] The device uses natural language processing software to analyze text data and understand the user's intent. It analyzes the grammatical structure and keywords of the text data, and outputs the analysis results. This analysis clarifies the user's instructions and requests.

[0597] Step 4:

[0598] The device uses its built-in camera to acquire image data of a specified object or area. This image data is output and sent to a server. One example is obtaining an image of a food item specified by the user from inside a refrigerator.

[0599] Step 5:

[0600] The server receives image data and detects specific information by applying image analysis algorithms. During this process, useful data points are extracted, and the analysis results are output. For example, the expiration date string can be extracted.

[0601] Step 6:

[0602] The server integrates information extracted from the acquired images with the analyzed user intent to generate a response. The generated response is output in text format, and audio playback software is also provided. The response content is provided in a format useful to the user as specific information in response to the user's question.

[0603] Step 7:

[0604] The device communicates the generated response to the user in either voice or text format. Text-to-speech software converts the text to speech, which is then played through the speaker. This final step allows the user to quickly receive actionable information.

[0605] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0606] To implement the present invention, it is necessary to configure a series of systems that include voice input, text conversion, intent recognition, desktop screen snapshots, image analysis, and an emotion engine that grasps the user's emotions. This system grasps the user's intent by analyzing text data obtained from the user's voice input, and further recognizes the emotions contained in that voice using the emotion engine.

[0607] This process begins with the device receiving the user's voice. A speech recognition engine converts the voice into text data, and then a natural language processing engine analyzes the text to identify the user's intent. Furthermore, an emotion engine infers the user's emotions from elements such as tone and tempo of the voice. This allows the system to not only provide information but also respond flexibly in accordance with the user's emotions.

[0608] The captured desktop screen snapshot is sent to the server, where image analysis extracts the information requested by the user. The server, along with the analyzed information, generates a response that takes into account the user's intent and emotions. This response is provided as text or audio in an emotionally appropriate tone, allowing the user to have a more personalized experience.

[0609] For example, if a user asks "How were my grades today?" in a slightly anxious tone, the emotion engine will detect this anxiety. The system will then retrieve the grade data and provide the user with a response that incorporates an encouraging tone. This allows the user not only to obtain information but also to receive mental support, enabling them to move on to the next action with confidence. In this way, the addition of the emotion engine enables a unique implementation that improves the user experience.

[0610] The following describes the processing flow.

[0611] Step 1:

[0612] The device receives voice input from the user. The user speaks into the voice input device, and the device records it in real time via the microphone.

[0613] Step 2:

[0614] The device uses a speech recognition engine to convert the recorded audio into text data. The converted text data is temporarily stored on the device for subsequent analysis.

[0615] Step 3:

[0616] The device uses a natural language processing engine to analyze text data and understand the intent behind the user's questions and requests. This analysis identifies exactly what the user wants to know.

[0617] Step 4:

[0618] The device uses an emotion engine to analyze elements such as voice tone and speed to recognize the user's emotions. This is an important step in taking the user's psychological state into account.

[0619] Step 5:

[0620] The device captures the user's desktop screen and saves the image as a snapshot. This snapshot is necessary to understand the user's current context.

[0621] Step 6:

[0622] The device sends the captured snapshot to the server, preparing it for image data analysis.

[0623] Step 7:

[0624] The server analyzes the snapshot and extracts the information the user is looking for from the image. This includes reading document content and application data.

[0625] Step 8:

[0626] The server combines the analyzed information with the user's intent and emotional data from the terminal to generate an appropriate response. This response incorporates both detailed content and an emotionally sensitive tone.

[0627] Step 9:

[0628] The server sends the generated response to the terminal. The terminal receives this response and prepares the output for the user.

[0629] Step 10:

[0630] The device uses a speech synthesis engine to convert the generated text responses into speech. Furthermore, it addresses questions by presenting information to the user in a way that reflects appropriate emotions.

[0631] (Example 2)

[0632] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0633] Conventional speech recognition systems could only recognize the user's intent, and struggled to generate responses that took into account emotional changes or the context on the desktop. As a result, users could not receive personalized responses that matched their emotions, leading to a decline in the quality of communication.

[0634] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0635] In this invention, the server includes means for converting voice input into text data, means for recognizing the user's intent and emotions, and means for analyzing images on the desktop screen to extract information. This makes it possible to provide flexible and personalized responses based on the user's intent and emotions.

[0636] "Voice input" is a data format for acquiring information through a user's voice.

[0637] "Text data" refers to data in string format that is converted from voice input.

[0638] "User intent" refers to the purpose or request for which the user is seeking an action or information through voice input.

[0639] "Emotion" refers to the emotional state inferred from the user's voice characteristics.

[0640] A "desktop image" is a snapshot of the visual information displayed on a computer screen.

[0641] A "response" is a message or action provided to the user, generated based on the analyzed information.

[0642] A "speech recognition engine" is a system that provides technology to convert speech input into text data.

[0643] A "natural language processing engine" is a system that provides technology to analyze text data and identify the user's intent.

[0644] One embodiment of this invention is a system that uses speech recognition and natural language processing technologies to precisely analyze user input and generate personalized responses. A specific example of this system is shown below.

[0645] The user provides information to the system via voice input. The terminal receives this voice input and converts the speech into text data using a speech recognition engine. In this process, open-source speech recognition software may be used.

[0646] The device further analyzes the converted text data using a natural language processing engine to identify the user's intent. Commonly available natural language processing models are utilized here. In addition, an emotion engine is used to analyze the user's emotions from the tone and tempo of their voice, preparing to generate an appropriate response.

[0647] The captured desktop screen snapshots are sent by the terminal to the server, where computer vision technology is used to analyze the images. This makes it possible to understand the information the user is requesting and the context of the actions they are performing on the desktop.

[0648] Based on the analyzed user intent, sentiment information, and desktop context, the server generates a response using a generative AI model. This response is delivered as voice or text, customized to the user's emotions and intent.

[0649] For example, if a user asks "What's on my schedule this week?" in an anxious tone, the emotion engine detects that anxiety. The system retrieves the week's calendar information and provides a response that is empathetic to the user's feelings, such as "Your schedule is full this week, but I'm sure you'll have a fulfilling week." An example of a prompt might be something like, "Respond to an anxious user with their schedule for the week in an encouraging tone." Through this process, the user feels better understood and can move on to the next action with confidence.

[0650] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0651] Step 1:

[0652] The user inputs a request to the system by voice. The terminal receives this voice input via its microphone. The input is raw voice data. The terminal activates its speech recognition engine and converts the voice data into digital text data. The output of this step is the string data converted from the voice.

[0653] Step 2:

[0654] The device sends text data to a natural language processing engine to analyze the user's intent. This input is the text data acquired by the device. The natural language processing engine uses parsing algorithms to analyze the context and grammatical structure of the text data and identify the information and actions the user is seeking. The output of this step is data indicating the user's intent.

[0655] Step 3:

[0656] Simultaneously, the device uses an emotion engine to analyze the user's emotions from the audio data. The input is the received audio data, and the emotion engine evaluates elements such as tone, pitch, and tempo of the voice to estimate the user's emotions. The output of this step is information about the identified user emotions.

[0657] Step 4:

[0658] The terminal takes a snapshot of the desktop screen. The input is the currently displayed desktop image. The snapshot is transferred to the server as image data. In this step, the information on the screen is captured for later analysis.

[0659] Step 5:

[0660] The server analyzes the received image data using computer vision technology and extracts useful information from the desktop screen. The input is snapshot image data, and the output is specific information relevant to the user. For example, the server identifies schedule information provided by task management software.

[0661] Step 6:

[0662] The server constructs a response using a generative AI model based on the analyzed intent, sentiment, and screen information. The input is data from steps 2, 3, and 5, which the generative AI model processes as prompts to generate a personalized response. The output of this step is a text or audio message to be presented to the user.

[0663] Step 7:

[0664] The terminal receives responses from the server and provides them to the user as voice or text. The input is the response data generated by the server, and the output is the result directly experienced by the user. By obtaining information and receiving emotionally resonant responses, the user develops a greater affinity with the system.

[0665] (Application Example 2)

[0666] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0667] Conventional voice dialogue systems have the drawback of being monotonous and uniform in their responses, as they do not take into account the user's emotions. As a result, users are often dissatisfied with their interactions with the system, and the quality of the dialogue experience deteriorates. In particular, for personal robots and applications, it is important to appropriately recognize the user's emotions and provide responses accordingly.

[0668] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0669] In this invention, the server includes means for acquiring voice information, means for converting it into text information, means for analyzing the text information to recognize the user's intent and emotions, means for analyzing acquired images to extract information, and means for generating a response based on the extracted information and the user's intent and emotions. This makes it possible to provide flexible and personalized responses that respond to the user's emotions.

[0670] "Audio information" refers to data obtained from the user's spoken words and sounds.

[0671] "Text information" refers to digital data that converts audio information into written form.

[0672] "User intent" refers to the purpose or request that the user wants to convey to the system through voice information.

[0673] "Emotional analysis" is the process of inferring a user's emotions and emotional state from their voice and text information.

[0674] A "display device" is a device that visually displays information from computers and electronic devices.

[0675] A "response" refers to a reply or reaction generated by a system based on the user's intentions and emotions.

[0676] A "speech recognition device" refers to hardware or software technology for converting speech information into text information.

[0677] A "natural language processing system" refers to algorithms and software technologies used to analyze text information and recognize intentions and emotions.

[0678] The system that realizes this application consists of a speech recognition device, a natural language processing device, an emotion analysis device, an image analysis device, and a response generation device. By coordinating these devices, the server analyzes the user's intent and emotions based on the voice information obtained from the user and generates an appropriate response.

[0679] The server first uses a speech recognition device to acquire the user's voice information and converts it into text. This converted text information is then analyzed by a natural language processing device to identify the user's intent. Next, an emotion analyzer infers the user's emotions from the text information. This process utilizes emotion analysis technologies such as IBM Watson Tone Analyzer.

[0680] The acquired intent and emotion information, along with image information from the display device of the user's terminal, is sent to an image analysis device. Necessary information is extracted from the display device's image. For example, the content displayed on the screen is analyzed using the Google Cloud Vision API, and data corresponding to the user's request is extracted.

[0681] Ultimately, the server uses a response generator to produce a response based on the user's intent and emotions. This response is delivered as voice or text, in a tone that matches the user's emotions. This allows the user to have a personalized experience.

[0682] As a concrete example, consider a scenario where a user asks, "What's the latest news?" If sentiment analysis reveals that the user is excited, the system will provide positive and interesting news in an upbeat tone. An example of a prompt using a generative AI model would be, "Considering the user's excitement, please provide positive news."

[0683] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0684] Step 1:

[0685] The user provides voice input. The terminal acquires this voice and inputs it into the speech recognition device. At this point, the voice information is the input data. The server uses the speech recognition device to convert this voice information into text information and outputs it.

[0686] Step 2:

[0687] The server inputs the acquired text information into a natural language processing unit. At this stage, the input is text information. The server performs natural language processing, analyzing the text information to identify the user's intent. The analyzed user intent is then output.

[0688] Step 3:

[0689] The server inputs text information into the sentiment analysis device and performs sentiment analysis. The input is the user's text information, and the sentiment analysis yields the user's emotional state. The server outputs this emotional state.

[0690] Step 4:

[0691] The terminal captures a snapshot of the display device's current screen and sends the image information to the server. The server uses an image analysis device to analyze this image information. The input is the image from the display device, and the server extracts the necessary information from it and outputs it.

[0692] Step 5:

[0693] The server receives the output of the natural language processing unit (user intent), the emotion analysis unit (user emotion), and the image analysis unit (necessary information), and inputs them into the response generation unit. Based on this data, the server generates an appropriate response and outputs it. The generated response is sent to the terminal as audio or text and provided to the user.

[0694] Step 6:

[0695] When providing a response to the user, the server inputs prompts into the generating AI model, which then completes and adjusts the response. These prompts are typically in the format of, "Considering the user's excitement, please provide positive news." This process ensures the response aligns with the user's emotions.

[0696] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0697] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0698] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0699] [Fourth Embodiment]

[0700] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0701] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0702] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0703] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0704] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0705] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0706] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0707] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0708] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0709] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0710] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0711] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0712] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0713] To implement this invention, a system is constructed in which a user, a terminal, and a server work together to perform a series of voice-based information processing tasks. This system begins with the user providing information in the form of voice input. When the user gives instructions or asks questions verbally, the terminal captures the audio in real time and converts it into text data using a speech recognition engine. This digitizes the user's voice information, enabling further processing.

[0714] Subsequently, the device uses a natural language processing engine to analyze the transcribed audio information and understand the user's intent. This analysis process involves analyzing the grammatical structure and key elements of the text data to identify the type of information the user is seeking from the system and the necessary actions.

[0715] Next, the device takes a screenshot of the desktop to understand the current context. The acquired image is sent to the server, which extracts information using image analysis algorithms. In this process, specific data points or document sections within the image are identified, and data that aligns with the user's request is extracted.

[0716] The server integrates the extracted information with the results of speech-to-text analysis to generate information useful to the user. This response is formatted in text and, if necessary, converted to speech using speech synthesis technology. The user receives the information output from the terminal as either audio or text, enabling them to smoothly perform necessary desk work.

[0717] For example, if a user asks, "What are the meetings scheduled for today?", the voice input is converted into text data, and subsequent analysis identifies keywords such as "meeting" and "schedule." The server analyzes the display screen of the schedule management application from a screenshot, extracts the relevant schedule information, and responds to the user. In this way, the present invention realizes a rapid and automated information provision process based on voice input.

[0718] The following describes the processing flow.

[0719] Step 1:

[0720] The device receives the user's voice input. When the user speaks into the voice input device, the device captures voice data in real time through the microphone.

[0721] Step 2:

[0722] The device uses a speech recognition engine to convert the captured audio into text data. The converted text data is temporarily stored in memory for subsequent analysis.

[0723] Step 3:

[0724] The device uses a natural language processing engine to analyze text data. Through this analysis, it examines the user's intent and the information they need, and identifies specific information requests.

[0725] Step 4:

[0726] The device takes a screenshot of the user's desktop screen. This screenshot is necessary to understand the current context.

[0727] Step 5:

[0728] The device sends a screenshot it has taken to the server. The server receives this image data and prepares it for processing.

[0729] Step 6:

[0730] The server analyzes the screenshot to extract the specific information the user is looking for. For example, it might identify numerical data or parts of documents within a particular application.

[0731] Step 7:

[0732] The server integrates the analysis results with the user's voice request and generates an appropriate response. This response includes detailed information tailored to the user's request.

[0733] Step 8:

[0734] The server sends the generated response to the terminal. The terminal receives this response and prepares for output.

[0735] Step 9:

[0736] The device uses a speech synthesis engine to convert text responses into speech. It provides information to the user in either voice or text and answers the user's questions.

[0737] (Example 1)

[0738] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0739] Conventional information acquisition systems have struggled to quickly and accurately retrieve necessary information from voice data and provide it to users. This challenge has led to inefficiencies, particularly in information integration and analysis, and has negatively impacted the user experience.

[0740] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0741] In this invention, the server includes means for acquiring voice data, means for converting the voice data into encoded information, and means for analyzing the encoded information and recognizing the user's request. This enables rapid and accurate processing of voice data and the provision of information based on the user's request.

[0742] "Audio data" refers to information that represents sound vibrations in digital format, and is the information collected from sound.

[0743] "Encoded information" refers to information obtained by analyzing audio or image data and converting it into a digitally understandable format.

[0744] A "display device" refers to a hardware device connected to a computer or electronic device that presents visual information to the user.

[0745] "User" refers to a person who operates or uses the system.

[0746] "Means of conversion" refers to the technical process of replacing data in one format with another.

[0747] "Means of analysis" refers to methods or techniques used to break down data and understand its structure and meaning.

[0748] "Means of acquisition" refers to the technical processes and equipment used to obtain data and information.

[0749] "Means of identification" refers to the process of identifying and extracting necessary data from the analyzed information.

[0750] This invention specifically demonstrates how to implement a system for acquiring and providing information to a user through a voice-based interface. First, the user gives instructions or questions by voice, and voice data is acquired through a microphone installed in the terminal. This terminal converts the voice data into encoded information using speech recognition technology. Google Cloud Speech-to-Text or other speech recognition technologies may be used in this process.

[0751] Next, the terminal analyzes the generated encoded information using natural language processing techniques, such as OpenAI's generative AI model, to interpret the user's request. The analysis identifies the type of information the user is seeking. To obtain this information, the terminal captures the current screen state and sends it to the server.

[0752] The server analyzes this screenshot using image analysis techniques, such as OpenCV. The server identifies specific information and data from the screenshot and extracts information that aligns with the user's request. This information is integrated with the analyzed encoded information to generate useful information for the user.

[0753] The server can then format the generated information into text and, if necessary, generate it as speech using speech synthesis technology. As a result, users can receive concise responses from their terminal in either voice or text, allowing them to efficiently continue with the required tasks.

[0754] For example, if a user prompts their device with "Check the preparation status for next week's meeting," this invention can analyze the voice input, retrieve information from the calendar and related applications, and provide the user with accurate information. In this way, the present invention highly streamlines voice-based information processing and user feedback.

[0755] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0756] Step 1:

[0757] The user gives instructions or asks questions to the device using voice commands.

[0758] Input: User's voice

[0759] Specific action: The user speaks into the microphone and says, "Please check the preparation status for next week's meeting."

[0760] Output: Audio signal input to the terminal

[0761] Step 2:

[0762] The device acquires audio data, sends that audio to a speech recognition engine, and converts it into text data.

[0763] Input: Audio signal

[0764] Data processing: The device uses a speech recognition engine to convert the speech signal into text information. Google Cloud Speech-to-Text, for example, is used.

[0765] Specific operation: The recorded audio is passed to the speech recognition software in real time.

[0766] Output: Text data "Please check the preparation status for next week's meeting."

[0767] Step 3:

[0768] The device uses natural language processing to analyze text data and understand user requests.

[0769] Input: Text data

[0770] Data processing: Using generative AI models such as GPT-3, we interpret user requests and intentions.

[0771] Specific actions: Extract keywords such as "meeting," "next week," and "preparation status" to identify the type of information the user is looking for.

[0772] Output: Interpreted request information

[0773] Step 4:

[0774] The device takes a screenshot to obtain an image of the current screen.

[0775] Input: Current screen state

[0776] Specific operation: The device uses the screen capture function to save the screen on the monitor as an image.

[0777] Output: Screenshot

[0778] Step 5:

[0779] The system sends screenshots to the server and extracts the necessary information through image analysis.

[0780] Input: Screenshot

[0781] Data processing: Extract necessary data from screenshots using image analysis techniques such as OpenCV.

[0782] Specific action: Retrieve schedule information for "Next week's meeting" from the calendar app screen.

[0783] Output: Extracted meeting information

[0784] Step 6:

[0785] The server integrates the request information and extracted information to generate a response for the user.

[0786] Input: Request information, Extraction information

[0787] Data processing: Generate appropriate response messages based on integrated information.

[0788] Specific actions: Create a detailed description including the date, time, and location of the meeting.

[0789] Output: Response message

[0790] Step 7:

[0791] The server converts the generated response into text or speech and sends it to the terminal.

[0792] Input: Response message

[0793] Data processing: It is also possible to convert the data into speech using speech synthesis technologies such as Amazon Polly.

[0794] Specific action: Generate the text "The next meeting will be held next Monday at 10:00 in Conference Room A."

[0795] Output: Text or voice response

[0796] Step 8:

[0797] The user receives responses through the terminal and makes decisions to carry out their tasks.

[0798] Input: Text or voice response

[0799] Specific action: The user reviews the information received via audio or text and begins preparing for the meeting.

[0800] Output: User guidelines and plans

[0801] (Application Example 1)

[0802] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0803] Voice-based interactions allow users to obtain a wide range of information simply by using voice input, but the process is typically limited to voice recognition and simple text responses. Such systems lack the ability to provide detailed information about the user's surroundings and specific context. Furthermore, in the home, there is a need for systems that allow users to quickly and easily obtain information based on actual objects and situations.

[0804] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0805] In this invention, the server includes means for obtaining voice data, means for converting the voice data into text form, means for analyzing the text data to understand the user's intent, means for acquiring image data, means for analyzing the acquired image data to detect information, means for creating a response based on the detected information and the user's intent, and means for communicating the response to the user in voice or text format. This enables the user to use voice commands to check their surroundings and obtain specific information.

[0806] "Audio data" refers to information that represents audio acquired from a user in digital format.

[0807] "Text format" refers to string-based information obtained as a result of recognizing and converting audio data.

[0808] "User intent" refers to the content that indicates the user's requests or objectives included in the voice input.

[0809] "Image data" refers to visual information acquired using devices such as cameras, represented in digital format.

[0810] "Means of detecting information" refers to the processing and methods used to analyze and extract useful information from acquired image data.

[0811] "Means of generating responses" refers to the process of generating answers or instructions for the user based on detected information and the user's intent.

[0812] "Means of communicating with the user in audio or text format" refers to methods of audio playback or text display that deliver the generated response content to the user in an easily understandable manner.

[0813] This system allows users to quickly obtain everyday information through voice input. The user provides voice data to the terminal, which then uses voice recognition software to convert the voice data into text. The terminal then analyzes the resulting text data using natural language processing software to understand the user's intent.

[0814] The system includes cameras mounted on terminals and robots, which acquire image data. The acquired image data is processed on a server using image analysis algorithms to detect specific information. This makes it possible to quickly obtain information such as the expiration date of food items in a home refrigerator, if the user asks about it by voice.

[0815] The server generates a response based on the detected information and the user's intent, and this response is communicated to the user through voice playback software or a display. This system allows users to obtain practical information on the spot simply by asking specific questions.

[0816] For example, a user might give a voice command such as, "Tell me the expiration date of the milk in the refrigerator." The system accurately recognizes and processes this command and provides relevant information. By using prompts for the generative AI model, such as, "Check the expiration date of the milk in the refrigerator using image analysis and provide the information via voice," it is possible to improve the accuracy of information retrieval and the user experience.

[0817] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0818] Step 1:

[0819] The user provides audio data to the device. What the user says is captured as digital audio data via the microphone. At this point, the audio data becomes the system's input.

[0820] Step 2:

[0821] The terminal uses speech recognition software to convert the audio data into text. The converted text data is output and used as input for the next processing step. This process yields the audio information as a parseable string.

[0822] Step 3:

[0823] The device uses natural language processing software to analyze text data and understand the user's intent. It analyzes the grammatical structure and keywords of the text data, and outputs the analysis results. This analysis clarifies the user's instructions and requests.

[0824] Step 4:

[0825] The device uses its built-in camera to acquire image data of a specified object or area. This image data is output and sent to a server. One example is obtaining an image of a food item specified by the user from inside a refrigerator.

[0826] Step 5:

[0827] The server receives image data and detects specific information by applying image analysis algorithms. During this process, useful data points are extracted, and the analysis results are output. For example, the expiration date string can be extracted.

[0828] Step 6:

[0829] The server integrates information extracted from the acquired images with the analyzed user intent to generate a response. The generated response is output in text format, and audio playback software is also provided. The response content is provided in a format useful to the user as specific information in response to the user's question.

[0830] Step 7:

[0831] The device communicates the generated response to the user in either voice or text format. Text-to-speech software converts the text to speech, which is then played through the speaker. This final step allows the user to quickly receive actionable information.

[0832] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0833] To implement the present invention, it is necessary to configure a series of systems that include voice input, text conversion, intent recognition, desktop screen snapshots, image analysis, and an emotion engine that grasps the user's emotions. This system grasps the user's intent by analyzing text data obtained from the user's voice input, and further recognizes the emotions contained in that voice using the emotion engine.

[0834] This process begins with the device receiving the user's voice. A speech recognition engine converts the voice into text data, and then a natural language processing engine analyzes the text to identify the user's intent. Furthermore, an emotion engine infers the user's emotions from elements such as tone and tempo of the voice. This allows the system to not only provide information but also respond flexibly in accordance with the user's emotions.

[0835] The captured desktop screen snapshot is sent to the server, where image analysis extracts the information requested by the user. The server, along with the analyzed information, generates a response that takes into account the user's intent and emotions. This response is provided as text or audio in an emotionally appropriate tone, allowing the user to have a more personalized experience.

[0836] For example, if a user asks "How were my grades today?" in a slightly anxious tone, the emotion engine will detect this anxiety. The system will then retrieve the grade data and provide the user with a response that incorporates an encouraging tone. This allows the user not only to obtain information but also to receive mental support, enabling them to move on to the next action with confidence. In this way, the addition of the emotion engine enables a unique implementation that improves the user experience.

[0837] The following describes the processing flow.

[0838] Step 1:

[0839] The device receives voice input from the user. The user speaks into the voice input device, and the device records it in real time via the microphone.

[0840] Step 2:

[0841] The device uses a speech recognition engine to convert the recorded audio into text data. The converted text data is temporarily stored on the device for subsequent analysis.

[0842] Step 3:

[0843] The device uses a natural language processing engine to analyze text data and understand the intent behind the user's questions and requests. This analysis identifies exactly what the user wants to know.

[0844] Step 4:

[0845] The device uses an emotion engine to analyze elements such as voice tone and speed to recognize the user's emotions. This is an important step in taking the user's psychological state into account.

[0846] Step 5:

[0847] The device captures the user's desktop screen and saves the image as a snapshot. This snapshot is necessary to understand the user's current context.

[0848] Step 6:

[0849] The device sends the captured snapshot to the server, preparing it for image data analysis.

[0850] Step 7:

[0851] The server analyzes the snapshot and extracts the information the user is looking for from the image. This includes reading document content and application data.

[0852] Step 8:

[0853] The server combines the analyzed information with the user's intent and emotional data from the terminal to generate an appropriate response. This response incorporates both detailed content and an emotionally sensitive tone.

[0854] Step 9:

[0855] The server sends the generated response to the terminal. The terminal receives this response and prepares the output for the user.

[0856] Step 10:

[0857] The device uses a speech synthesis engine to convert the generated text responses into speech. Furthermore, it addresses questions by presenting information to the user in a way that reflects appropriate emotions.

[0858] (Example 2)

[0859] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0860] Conventional speech recognition systems could only recognize the user's intent, and struggled to generate responses that took into account emotional changes or the context on the desktop. As a result, users could not receive personalized responses that matched their emotions, leading to a decline in the quality of communication.

[0861] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0862] In this invention, the server includes means for converting voice input into text data, means for recognizing the user's intent and emotions, and means for analyzing images on the desktop screen to extract information. This makes it possible to provide flexible and personalized responses based on the user's intent and emotions.

[0863] "Voice input" is a data format for acquiring information through a user's voice.

[0864] "Text data" refers to data in string format that is converted from voice input.

[0865] "User intent" refers to the purpose or request for which the user is seeking an action or information through voice input.

[0866] "Emotion" refers to the emotional state inferred from the user's voice characteristics.

[0867] A "desktop image" is a snapshot of the visual information displayed on a computer screen.

[0868] A "response" is a message or action provided to the user, generated based on the analyzed information.

[0869] A "speech recognition engine" is a system that provides technology to convert speech input into text data.

[0870] A "natural language processing engine" is a system that provides technology to analyze text data and identify the user's intent.

[0871] One embodiment of this invention is a system that uses speech recognition and natural language processing technologies to precisely analyze user input and generate personalized responses. A specific example of this system is shown below.

[0872] The user provides information to the system via voice input. The terminal receives this voice input and converts the speech into text data using a speech recognition engine. In this process, open-source speech recognition software may be used.

[0873] The device further analyzes the converted text data using a natural language processing engine to identify the user's intent. Commonly available natural language processing models are utilized here. In addition, an emotion engine is used to analyze the user's emotions from the tone and tempo of their voice, preparing to generate an appropriate response.

[0874] The captured desktop screen snapshots are sent by the terminal to the server, where computer vision technology is used to analyze the images. This makes it possible to understand the information the user is requesting and the context of the actions they are performing on the desktop.

[0875] Based on the analyzed user intent, sentiment information, and desktop context, the server generates a response using a generative AI model. This response is delivered as voice or text, customized to the user's emotions and intent.

[0876] For example, if a user asks "What's on my schedule this week?" in an anxious tone, the emotion engine detects that anxiety. The system retrieves the week's calendar information and provides a response that is empathetic to the user's feelings, such as "Your schedule is full this week, but I'm sure you'll have a fulfilling week." An example of a prompt might be something like, "Respond to an anxious user with their schedule for the week in an encouraging tone." Through this process, the user feels better understood and can move on to the next action with confidence.

[0877] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0878] Step 1:

[0879] The user inputs a request to the system by voice. The terminal receives this voice input via its microphone. The input is raw voice data. The terminal activates its speech recognition engine and converts the voice data into digital text data. The output of this step is the string data converted from the voice.

[0880] Step 2:

[0881] The device sends text data to a natural language processing engine to analyze the user's intent. This input is the text data acquired by the device. The natural language processing engine uses parsing algorithms to analyze the context and grammatical structure of the text data and identify the information and actions the user is seeking. The output of this step is data indicating the user's intent.

[0882] Step 3:

[0883] Simultaneously, the device uses an emotion engine to analyze the user's emotions from the audio data. The input is the received audio data, and the emotion engine evaluates elements such as tone, pitch, and tempo of the voice to estimate the user's emotions. The output of this step is information about the identified user emotions.

[0884] Step 4:

[0885] The terminal takes a snapshot of the desktop screen. The input is the currently displayed desktop image. The snapshot is transferred to the server as image data. In this step, the information on the screen is captured for later analysis.

[0886] Step 5:

[0887] The server analyzes the received image data using computer vision technology and extracts useful information from the desktop screen. The input is snapshot image data, and the output is specific information relevant to the user. For example, the server identifies schedule information provided by task management software.

[0888] Step 6:

[0889] The server constructs a response using a generative AI model based on the analyzed intent, sentiment, and screen information. The input is data from steps 2, 3, and 5, which the generative AI model processes as prompts to generate a personalized response. The output of this step is a text or audio message to be presented to the user.

[0890] Step 7:

[0891] The terminal receives responses from the server and provides them to the user as voice or text. The input is the response data generated by the server, and the output is the result directly experienced by the user. By obtaining information and receiving emotionally resonant responses, the user develops a greater affinity with the system.

[0892] (Application Example 2)

[0893] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0894] Conventional voice dialogue systems have the drawback of being monotonous and uniform in their responses, as they do not take into account the user's emotions. As a result, users are often dissatisfied with their interactions with the system, and the quality of the dialogue experience deteriorates. In particular, for personal robots and applications, it is important to appropriately recognize the user's emotions and provide responses accordingly.

[0895] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0896] In this invention, the server includes means for acquiring voice information, means for converting it into text information, means for analyzing the text information to recognize the user's intent and emotions, means for analyzing acquired images to extract information, and means for generating a response based on the extracted information and the user's intent and emotions. This makes it possible to provide flexible and personalized responses that respond to the user's emotions.

[0897] "Audio information" refers to data obtained from the user's spoken words and sounds.

[0898] "Text information" refers to digital data that converts audio information into written form.

[0899] "User intent" refers to the purpose or request that the user wants to convey to the system through voice information.

[0900] "Emotional analysis" is the process of inferring a user's emotions and emotional state from their voice and text information.

[0901] A "display device" is a device that visually displays information from computers and electronic devices.

[0902] A "response" refers to a reply or reaction generated by a system based on the user's intentions and emotions.

[0903] A "speech recognition device" refers to hardware or software technology for converting speech information into text information.

[0904] A "natural language processing system" refers to algorithms and software technologies used to analyze text information and recognize intentions and emotions.

[0905] The system that realizes this application consists of a speech recognition device, a natural language processing device, an emotion analysis device, an image analysis device, and a response generation device. By coordinating these devices, the server analyzes the user's intent and emotions based on the voice information obtained from the user and generates an appropriate response.

[0906] The server first uses a speech recognition device to acquire the user's voice information and converts it into text. This converted text information is then analyzed by a natural language processing device to identify the user's intent. Next, an emotion analyzer infers the user's emotions from the text information. This process utilizes emotion analysis technologies such as IBM Watson Tone Analyzer.

[0907] The acquired intent and emotion information, along with image information from the display device of the user's terminal, is sent to an image analysis device. Necessary information is extracted from the display device's image. For example, the content displayed on the screen is analyzed using the Google Cloud Vision API, and data corresponding to the user's request is extracted.

[0908] Ultimately, the server uses a response generator to produce a response based on the user's intent and emotions. This response is delivered as voice or text, in a tone that matches the user's emotions. This allows the user to have a personalized experience.

[0909] As a concrete example, consider a scenario where a user asks, "What's the latest news?" If sentiment analysis reveals that the user is excited, the system will provide positive and interesting news in an upbeat tone. An example of a prompt using a generative AI model would be, "Considering the user's excitement, please provide positive news."

[0910] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0911] Step 1:

[0912] The user provides voice input. The terminal acquires this voice and inputs it into the speech recognition device. At this point, the voice information is the input data. The server uses the speech recognition device to convert this voice information into text information and outputs it.

[0913] Step 2:

[0914] The server inputs the acquired text information into a natural language processing unit. At this stage, the input is text information. The server performs natural language processing, analyzing the text information to identify the user's intent. The analyzed user intent is then output.

[0915] Step 3:

[0916] The server inputs text information into the sentiment analysis device and performs sentiment analysis. The input is the user's text information, and the sentiment analysis yields the user's emotional state. The server outputs this emotional state.

[0917] Step 4:

[0918] The terminal captures a snapshot of the display device's current screen and sends the image information to the server. The server uses an image analysis device to analyze this image information. The input is the image from the display device, and the server extracts the necessary information from it and outputs it.

[0919] Step 5:

[0920] The server receives the output of the natural language processing unit (user intent), the emotion analysis unit (user emotion), and the image analysis unit (necessary information), and inputs them into the response generation unit. Based on this data, the server generates an appropriate response and outputs it. The generated response is sent to the terminal as audio or text and provided to the user.

[0921] Step 6:

[0922] When providing a response to the user, the server inputs prompts into the generating AI model, which then completes and adjusts the response. These prompts are typically in the format of, "Considering the user's excitement, please provide positive news." This process ensures the response aligns with the user's emotions.

[0923] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0924] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0925] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0926] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0927] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0928] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0929] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0930] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0931] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0932] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0933] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0934] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0935] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0936] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0937] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0938] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using this memory.

[0939] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0940] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0941] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0942] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0943] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0944] The following is further disclosed regarding the embodiments described above.

[0945] (Claim 1)

[0946] A means of acquiring voice input,

[0947] means for converting the aforementioned voice input into text data,

[0948] A means for analyzing the aforementioned text data and recognizing the user's intent,

[0949] A means of obtaining an image of the desktop screen,

[0950] A method for analyzing the acquired desktop screen image and extracting information,

[0951] A means of generating a response based on extracted information and user intent,

[0952] Means for providing the aforementioned response to the user as audio or text,

[0953] A system that includes this.

[0954] (Claim 2)

[0955] The system according to claim 1, further comprising means for converting the voice input into text data using a voice recognition engine.

[0956] (Claim 3)

[0957] The system according to claim 1, further comprising means for analyzing the text data using a natural language processing engine.

[0958] "Example 1"

[0959] (Claim 1)

[0960] Means for acquiring audio data,

[0961] means for converting the aforementioned audio data into encoded information,

[0962] A means for analyzing the encoded information and recognizing the user's request,

[0963] Means for acquiring an image from a display device,

[0964] A means of analyzing the image of the acquired display device to identify information,

[0965] A means of generating a response based on identified information and user requests,

[0966] Means for providing the aforementioned response to the user as audio or encoded information,

[0967] A system that includes this.

[0968] (Claim 2)

[0969] The system according to claim 1, further comprising means for converting the audio data into encoded information using speech conversion technology.

[0970] (Claim 3)

[0971] The system according to claim 1, further comprising means for analyzing the encoded information using text analysis technology.

[0972] "Application Example 1"

[0973] (Claim 1)

[0974] Means of obtaining audio data,

[0975] Means for converting the aforementioned audio data into text format,

[0976] A means for analyzing the aforementioned text data and understanding the user's intent,

[0977] Means for acquiring image data,

[0978] A means of analyzing acquired image data to detect information,

[0979] Means for creating a response based on detected information and user intent,

[0980] Means for conveying the aforementioned response to the user in audio or text format,

[0981] A device that includes this.

[0982] (Claim 2)

[0983] The apparatus according to claim 1, further comprising means for converting the voice data into text form using voice recognition software.

[0984] (Claim 3)

[0985] The apparatus according to claim 1, further comprising means for analyzing the text data using natural language processing software.

[0986] "Example 2 of combining an emotion engine"

[0987] (Claim 1)

[0988] A means of acquiring voice input,

[0989] means for converting the aforementioned voice input into text data,

[0990] A means for analyzing the aforementioned text data and recognizing the user's intent,

[0991] Methods for analyzing emotions from voice input,

[0992] A means of obtaining an image of the desktop screen,

[0993] A method for analyzing the acquired desktop screen image and extracting information,

[0994] A means for generating a response based on extracted information and the user's intent and emotions,

[0995] Means for providing the aforementioned response to the user as audio or text,

[0996] A system that includes this.

[0997] (Claim 2)

[0998] The system according to claim 1, further comprising means for converting the voice input into text data using a voice recognition engine.

[0999] (Claim 3)

[1000] The system according to claim 1, further comprising means for analyzing the text data using a natural language processing engine.

[1001] "Application example 2 when combining with an emotional engine"

[1002] (Claim 1)

[1003] Means for acquiring audio information,

[1004] Means for converting the aforementioned audio information into text information,

[1005] A means for analyzing the aforementioned text information and recognizing the user's intent,

[1006] Means for performing emotion analysis,

[1007] Means for acquiring an image from a display device,

[1008] A means for analyzing the image of the acquired display device and extracting information,

[1009] A means of generating a response based on extracted information and the user's intentions and emotions,

[1010] Means for providing the aforementioned response to the user as audio or text,

[1011] A system that includes this.

[1012] (Claim 2)

[1013] The system according to claim 1, further comprising means for converting the voice information into text information using a voice recognition device.

[1014] (Claim 3)

[1015] The system according to claim 1, further comprising means for analyzing the text information using a natural language processing device. [Explanation of symbols]

[1016] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of acquiring voice input, means for converting the aforementioned voice input into text data, A means for analyzing the aforementioned text data and recognizing the user's intent, A means of obtaining an image of the desktop screen, A method for analyzing the acquired desktop screen image and extracting information, A means of generating a response based on extracted information and user intent, Means for providing the aforementioned response to the user as audio or text, A system that includes this.

2. The system according to claim 1, further comprising means for converting the voice input into text data using a voice recognition engine.

3. The system according to claim 1, further comprising means for analyzing the text data using a natural language processing engine.

Images