system

Abstract

We provide the system. [Solution] An input means for collecting voice information from the user, Processing means for preprocessing the aforementioned audio information and extracting emotion data, A generation means for analyzing the aforementioned emotional data and generating a response based on the user's emotional state, Output means for outputting the aforementioned response to the user, A coordination means that shares the aforementioned emotional data among multiple devices and operates in a coordinated manner with each other, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] With the diversification of digital devices, users are facing the complexity of operation and setting of each device, making smooth use difficult. In addition, the lack of emotional connection in digital interfaces contributes to reducing the user experience. Therefore, there is a need for a technology that empathizes with users' emotions and provides natural and emotional interactions.

Means for Solving the Problems

[0005] This invention provides a system that collects voice information from users, preprocesses it, and analyzes the emotional data to generate responses based on the user's emotional state. Furthermore, by sharing emotional data among multiple devices over a network and enabling these devices to operate in coordination, it becomes possible to provide users with a comfortable and emotional experience. This system achieves a multidimensional user interface by using not only voice output but also visual display.

[0006] "Input means" refers to devices or components used to collect information from users, and are responsible for acquiring data such as audio and visual information.

[0007] "Processing means" refers to devices or systems that perform preprocessing on acquired raw data and convert it into an analyzable format, including noise reduction and feature extraction.

[0008] "Generation means" refers to devices or programs that create responses to the user based on analyzed emotional data, and may include audio or visual feedback.

[0009] "Output means" refers to devices or systems that convey the generated response to the user, and provide information using speakers or displays.

[0010] "Coordination means" refers to devices or software that share emotional data among multiple devices and control their actions in a coordinated manner, synchronizing information through network communication. [Brief explanation of the drawing]

[0011] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3]This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0012] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0013] First, let's explain the terminology used in the following explanation.

[0014] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0015] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0016] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0017] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0019] [First Embodiment]

[0020] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0021] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0022] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0024] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0025] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0026] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0027] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0028] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0029] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0030] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0031] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0032] This invention introduces a configuration in which multiple devices work in cooperation to implement a system including an emotion-empathetic AI agent. The system is primarily composed of interactions between three parties: a terminal, a server, and a user.

[0033] The device first collects voice information from the user via the microphone. The collected voice data is immediately preprocessed, undergoing noise reduction and volume normalization. This preprocessing makes the voice data clean input data that can be analyzed by machine learning models. Typically, feature extraction such as Mel-frequency cepstrum coefficients (MFCCs) is performed.

[0034] The pre-processed audio data is analyzed by an emotion analysis algorithm within the device. This algorithm utilizes a deep learning model to infer and classify the user's emotions from their voice tone and word choice. Emotion determination is performed in real time, and a class label (e.g., joy, sadness, surprise) is assigned immediately.

[0035] The analyzed emotion data is sent to a server. The server aggregates the received data and synchronizes it with other devices on the network. This allows multiple devices to operate based on unified information, enabling coordinated operation. For example, if a device detects user stress, the server transmits that information to all related devices and instructs them to change their settings.

[0036] The terminal generates a voice response according to instructions received from the server. This process involves matching it with emotional data to synthesize voice with appropriate tone and content. The generated voice is immediately presented to the user through the speaker, and related information may also be displayed on the screen.

[0037] As a concrete example, consider a scenario where a user asks the device, "Tell me today's schedule." The device inputs the voice and analyzes it to determine that the user is fatigued. Based on this information, the server instructs other devices to dim the lighting slightly and create a more comfortable environment. The device then generates and provides an empathetic response such as, "That's all for today's work. I'll play some relaxing music now."

[0038] Thus, the present invention realizes a concrete form of a system that takes user emotions into consideration and enables natural and seamless interaction.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The device collects voice information from the user through its microphone. The voice input process begins when the user speaks into the device, and voice data is recorded.

[0042] Step 2:

[0043] The terminal performs preprocessing on the acquired audio data. This preprocessing includes noise reduction, sampling rate adjustment, and volume normalization, which prepares the data for analysis.

[0044] Step 3:

[0045] The terminal extracts features from the pre-processed audio data. Here, it calculates Mel-frequency cepstrum coefficients (MFCCs), etc., to quantify the characteristics of the audio signal.

[0046] Step 4:

[0047] The device uses a deep learning model to analyze the user's emotions based on extracted features. Emotions are categorized into states such as joy, sadness, and anger, and output as probability values.

[0048] Step 5:

[0049] The device sends the analyzed emotion data to the server. Here, the emotion information is shared with other network devices via an API.

[0050] Step 6:

[0051] The server aggregates the received emotional data and sends coordinated action instructions to all relevant devices on the network. These instructions include changing device settings and determining the best way to respond to the user.

[0052] Step 7:

[0053] The terminal generates a voice response to the user based on instructions from the server. It determines the content of the response using natural language and tone, taking into account emotional information.

[0054] Step 8:

[0055] The device delivers the generated voice response to the user through its speaker. It also uses the display to provide visual information as needed, complementing the user experience.

[0056] Through these steps, users can experience an interactive experience that resonates with their emotions.

[0057] (Example 1)

[0058] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0059] In modern information processing systems, appropriately identifying user emotions and providing interactive responses based on those emotions is a challenging task. Furthermore, this must be done in real time and with consistent responses across multiple terminals. This invention aims to solve these problems and provide a system that enables natural interactions that respond to user emotions.

[0060] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0061] In this invention, the server includes means for acquiring audio signals from a user, signal processing means for preprocessing the audio signals and extracting feature quantities, and analysis means for analyzing and classifying the user's emotions using the feature quantities. This makes it possible to provide appropriate responses based on the user's emotions.

[0062] An "input means" is a device that acquires audio signals from the user.

[0063] A "signal processing means" is a device for preprocessing acquired audio signals and extracting feature quantities.

[0064] An "analysis tool" is a device that analyzes and classifies user emotions using pre-processed features.

[0065] "Synchronization means" refers to a system function for sharing and integrating classified emotion data across multiple devices.

[0066] A "generation means" is a device for generating and providing responses based on user emotional data.

[0067] "Output means" refers to a device for conveying the generated response to the user audibly or visually.

[0068] The embodiments for carrying out the present invention will be described below.

[0069] The device acquires the user's voice signal using a highly sensitive microphone. This voice signal is converted to a digital format and processed using a noise reduction filter and gain control. Specifically, DSP (Digital Signal Processing) technology is used for noise reduction to ensure clear voice quality. Furthermore, Mel-frequency cepstrum coefficients (MFCCs) are extracted to obtain characteristic features of the voice.

[0070] The server receives feature data sent from the terminal and performs sentiment analysis using a deep learning algorithm. Here, emotional states are classified into classes such as "joy," "sadness," and "surprise." This analysis utilizes a generative AI model specifically designed for sentiment analysis.

[0071] The analyzed emotional data is synchronized with other related devices by the server. This synchronization allows multiple devices to understand the same user's emotional state and respond consistently. Based on this data, the server sends commands to each device for environmental adjustments and response generation.

[0072] The terminal generates voice responses based on instructions from the server. In speech synthesis, a tone and content that takes emotion into account are combined, and the resulting voice is delivered to the user through the speaker. Visual feedback may also be displayed on the screen as relevant information.

[0073] As a concrete example, consider a scenario where a user asks the device, "Tell me what my schedule is for today." The device receives the voice and analyzes the user's emotion as "fatigue." Based on this information, the server instructs other devices to adjust the lighting to a relaxing brightness. The device then generates an empathetic response such as, "That's all for today's work. I'll play some relaxing music now."

[0074] Example of a prompt:

[0075] "Analyze the user's emotions from their voice and suggest appropriate responses."

[0076] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0077] Step 1:

[0078] The device acquires audio signals from the user through the microphone. The input audio signal is raw digital data. This data is then subjected to noise reduction and volume normalization processing. Noise reduction filters out background noise to make the audio clearer. Volume normalization unifies the volume levels of the audio signals to maintain consistent quality. The output is clean audio data with reduced noise and unified volume.

[0079] Step 2:

[0080] The device extracts features from pre-processed audio data. The input is audio data that has undergone noise reduction and volume normalization. Mel-frequency cepstrum coefficients (MFCCs) are extracted from this data. MFCCs reflect the spectral characteristics of the audio and can smoothly extract audio features. The output is the features as numerical data that can be analyzed by a deep learning model.

[0081] Step 3:

[0082] The terminal sends extracted features to the server. The server receives them and inputs them into an emotion analysis algorithm. The input is feature data in the form of a numerical sequence. The server uses a deep learning model to analyze emotions from the speech and determines a specific emotion class (e.g., joy, sadness, surprise). The output is a class label indicating the user's emotion.

[0083] Step 4:

[0084] The server shares and synchronizes analyzed sentiment data with other devices on the network. The input is information about class labels and their intensity, which are the results of the sentiment analysis. Based on this information, the server synchronizes the data to ensure consistent operation across all relevant devices. The output is unified sentiment information to achieve a uniform user experience.

[0085] Step 5:

[0086] The terminal generates a voice response based on synchronization information from the server. The input consists of response instruction data and updated emotion information from the server. A speech synthesis engine is used to generate speech with a tone and content that matches the emotion. The output is a voice response that resonates with the user's emotions, played back through the speaker.

[0087] Step 6:

[0088] The device provides the user with generated audio while displaying related information on its screen. Input consists of the voice response and associated visual data. The audio is transmitted to the user through the speaker, and corresponding information is visually presented on the display. Output is the comprehensive user interaction obtained through auditory and visual means.

[0089] (Application Example 1)

[0090] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0091] In recent years, the number of consumer electronic devices and personal robots used in homes has increased, but these devices have not yet developed sufficiently in terms of understanding user emotions and providing appropriate responses. Often, these devices fail to detect when a user is stressed or wants to relax, and are unable to provide personalized responses. This challenge needs to be addressed to improve the quality of user interaction and enable more natural and effective communication.

[0092] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0093] In this invention, the server includes means for preprocessing voice data from the user and extracting feature data, means for analyzing the feature data and generating a response based on the user's emotional state, and means for outputting the generated response to the user. This enables consumer electronic devices to sense the user's emotions in real time and provide responses and environmental adjustments tailored to individual needs.

[0094] A "user" refers to an individual who interacts with a system or machine and provides voice information.

[0095] "Audio data" refers to recordings of human speech acquired through input units such as microphones.

[0096] An "input unit" refers to equipment that includes hardware and software components for collecting audio and visual data.

[0097] A "processing unit" refers to a module that extracts necessary characteristics from collected data and performs preprocessing such as noise reduction and normalization.

[0098] "Feature data" refers to specific parameters extracted from the original audio data that enable sentiment analysis.

[0099] A "generation unit" refers to a module that constructs a response to the user based on the analyzed feature data.

[0100] An "output unit" refers to a device that communicates the generated response to the user in the form of audio or visuals.

[0101] A "coordination unit" refers to a module that synchronizes feature data between multiple systems, ensuring coordinated operation as a whole.

[0102] An "interaction unit" refers to a configuration designed to optimize user interaction and provide individually customized operations and responses.

[0103] "Consumer machinery and equipment" refers to automated machines used by general consumers in homes and other similar settings, and includes robots and smart devices.

[0104] The system for realizing this invention aims to improve the user experience by recognizing the user's emotions and generating a corresponding response. The system mainly consists of an input unit, a processing unit, a generation unit, an output unit, a adjustment unit, and an interaction unit.

[0105] First, the device is equipped with a microphone, which acts as an "input unit" to collect voice data from the user in real time. This voice data is sent to a "processing unit" where noise reduction and voice normalization are performed. Specifically, the voice is converted into text data using the Google® Cloud Speech-to-Text API, and then Mel-frequency cepstrum coefficients (MFCCs) are extracted.

[0106] Next, the server, acting as a "generation unit," analyzes the acquired feature data and uses a deep learning model, such as TENSORFLOW®, to infer emotions. Based on this analysis, it constructs an appropriate response using OpenAI®'s GPT model.

[0107] The generated response is presented to the user via a speaker or display by an "output unit." This allows the user to receive not only an audio response but also visual feedback.

[0108] Furthermore, the adjustment unit plays a role in sharing this emotional data among multiple related devices. For example, when a user feels stressed, the server communicates this to consumer electronics, instructing them to adjust the interior lighting or play relaxing music.

[0109] For example, if a user says, "I was busy today, so I want to relax," the system processes the voice to sense the user's fatigue level and responds, "I'll create a relaxing environment for you." Another example of a prompt is, "When the user uses words that indicate fatigue, generate words to encourage relaxation," enabling natural dialogue that is in line with the user's emotional state.

[0110] This system allows users to enjoy a more personalized experience, and enables machines to become more than just home appliances—they become interactive entities.

[0111] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0112] Step 1:

[0113] The device uses a microphone to input audio data from the user in real time. Since this input audio data contains noise, noise reduction filtering is applied and the volume is normalized to output clean audio data.

[0114] Step 2:

[0115] The terminal extracts Mel-frequency cepstrum coefficients (MFCCs) from the audio data using a processing unit. This feature extraction optimizes the audio data as feature data for sentiment analysis in a deep learning model. The extracted MFCC data is obtained as output.

[0116] Step 3:

[0117] The server receives MFCC data sent from the terminal as input and uses a deep learning model to infer the emotional state. Specifically, it analyzes the voice tone using a TensorFlow-based model and outputs emotional labels such as joy and sadness. These emotional labels are then used as input for the next step.

[0118] Step 4:

[0119] The server uses OpenAI's GPT, an AI model for generating responses, to create appropriate voice responses based on emotion labels as input. It generates response sentences based on prompts and outputs them in text format. For example, if the prompt is "When the user uses words that indicate fatigue, generate words that promote relaxation," a relaxing response will be formed.

[0120] Step 5:

[0121] The terminal converts the response text received from the server into speech data using speech synthesis software and outputs it to the user through the speaker. Additionally, a visual display unit provides user feedback by displaying supplementary information.

[0122] Step 6:

[0123] The server shares emotion labels and response content with other relevant terminals via a coordination unit to ensure the overall system works in a coordinated manner. For example, if the server determines that a user is tired, it instructs other terminals to adjust the environment, such as lighting or music.

[0124] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0125] This invention describes an empathetic AI agent system that incorporates an emotion engine for recognizing user emotions. This system consists of terminal devices, server devices, and the emotion engines installed in them.

[0126] First, the device collects the user's voice information through a microphone. This voice information is incorporated into the system in real time and used as input data for analysis by the emotion engine. Simultaneously, visual information is also acquired using a camera and preprocessed as multidimensional data including the environment.

[0127] Next, the emotion engine installed in the device analyzes the acquired audio and visual information to recognize the user's emotions. This emotion engine uses a deep learning model to identify various emotion categories (e.g., joy, sadness, surprise, etc.) based on audio and visual features. In particular, it achieves highly accurate emotion recognition by fusing visual and audio information.

[0128] The recognized emotion data is then sent to a server. The server aggregates this data and shares it with multiple related devices. Integrating emotion data over the network enables coordination between devices and maintains consistency in how users are interacted with.

[0129] The device receives instructions from the server and generates the most appropriate response for the user based on those instructions. By adapting emotion-sensitive voice responses to the context, it provides users with natural and friendly interaction. Because these voice responses are conveyed to the user in combination with visual displays, a deeper understanding and feedback can be expected.

[0130] As a concrete example, consider a scenario where a user asks the device, "Tell me the weather." The device detects depression from the user's tone of voice and facial expression, and an emotion engine analyzes this. As a result, information is transmitted to other devices via the server, and instructions are sent to change the indoor lighting to a relaxing setting. The device generates an empathetic response such as, "It looks like it's going to rain today. Why not relax with your favorite movie?" and delivers it to the user, providing advice that suits the user's mood.

[0131] Thus, the system in this invention centers around an emotion engine, accurately recognizing the user's emotions and thereby possessing the ability to adjust the real-world environment. Therefore, it can provide a more comfortable and personalized user experience.

[0132] The following describes the processing flow.

[0133] Step 1:

[0134] The system interface is initiated when the user speaks to the device. The user requests, "I want to listen to music that suits my mood right now."

[0135] Step 2:

[0136] The device uses a built-in microphone to acquire voice data from the user. It also uses a camera to capture the user's facial expressions and surrounding environment.

[0137] Step 3:

[0138] The device performs noise reduction and normalization on the collected audio data, while simultaneously processing the visual data to extract features. This converts the audio and image signals into a data format that is easy to analyze.

[0139] Step 4:

[0140] The device inputs pre-processed audio and visual data into the emotion engine. The emotion engine uses deep learning technology to analyze the user's emotions and classify them into categories such as joy, sadness, and surprise.

[0141] Step 5:

[0142] The device sends emotional data obtained from the emotion engine to the server. Here, an API is used to share emotional information with other devices on the network.

[0143] Step 6:

[0144] The server aggregates emotional data and, as needed, sends environmental adjustment instructions to the relevant devices. For example, it might instruct devices to adjust lighting and sound settings appropriately.

[0145] Step 7:

[0146] The device generates a voice response optimized for the user based on instructions received from the server. Using a voice that resonates with the user's emotions, it might suggest, "Today, I'll play some uplifting music."

[0147] Step 8:

[0148] The device delivers the generated voice response to the user through the speaker and simultaneously plays the selected music. If visual display is possible, it displays song information, related videos, etc., on the screen.

[0149] Through these steps, the system understands the user's emotions and provides appropriate feedback and suggestions based on those emotions, resulting in more human-like interactions.

[0150] (Example 2)

[0151] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0152] In user interaction, emotion recognition that relies on a single source of information has limitations in terms of accuracy and flexibility. Therefore, it is necessary to integrate not only audio information but also visual and environmental information to provide natural and empathetic responses that are tailored to the user's emotions and state. Furthermore, there is a need for systems where multiple devices can work together to adjust the user's environment.

[0153] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0154] In this invention, the server includes an input configuration for acquiring voice information from the user, a data processing configuration for preprocessing the voice and visual information and extracting emotion data, and a generation configuration for analyzing the emotion data using a deep learning model and generating a response based on the user's emotional state. This enables more accurate emotion recognition and consistent, effective responses to the user.

[0155] A "user" is a person who provides audio or visual information and operates or utilizes the system.

[0156] An "input configuration" is a collection of devices and sensors used to acquire audio and visual information from the user and incorporate it into the system.

[0157] A "data processing configuration" refers to a technical means for preprocessing acquired audio and visual information and converting it into an analyzable format.

[0158] "Emotional data" refers to digital data extracted from audio and visual information that represents the user's emotional state.

[0159] A "deep learning model" is a machine learning model that uses a multi-layered artificial neural network and is used to identify patterns in data.

[0160] A "generative configuration" is a program or algorithm that creates user-appropriate responses and outputs based on analyzed sentiment data.

[0161] An "output configuration" is a set of devices and interfaces used to present the generated response to the user as audio or visual information.

[0162] "Cooperative configuration" refers to a technical means by which multiple devices or systems work together to achieve a unified objective.

[0163] This invention relates to an empathetic AI agent system that accurately recognizes a user's emotions and responds appropriately. The system consists of a terminal, a server, and multiple devices that work in coordination to provide the information the user requests.

[0164] The device uses a microphone and camera to collect audio information from the user. The microphone captures the user's speech and inputs it into the system as audio data. The camera also captures visual information from the user's facial expressions. This data is sent to the device's emotion engine, where the audio and visual information is preprocessed.

[0165] The emotion engine incorporates a deep learning model and can identify various emotion categories of the user by analyzing audio and visual features. Visual and audio information are fused to achieve highly accurate emotion recognition. Furthermore, the recognized emotion data is transmitted from the device to the server.

[0166] The server aggregates this emotional data and processes it as needed. The server has the ability to integrate emotional data across the network and share it with multiple devices. This facilitates coordination between devices and ensures consistent responses to the user.

[0167] As a concrete example, let's consider a scenario where a user asks their device, "Tell me the weather." In this case, the device detects a depressed mood from the user's tone of voice and facial expression, and analyzes it using an emotion engine. As a result, a notification is sent to other devices via the server, which could, for example, adjust the room lighting to a relaxing setting. The device generates and delivers an empathetic response to the user, such as, "It looks like it's going to rain today. Why not relax with your favorite movie?"

[0168] As an example of a prompt, one might input the instruction "How would you adjust the lighting when the user is feeling down?" to the generative AI model. In this way, by utilizing the emotion engine and collaborative configuration, a more personalized user experience can be provided.

[0169] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0170] Step 1:

[0171] The device collects audio and visual information from the user. It acquires audio data via a microphone and visual data via a camera as input. Specifically, the microphone captures the frequency characteristics of the audio, and the camera extracts facial expressions using a face recognition algorithm. This data undergoes preprocessing steps such as noise reduction and face detection before being prepared as input data for the emotion engine.

[0172] Step 2:

[0173] The emotion engine within the device analyzes pre-processed audio and visual information to generate emotion data. Audio and visual features are input to a deep learning model. Specifically, the model identifies the user's emotions from changes in speech pitch, tone, speed, and facial expressions, classifying them into emotion categories such as joy, sadness, and surprise. The identified emotion data is then retrieved as output.

[0174] Step 3:

[0175] The device sends the analyzed emotion data to the server. The input is the emotion data generated in step 2. Specifically, the data is securely transmitted to the server using an encryption protocol. The server aggregates the received data and prepares it to be shared with other devices as needed.

[0176] Step 4:

[0177] The server executes a process to issue instructions to other related devices based on the received emotion data. The input is the emotion data sent in the previous step. Specifically, the server adjusts the corresponding real-world environment while maintaining data integrity between devices. As an output, for example, commands are issued to change the brightness or color settings of smart lights.

[0178] Step 5:

[0179] The terminal receives instructions from the server and generates responses for the user. Inputs include server instructions and emotion data. Specifically, it uses a speech synthesis system to generate voice responses in a tone appropriate to the emotion. Visual feedback is also provided through screen displays. The response is communicated to the user through both audio and visual means.

[0180] Step 6:

[0181] The user reacts to the generated responses and environmental changes. The user's feedback is then incorporated back into the system as input data for the next cycle. Specifically, the device collects the user's impressions and opinions again as voice and facial expressions, and uses this data as new information. This cycle is repeated, enabling a higher level of personalization.

[0182] (Application Example 2)

[0183] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0184] In modern society, addressing the anxiety and loneliness experienced by the elderly and providing them with a sense of security has become a crucial challenge. Furthermore, accurately capturing the emotional changes of the elderly and providing individualized support is not easy. Conventional systems lacked the flexibility to grasp the diverse emotions of the elderly in real time and respond immediately accordingly. In particular, there was a lack of mechanisms to provide audio and visual feedback that resonated with their emotions.

[0185] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0186] In this invention, the server includes means for collecting audio and visual information from the user, means for preprocessing the audio and visual information and extracting emotional data, and means for analyzing the emotional data and generating a response based on the user's emotional state. This makes it possible to constantly monitor changes in the emotional state of elderly people and provide appropriate support according to their individual emotional state. Specifically, by generating and providing reassuring audio and visual responses to the user, personalized support tailored to individual needs can be realized.

[0187] "Audio information" refers to data used to extract linguistic or emotional characteristics based on the waveform of sounds emitted by the user.

[0188] "Visual information" refers to visual data collected to analyze a user's facial expressions and movements in order to identify their emotional state.

[0189] "Emotional data" refers to a dataset used to express emotions, obtained by analyzing user voice and visual information.

[0190] A "generation method" refers to a device or software that processes emotional data to create the most suitable response for the user and outputs it as audio or visual feedback.

[0191] A "coordination mechanism" is a system that shares emotional data among multiple devices and adjusts their operation to achieve consistent responses and support activities.

[0192] "Output means" refers to devices or systems that communicate generated responses to the user in audio and visual form, thereby enabling interaction.

[0193] "Emotional change" is a concept that captures shifts in a user's emotional state, and is usually recognized through the comparison of analyzed data.

[0194] "A sense of security" refers to a feeling or state that a system provides to a user to promote psychological stability.

[0195] "Individual needs" refer to the unique requirements and expectations of each user, and serve as the criteria for the support the system provides.

[0196] The system for implementing this invention is an empathetic care support system that utilizes the user's voice and visual information. The terminal uses a Logitech C920 camera and a Blue Yeti microphone to collect the user's voice and visual information in real time. This collected data is preprocessed within the system, and emotional data is extracted using a generative AI model.

[0197] The server uses EmotionRecognizer and OpenCV, implemented in Python, to analyze this emotional data with high accuracy. In particular, a deep learning model combining audio and visual features allows for detailed classification of the user's emotional state and generates the optimal response to provide reassurance.

[0198] The generated responses are displayed visually along with audio output, providing a natural interaction for the user. This makes it possible to provide a sense of security to the elderly in emergencies or when they are feeling anxious. Furthermore, this emotional data can be shared among other caregivers and devices, enabling collaborative responses and the creation of a more personalized support system.

[0199] For example, if a user is feeling lonely, the system can detect that emotion and generate a voice message such as, "Hello, would you like to go for a walk? The weather is lovely," to help lift the user's spirits.

[0200] Examples of prompt statements for a generative AI model are as follows:

[0201] User: "I'm feeling a bit lonely today."

[0202] Emotion Engine: Analyze user's tone and facial expression for emotional state.

[0203] This specification will provide a method for specifically implementing the invention, making it possible to realize a system that provides a sense of security in a way that is sensitive to the user's emotions.

[0204] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0205] Step 1:

[0206] The device uses a Logitech C920 camera to capture the user's facial expressions and a Blue Yeti microphone to capture the user's voice. The input consists of the collected raw audio and image data. This data is taken into the system in real time. Here, data preprocessing is performed, particularly noise filtering and image clarification.

[0207] Step 2:

[0208] The server receives pre-processed audio and visual data and analyzes it using a generative AI model. The input is pre-processed data, and the output is emotion data indicating the user's emotions. Specifically, a deep learning algorithm analyzes the tone of voice and facial features to determine emotions such as joy, sadness, and surprise.

[0209] Step 3:

[0210] The server generates the most appropriate response for the user's emotional state based on analyzed emotion data. The input is emotion data, and the output is data for voice messages and visual displays. It utilizes a generative AI model to generate emotionally resonant messages. In this process, prompts are used to create user-friendly responses.

[0211] Step 4:

[0212] The terminal outputs the generated voice response through its speaker and displays visual feedback on its screen. The input is response data sent from the server. The user receives support from the system through specific voice and video interactions. This allows for the provision of direct reassurance to the user.

[0213] Step 5:

[0214] The server shares emotional data with other supporters and devices, coordinating the entire system to work together. Inputs are emotional data and generated response information, while outputs are instructions to other devices and data sharing. This creates a consistent support system overall, providing a relaxed environment for the elderly.

[0215] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0216] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0217] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0218] [Second Embodiment]

[0219] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0220] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0221] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0222] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0223] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0224] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0225] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0226] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0227] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0228] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0229] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0230] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0231] This invention introduces a configuration in which multiple devices work in cooperation to implement a system including an emotion-empathetic AI agent. The system is primarily composed of interactions between three parties: a terminal, a server, and a user.

[0232] The device first collects voice information from the user via the microphone. The collected voice data is immediately preprocessed, undergoing noise reduction and volume normalization. This preprocessing makes the voice data clean input data that can be analyzed by machine learning models. Typically, feature extraction such as Mel-frequency cepstrum coefficients (MFCCs) is performed.

[0233] The pre-processed audio data is analyzed by an emotion analysis algorithm within the device. This algorithm utilizes a deep learning model to infer and classify the user's emotions from their voice tone and word choice. Emotion determination is performed in real time, and a class label (e.g., joy, sadness, surprise) is assigned immediately.

[0234] The analyzed emotion data is sent to a server. The server aggregates the received data and synchronizes it with other devices on the network. This allows multiple devices to operate based on unified information, enabling coordinated operation. For example, if a device detects user stress, the server transmits that information to all related devices and instructs them to change their settings.

[0235] The terminal generates a voice response according to instructions received from the server. This process involves matching it with emotional data to synthesize voice with appropriate tone and content. The generated voice is immediately presented to the user through the speaker, and related information may also be displayed on the screen.

[0236] As a concrete example, consider a scenario where a user asks the device, "Tell me today's schedule." The device inputs the voice and analyzes it to determine that the user is fatigued. Based on this information, the server instructs other devices to dim the lighting slightly and create a more comfortable environment. The device then generates and provides an empathetic response such as, "That's all for today's work. I'll play some relaxing music now."

[0237] Thus, the present invention realizes a concrete form of a system that takes user emotions into consideration and enables natural and seamless interaction.

[0238] The following describes the processing flow.

[0239] Step 1:

[0240] The device collects voice information from the user through its microphone. The voice input process begins when the user speaks into the device, and voice data is recorded.

[0241] Step 2:

[0242] The terminal performs preprocessing on the acquired audio data. This preprocessing includes noise reduction, sampling rate adjustment, and volume normalization, which prepares the data for analysis.

[0243] Step 3:

[0244] The terminal extracts features from the pre-processed audio data. Here, it calculates Mel-frequency cepstrum coefficients (MFCCs), etc., to quantify the characteristics of the audio signal.

[0245] Step 4:

[0246] The device uses a deep learning model to analyze the user's emotions based on extracted features. Emotions are categorized into states such as joy, sadness, and anger, and output as probability values.

[0247] Step 5:

[0248] The device sends the analyzed emotion data to the server. Here, the emotion information is shared with other network devices via an API.

[0249] Step 6:

[0250] The server aggregates the received emotional data and sends coordinated action instructions to all relevant devices on the network. These instructions include changing device settings and determining the best way to respond to the user.

[0251] Step 7:

[0252] The terminal generates a voice response to the user based on instructions from the server. It determines the content of the response using natural language and tone, taking into account emotional information.

[0253] Step 8:

[0254] The device delivers the generated voice response to the user through its speaker. It also uses the display to provide visual information as needed, complementing the user experience.

[0255] Through these steps, users can experience an interactive experience that resonates with their emotions.

[0256] (Example 1)

[0257] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0258] In modern information processing systems, appropriately identifying user emotions and providing interactive responses based on those emotions is a challenging task. Furthermore, this must be done in real time and with consistent responses across multiple terminals. This invention aims to solve these problems and provide a system that enables natural interactions that respond to user emotions.

[0259] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0260] In this invention, the server includes means for acquiring audio signals from a user, signal processing means for preprocessing the audio signals and extracting feature quantities, and analysis means for analyzing and classifying the user's emotions using the feature quantities. This makes it possible to provide appropriate responses based on the user's emotions.

[0261] An "input means" is a device that acquires audio signals from the user.

[0262] A "signal processing means" is a device for preprocessing acquired audio signals and extracting feature quantities.

[0263] An "analysis tool" is a device that analyzes and classifies user emotions using pre-processed features.

[0264] "Synchronization means" refers to a system function for sharing and integrating classified emotion data across multiple devices.

[0265] A "generation means" is a device for generating and providing responses based on user emotional data.

[0266] "Output means" refers to a device for conveying the generated response to the user audibly or visually.

[0267] The embodiments for carrying out the present invention will be described below.

[0268] The device acquires the user's voice signal using a highly sensitive microphone. This voice signal is converted to a digital format and processed using a noise reduction filter and gain control. Specifically, DSP (Digital Signal Processing) technology is used for noise reduction to ensure clear voice quality. Furthermore, Mel-frequency cepstrum coefficients (MFCCs) are extracted to obtain characteristic features of the voice.

[0269] The server receives feature data sent from the terminal and performs sentiment analysis using a deep learning algorithm. Here, emotional states are classified into classes such as "joy," "sadness," and "surprise." This analysis utilizes a generative AI model specifically designed for sentiment analysis.

[0270] The analyzed emotional data is synchronized with other related devices by the server. This synchronization allows multiple devices to understand the same user's emotional state and respond consistently. Based on this data, the server sends commands to each device for environmental adjustments and response generation.

[0271] The terminal generates voice responses based on instructions from the server. In speech synthesis, a tone and content that takes emotion into account are combined, and the resulting voice is delivered to the user through the speaker. Visual feedback may also be displayed on the screen as relevant information.

[0272] As a concrete example, consider a scenario where a user asks the device, "Tell me what my schedule is for today." The device receives the voice and analyzes the user's emotion as "fatigue." Based on this information, the server instructs other devices to adjust the lighting to a relaxing brightness. The device then generates an empathetic response such as, "That's all for today's work. I'll play some relaxing music now."

[0273] Example of a prompt:

[0274] "Analyze the user's emotions from their voice and suggest appropriate responses."

[0275] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0276] Step 1:

[0277] The device acquires audio signals from the user through the microphone. The input audio signal is raw digital data. This data is then subjected to noise reduction and volume normalization processing. Noise reduction filters out background noise to make the audio clearer. Volume normalization unifies the volume levels of the audio signals to maintain consistent quality. The output is clean audio data with reduced noise and unified volume.

[0278] Step 2:

[0279] The terminal extracts feature quantities from the preprocessed voice data. The input is voice data that has undergone noise removal and volume normalization. For this data, the extraction of Mel Frequency Cepstral Coefficients (MFCC) is performed. MFCC reflects the spectral characteristics of the voice and can smoothly extract the feature quantities of the voice. The output is the feature quantities as numerical data that can be analyzed by the deep learning model.

[0280] Step 3:

[0281] The terminal transmits the extracted feature quantities to the server. The server receives it and inputs it into the emotion analysis algorithm. The input is the feature quantity data in the form of a sequence of numbers. The server analyzes the emotion from the voice using the deep learning model and determines a specific emotion class (e.g., joy, sadness, surprise). As the output, a class label indicating the user's emotion is obtained.

[0282] Step 4:

[0283] The server shares and synchronizes the analyzed emotion data with other terminals on the network. The input is the class label, which is the result of the emotion analysis, and information regarding its intensity. Based on this information, the server synchronizes the data so that all related terminals can perform consistent operations. The output is the unified emotion information for realizing a uniform user experience.

[0284] Step 5:

[0285] The terminal generates a voice response based on the synchronization information from the server. The input is the response instruction data from the server and the updated emotion information. Using the voice synthesis engine, voice is generated with a tone and content that match the emotion. The output is the voice response that conforms to the user's emotion and is played through the speaker.

[0286] Step 6:

[0287] The device provides the user with generated audio while displaying related information on its screen. Input consists of the voice response and associated visual data. The audio is transmitted to the user through the speaker, and corresponding information is visually presented on the display. Output is the comprehensive user interaction obtained through auditory and visual means.

[0288] (Application Example 1)

[0289] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0290] In recent years, the number of consumer electronic devices and personal robots used in homes has increased, but these devices have not yet developed sufficiently in terms of understanding user emotions and providing appropriate responses. Often, these devices fail to detect when a user is stressed or wants to relax, and are unable to provide personalized responses. This challenge needs to be addressed to improve the quality of user interaction and enable more natural and effective communication.

[0291] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0292] In this invention, the server includes means for preprocessing voice data from the user and extracting feature data, means for analyzing the feature data and generating a response based on the user's emotional state, and means for outputting the generated response to the user. This enables consumer electronic devices to sense the user's emotions in real time and provide responses and environmental adjustments tailored to individual needs.

[0293] A "user" refers to an individual who interacts with a system or machine and provides voice information.

[0294] "Audio data" refers to recordings of human speech acquired through input units such as microphones.

[0295] An "input unit" refers to equipment that includes hardware and software components for collecting audio and visual data.

[0296] A "processing unit" refers to a module that extracts necessary characteristics from collected data and performs preprocessing such as noise reduction and normalization.

[0297] "Feature data" refers to specific parameters extracted from the original audio data that enable sentiment analysis.

[0298] A "generation unit" refers to a module that constructs a response to the user based on the analyzed feature data.

[0299] An "output unit" refers to a device that communicates the generated response to the user in the form of audio or visuals.

[0300] A "coordination unit" refers to a module that synchronizes feature data between multiple systems, ensuring coordinated operation as a whole.

[0301] An "interaction unit" refers to a configuration designed to optimize user interaction and provide individually customized operations and responses.

[0302] "Consumer machinery and equipment" refers to automated machines used by general consumers in homes and other similar settings, and includes robots and smart devices.

[0303] The system for realizing this invention aims to improve the user experience by recognizing the user's emotions and generating a corresponding response. The system mainly consists of an input unit, a processing unit, a generation unit, an output unit, a adjustment unit, and an interaction unit.

[0304] First, the terminal is equipped with a microphone, which, as an "input unit", collects voice data from the user in real time. This voice data is sent to a "processing unit" where noise removal and voice normalization are performed. Specifically, after converting the voice into text data using the Google Cloud Speech-to-Text API, extraction of Mel-frequency cepstral coefficients (MFCC) is carried out.

[0305] Next, as a "generation unit", the server analyzes the acquired feature data and uses a deep learning model, such as TensorFlow, to infer emotions. Based on this analysis result, an appropriate response is constructed using OpenAI's GPT model.

[0306] The constructed response is presented to the user by the "output unit" through a speaker or a display. As a result, the user can obtain not only a voice response but also visual feedback.

[0307] Furthermore, the adjustment unit has the role of sharing this emotion data among multiple related devices. For example, when the user feels stressed, the server conveys this to consumer mechanical devices and gives instructions to adjust the interior lighting or play relaxing music.

[0308] As a specific example, when the user says "I was busy today, so I want to relax", the system processes the voice to sense the user's fatigue state and responds with "I'll create a relaxing environment for you". Also, as an example of a prompt sentence, by setting "When the user uses words with a tired expression, generate words that encourage relaxation", natural conversation according to the user's emotional state becomes possible.

[0309] With this system, the user can enjoy a more personalized experience, and the machine can become an interactive presence beyond just an appliance.

[0310] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0311] Step 1:

[0312] The device uses a microphone to input audio data from the user in real time. Since this input audio data contains noise, noise reduction filtering is applied and the volume is normalized to output clean audio data.

[0313] Step 2:

[0314] The terminal extracts Mel-frequency cepstrum coefficients (MFCCs) from the audio data using a processing unit. This feature extraction optimizes the audio data as feature data for sentiment analysis in a deep learning model. The extracted MFCC data is obtained as output.

[0315] Step 3:

[0316] The server receives MFCC data sent from the terminal as input and uses a deep learning model to infer the emotional state. Specifically, it analyzes the voice tone using a TensorFlow-based model and outputs emotional labels such as joy and sadness. These emotional labels are then used as input for the next step.

[0317] Step 4:

[0318] The server uses OpenAI's GPT, an AI model for generating responses, to create appropriate voice responses based on emotion labels as input. It generates response sentences based on prompts and outputs them in text format. For example, if the prompt is "When the user uses words that indicate fatigue, generate words that promote relaxation," a relaxing response will be formed.

[0319] Step 5:

[0320] The terminal converts the response text received from the server into speech data using speech synthesis software and outputs it to the user through the speaker. Additionally, a visual display unit provides user feedback by displaying supplementary information.

[0321] Step 6:

[0322] The server shares emotion labels and response content with other relevant terminals via a coordination unit to ensure the overall system works in a coordinated manner. For example, if the server determines that a user is tired, it instructs other terminals to adjust the environment, such as lighting or music.

[0323] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0324] This invention describes an empathetic AI agent system that incorporates an emotion engine for recognizing user emotions. This system consists of terminal devices, server devices, and the emotion engines installed in them.

[0325] First, the device collects the user's voice information through a microphone. This voice information is incorporated into the system in real time and used as input data for analysis by the emotion engine. Simultaneously, visual information is also acquired using a camera and preprocessed as multidimensional data including the environment.

[0326] Next, the emotion engine installed in the device analyzes the acquired audio and visual information to recognize the user's emotions. This emotion engine uses a deep learning model to identify various emotion categories (e.g., joy, sadness, surprise, etc.) based on audio and visual features. In particular, it achieves highly accurate emotion recognition by fusing visual and audio information.

[0327] The recognized emotion data is then sent to a server. The server aggregates this data and shares it with multiple related devices. Integrating emotion data over the network enables coordination between devices and maintains consistency in how users are interacted with.

[0328] The device receives instructions from the server and generates the most appropriate response for the user based on those instructions. By adapting emotion-sensitive voice responses to the context, it provides users with natural and friendly interaction. Because these voice responses are conveyed to the user in combination with visual displays, a deeper understanding and feedback can be expected.

[0329] As a concrete example, consider a scenario where a user asks the device, "Tell me the weather." The device detects depression from the user's tone of voice and facial expression, and an emotion engine analyzes this. As a result, information is transmitted to other devices via the server, and instructions are sent to change the indoor lighting to a relaxing setting. The device generates an empathetic response such as, "It looks like it's going to rain today. Why not relax with your favorite movie?" and delivers it to the user, providing advice that suits the user's mood.

[0330] Thus, the system in this invention centers around an emotion engine, accurately recognizing the user's emotions and thereby possessing the ability to adjust the real-world environment. Therefore, it can provide a more comfortable and personalized user experience.

[0331] The following describes the processing flow.

[0332] Step 1:

[0333] The system interface is initiated when the user speaks to the device. The user requests, "I want to listen to music that suits my mood right now."

[0334] Step 2:

[0335] The device uses a built-in microphone to acquire voice data from the user. It also uses a camera to capture the user's facial expressions and surrounding environment.

[0336] Step 3:

[0337] The device performs noise reduction and normalization on the collected audio data, while simultaneously processing the visual data to extract features. This converts the audio and image signals into a data format that is easy to analyze.

[0338] Step 4:

[0339] The device inputs pre-processed audio and visual data into the emotion engine. The emotion engine uses deep learning technology to analyze the user's emotions and classify them into categories such as joy, sadness, and surprise.

[0340] Step 5:

[0341] The device sends emotional data obtained from the emotion engine to the server. Here, an API is used to share emotional information with other devices on the network.

[0342] Step 6:

[0343] The server aggregates emotional data and, as needed, sends environmental adjustment instructions to the relevant devices. For example, it might instruct devices to adjust lighting and sound settings appropriately.

[0344] Step 7:

[0345] The device generates a voice response optimized for the user based on instructions received from the server. Using a voice that resonates with the user's emotions, it might suggest, "Today, I'll play some uplifting music."

[0346] Step 8:

[0347] The device delivers the generated voice response to the user through the speaker and simultaneously plays the selected music. If visual display is possible, it displays song information, related videos, etc., on the screen.

[0348] Through these steps, the system understands the user's emotions and provides appropriate feedback and suggestions based on those emotions, resulting in more human-like interactions.

[0349] (Example 2)

[0350] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0351] In user interaction, emotion recognition that relies on a single source of information has limitations in terms of accuracy and flexibility. Therefore, it is necessary to integrate not only audio information but also visual and environmental information to provide natural and empathetic responses that are tailored to the user's emotions and state. Furthermore, there is a need for systems where multiple devices can work together to adjust the user's environment.

[0352] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0353] In this invention, the server includes an input configuration for acquiring voice information from the user, a data processing configuration for preprocessing the voice and visual information and extracting emotion data, and a generation configuration for analyzing the emotion data using a deep learning model and generating a response based on the user's emotional state. This enables more accurate emotion recognition and consistent, effective responses to the user.

[0354] A "user" is a person who provides audio or visual information and operates or utilizes the system.

[0355] An "input configuration" is a collection of devices and sensors used to acquire audio and visual information from the user and incorporate it into the system.

[0356] A "data processing configuration" refers to a technical means for preprocessing acquired audio and visual information and converting it into an analyzable format.

[0357] "Emotional data" refers to digital data extracted from audio and visual information that represents the user's emotional state.

[0358] A "deep learning model" is a machine learning model that uses a multi-layered artificial neural network and is used to identify patterns in data.

[0359] A "generative configuration" is a program or algorithm that creates user-appropriate responses and outputs based on analyzed sentiment data.

[0360] An "output configuration" is a set of devices and interfaces used to present the generated response to the user as audio or visual information.

[0361] "Cooperative configuration" refers to a technical means by which multiple devices or systems work together to achieve a unified objective.

[0362] This invention relates to an empathetic AI agent system that accurately recognizes a user's emotions and responds appropriately. The system consists of a terminal, a server, and multiple devices that work in coordination to provide the information the user requests.

[0363] The device uses a microphone and camera to collect audio information from the user. The microphone captures the user's speech and inputs it into the system as audio data. The camera also captures visual information from the user's facial expressions. This data is sent to the device's emotion engine, where the audio and visual information is preprocessed.

[0364] The emotion engine incorporates a deep learning model and can identify various emotion categories of the user by analyzing audio and visual features. Visual and audio information are fused to achieve highly accurate emotion recognition. Furthermore, the recognized emotion data is transmitted from the device to the server.

[0365] The server aggregates this emotional data and processes it as needed. The server has the ability to integrate emotional data across the network and share it with multiple devices. This facilitates coordination between devices and ensures consistent responses to the user.

[0366] As a concrete example, let's consider a scenario where a user asks their device, "Tell me the weather." In this case, the device detects a depressed mood from the user's tone of voice and facial expression, and analyzes it using an emotion engine. As a result, a notification is sent to other devices via the server, which could, for example, adjust the room lighting to a relaxing setting. The device generates and delivers an empathetic response to the user, such as, "It looks like it's going to rain today. Why not relax with your favorite movie?"

[0367] As an example of a prompt, one might input the instruction "How would you adjust the lighting when the user is feeling down?" to the generative AI model. In this way, by utilizing the emotion engine and collaborative configuration, a more personalized user experience can be provided.

[0368] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0369] Step 1:

[0370] The device collects audio and visual information from the user. It acquires audio data via a microphone and visual data via a camera as input. Specifically, the microphone captures the frequency characteristics of the audio, and the camera extracts facial expressions using a face recognition algorithm. This data undergoes preprocessing steps such as noise reduction and face detection before being prepared as input data for the emotion engine.

[0371] Step 2:

[0372] The emotion engine within the device analyzes pre-processed audio and visual information to generate emotion data. Audio and visual features are input to a deep learning model. Specifically, the model identifies the user's emotions from changes in speech pitch, tone, speed, and facial expressions, classifying them into emotion categories such as joy, sadness, and surprise. The identified emotion data is then retrieved as output.

[0373] Step 3:

[0374] The device sends the analyzed emotion data to the server. The input is the emotion data generated in step 2. Specifically, the data is securely transmitted to the server using an encryption protocol. The server aggregates the received data and prepares it to be shared with other devices as needed.

[0375] Step 4:

[0376] The server executes a process to issue instructions to other related devices based on the received emotion data. The input is the emotion data sent in the previous step. Specifically, the server adjusts the corresponding real-world environment while maintaining data integrity between devices. As an output, for example, commands are issued to change the brightness or color settings of smart lights.

[0377] Step 5:

[0378] The terminal receives instructions from the server and generates responses for the user. Inputs include server instructions and emotion data. Specifically, it uses a speech synthesis system to generate voice responses in a tone appropriate to the emotion. Visual feedback is also provided through screen displays. The response is communicated to the user through both audio and visual means.

[0379] Step 6:

[0380] The user reacts to the generated responses and environmental changes. The user's feedback is then incorporated back into the system as input data for the next cycle. Specifically, the device collects the user's impressions and opinions again as voice and facial expressions, and uses this data as new information. This cycle is repeated, enabling a higher level of personalization.

[0381] (Application Example 2)

[0382] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0383] In modern society, addressing the anxiety and loneliness experienced by the elderly and providing them with a sense of security has become a crucial challenge. Furthermore, accurately capturing the emotional changes of the elderly and providing individualized support is not easy. Conventional systems lacked the flexibility to grasp the diverse emotions of the elderly in real time and respond immediately accordingly. In particular, there was a lack of mechanisms to provide audio and visual feedback that resonated with their emotions.

[0384] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0385] In this invention, the server includes means for collecting audio and visual information from the user, means for preprocessing the audio and visual information and extracting emotional data, and means for analyzing the emotional data and generating a response based on the user's emotional state. This makes it possible to constantly monitor changes in the emotional state of elderly people and provide appropriate support according to their individual emotional state. Specifically, by generating and providing reassuring audio and visual responses to the user, personalized support tailored to individual needs can be realized.

[0386] "Audio information" refers to data used to extract linguistic or emotional characteristics based on the waveform of sounds emitted by the user.

[0387] "Visual information" refers to visual data collected to analyze a user's facial expressions and movements in order to identify their emotional state.

[0388] "Emotional data" refers to a dataset used to express emotions, obtained by analyzing user voice and visual information.

[0389] A "generation method" refers to a device or software that processes emotional data to create the most suitable response for the user and outputs it as audio or visual feedback.

[0390] A "coordination mechanism" is a system that shares emotional data among multiple devices and adjusts their operation to achieve consistent responses and support activities.

[0391] "Output means" refers to devices or systems that communicate generated responses to the user in audio and visual form, thereby enabling interaction.

[0392] "Emotional change" is a concept that captures shifts in a user's emotional state, and is usually recognized through the comparison of analyzed data.

[0393] "A sense of security" refers to a feeling or state that a system provides to a user to promote psychological stability.

[0394] "Individual needs" refer to the unique requirements and expectations of each user, and serve as the criteria for the support the system provides.

[0395] The system for implementing this invention is an empathetic care support system that utilizes the user's voice and visual information. The terminal uses a Logitech C920 camera and a Blue Yeti microphone to collect the user's voice and visual information in real time. This collected data is preprocessed within the system, and emotional data is extracted using a generative AI model.

[0396] The server uses EmotionRecognizer and OpenCV, implemented in Python, to analyze this emotional data with high accuracy. In particular, a deep learning model combining audio and visual features allows for detailed classification of the user's emotional state and generates the optimal response to provide reassurance.

[0397] The generated responses are displayed visually along with audio output, providing a natural interaction for the user. This makes it possible to provide a sense of security to the elderly in emergencies or when they are feeling anxious. Furthermore, this emotional data can be shared among other caregivers and devices, enabling collaborative responses and the creation of a more personalized support system.

[0398] For example, if a user is feeling lonely, the system can detect that emotion and generate a voice message such as, "Hello, would you like to go for a walk? The weather is lovely," to help lift the user's spirits.

[0399] Examples of prompt statements for a generative AI model are as follows:

[0400] User: "I'm feeling a bit lonely today."

[0401] Emotion Engine: Analyze user's tone and facial expression for emotional state.

[0402] This specification will provide a method for specifically implementing the invention, making it possible to realize a system that provides a sense of security in a way that is sensitive to the user's emotions.

[0403] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0404] Step 1:

[0405] The device uses a Logitech C920 camera to capture the user's facial expressions and a Blue Yeti microphone to capture the user's voice. The input consists of the collected raw audio and image data. This data is taken into the system in real time. Here, data preprocessing is performed, particularly noise filtering and image clarification.

[0406] Step 2:

[0407] The server receives pre-processed audio and visual data and analyzes it using a generative AI model. The input is pre-processed data, and the output is emotion data indicating the user's emotions. Specifically, a deep learning algorithm analyzes the tone of voice and facial features to determine emotions such as joy, sadness, and surprise.

[0408] Step 3:

[0409] The server generates the most appropriate response for the user's emotional state based on analyzed emotion data. The input is emotion data, and the output is data for voice messages and visual displays. It utilizes a generative AI model to generate emotionally resonant messages. In this process, prompts are used to create user-friendly responses.

[0410] Step 4:

[0411] The terminal outputs the generated voice response through its speaker and displays visual feedback on its screen. The input is response data sent from the server. The user receives support from the system through specific voice and video interactions. This allows for the provision of direct reassurance to the user.

[0412] Step 5:

[0413] The server shares emotional data with other supporters and devices, coordinating the entire system to work together. Inputs are emotional data and generated response information, while outputs are instructions to other devices and data sharing. This creates a consistent support system overall, providing a relaxed environment for the elderly.

[0414] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0415] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0416] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0417] [Third Embodiment]

[0418] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0419] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0420] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0421] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0422] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0423] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0424] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0425] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0426] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0427] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0428] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0429] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0430] This invention introduces a configuration in which multiple devices work in cooperation to implement a system including an emotion-empathetic AI agent. The system is primarily composed of interactions between three parties: a terminal, a server, and a user.

[0431] The device first collects voice information from the user via the microphone. The collected voice data is immediately preprocessed, undergoing noise reduction and volume normalization. This preprocessing makes the voice data clean input data that can be analyzed by machine learning models. Typically, feature extraction such as Mel-frequency cepstrum coefficients (MFCCs) is performed.

[0432] The pre-processed audio data is analyzed by an emotion analysis algorithm within the device. This algorithm utilizes a deep learning model to infer and classify the user's emotions from their voice tone and word choice. Emotion determination is performed in real time, and a class label (e.g., joy, sadness, surprise) is assigned immediately.

[0433] The analyzed emotion data is sent to a server. The server aggregates the received data and synchronizes it with other devices on the network. This allows multiple devices to operate based on unified information, enabling coordinated operation. For example, if a device detects user stress, the server transmits that information to all related devices and instructs them to change their settings.

[0434] The terminal generates a voice response according to instructions received from the server. This process involves matching it with emotional data to synthesize voice with appropriate tone and content. The generated voice is immediately presented to the user through the speaker, and related information may also be displayed on the screen.

[0435] As a concrete example, consider a scenario where a user asks the device, "Tell me today's schedule." The device inputs the voice and analyzes it to determine that the user is fatigued. Based on this information, the server instructs other devices to dim the lighting slightly and create a more comfortable environment. The device then generates and provides an empathetic response such as, "That's all for today's work. I'll play some relaxing music now."

[0436] Thus, the present invention realizes a concrete form of a system that takes user emotions into consideration and enables natural and seamless interaction.

[0437] The following describes the processing flow.

[0438] Step 1:

[0439] The device collects voice information from the user through its microphone. The voice input process begins when the user speaks into the device, and voice data is recorded.

[0440] Step 2:

[0441] The terminal performs preprocessing on the acquired audio data. This preprocessing includes noise reduction, sampling rate adjustment, and volume normalization, which prepares the data for analysis.

[0442] Step 3:

[0443] The terminal extracts features from the pre-processed audio data. Here, it calculates Mel-frequency cepstrum coefficients (MFCCs), etc., to quantify the characteristics of the audio signal.

[0444] Step 4:

[0445] The device uses a deep learning model to analyze the user's emotions based on extracted features. Emotions are categorized into states such as joy, sadness, and anger, and output as probability values.

[0446] Step 5:

[0447] The device sends the analyzed emotion data to the server. Here, the emotion information is shared with other network devices via an API.

[0448] Step 6:

[0449] The server aggregates the received emotional data and sends coordinated action instructions to all relevant devices on the network. These instructions include changing device settings and determining the best way to respond to the user.

[0450] Step 7:

[0451] The terminal generates a voice response to the user based on instructions from the server. It determines the content of the response using natural language and tone, taking into account emotional information.

[0452] Step 8:

[0453] The device delivers the generated voice response to the user through its speaker. It also uses the display to provide visual information as needed, complementing the user experience.

[0454] Through these steps, users can experience an interactive experience that resonates with their emotions.

[0455] (Example 1)

[0456] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0457] In modern information processing systems, appropriately identifying user emotions and providing interactive responses based on those emotions is a challenging task. Furthermore, this must be done in real time and with consistent responses across multiple terminals. This invention aims to solve these problems and provide a system that enables natural interactions that respond to user emotions.

[0458] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0459] In this invention, the server includes means for acquiring audio signals from a user, signal processing means for preprocessing the audio signals and extracting feature quantities, and analysis means for analyzing and classifying the user's emotions using the feature quantities. This makes it possible to provide appropriate responses based on the user's emotions.

[0460] An "input means" is a device that acquires audio signals from the user.

[0461] A "signal processing means" is a device for preprocessing acquired audio signals and extracting feature quantities.

[0462] An "analysis tool" is a device that analyzes and classifies user emotions using pre-processed features.

[0463] "Synchronization means" refers to a system function for sharing and integrating classified emotion data across multiple devices.

[0464] A "generation means" is a device for generating and providing responses based on user emotional data.

[0465] "Output means" refers to a device for conveying the generated response to the user audibly or visually.

[0466] The embodiments for carrying out the present invention will be described below.

[0467] The device acquires the user's voice signal using a highly sensitive microphone. This voice signal is converted to a digital format and processed using a noise reduction filter and gain control. Specifically, DSP (Digital Signal Processing) technology is used for noise reduction to ensure clear voice quality. Furthermore, Mel-frequency cepstrum coefficients (MFCCs) are extracted to obtain characteristic features of the voice.

[0468] The server receives feature data sent from the terminal and performs sentiment analysis using a deep learning algorithm. Here, emotional states are classified into classes such as "joy," "sadness," and "surprise." This analysis utilizes a generative AI model specifically designed for sentiment analysis.

[0469] The analyzed emotional data is synchronized with other related devices by the server. This synchronization allows multiple devices to understand the same user's emotional state and respond consistently. Based on this data, the server sends commands to each device for environmental adjustments and response generation.

[0470] The terminal generates voice responses based on instructions from the server. In speech synthesis, a tone and content that takes emotion into account are combined, and the resulting voice is delivered to the user through the speaker. Visual feedback may also be displayed on the screen as relevant information.

[0471] As a concrete example, consider a scenario where a user asks the device, "Tell me what my schedule is for today." The device receives the voice and analyzes the user's emotion as "fatigue." Based on this information, the server instructs other devices to adjust the lighting to a relaxing brightness. The device then generates an empathetic response such as, "That's all for today's work. I'll play some relaxing music now."

[0472] Example of a prompt:

[0473] "Analyze the user's emotions from their voice and suggest appropriate responses."

[0474] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0475] Step 1:

[0476] The device acquires audio signals from the user through the microphone. The input audio signal is raw digital data. This data is then subjected to noise reduction and volume normalization processing. Noise reduction filters out background noise to make the audio clearer. Volume normalization unifies the volume levels of the audio signals to maintain consistent quality. The output is clean audio data with reduced noise and unified volume.

[0477] Step 2:

[0478] The device extracts features from pre-processed audio data. The input is audio data that has undergone noise reduction and volume normalization. Mel-frequency cepstrum coefficients (MFCCs) are extracted from this data. MFCCs reflect the spectral characteristics of the audio and can smoothly extract audio features. The output is the features as numerical data that can be analyzed by a deep learning model.

[0479] Step 3:

[0480] The terminal sends extracted features to the server. The server receives them and inputs them into an emotion analysis algorithm. The input is feature data in the form of a numerical sequence. The server uses a deep learning model to analyze emotions from the speech and determines a specific emotion class (e.g., joy, sadness, surprise). The output is a class label indicating the user's emotion.

[0481] Step 4:

[0482] The server shares and synchronizes analyzed sentiment data with other devices on the network. The input is information about class labels and their intensity, which are the results of the sentiment analysis. Based on this information, the server synchronizes the data to ensure consistent operation across all relevant devices. The output is unified sentiment information to achieve a uniform user experience.

[0483] Step 5:

[0484] The terminal generates a voice response based on synchronization information from the server. The input consists of response instruction data and updated emotion information from the server. A speech synthesis engine is used to generate speech with a tone and content that matches the emotion. The output is a voice response that resonates with the user's emotions, played back through the speaker.

[0485] Step 6:

[0486] The device provides the user with generated audio while displaying related information on its screen. Input consists of the voice response and associated visual data. The audio is transmitted to the user through the speaker, and corresponding information is visually presented on the display. Output is the comprehensive user interaction obtained through auditory and visual means.

[0487] (Application Example 1)

[0488] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0489] In recent years, the number of consumer electronic devices and personal robots used in homes has increased, but these devices have not yet developed sufficiently in terms of understanding user emotions and providing appropriate responses. Often, these devices fail to detect when a user is stressed or wants to relax, and are unable to provide personalized responses. This challenge needs to be addressed to improve the quality of user interaction and enable more natural and effective communication.

[0490] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0491] In this invention, the server includes means for preprocessing voice data from the user and extracting feature data, means for analyzing the feature data and generating a response based on the user's emotional state, and means for outputting the generated response to the user. This enables consumer electronic devices to sense the user's emotions in real time and provide responses and environmental adjustments tailored to individual needs.

[0492] A "user" refers to an individual who interacts with a system or machine and provides voice information.

[0493] "Audio data" refers to recordings of human speech acquired through input units such as microphones.

[0494] An "input unit" refers to equipment that includes hardware and software components for collecting audio and visual data.

[0495] A "processing unit" refers to a module that extracts necessary characteristics from collected data and performs preprocessing such as noise reduction and normalization.

[0496] "Feature data" refers to specific parameters extracted from the original audio data that enable sentiment analysis.

[0497] A "generation unit" refers to a module that constructs a response to the user based on the analyzed feature data.

[0498] An "output unit" refers to a device that communicates the generated response to the user in the form of audio or visuals.

[0499] A "coordination unit" refers to a module that synchronizes feature data between multiple systems, ensuring coordinated operation as a whole.

[0500] An "interaction unit" refers to a configuration designed to optimize user interaction and provide individually customized operations and responses.

[0501] "Consumer machinery and equipment" refers to automated machines used by general consumers in homes and other similar settings, and includes robots and smart devices.

[0502] The system for realizing this invention aims to improve the user experience by recognizing the user's emotions and generating a corresponding response. The system mainly consists of an input unit, a processing unit, a generation unit, an output unit, a adjustment unit, and an interaction unit.

[0503] First, the device is equipped with a microphone, which acts as an "input unit" to collect voice data from the user in real time. This voice data is sent to a "processing unit" where noise reduction and voice normalization are performed. Specifically, the voice is converted into text data using the Google Cloud Speech-to-Text API, and then Mel-frequency cepstrum coefficients (MFCCs) are extracted.

[0504] Next, the server, acting as a "generation unit," analyzes the acquired feature data and uses a deep learning model, such as TensorFlow, to infer emotions. Based on this analysis, it constructs an appropriate response using OpenAI's GPT model.

[0505] The generated response is presented to the user via a speaker or display by an "output unit." This allows the user to receive not only an audio response but also visual feedback.

[0506] Furthermore, the adjustment unit plays a role in sharing this emotional data among multiple related devices. For example, when a user feels stressed, the server communicates this to consumer electronics, instructing them to adjust the interior lighting or play relaxing music.

[0507] For example, if a user says, "I was busy today, so I want to relax," the system processes the voice to sense the user's fatigue level and responds, "I'll create a relaxing environment for you." Another example of a prompt is, "When the user uses words that indicate fatigue, generate words to encourage relaxation," enabling natural dialogue that is in line with the user's emotional state.

[0508] This system allows users to enjoy a more personalized experience, and enables machines to become more than just home appliances—they become interactive entities.

[0509] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0510] Step 1:

[0511] The device uses a microphone to input audio data from the user in real time. Since this input audio data contains noise, noise reduction filtering is applied and the volume is normalized to output clean audio data.

[0512] Step 2:

[0513] The terminal extracts Mel-frequency cepstrum coefficients (MFCCs) from the audio data using a processing unit. This feature extraction optimizes the audio data as feature data for sentiment analysis in a deep learning model. The extracted MFCC data is obtained as output.

[0514] Step 3:

[0515] The server receives MFCC data sent from the terminal as input and uses a deep learning model to infer the emotional state. Specifically, it analyzes the voice tone using a TensorFlow-based model and outputs emotional labels such as joy and sadness. These emotional labels are then used as input for the next step.

[0516] Step 4:

[0517] The server uses OpenAI's GPT, an AI model for generating responses, to create appropriate voice responses based on emotion labels as input. It generates response sentences based on prompts and outputs them in text format. For example, if the prompt is "When the user uses words that indicate fatigue, generate words that promote relaxation," a relaxing response will be formed.

[0518] Step 5:

[0519] The terminal converts the response text received from the server into speech data using speech synthesis software and outputs it to the user through the speaker. Additionally, a visual display unit provides user feedback by displaying supplementary information.

[0520] Step 6:

[0521] The server shares emotion labels and response content with other relevant terminals via a coordination unit to ensure the overall system works in a coordinated manner. For example, if the server determines that a user is tired, it instructs other terminals to adjust the environment, such as lighting or music.

[0522] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0523] This invention describes an empathetic AI agent system that incorporates an emotion engine for recognizing user emotions. This system consists of terminal devices, server devices, and the emotion engines installed in them.

[0524] First, the device collects the user's voice information through a microphone. This voice information is incorporated into the system in real time and used as input data for analysis by the emotion engine. Simultaneously, visual information is also acquired using a camera and preprocessed as multidimensional data including the environment.

[0525] Next, the emotion engine installed in the device analyzes the acquired audio and visual information to recognize the user's emotions. This emotion engine uses a deep learning model to identify various emotion categories (e.g., joy, sadness, surprise, etc.) based on audio and visual features. In particular, it achieves highly accurate emotion recognition by fusing visual and audio information.

[0526] The recognized emotion data is then sent to a server. The server aggregates this data and shares it with multiple related devices. Integrating emotion data over the network enables coordination between devices and maintains consistency in how users are interacted with.

[0527] The device receives instructions from the server and generates the most appropriate response for the user based on those instructions. By adapting emotion-sensitive voice responses to the context, it provides users with natural and friendly interaction. Because these voice responses are conveyed to the user in combination with visual displays, a deeper understanding and feedback can be expected.

[0528] As a concrete example, consider a scenario where a user asks the device, "Tell me the weather." The device detects depression from the user's tone of voice and facial expression, and an emotion engine analyzes this. As a result, information is transmitted to other devices via the server, and instructions are sent to change the indoor lighting to a relaxing setting. The device generates an empathetic response such as, "It looks like it's going to rain today. Why not relax with your favorite movie?" and delivers it to the user, providing advice that suits the user's mood.

[0529] Thus, the system in this invention centers around an emotion engine, accurately recognizing the user's emotions and thereby possessing the ability to adjust the real-world environment. Therefore, it can provide a more comfortable and personalized user experience.

[0530] The following describes the processing flow.

[0531] Step 1:

[0532] The system interface is initiated when the user speaks to the device. The user requests, "I want to listen to music that suits my mood right now."

[0533] Step 2:

[0534] The device uses a built-in microphone to acquire voice data from the user. It also uses a camera to capture the user's facial expressions and surrounding environment.

[0535] Step 3:

[0536] The device performs noise reduction and normalization on the collected audio data, while simultaneously processing the visual data to extract features. This converts the audio and image signals into a data format that is easy to analyze.

[0537] Step 4:

[0538] The device inputs pre-processed audio and visual data into the emotion engine. The emotion engine uses deep learning technology to analyze the user's emotions and classify them into categories such as joy, sadness, and surprise.

[0539] Step 5:

[0540] The device sends emotional data obtained from the emotion engine to the server. Here, an API is used to share emotional information with other devices on the network.

[0541] Step 6:

[0542] The server aggregates emotional data and, as needed, sends environmental adjustment instructions to the relevant devices. For example, it might instruct devices to adjust lighting and sound settings appropriately.

[0543] Step 7:

[0544] The device generates a voice response optimized for the user based on instructions received from the server. Using a voice that resonates with the user's emotions, it might suggest, "Today, I'll play some uplifting music."

[0545] Step 8:

[0546] The device delivers the generated voice response to the user through the speaker and simultaneously plays the selected music. If visual display is possible, it displays song information, related videos, etc., on the screen.

[0547] Through these steps, the system understands the user's emotions and provides appropriate feedback and suggestions based on those emotions, resulting in more human-like interactions.

[0548] (Example 2)

[0549] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0550] In user interaction, emotion recognition that relies on a single source of information has limitations in terms of accuracy and flexibility. Therefore, it is necessary to integrate not only audio information but also visual and environmental information to provide natural and empathetic responses that are tailored to the user's emotions and state. Furthermore, there is a need for systems where multiple devices can work together to adjust the user's environment.

[0551] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0552] In this invention, the server includes an input configuration for acquiring voice information from the user, a data processing configuration for preprocessing the voice and visual information and extracting emotion data, and a generation configuration for analyzing the emotion data using a deep learning model and generating a response based on the user's emotional state. This enables more accurate emotion recognition and consistent, effective responses to the user.

[0553] A "user" is a person who provides audio or visual information and operates or utilizes the system.

[0554] An "input configuration" is a collection of devices and sensors used to acquire audio and visual information from the user and incorporate it into the system.

[0555] A "data processing configuration" refers to a technical means for preprocessing acquired audio and visual information and converting it into an analyzable format.

[0556] "Emotional data" refers to digital data extracted from audio and visual information that represents the user's emotional state.

[0557] A "deep learning model" is a machine learning model that uses a multi-layered artificial neural network and is used to identify patterns in data.

[0558] A "generative configuration" is a program or algorithm that creates user-appropriate responses and outputs based on analyzed sentiment data.

[0559] An "output configuration" is a set of devices and interfaces used to present the generated response to the user as audio or visual information.

[0560] "Cooperative configuration" refers to a technical means by which multiple devices or systems work together to achieve a unified objective.

[0561] This invention relates to an empathetic AI agent system that accurately recognizes a user's emotions and responds appropriately. The system consists of a terminal, a server, and multiple devices that work in coordination to provide the information the user requests.

[0562] The device uses a microphone and camera to collect audio information from the user. The microphone captures the user's speech and inputs it into the system as audio data. The camera also captures visual information from the user's facial expressions. This data is sent to the device's emotion engine, where the audio and visual information is preprocessed.

[0563] The emotion engine incorporates a deep learning model and can identify various emotion categories of the user by analyzing audio and visual features. Visual and audio information are fused to achieve highly accurate emotion recognition. Furthermore, the recognized emotion data is transmitted from the device to the server.

[0564] The server aggregates this emotional data and processes it as needed. The server has the ability to integrate emotional data across the network and share it with multiple devices. This facilitates coordination between devices and ensures consistent responses to the user.

[0565] As a concrete example, let's consider a scenario where a user asks their device, "Tell me the weather." In this case, the device detects a depressed mood from the user's tone of voice and facial expression, and analyzes it using an emotion engine. As a result, a notification is sent to other devices via the server, which could, for example, adjust the room lighting to a relaxing setting. The device generates and delivers an empathetic response to the user, such as, "It looks like it's going to rain today. Why not relax with your favorite movie?"

[0566] As an example of a prompt, one might input the instruction "How would you adjust the lighting when the user is feeling down?" to the generative AI model. In this way, by utilizing the emotion engine and collaborative configuration, a more personalized user experience can be provided.

[0567] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0568] Step 1:

[0569] The device collects audio and visual information from the user. It acquires audio data via a microphone and visual data via a camera as input. Specifically, the microphone captures the frequency characteristics of the audio, and the camera extracts facial expressions using a face recognition algorithm. This data undergoes preprocessing steps such as noise reduction and face detection before being prepared as input data for the emotion engine.

[0570] Step 2:

[0571] The emotion engine within the device analyzes pre-processed audio and visual information to generate emotion data. Audio and visual features are input to a deep learning model. Specifically, the model identifies the user's emotions from changes in speech pitch, tone, speed, and facial expressions, classifying them into emotion categories such as joy, sadness, and surprise. The identified emotion data is then retrieved as output.

[0572] Step 3:

[0573] The device sends the analyzed emotion data to the server. The input is the emotion data generated in step 2. Specifically, the data is securely transmitted to the server using an encryption protocol. The server aggregates the received data and prepares it to be shared with other devices as needed.

[0574] Step 4:

[0575] The server executes a process to issue instructions to other related devices based on the received emotion data. The input is the emotion data sent in the previous step. Specifically, the server adjusts the corresponding real-world environment while maintaining data integrity between devices. As an output, for example, commands are issued to change the brightness or color settings of smart lights.

[0576] Step 5:

[0577] The terminal receives instructions from the server and generates responses for the user. Inputs include server instructions and emotion data. Specifically, it uses a speech synthesis system to generate voice responses in a tone appropriate to the emotion. Visual feedback is also provided through screen displays. The response is communicated to the user through both audio and visual means.

[0578] Step 6:

[0579] The user reacts to the generated responses and environmental changes. The user's feedback is then incorporated back into the system as input data for the next cycle. Specifically, the device collects the user's impressions and opinions again as voice and facial expressions, and uses this data as new information. This cycle is repeated, enabling a higher level of personalization.

[0580] (Application Example 2)

[0581] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0582] In modern society, addressing the anxiety and loneliness experienced by the elderly and providing them with a sense of security has become a crucial challenge. Furthermore, accurately capturing the emotional changes of the elderly and providing individualized support is not easy. Conventional systems lacked the flexibility to grasp the diverse emotions of the elderly in real time and respond immediately accordingly. In particular, there was a lack of mechanisms to provide audio and visual feedback that resonated with their emotions.

[0583] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0584] In this invention, the server includes means for collecting audio and visual information from the user, means for preprocessing the audio and visual information and extracting emotional data, and means for analyzing the emotional data and generating a response based on the user's emotional state. This makes it possible to constantly monitor changes in the emotional state of elderly people and provide appropriate support according to their individual emotional state. Specifically, by generating and providing reassuring audio and visual responses to the user, personalized support tailored to individual needs can be realized.

[0585] "Audio information" refers to data used to extract linguistic or emotional characteristics based on the waveform of sounds emitted by the user.

[0586] "Visual information" refers to visual data collected to analyze a user's facial expressions and movements in order to identify their emotional state.

[0587] "Emotional data" refers to a dataset used to express emotions, obtained by analyzing user voice and visual information.

[0588] A "generation method" refers to a device or software that processes emotional data to create the most suitable response for the user and outputs it as audio or visual feedback.

[0589] A "coordination mechanism" is a system that shares emotional data among multiple devices and adjusts their operation to achieve consistent responses and support activities.

[0590] "Output means" refers to devices or systems that communicate generated responses to the user in audio and visual form, thereby enabling interaction.

[0591] "Emotional change" is a concept that captures shifts in a user's emotional state, and is usually recognized through the comparison of analyzed data.

[0592] "A sense of security" refers to a feeling or state that a system provides to a user to promote psychological stability.

[0593] "Individual needs" refer to the unique requirements and expectations of each user, and serve as the criteria for the support the system provides.

[0594] The system for implementing this invention is an empathetic care support system that utilizes the user's voice and visual information. The terminal uses a Logitech C920 camera and a Blue Yeti microphone to collect the user's voice and visual information in real time. This collected data is preprocessed within the system, and emotional data is extracted using a generative AI model.

[0595] The server uses EmotionRecognizer and OpenCV, implemented in Python, to analyze this emotional data with high accuracy. In particular, a deep learning model combining audio and visual features allows for detailed classification of the user's emotional state and generates the optimal response to provide reassurance.

[0596] The generated responses are displayed visually along with audio output, providing a natural interaction for the user. This makes it possible to provide a sense of security to the elderly in emergencies or when they are feeling anxious. Furthermore, this emotional data can be shared among other caregivers and devices, enabling collaborative responses and the creation of a more personalized support system.

[0597] For example, if a user is feeling lonely, the system can detect that emotion and generate a voice message such as, "Hello, would you like to go for a walk? The weather is lovely," to help lift the user's spirits.

[0598] Examples of prompt statements for a generative AI model are as follows:

[0599] User: "I'm feeling a bit lonely today."

[0600] Emotion Engine: Analyze user's tone and facial expression for emotional state.

[0601] This specification will provide a method for specifically implementing the invention, making it possible to realize a system that provides a sense of security in a way that is sensitive to the user's emotions.

[0602] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0603] Step 1:

[0604] The device uses a Logitech C920 camera to capture the user's facial expressions and a Blue Yeti microphone to capture the user's voice. The input consists of the collected raw audio and image data. This data is taken into the system in real time. Here, data preprocessing is performed, particularly noise filtering and image clarification.

[0605] Step 2:

[0606] The server receives pre-processed audio and visual data and analyzes it using a generative AI model. The input is pre-processed data, and the output is emotion data indicating the user's emotions. Specifically, a deep learning algorithm analyzes the tone of voice and facial features to determine emotions such as joy, sadness, and surprise.

[0607] Step 3:

[0608] The server generates the most appropriate response for the user's emotional state based on analyzed emotion data. The input is emotion data, and the output is data for voice messages and visual displays. It utilizes a generative AI model to generate emotionally resonant messages. In this process, prompts are used to create user-friendly responses.

[0609] Step 4:

[0610] The terminal outputs the generated voice response through its speaker and displays visual feedback on its screen. The input is response data sent from the server. The user receives support from the system through specific voice and video interactions. This allows for the provision of direct reassurance to the user.

[0611] Step 5:

[0612] The server shares emotional data with other supporters and devices, coordinating the entire system to work together. Inputs are emotional data and generated response information, while outputs are instructions to other devices and data sharing. This creates a consistent support system overall, providing a relaxed environment for the elderly.

[0613] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0614] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0615] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0616] [Fourth Embodiment]

[0617] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0618] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0619] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0620] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0621] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0622] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0623] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0624] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0625] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0626] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0627] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0628] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0629] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0630] This invention introduces a configuration in which multiple devices work in cooperation to implement a system including an emotion-empathetic AI agent. The system is primarily composed of interactions between three parties: a terminal, a server, and a user.

[0631] The device first collects voice information from the user via the microphone. The collected voice data is immediately preprocessed, undergoing noise reduction and volume normalization. This preprocessing makes the voice data clean input data that can be analyzed by machine learning models. Typically, feature extraction such as Mel-frequency cepstrum coefficients (MFCCs) is performed.

[0632] The pre-processed audio data is analyzed by an emotion analysis algorithm within the device. This algorithm utilizes a deep learning model to infer and classify the user's emotions from their voice tone and word choice. Emotion determination is performed in real time, and a class label (e.g., joy, sadness, surprise) is assigned immediately.

[0633] The analyzed emotion data is sent to a server. The server aggregates the received data and synchronizes it with other devices on the network. This allows multiple devices to operate based on unified information, enabling coordinated operation. For example, if a device detects user stress, the server transmits that information to all related devices and instructs them to change their settings.

[0634] The terminal generates a voice response according to instructions received from the server. This process involves matching it with emotional data to synthesize voice with appropriate tone and content. The generated voice is immediately presented to the user through the speaker, and related information may also be displayed on the screen.

[0635] As a concrete example, consider a scenario where a user asks the device, "Tell me today's schedule." The device inputs the voice and analyzes it to determine that the user is fatigued. Based on this information, the server instructs other devices to dim the lighting slightly and create a more comfortable environment. The device then generates and provides an empathetic response such as, "That's all for today's work. I'll play some relaxing music now."

[0636] Thus, the present invention realizes a concrete form of a system that takes user emotions into consideration and enables natural and seamless interaction.

[0637] The following describes the processing flow.

[0638] Step 1:

[0639] The device collects voice information from the user through its microphone. The voice input process begins when the user speaks into the device, and voice data is recorded.

[0640] Step 2:

[0641] The terminal performs preprocessing on the acquired audio data. This preprocessing includes noise reduction, sampling rate adjustment, and volume normalization, which prepares the data for analysis.

[0642] Step 3:

[0643] The terminal extracts features from the pre-processed audio data. Here, it calculates Mel-frequency cepstrum coefficients (MFCCs), etc., to quantify the characteristics of the audio signal.

[0644] Step 4:

[0645] The device uses a deep learning model to analyze the user's emotions based on extracted features. Emotions are categorized into states such as joy, sadness, and anger, and output as probability values.

[0646] Step 5:

[0647] The device sends the analyzed emotion data to the server. Here, the emotion information is shared with other network devices via an API.

[0648] Step 6:

[0649] The server aggregates the received emotional data and sends coordinated action instructions to all relevant devices on the network. These instructions include changing device settings and determining the best way to respond to the user.

[0650] Step 7:

[0651] The terminal generates a voice response to the user based on instructions from the server. It determines the content of the response using natural language and tone, taking into account emotional information.

[0652] Step 8:

[0653] The device delivers the generated voice response to the user through its speaker. It also uses the display to provide visual information as needed, complementing the user experience.

[0654] Through these steps, users can experience an interactive experience that resonates with their emotions.

[0655] (Example 1)

[0656] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0657] In modern information processing systems, appropriately identifying user emotions and providing interactive responses based on those emotions is a challenging task. Furthermore, this must be done in real time and with consistent responses across multiple terminals. This invention aims to solve these problems and provide a system that enables natural interactions that respond to user emotions.

[0658] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0659] In this invention, the server includes means for acquiring audio signals from a user, signal processing means for preprocessing the audio signals and extracting feature quantities, and analysis means for analyzing and classifying the user's emotions using the feature quantities. This makes it possible to provide appropriate responses based on the user's emotions.

[0660] An "input means" is a device that acquires audio signals from the user.

[0661] A "signal processing means" is a device for preprocessing acquired audio signals and extracting feature quantities.

[0662] An "analysis tool" is a device that analyzes and classifies user emotions using pre-processed features.

[0663] "Synchronization means" refers to a system function for sharing and integrating classified emotion data across multiple devices.

[0664] A "generation means" is a device for generating and providing responses based on user emotional data.

[0665] "Output means" refers to a device for conveying the generated response to the user audibly or visually.

[0666] The embodiments for carrying out the present invention will be described below.

[0667] The device acquires the user's voice signal using a highly sensitive microphone. This voice signal is converted to a digital format and processed using a noise reduction filter and gain control. Specifically, DSP (Digital Signal Processing) technology is used for noise reduction to ensure clear voice quality. Furthermore, Mel-frequency cepstrum coefficients (MFCCs) are extracted to obtain characteristic features of the voice.

[0668] The server receives feature data sent from the terminal and performs sentiment analysis using a deep learning algorithm. Here, emotional states are classified into classes such as "joy," "sadness," and "surprise." This analysis utilizes a generative AI model specifically designed for sentiment analysis.

[0669] The analyzed emotional data is synchronized with other related devices by the server. This synchronization allows multiple devices to understand the same user's emotional state and respond consistently. Based on this data, the server sends commands to each device for environmental adjustments and response generation.

[0670] The terminal generates voice responses based on instructions from the server. In speech synthesis, a tone and content that takes emotion into account are combined, and the resulting voice is delivered to the user through the speaker. Visual feedback may also be displayed on the screen as relevant information.

[0671] As a concrete example, consider a scenario where a user asks the device, "Tell me what my schedule is for today." The device receives the voice and analyzes the user's emotion as "fatigue." Based on this information, the server instructs other devices to adjust the lighting to a relaxing brightness. The device then generates an empathetic response such as, "That's all for today's work. I'll play some relaxing music now."

[0672] Example of a prompt:

[0673] "Analyze the user's emotions from their voice and suggest appropriate responses."

[0674] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0675] Step 1:

[0676] The device acquires audio signals from the user through the microphone. The input audio signal is raw digital data. This data is then subjected to noise reduction and volume normalization processing. Noise reduction filters out background noise to make the audio clearer. Volume normalization unifies the volume levels of the audio signals to maintain consistent quality. The output is clean audio data with reduced noise and unified volume.

[0677] Step 2:

[0678] The device extracts features from pre-processed audio data. The input is audio data that has undergone noise reduction and volume normalization. Mel-frequency cepstrum coefficients (MFCCs) are extracted from this data. MFCCs reflect the spectral characteristics of the audio and can smoothly extract audio features. The output is the features as numerical data that can be analyzed by a deep learning model.

[0679] Step 3:

[0680] The terminal sends extracted features to the server. The server receives them and inputs them into an emotion analysis algorithm. The input is feature data in the form of a numerical sequence. The server uses a deep learning model to analyze emotions from the speech and determines a specific emotion class (e.g., joy, sadness, surprise). The output is a class label indicating the user's emotion.

[0681] Step 4:

[0682] The server shares and synchronizes analyzed sentiment data with other devices on the network. The input is information about class labels and their intensity, which are the results of the sentiment analysis. Based on this information, the server synchronizes the data to ensure consistent operation across all relevant devices. The output is unified sentiment information to achieve a uniform user experience.

[0683] Step 5:

[0684] The terminal generates a voice response based on synchronization information from the server. The input consists of response instruction data and updated emotion information from the server. A speech synthesis engine is used to generate speech with a tone and content that matches the emotion. The output is a voice response that resonates with the user's emotions, played back through the speaker.

[0685] Step 6:

[0686] The device provides the user with generated audio while displaying related information on its screen. Input consists of the voice response and associated visual data. The audio is transmitted to the user through the speaker, and corresponding information is visually presented on the display. Output is the comprehensive user interaction obtained through auditory and visual means.

[0687] (Application Example 1)

[0688] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0689] In recent years, the number of consumer electronic devices and personal robots used in homes has increased, but these devices have not yet developed sufficiently in terms of understanding user emotions and providing appropriate responses. Often, these devices fail to detect when a user is stressed or wants to relax, and are unable to provide personalized responses. This challenge needs to be addressed to improve the quality of user interaction and enable more natural and effective communication.

[0690] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0691] In this invention, the server includes means for preprocessing voice data from the user and extracting feature data, means for analyzing the feature data and generating a response based on the user's emotional state, and means for outputting the generated response to the user. This enables consumer electronic devices to sense the user's emotions in real time and provide responses and environmental adjustments tailored to individual needs.

[0692] A "user" refers to an individual who interacts with a system or machine and provides voice information.

[0693] "Audio data" refers to recordings of human speech acquired through input units such as microphones.

[0694] An "input unit" refers to equipment that includes hardware and software components for collecting audio and visual data.

[0695] A "processing unit" refers to a module that extracts necessary characteristics from collected data and performs preprocessing such as noise reduction and normalization.

[0696] "Feature data" refers to specific parameters extracted from the original audio data that enable sentiment analysis.

[0697] A "generation unit" refers to a module that constructs a response to the user based on the analyzed feature data.

[0698] An "output unit" refers to a device that communicates the generated response to the user in the form of audio or visuals.

[0699] A "coordination unit" refers to a module that synchronizes feature data between multiple systems, ensuring coordinated operation as a whole.

[0700] An "interaction unit" refers to a configuration designed to optimize user interaction and provide individually customized operations and responses.

[0701] "Consumer machinery and equipment" refers to automated machines used by general consumers in homes and other similar settings, and includes robots and smart devices.

[0702] The system for realizing this invention aims to improve the user experience by recognizing the user's emotions and generating a corresponding response. The system mainly consists of an input unit, a processing unit, a generation unit, an output unit, a adjustment unit, and an interaction unit.

[0703] First, the device is equipped with a microphone, which acts as an "input unit" to collect voice data from the user in real time. This voice data is sent to a "processing unit" where noise reduction and voice normalization are performed. Specifically, the voice is converted into text data using the Google Cloud Speech-to-Text API, and then Mel-frequency cepstrum coefficients (MFCCs) are extracted.

[0704] Next, the server, acting as a "generation unit," analyzes the acquired feature data and uses a deep learning model, such as TensorFlow, to infer emotions. Based on this analysis, it constructs an appropriate response using OpenAI's GPT model.

[0705] The generated response is presented to the user via a speaker or display by an "output unit." This allows the user to receive not only an audio response but also visual feedback.

[0706] Furthermore, the adjustment unit plays a role in sharing this emotional data among multiple related devices. For example, when a user feels stressed, the server communicates this to consumer electronics, instructing them to adjust the interior lighting or play relaxing music.

[0707] For example, if a user says, "I was busy today, so I want to relax," the system processes the voice to sense the user's fatigue level and responds, "I'll create a relaxing environment for you." Another example of a prompt is, "When the user uses words that indicate fatigue, generate words to encourage relaxation," enabling natural dialogue that is in line with the user's emotional state.

[0708] This system allows users to enjoy a more personalized experience, and enables machines to become more than just home appliances—they become interactive entities.

[0709] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0710] Step 1:

[0711] The device uses a microphone to input audio data from the user in real time. Since this input audio data contains noise, noise reduction filtering is applied and the volume is normalized to output clean audio data.

[0712] Step 2:

[0713] The terminal extracts Mel-frequency cepstrum coefficients (MFCCs) from the audio data using a processing unit. This feature extraction optimizes the audio data as feature data for sentiment analysis in a deep learning model. The extracted MFCC data is obtained as output.

[0714] Step 3:

[0715] The server receives MFCC data sent from the terminal as input and uses a deep learning model to infer the emotional state. Specifically, it analyzes the voice tone using a TensorFlow-based model and outputs emotional labels such as joy and sadness. These emotional labels are then used as input for the next step.

[0716] Step 4:

[0717] The server uses OpenAI's GPT, an AI model for generating responses, to create appropriate voice responses based on emotion labels as input. It generates response sentences based on prompts and outputs them in text format. For example, if the prompt is "When the user uses words that indicate fatigue, generate words that promote relaxation," a relaxing response will be formed.

[0718] Step 5:

[0719] The terminal converts the response text received from the server into speech data using speech synthesis software and outputs it to the user through the speaker. Additionally, a visual display unit provides user feedback by displaying supplementary information.

[0720] Step 6:

[0721] The server shares emotion labels and response content with other relevant terminals via a coordination unit to ensure the overall system works in a coordinated manner. For example, if the server determines that a user is tired, it instructs other terminals to adjust the environment, such as lighting or music.

[0722] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0723] This invention describes an empathetic AI agent system that incorporates an emotion engine for recognizing user emotions. This system consists of terminal devices, server devices, and the emotion engines installed in them.

[0724] First, the device collects the user's voice information through a microphone. This voice information is incorporated into the system in real time and used as input data for analysis by the emotion engine. Simultaneously, visual information is also acquired using a camera and preprocessed as multidimensional data including the environment.

[0725] Next, the emotion engine installed in the device analyzes the acquired audio and visual information to recognize the user's emotions. This emotion engine uses a deep learning model to identify various emotion categories (e.g., joy, sadness, surprise, etc.) based on audio and visual features. In particular, it achieves highly accurate emotion recognition by fusing visual and audio information.

[0726] The recognized emotion data is then sent to a server. The server aggregates this data and shares it with multiple related devices. Integrating emotion data over the network enables coordination between devices and maintains consistency in how users are interacted with.

[0727] The device receives instructions from the server and generates the most appropriate response for the user based on those instructions. By adapting emotion-sensitive voice responses to the context, it provides users with natural and friendly interaction. Because these voice responses are conveyed to the user in combination with visual displays, a deeper understanding and feedback can be expected.

[0728] As a concrete example, consider a scenario where a user asks the device, "Tell me the weather." The device detects depression from the user's tone of voice and facial expression, and an emotion engine analyzes this. As a result, information is transmitted to other devices via the server, and instructions are sent to change the indoor lighting to a relaxing setting. The device generates an empathetic response such as, "It looks like it's going to rain today. Why not relax with your favorite movie?" and delivers it to the user, providing advice that suits the user's mood.

[0729] Thus, the system in this invention centers around an emotion engine, accurately recognizing the user's emotions and thereby possessing the ability to adjust the real-world environment. Therefore, it can provide a more comfortable and personalized user experience.

[0730] The following describes the processing flow.

[0731] Step 1:

[0732] The system interface is initiated when the user speaks to the device. The user requests, "I want to listen to music that suits my mood right now."

[0733] Step 2:

[0734] The device uses a built-in microphone to acquire voice data from the user. It also uses a camera to capture the user's facial expressions and surrounding environment.

[0735] Step 3:

[0736] The device performs noise reduction and normalization on the collected audio data, while simultaneously processing the visual data to extract features. This converts the audio and image signals into a data format that is easy to analyze.

[0737] Step 4:

[0738] The device inputs pre-processed audio and visual data into the emotion engine. The emotion engine uses deep learning technology to analyze the user's emotions and classify them into categories such as joy, sadness, and surprise.

[0739] Step 5:

[0740] The device sends emotional data obtained from the emotion engine to the server. Here, an API is used to share emotional information with other devices on the network.

[0741] Step 6:

[0742] The server aggregates emotional data and, as needed, sends environmental adjustment instructions to the relevant devices. For example, it might instruct devices to adjust lighting and sound settings appropriately.

[0743] Step 7:

[0744] The device generates a voice response optimized for the user based on instructions received from the server. Using a voice that resonates with the user's emotions, it might suggest, "Today, I'll play some uplifting music."

[0745] Step 8:

[0746] The device delivers the generated voice response to the user through the speaker and simultaneously plays the selected music. If visual display is possible, it displays song information, related videos, etc., on the screen.

[0747] Through these steps, the system understands the user's emotions and provides appropriate feedback and suggestions based on those emotions, resulting in more human-like interactions.

[0748] (Example 2)

[0749] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0750] In user interaction, emotion recognition that relies on a single source of information has limitations in terms of accuracy and flexibility. Therefore, it is necessary to integrate not only audio information but also visual and environmental information to provide natural and empathetic responses that are tailored to the user's emotions and state. Furthermore, there is a need for systems where multiple devices can work together to adjust the user's environment.

[0751] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0752] In this invention, the server includes an input configuration for acquiring voice information from the user, a data processing configuration for preprocessing the voice and visual information and extracting emotion data, and a generation configuration for analyzing the emotion data using a deep learning model and generating a response based on the user's emotional state. This enables more accurate emotion recognition and consistent, effective responses to the user.

[0753] A "user" is a person who provides audio or visual information and operates or utilizes the system.

[0754] An "input configuration" is a collection of devices and sensors used to acquire audio and visual information from the user and incorporate it into the system.

[0755] A "data processing configuration" refers to a technical means for preprocessing acquired audio and visual information and converting it into an analyzable format.

[0756] "Emotional data" refers to digital data extracted from audio and visual information that represents the user's emotional state.

[0757] A "deep learning model" is a machine learning model that uses a multi-layered artificial neural network and is used to identify patterns in data.

[0758] A "generative configuration" is a program or algorithm that creates user-appropriate responses and outputs based on analyzed sentiment data.

[0759] An "output configuration" is a set of devices and interfaces used to present the generated response to the user as audio or visual information.

[0760] "Cooperative configuration" refers to a technical means by which multiple devices or systems work together to achieve a unified objective.

[0761] This invention relates to an empathetic AI agent system that accurately recognizes a user's emotions and responds appropriately. The system consists of a terminal, a server, and multiple devices that work in coordination to provide the information the user requests.

[0762] The device uses a microphone and camera to collect audio information from the user. The microphone captures the user's speech and inputs it into the system as audio data. The camera also captures visual information from the user's facial expressions. This data is sent to the device's emotion engine, where the audio and visual information is preprocessed.

[0763] The emotion engine incorporates a deep learning model and can identify various emotion categories of the user by analyzing audio and visual features. Visual and audio information are fused to achieve highly accurate emotion recognition. Furthermore, the recognized emotion data is transmitted from the device to the server.

[0764] The server aggregates this emotional data and processes it as needed. The server has the ability to integrate emotional data across the network and share it with multiple devices. This facilitates coordination between devices and ensures consistent responses to the user.

[0765] As a concrete example, let's consider a scenario where a user asks their device, "Tell me the weather." In this case, the device detects a depressed mood from the user's tone of voice and facial expression, and analyzes it using an emotion engine. As a result, a notification is sent to other devices via the server, which could, for example, adjust the room lighting to a relaxing setting. The device generates and delivers an empathetic response to the user, such as, "It looks like it's going to rain today. Why not relax with your favorite movie?"

[0766] As an example of a prompt, one might input the instruction "How would you adjust the lighting when the user is feeling down?" to the generative AI model. In this way, by utilizing the emotion engine and collaborative configuration, a more personalized user experience can be provided.

[0767] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0768] Step 1:

[0769] The device collects audio and visual information from the user. It acquires audio data via a microphone and visual data via a camera as input. Specifically, the microphone captures the frequency characteristics of the audio, and the camera extracts facial expressions using a face recognition algorithm. This data undergoes preprocessing steps such as noise reduction and face detection before being prepared as input data for the emotion engine.

[0770] Step 2:

[0771] The emotion engine within the device analyzes pre-processed audio and visual information to generate emotion data. Audio and visual features are input to a deep learning model. Specifically, the model identifies the user's emotions from changes in speech pitch, tone, speed, and facial expressions, classifying them into emotion categories such as joy, sadness, and surprise. The identified emotion data is then retrieved as output.

[0772] Step 3:

[0773] The device sends the analyzed emotion data to the server. The input is the emotion data generated in step 2. Specifically, the data is securely transmitted to the server using an encryption protocol. The server aggregates the received data and prepares it to be shared with other devices as needed.

[0774] Step 4:

[0775] The server executes a process to issue instructions to other related devices based on the received emotion data. The input is the emotion data sent in the previous step. Specifically, the server adjusts the corresponding real-world environment while maintaining data integrity between devices. As an output, for example, commands are issued to change the brightness or color settings of smart lights.

[0776] Step 5:

[0777] The terminal receives instructions from the server and generates responses for the user. Inputs include server instructions and emotion data. Specifically, it uses a speech synthesis system to generate voice responses in a tone appropriate to the emotion. Visual feedback is also provided through screen displays. The response is communicated to the user through both audio and visual means.

[0778] Step 6:

[0779] The user reacts to the generated responses and environmental changes. The user's feedback is then incorporated back into the system as input data for the next cycle. Specifically, the device collects the user's impressions and opinions again as voice and facial expressions, and uses this data as new information. This cycle is repeated, enabling a higher level of personalization.

[0780] (Application Example 2)

[0781] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0782] In modern society, addressing the anxiety and loneliness experienced by the elderly and providing them with a sense of security has become a crucial challenge. Furthermore, accurately capturing the emotional changes of the elderly and providing individualized support is not easy. Conventional systems lacked the flexibility to grasp the diverse emotions of the elderly in real time and respond immediately accordingly. In particular, there was a lack of mechanisms to provide audio and visual feedback that resonated with their emotions.

[0783] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0784] In this invention, the server includes means for collecting audio and visual information from the user, means for preprocessing the audio and visual information and extracting emotional data, and means for analyzing the emotional data and generating a response based on the user's emotional state. This makes it possible to constantly monitor changes in the emotional state of elderly people and provide appropriate support according to their individual emotional state. Specifically, by generating and providing reassuring audio and visual responses to the user, personalized support tailored to individual needs can be realized.

[0785] "Audio information" refers to data used to extract linguistic or emotional characteristics based on the waveform of sounds emitted by the user.

[0786] "Visual information" refers to visual data collected to analyze a user's facial expressions and movements in order to identify their emotional state.

[0787] "Emotional data" refers to a dataset used to express emotions, obtained by analyzing user voice and visual information.

[0788] A "generation method" refers to a device or software that processes emotional data to create the most suitable response for the user and outputs it as audio or visual feedback.

[0789] A "coordination mechanism" is a system that shares emotional data among multiple devices and adjusts their operation to achieve consistent responses and support activities.

[0790] "Output means" refers to devices or systems that communicate generated responses to the user in audio and visual form, thereby enabling interaction.

[0791] "Emotional change" is a concept that captures shifts in a user's emotional state, and is usually recognized through the comparison of analyzed data.

[0792] "A sense of security" refers to a feeling or state that a system provides to a user to promote psychological stability.

[0793] "Individual needs" refer to the unique requirements and expectations of each user, and serve as the criteria for the support the system provides.

[0794] The system for implementing this invention is an empathetic care support system that utilizes the user's voice and visual information. The terminal uses a Logitech C920 camera and a Blue Yeti microphone to collect the user's voice and visual information in real time. This collected data is preprocessed within the system, and emotional data is extracted using a generative AI model.

[0795] The server uses EmotionRecognizer and OpenCV, implemented in Python, to analyze this emotional data with high accuracy. In particular, a deep learning model combining audio and visual features allows for detailed classification of the user's emotional state and generates the optimal response to provide reassurance.

[0796] The generated responses are displayed visually along with audio output, providing a natural interaction for the user. This makes it possible to provide a sense of security to the elderly in emergencies or when they are feeling anxious. Furthermore, this emotional data can be shared among other caregivers and devices, enabling collaborative responses and the creation of a more personalized support system.

[0797] For example, if a user is feeling lonely, the system can detect that emotion and generate a voice message such as, "Hello, would you like to go for a walk? The weather is lovely," to help lift the user's spirits.

[0798] Examples of prompt statements for a generative AI model are as follows:

[0799] User: "I'm feeling a bit lonely today."

[0800] Emotion Engine: Analyze user's tone and facial expression for emotional state.

[0801] This specification will provide a method for specifically implementing the invention, making it possible to realize a system that provides a sense of security in a way that is sensitive to the user's emotions.

[0802] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0803] Step 1:

[0804] The device uses a Logitech C920 camera to capture the user's facial expressions and a Blue Yeti microphone to capture the user's voice. The input consists of the collected raw audio and image data. This data is taken into the system in real time. Here, data preprocessing is performed, particularly noise filtering and image clarification.

[0805] Step 2:

[0806] The server receives pre-processed audio and visual data and analyzes it using a generative AI model. The input is pre-processed data, and the output is emotion data indicating the user's emotions. Specifically, a deep learning algorithm analyzes the tone of voice and facial features to determine emotions such as joy, sadness, and surprise.

[0807] Step 3:

[0808] The server generates the most appropriate response for the user's emotional state based on analyzed emotion data. The input is emotion data, and the output is data for voice messages and visual displays. It utilizes a generative AI model to generate emotionally resonant messages. In this process, prompts are used to create user-friendly responses.

[0809] Step 4:

[0810] The terminal outputs the generated voice response through its speaker and displays visual feedback on its screen. The input is response data sent from the server. The user receives support from the system through specific voice and video interactions. This allows for the provision of direct reassurance to the user.

[0811] Step 5:

[0812] The server shares emotional data with other supporters and devices, coordinating the entire system to work together. Inputs are emotional data and generated response information, while outputs are instructions to other devices and data sharing. This creates a consistent support system overall, providing a relaxed environment for the elderly.

[0813] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0814] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0815] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0816] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0817] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0818] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0819] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0820] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0821] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0822] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0823] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0824] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0825] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0826] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0827] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0828] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0829] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0830] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0831] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0832] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0833] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0834] The following is further disclosed regarding the embodiments described above.

[0835] (Claim 1)

[0836] An input means for collecting voice information from the user,

[0837] Processing means for preprocessing the aforementioned audio information and extracting emotion data,

[0838] A generation means for analyzing the aforementioned emotional data and generating a response based on the user's emotional state,

[0839] Output means for outputting the aforementioned response to the user,

[0840] A coordination means that shares the aforementioned emotional data among multiple devices and operates in a coordinated manner with each other,

[0841] A system that includes this.

[0842] (Claim 2)

[0843] The system according to claim 1, wherein the input means includes a sensor that collects visual information and environmental information in addition to sound.

[0844] (Claim 3)

[0845] The system according to claim 1, wherein the output means includes a visual display device in addition to an audio output.

[0846] "Example 1"

[0847] (Claim 1)

[0848] An input means for acquiring audio signals from the user,

[0849] A signal processing means for preprocessing the aforementioned audio signal and extracting feature quantities,

[0850] An analytical means for analyzing and classifying user emotions using the aforementioned features,

[0851] A synchronization means for sharing and integrating the classified emotion data among multiple terminals,

[0852] A generation means that generates and provides a response to the user based on the aforementioned sentiment data,

[0853] Output means that conveys the response to the user by acoustic output or visual output,

[0854] An information processing system that includes this.

[0855] (Claim 2)

[0856] The information processing system according to claim 1, wherein the input means includes a sensing device that collects image information and environmental data in addition to audio information.

[0857] (Claim 3)

[0858] The information processing system according to claim 1, wherein the output means includes a visual display device in addition to sound generation.

[0859] "Application Example 1"

[0860] (Claim 1)

[0861] An input unit that collects voice data from users,

[0862] A processing unit that preprocesses the aforementioned audio data and extracts feature data,

[0863] A generation unit that analyzes the aforementioned feature data and generates a response based on the user's emotional state,

[0864] An output unit that outputs the aforementioned response to the user,

[0865] A coordination unit that shares the aforementioned feature data among multiple systems and operates in a coordinated manner,

[0866] An interaction unit implemented in consumer electronics that provides emotion-responsive operation and output,

[0867] A system that includes this.

[0868] (Claim 2)

[0869] The system according to claim 1, wherein the input unit includes a sensor that acquires visual data and environmental data in addition to sound.

[0870] (Claim 3)

[0871] The system according to claim 1, wherein the output unit includes a visual display unit in addition to an audio output unit, and the interaction unit provides diverse emotion-based feedback.

[0872] "Example 2 of combining an emotion engine"

[0873] (Claim 1)

[0874] An input configuration for acquiring voice information from the user,

[0875] A data processing configuration that preprocesses the aforementioned audio and visual information and extracts emotional data,

[0876] A generation configuration that uses a deep learning model to analyze the emotion data and generates a response based on the user's emotional state,

[0877] An output configuration that provides the aforementioned response to the user,

[0878] A cooperative configuration that integrates the aforementioned emotional data among multiple devices and operates in harmony with each other,

[0879] A system that includes this.

[0880] (Claim 2)

[0881] The system according to claim 1, wherein the input configuration includes sensors that acquire visual information and environmental information in addition to audio information.

[0882] (Claim 3)

[0883] The system according to claim 1, wherein the output configuration includes a visual display device in addition to an auditory output.

[0884] "Application example 2 when combining with an emotional engine"

[0885] (Claim 1)

[0886] An input means for collecting audio and visual information from the user,

[0887] Processing means for preprocessing the aforementioned audio and visual information and extracting emotion data,

[0888] A generation means that analyzes the aforementioned emotional data, generates a response based on the user's emotional state, and provides a sense of security through visual and auditory means,

[0889] An output means for outputting the aforementioned response to the user and supporting the user's individual needs,

[0890] A coordination means that operates in cooperation with other devices by sharing the aforementioned emotion data among multiple devices and transmitting information based on the user's emotions to other supporters,

[0891] A system that includes this.

[0892] (Claim 2)

[0893] The system according to claim 1, wherein the input means includes a device capable of continuously monitoring emotional changes based on voice and visual information.

[0894] (Claim 3)

[0895] The system according to claim 1, wherein the output means includes a device that provides audio output and visual display corresponding to the emotions of an elderly person. [Explanation of Symbols]

[0896] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. An input means for collecting voice information from the user, Processing means for preprocessing the aforementioned audio information and extracting emotion data, A generation means for analyzing the aforementioned emotional data and generating a response based on the user's emotional state, Output means for outputting the aforementioned response to the user, A coordination means that shares the aforementioned emotional data among multiple devices and operates in a coordinated manner with each other, A system that includes this.

2. The system according to claim 1, wherein the input means includes a sensor that collects visual information and environmental information in addition to sound.

3. The system according to claim 1, wherein the output means includes a visual display device in addition to an audio output.

Images