system

Abstract

We provide the system. [Solution] A means of acquiring voice data, facial expression data, and body movement data from a mobile device in real time, A means for preprocessing acquired data and performing multidimensional analysis to predict emotional states and behaviors, A means for generating an appropriate strategy for the user based on the aforementioned analysis results, A means of providing an interface for visually presenting the generated strategy to the user, Means for encrypting data and implementing access control to protect privacy, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern business and personal negotiations, it is difficult to accurately grasp the feelings and intentions of the other party, and it is a problem that dynamic strategic changes in real time are difficult with conventional methods. In addition, the risk of information leakage and privacy infringement is also high, and appropriate means for efficiently managing these are required.

Means for Solving the Problems

[0005] In the present invention, voice data, facial expression data, and body movement data are acquired in real time using a portable information terminal, and these are analyzed in a multivariate manner. Based on the analysis results, an interface is provided that generates an appropriate strategy for the user and visually presents it. In addition, a system is constructed to prevent information leakage by thoroughly protecting privacy through data encryption and access control.

[0006] A "personal information terminal" is a mobile device that can collect data such as voice, images, and sensor information, and connect to external systems via a communication network.

[0007] "Audio data" refers to digital acoustic information related to human speech, acquired through a mobile information terminal.

[0008] "Facial expression data" refers to digital image information about the movements and changes in facial expressions of a person, acquired using a camera device.

[0009] "Body motion data" refers to digital information about the movements of the human body and limbs, acquired using sensors such as accelerometers and gyroscopes.

[0010] "Multidimensional analysis" is an analytical method that integrates data from voice, facial expressions, and body movements, and evaluates their interrelationships to predict the emotions and behaviors of others.

[0011] "Strategy generation" is the process of designing action guidelines based on analyzed data to help users gain an advantage in negotiations with other parties.

[0012] An "interface" refers to the screen display and operation methods that provide visual information to help users understand and utilize a generated strategy.

[0013] "Encryption" is an information security technique that applies specific transformations to transmitted or stored data to prevent it from being deciphered by third parties.

[0014] "Access control" is a management method that defines access rights to data and functions within a system, ensuring that only authorized users can use them. [Brief explanation of the drawing]

[0015] [Figure 1]It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0021] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention is a system that uses a mobile information terminal to collect and analyze data on voice, facial expressions, and body movements in real time, thereby presenting the user with effective strategies based on the other party's emotional state and predicted behavior. This allows the user to gain a greater advantage in negotiations and interpersonal situations.

[0037] First, the device acquires data from sensors in smartphones or wearable devices. This data is compressed and then sent to the server. The server receives the data, performs preprocessing such as noise reduction and normalization, and then uses generative AI to perform multi-dimensional analysis. In the analysis, speech recognition technology is used to extract emotions from speech, facial recognition technology is used to analyze facial expression data, and motion analysis technology is used to analyze motion data.

[0038] As a result, the other party's emotional state and future actions are predicted, and strategies are generated to support the user's decision-making. The generated strategies are presented visually through a user-friendly interface, making it easy for users to understand and adopt them.

[0039] For example, in business negotiations, the server can analyze the other party's tone of voice and facial movements to instantly predict whether they have doubts, and then offer appropriate suggestions to gain an advantage in the negotiation. Similarly, in job interviews, it can suggest strategies to put a nervous job applicant at ease. This allows users to take the most appropriate action for each situation.

[0040] Furthermore, the server encrypts data and controls access to ensure the privacy of information. Thus, the present invention achieves both data protection and analytical accuracy while directly influencing the actual actions of users.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The device uses the microphone, camera, and sensors of a smartphone or wearable device to collect voice data, facial expression data, and body movement data of the user and the other party in real time. The data is temporarily stored locally and transmitted to a server via an available communication network.

[0044] Step 2:

[0045] The server receives data sent from the terminal and performs preprocessing to improve data quality. This preprocessing includes noise reduction, data standardization, and removal of outliers.

[0046] Step 3:

[0047] The server uses a generation AI to perform multimodal analysis on pre-processed data. For audio data, natural language processing techniques are used to extract emotional keywords, and for facial expression data, computer vision techniques are used to analyze reactions. For body movement data, movement patterns are recognized and their intentions are inferred.

[0048] Step 4:

[0049] The server predicts the other party's emotional state and behavioral tendencies based on the analysis results. For this prediction, machine learning algorithms are used to learn from similar past cases and estimate the likely next occurrence.

[0050] Step 5:

[0051] The server generates strategies to offer the user based on predicted emotional states and behavioral tendencies. These proposed strategies cover a wide range of areas, including negotiation techniques and communication tactics.

[0052] Step 6:

[0053] The terminal receives strategies generated from the server and presents them visually to the user. The user interface is designed to be intuitive and easy to use, allowing the user to quickly understand and execute the presented strategies.

[0054] Step 7:

[0055] The server encrypts data to ensure its security and protects privacy. User data is accessible only to authorized personnel through access control.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] In modern society, communication in negotiations and interpersonal relationships is highly valued, but accurately understanding the emotional state of others remains a challenge. Furthermore, there is a lack of means to efficiently predict the emotions and actions of others while ensuring privacy protection, and to provide users with appropriate strategies.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes means for instantly acquiring voice signals, facial expression information, and physical information from an information acquisition device; means for pre-processing the acquired information and performing multidimensional analysis to predict emotional states and behaviors; and means for generating appropriate strategies for the user based on the analysis results. This makes it possible to predict the emotions and behaviors of others in real time with high accuracy and to quickly provide useful strategies to the user.

[0061] An "information acquisition device" is a device that instantly collects voice signals, facial expression information, and physical information.

[0062] An "audio signal" is data in the form of sound, including the tone and volume of the other person's voice, as well as linguistic elements.

[0063] "Facial expression information" refers to data used to analyze emotions based on the movements and changes in a person's face.

[0064] "Physical information" refers to data used for analysis based on the opponent's physical movements and changes in posture.

[0065] "Multidimensional analysis" is a technique that integrates and analyzes different types of data to predict emotional states and behaviors.

[0066] "Encryption" is a technology that converts data into a format that cannot be viewed by third parties in order to protect the confidentiality and security of the data.

[0067] "Access control" is a method of ensuring data security by granting or restricting access to information.

[0068] "Natural language processing technology" is a computational technique used to analyze human emotions and intentions from speech and text.

[0069] "Machine vision technology" is a technology that uses cameras and sensors to analyze visual information and recognize objects and facial expressions.

[0070] A "strategy" refers to a set of strategies or suggestions designed to support users in taking appropriate action based on the analysis results.

[0071] "Immediately" means that the process from information acquisition to analysis and proposal of strategies is carried out in real time without any delay.

[0072] One embodiment of this invention is a system that uses a portable information terminal to acquire voice, facial expression, and body movement data in real time, and provides the user with useful strategies based on that data. This allows the user to gain an advantage in negotiation and interpersonal relationship scenarios.

[0073] The devices include smartphones and wearable devices, which collect data through microphones, cameras, accelerometers, etc. Voice signals, facial expression information, and body information are compressed by the devices before being sent to the server. The server receives this data and performs preprocessing such as noise reduction and data normalization. Technologies used include speech recognition, computer vision, and motion analysis, and a generative AI model combines these technologies to analyze the data from multiple perspectives.

[0074] Specifically, the server extracts emotions and keywords from voice data using natural language processing technology. Furthermore, it applies machine vision technology to analyze facial expressions and uses motion analysis technology to predict movement patterns. This allows the user to be presented with visual strategies using cursors and icons, assisting them in selecting the best course of action in each situation.

[0075] As a concrete example, in a business negotiation scenario, the server analyzes the other party's tone of voice and facial movements to instantly determine if they have any doubts or concerns. Based on these results, a prompt message such as "Analyze the facial expressions and vocal characteristics of the negotiating party when they have doubts, and suggest a strategy to propose" is input into the AI ​​model, allowing the negotiator to make the most appropriate proposal and gain an advantage.

[0076] Furthermore, to protect user privacy, data is encrypted and access is strictly controlled. This makes it possible to deliver effective results to users while simultaneously protecting data and achieving high analytical accuracy.

[0077] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0078] Step 1:

[0079] The device uses sensors from smartphones and wearable devices to acquire voice, facial expression, and body movement data in real time. This data is in its raw, uncompressed state. Specifically, the input consists of voice picked up by the microphone, facial expressions captured by the camera, and body movements measured by the accelerometer. Because it is difficult to send this data to a server simultaneously, it is compressed within the device to improve communication efficiency. The output is the compressed data.

[0080] Step 2:

[0081] The device sends the compressed data to the server via the internet. To ensure the security of data transmission, the data is encrypted. The main inputs here are the compressed audio, facial expression, and body movement data compressed in step 1. The output maintains a secure transmission state to the server.

[0082] Step 3:

[0083] The server performs preprocessing on the received data, such as noise reduction and data normalization. The input here is encrypted, compressed data sent from the terminal. Specifically, background noise is filtered from audio data, color correction is performed on facial data, and motion data is smoothed. These processes result in clean data suitable for analysis. The output is the preprocessed data.

[0084] Step 4:

[0085] The server performs multidimensional analysis using pre-processed data. This analysis utilizes a generative AI model to extract emotions and keywords from voice data using natural language processing techniques, and applies machine vision techniques to facial and motion data for analysis. The input is the pre-processed data obtained in step 3, and the output is the result of emotional state and behavioral predictions. This allows the server to predict the other party's potential emotions and behaviors.

[0086] Step 5:

[0087] The server generates optimal strategies and suggestions for the user based on the analysis results. This process is carried out through prompts generated by the generative AI model. The input is the analysis results obtained in step 4, and the output is text data containing specific suggestions and strategies. This allows the server to construct strategies to support the user.

[0088] Step 6:

[0089] Users visually retrieve strategies generated by the server through their device interface. The optimal strategy is displayed, allowing users to adjust their responses in real time and gain an advantage in negotiations and communication. Input is the server's suggestions and strategy data, while output is the user's decision and the resulting actions. This enables users to take appropriate responses based on the situation.

[0090] (Application Example 1)

[0091] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0092] In modern family environments, communication among family members is not always efficient. Furthermore, it is not easy to assess individual emotional and health conditions and respond appropriately. Moreover, in today's world where the protection of personal information is increasingly important, accurate data analysis while simultaneously protecting privacy is required.

[0093] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0094] In this invention, the server includes means for preprocessing acquired data and performing multi-dimensional analysis to predict emotional states and behaviors; means for analyzing emotional states from changes in voice and facial expressions in a home autonomous device and providing corresponding response methods; and means for making suggestions for health management purposes. This makes it possible to facilitate communication within the home, provide appropriate advice and health management suggestions to each individual, while ensuring the privacy of information.

[0095] A "portable information terminal" is an information processing device that is portable by the user and has the function of collecting and transmitting various types of data.

[0096] "Audio data" is a form of information that includes human speech and intonation, and is a digital signal used to analyze emotions and intentions.

[0097] "Facial expression data" refers to information that digitally records the characteristic movements and expressions of an individual's face, and is used to estimate emotions.

[0098] "Body movement data" refers to digital information that represents the movements and postures of the human body, and is used to analyze intentions and states.

[0099] "Multidimensional analysis" is a method that comprehensively analyzes multiple types of data to extract correlations and hidden patterns.

[0100] "Home autonomous devices" are devices designed to operate within the home and automatically perform specific tasks.

[0101] "Health management suggestions" refer to information that provides advice and action guidelines tailored to an individual's health condition.

[0102] "Privacy protection" refers to technical and organizational measures to prevent the leakage or unauthorized access of personal information.

[0103] The system implementing this invention is built around a portable information terminal. The terminal uses built-in sensors to acquire voice, facial expression, and body movement data in real time. The data acquired from this terminal is transmitted to a server in the cloud via a communication module, where data preprocessing is performed.

[0104] The server uses natural language processing techniques to process audio data, and this process can utilize speech recognition software such as Google Cloud Speech-to-Text. This technology is used to extract emotions and keywords from the audio. Simultaneously, facial expression data is analyzed using machine vision technologies such as Amazon Rekognition. Body movement data is analyzed using motion analysis technologies such as the OpenPose library. Through these multi-faceted analyses, the emotional state and behavior of the other party are predicted.

[0105] The server uses a generative AI model to generate appropriate response strategies based on these analysis results. The generated strategies are visually presented on the displays of mobile devices and autonomous devices through a user-friendly interface. In situations where voice guidance is used, information is provided to the user audibly.

[0106] Furthermore, the server encrypts data and uses the SSL / TLS protocol for access control. This makes the system secure from a privacy perspective.

[0107] As a concrete example, consider a scenario where a home-use autonomous device analyzes the emotional state of family members during dinner each day and provides appropriate advice. If the voice data detects that someone is feeling tired, suggestions for relieving fatigue will be offered. This can facilitate smoother communication within the family and contribute to improving individual health.

[0108] An example of a prompt might be: "Analyze the emotions from the following audio data and generate a safe strategy to suggest. The emotion data based on the audio indicates a normal level of anger and moderate interest. Provide advice on what to do next."

[0109] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0110] Step 1:

[0111] The device uses its built-in microphone and camera to acquire voice, facial expression, and body movement data in real time. The input in this step consists of the user's voice and visual data. The acquired data is temporarily stored digitally within the device.

[0112] Step 2:

[0113] The terminal compresses the acquired data and sends it to the server using an encryption protocol. The input is the data acquired in step 1, and the output is a compressed and encrypted data stream. This data is sent to the server via a secure communication channel.

[0114] Step 3:

[0115] The server performs noise reduction and normalization processing on the received data. The input is compressed data received from the terminal, and the output is clean data suitable for analysis. Signal processing software is used for noise reduction to improve data quality.

[0116] Step 4:

[0117] The server uses natural language processing techniques to analyze audio data and extract emotions and keywords. The input is audio data, and the output is extracted emotion information and keywords. A speech recognition engine performs these extractions, and a generative AI model supports the data analysis.

[0118] Step 5:

[0119] The server uses machine vision technology to analyze facial expression data and body movement data. The inputs are facial expression data and body movement data, and the outputs are the analyzed emotional state and predicted movement. Computer vision algorithms process the data and perform movement estimation.

[0120] Step 6:

[0121] The server uses a generated AI model to generate an appropriate response strategy based on the analysis results. The input in this step is the analysis results from steps 4 and 5, and the output is the response strategy. The generated strategy is instructed to the AI ​​model using prompt statements.

[0122] Step 7:

[0123] The server sends the generated response strategy to the terminal and presents it to the user visually or audibly. The input is the generated response strategy, and the output is the information provided to the user through the terminal's display or audio functions. The user can choose an action based on the presented information.

[0124] Step 8:

[0125] The server continuously implements encryption and access control to protect data privacy throughout the entire process. Inputs are the data at each step, and outputs are securely protected data. Appropriate encryption is performed using the SSL / TLS protocol.

[0126] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0127] This invention is a system that analyzes voice, facial expressions, and body movement data collected from a mobile information terminal in a multi-faceted manner to recognize the user's emotions in real time and generate and present appropriate strategies according to the situation. This system incorporates an emotion engine, enabling a more accurate understanding of the emotional state of the user and their negotiating partner.

[0128] First, the device continuously collects voice, facial expression, and movement data using a smartphone or wearable device. Voice data is acquired through a microphone, and facial expression data is captured by a camera. Movement data is acquired in real time using a gyroscope and accelerometer.

[0129] Next, the server receives this data and performs preprocessing such as noise reduction and standardization. The preprocessed data is then subjected to multi-dimensional analysis by a generative AI equipped with an emotion engine. Here, natural language processing techniques are applied to the audio data to extract emotion keywords. Computer vision techniques are used for facial expression data to assign emotion labels. Motion data is interpreted by a motion analysis algorithm.

[0130] The emotion engine uses this data to recognize and predict the user's emotional state and the behavioral tendencies of the negotiating partner. For example, in business negotiations, if the other party is feeling anxious, the engine immediately reflects data indicating that emotion and presents a strategy to proceed with the negotiation calmly. The user receives the strategy visually through the device interface and can take appropriate countermeasures.

[0131] Furthermore, the system receives user feedback and continuously improves the accuracy of its emotion engine's analysis. As data is accumulated on the results of users acting according to the system's suggestions, the accuracy of subsequent suggestions improves.

[0132] This system ensures user peace of mind by strictly considering privacy protection, encrypting data, and implementing access control. This provides an effective and secure form of the invention.

[0133] The following describes the processing flow.

[0134] Step 1:

[0135] The device collects audio data using the microphone of a smartphone or wearable device, records facial expressions with a camera, and captures body movements through an accelerometer and gyroscope. The data is temporarily stored in local storage.

[0136] Step 2:

[0137] The device compresses the collected voice, facial expression, and motion data and transmits it to the server using an available communication network (such as Wi-Fi or mobile data).

[0138] Step 3:

[0139] The server receives data sent from the terminal and performs preprocessing such as noise reduction, standardization, and outlier removal to improve its quality.

[0140] Step 4:

[0141] The server analyzes pre-processed data using a generative AI equipped with an emotion engine. For audio data, natural language processing (NLP) is used to analyze emotional keywords and tones, and for facial expression data, computer vision technology is used to classify emotional labels. Motion data undergoes historical analysis using a motion analysis algorithm.

[0142] Step 5:

[0143] The server models the current emotional state of the user and their negotiating partner based on the results analyzed by the emotion engine, and uses this information to predict future actions.

[0144] Step 6:

[0145] The server generates effective strategies for the user based on recognized emotional states and behavioral predictions. These strategies are designed taking negotiation techniques and communication approaches into consideration.

[0146] Step 7:

[0147] The device receives strategies generated from the server and presents them to the user visually using an intuitive user interface. The user can then adjust their actions in real time based on this information.

[0148] Step 8:

[0149] The server encrypts user data and implements strict access control. Privacy is protected, and only users with legitimate permission can view the data.

[0150] (Example 2)

[0151] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0152] In today's diverse communication landscape, quickly and accurately understanding a person's emotional state and behavioral tendencies is crucial, but conventional technologies have struggled to achieve this in real time. Furthermore, there is a need for methods that efficiently utilize feedback to improve analysis accuracy while ensuring the privacy of acquired data.

[0153] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0154] In this invention, the server includes means for acquiring voice data, facial expression data, and motion data from information and communication devices in real time; means for preprocessing the acquired data and performing multifaceted analysis to recognize and predict emotions and behavioral tendencies; and means for collecting user feedback to improve the accuracy of the analysis. This makes it possible to grasp the emotional state of the other party in real time and propose appropriate communication strategies.

[0155] "Information and communication equipment" is a general term for electronic devices that collect data such as the user's voice, facial expressions, and movements in real time.

[0156] "Voice data" refers to data that includes the user's voice characteristics, collected through sensors such as microphones.

[0157] "Facial expression data" refers to data that shows the state of the user's face as captured by a camera.

[0158] "Motion data" refers to data about the user's body movements acquired using gyroscopes and accelerometers.

[0159] "Preprocessing" refers to the process of preparing data for analysis, such as removing noise and standardizing it, to make it suitable for analysis.

[0160] "Multifaceted analysis" is a method that integrates and analyzes multiple data sources to extract more advanced information.

[0161] "Emotion recognition" is the process of identifying a user's emotional state from collected data.

[0162] "Strategy generation" is the process of designing the optimal communication strategy for users based on analyzed emotions and behavioral tendencies.

[0163] "Feedback" refers to the collection of data based on information and results obtained from users, in order to improve the system's performance and suggested features.

[0164] Encryption is a technology that transforms information to protect it from unauthorized access.

[0165] "Access control" is a management method that restricts access to data and functions within a system, allowing only authorized users to use them.

[0166] This invention is a system that uses information and communication equipment to collect and analyze data on the user's voice, facial expressions, and movements in real time, thereby accurately recognizing the user's emotional state and suggesting appropriate countermeasures. This system is mainly composed of a terminal and a server working together.

[0167] The terminal consists of information and communication devices such as smartphones and wearable devices, and acquires the user's voice in real time using a microphone, facial expressions with a camera, and movements with a gyroscope and accelerometer. This acquired data is then transmitted to a server.

[0168] The server processes the received data. First, it performs preprocessing such as noise reduction on the data to prepare it for analysis. Next, it applies natural language understanding techniques to the audio data using an emotion engine to extract emotion keywords. For facial expression data, it applies image processing techniques to identify emotion labels. For motion data, it uses a motion analysis algorithm to analyze the user's behavioral tendencies in detail. Based on this, a generative AI model constructs an appropriate strategy and presents it to the user.

[0169] Users review strategies presented by the server via their terminals and apply them to their actual communication. For example, if the other party in a negotiation appears anxious, the system might suggest a strategy such as, "It would be best to continue explaining in a calm tone." This allows users to conduct negotiations more smoothly.

[0170] In terms of security, data encryption and access control are thoroughly implemented throughout the entire system, enabling safe operation while protecting privacy.

[0171] A concrete example is a business meeting where understanding the emotions of others and facilitating smooth conversation based on those emotions is required. An example of a prompt for a generative AI model would be, "Assuming the user is participating in the meeting, please suggest the next topic to discuss based on the participants' emotions."

[0172] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0173] Step 1:

[0174] The device acquires data on the user's voice, facial expressions, and movements in real time. Specifically, it records voice with a microphone, captures facial expressions as video with a camera, and detects movements with a gyroscope and accelerometer. Since this input data contains noise, preprocessing is required.

[0175] Step 2:

[0176] The server preprocesses the audio, facial expression, and motion data received from the terminal. It applies a noise reduction filter to improve the clarity of the audio data by removing irrelevant data. Facial expression data is standardized using a video conversion algorithm, and motion data is normalized based on a baseline. The output of this step is clean data ready for analysis.

[0177] Step 3:

[0178] The server performs a multifaceted analysis of the pre-processed data. Natural language understanding techniques are applied to audio data to extract emotional keywords. Image processing algorithms are used to detect emotional labels in facial expression data. Motion data is analyzed using a motion analysis model to evaluate user behavioral tendencies. The output obtained through these analyses provides detailed information about the user's emotional state and behavior.

[0179] Step 4:

[0180] The server utilizes an emotion engine to generate the optimal strategy for the user based on the analysis results. The generating AI model designs countermeasures tailored to the situation and emotions, creating a concrete plan to propose to the user. This output includes clear action guidelines to present to the user.

[0181] Step 5:

[0182] Users receive strategies provided by the server via their devices and utilize them in real-world situations. For example, they might adjust their speaking style and gestures during a meeting according to a suggested strategy. The results and impressions obtained from this use are sent to the system as feedback and used for future analysis.

[0183] (Application Example 2)

[0184] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0185] In modern society, managing stress within the family and providing effective communication methods are crucial. However, conventional technologies have difficulty accurately understanding users' emotional states and responding appropriately based on that understanding.

[0186] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0187] In this invention, the server includes means for continuously acquiring voice information, facial expression information, and body movement information from a portable information device; means for preprocessing the acquired information and performing multi-dimensional analysis to predict emotional state and behavior; and means for playing audio media according to the user's emotional state and suggesting refreshing activities. This enables real-time understanding of emotional state within the home and appropriate responses based on that understanding.

[0188] A "portable information device" is a portable electronic device used to continuously acquire voice information, facial expression information, and body movement information.

[0189] "Audio information" refers to data of human voices and sounds acquired through a microphone.

[0190] "Facial expression information" refers to data about a person's facial expressions captured by a camera.

[0191] "Body movement information" refers to data about a person's movements acquired via gyroscopes and accelerometers.

[0192] "Multidimensional analysis" is an analytical technique that analyzes acquired information from various perspectives to predict emotional states and behaviors.

[0193] A "visual presentation mechanism" is an interface designed to display the generated strategy in an easy-to-understand manner for the user.

[0194] "Playing audio media" means playing appropriate audio content through an output device based on the user's emotional state.

[0195] "Personal information protection" refers to security measures that encrypt and properly manage information for the purpose of protecting privacy.

[0196] "Machine vision technology" is a technology that uses computers to analyze image data to understand people's facial expressions and scenes.

[0197] "Natural language processing technology" is a technique that analyzes language data to extract emotions and keywords from audio information.

[0198] The system implementing this invention has the function of continuously acquiring voice information, facial expression information, and body movement information using a smartphone or wearable device. By collecting data using a microphone, camera, gyroscope, and accelerometer built into the terminal, it becomes possible to acquire information in real time.

[0199] The server receives the acquired data, performs preprocessing such as noise reduction and standardization, and then performs multidimensional analysis using a generative AI model equipped with an emotion engine. In this analysis, natural language processing techniques are applied to the audio data to extract emotions and related keywords, and machine vision techniques are used for facial expression data. Furthermore, motion data is interpreted by a motion analysis algorithm. Specific technologies used include deep learning models used in natural language processing and image recognition algorithms used in machine vision techniques.

[0200] Based on the generated emotional state, the server generates the optimal strategy for the user and presents it through a visual interface. This allows users to receive information and suggestions tailored to their emotional state in real time. Furthermore, to support refreshing activities, audio media can be played; for example, relaxing music can be played when the user is feeling stressed, improving their comfort level.

[0201] Furthermore, the system is designed to protect personal information by encrypting all personal data and implementing thorough access controls, ensuring that users can use it with peace of mind while protecting their privacy.

[0202] As a concrete example, if the system detects stress during a casual conversation, it will suggest activities to help the user relax. This feature aims to accurately understand the user's emotional state and provide a better living environment.

[0203] An example of a prompt to input into the generating AI model is: "Based on voice data and facial expression data, please analyze the current emotional state and suggest appropriate relaxation techniques. If stress or anxiety is present, please also provide practical solutions."

[0204] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0205] Step 1:

[0206] The device acquires audio information, facial expression information, and body movement information using a microphone, camera, gyroscope, and accelerometer. Input is real-time audio, image, and motion data, and output is the raw data for each. This allows the device to prepare a variety of data.

[0207] Step 2:

[0208] The server receives raw data sent from the terminal. The input consists of raw data of voice, facial expressions, and body movements, which are preprocessed. The output is noise-removed and normalized data. The server improves the quality of the data.

[0209] Step 3:

[0210] The server runs a generative AI model to analyze pre-processed data. The input is pre-processed audio, facial expression, and body movement data, and the output is emotional state and behavioral prediction derived from the analysis. The server analyzes the data from multiple perspectives to understand the emotional state.

[0211] Step 4:

[0212] The server generates the optimal strategy for the user based on the analysis results. The input is emotional state and behavioral prediction, and the output is a specific strategic proposal. The server devises appropriate countermeasures depending on the situation.

[0213] Step 5:

[0214] Users review the generated strategies through a visual interface on their devices. The input is the strategy proposal, and the output is visualized information. Users receive the information visually and make decisions about their actions.

[0215] Step 6:

[0216] The server plays audio media based on the user's emotional state. The input is emotional state data, and the output is the playback of audio media. The server selects appropriate audio content to help alleviate the user's stress.

[0217] Step 7:

[0218] The server encrypts all personal data and implements appropriate access controls. Inputs consist of all collected and analyzed data, while output is securely protected data. The server protects data privacy.

[0219] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0220] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0221] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0222] [Second Embodiment]

[0223] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0224] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0225] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0226] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0227] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0228] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0229] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0230] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0231] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0232] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0233] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0234] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0235] This invention is a system that uses a mobile information terminal to collect and analyze data on voice, facial expressions, and body movements in real time, thereby presenting the user with effective strategies based on the other party's emotional state and predicted behavior. This allows the user to gain a greater advantage in negotiations and interpersonal situations.

[0236] First, the device acquires data from sensors in smartphones or wearable devices. This data is compressed and then sent to the server. The server receives the data, performs preprocessing such as noise reduction and normalization, and then uses generative AI to perform multi-dimensional analysis. In the analysis, speech recognition technology is used to extract emotions from speech, facial recognition technology is used to analyze facial expression data, and motion analysis technology is used to analyze motion data.

[0237] As a result, the other party's emotional state and future actions are predicted, and strategies are generated to support the user's decision-making. The generated strategies are presented visually through a user-friendly interface, making it easy for users to understand and adopt them.

[0238] For example, in business negotiations, the server can analyze the other party's tone of voice and facial movements to instantly predict whether they have doubts, and then offer appropriate suggestions to gain an advantage in the negotiation. Similarly, in job interviews, it can suggest strategies to put a nervous job applicant at ease. This allows users to take the most appropriate action for each situation.

[0239] Furthermore, the server encrypts data and controls access to ensure the privacy of information. Thus, the present invention achieves both data protection and analytical accuracy while directly influencing the actual actions of users.

[0240] The following describes the processing flow.

[0241] Step 1:

[0242] The device uses the microphone, camera, and sensors of a smartphone or wearable device to collect voice data, facial expression data, and body movement data of the user and the other party in real time. The data is temporarily stored locally and transmitted to a server via an available communication network.

[0243] Step 2:

[0244] The server receives data sent from the terminal and performs preprocessing to improve data quality. This preprocessing includes noise reduction, data standardization, and removal of outliers.

[0245] Step 3:

[0246] The server uses a generation AI to perform multimodal analysis on pre-processed data. For audio data, natural language processing techniques are used to extract emotional keywords, and for facial expression data, computer vision techniques are used to analyze reactions. For body movement data, movement patterns are recognized and their intentions are inferred.

[0247] Step 4:

[0248] The server predicts the other party's emotional state and behavioral tendencies based on the analysis results. For this prediction, machine learning algorithms are used to learn from similar past cases and estimate the likely next occurrence.

[0249] Step 5:

[0250] The server generates strategies to offer the user based on predicted emotional states and behavioral tendencies. These proposed strategies cover a wide range of areas, including negotiation techniques and communication tactics.

[0251] Step 6:

[0252] The terminal receives strategies generated from the server and presents them visually to the user. The user interface is designed to be intuitive and easy to use, allowing the user to quickly understand and execute the presented strategies.

[0253] Step 7:

[0254] The server encrypts data to ensure its security and protects privacy. User data is accessible only to authorized personnel through access control.

[0255] (Example 1)

[0256] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0257] In modern society, communication in negotiations and interpersonal relationships is highly valued, but accurately understanding the emotional state of others remains a challenge. Furthermore, there is a lack of means to efficiently predict the emotions and actions of others while ensuring privacy protection, and to provide users with appropriate strategies.

[0258] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0259] In this invention, the server includes means for instantly acquiring voice signals, facial expression information, and physical information from an information acquisition device; means for pre-processing the acquired information and performing multidimensional analysis to predict emotional states and behaviors; and means for generating appropriate strategies for the user based on the analysis results. This makes it possible to predict the emotions and behaviors of others in real time with high accuracy and to quickly provide useful strategies to the user.

[0260] An "information acquisition device" is a device that instantly collects voice signals, facial expression information, and physical information.

[0261] An "audio signal" is data in the form of sound, including the tone and volume of the other person's voice, as well as linguistic elements.

[0262] "Facial expression information" refers to data used to analyze emotions based on the movements and changes in a person's face.

[0263] "Physical information" refers to data used for analysis based on the opponent's physical movements and changes in posture.

[0264] "Multidimensional analysis" is a technique that integrates and analyzes different types of data to predict emotional states and behaviors.

[0265] "Encryption" is a technology that converts data into a format that cannot be viewed by third parties in order to protect the confidentiality and security of the data.

[0266] "Access control" is a method of ensuring data security by granting or restricting access to information.

[0267] "Natural language processing technology" is a computational technique used to analyze human emotions and intentions from speech and text.

[0268] "Machine vision technology" is a technology that uses cameras and sensors to analyze visual information and recognize objects and facial expressions.

[0269] A "strategy" refers to a set of strategies or suggestions designed to support users in taking appropriate action based on the analysis results.

[0270] "Immediately" means that the process from information acquisition to analysis and proposal of strategies is carried out in real time without any delay.

[0271] One embodiment of this invention is a system that uses a portable information terminal to acquire voice, facial expression, and body movement data in real time, and provides the user with useful strategies based on that data. This allows the user to gain an advantage in negotiation and interpersonal relationship scenarios.

[0272] The devices include smartphones and wearable devices, which collect data through microphones, cameras, accelerometers, etc. Voice signals, facial expression information, and body information are compressed by the devices before being sent to the server. The server receives this data and performs preprocessing such as noise reduction and data normalization. Technologies used include speech recognition, computer vision, and motion analysis, and a generative AI model combines these technologies to analyze the data from multiple perspectives.

[0273] Specifically, the server extracts emotions and keywords from voice data using natural language processing technology. Furthermore, it applies machine vision technology to analyze facial expressions and uses motion analysis technology to predict movement patterns. This allows the user to be presented with visual strategies using cursors and icons, assisting them in selecting the best course of action in each situation.

[0274] As a concrete example, in a business negotiation scenario, the server analyzes the other party's tone of voice and facial movements to instantly determine if they have any doubts or concerns. Based on these results, a prompt message such as "Analyze the facial expressions and vocal characteristics of the negotiating party when they have doubts, and suggest a strategy to propose" is input into the AI ​​model, allowing the negotiator to make the most appropriate proposal and gain an advantage.

[0275] Furthermore, to protect user privacy, data is encrypted and access is strictly controlled. This makes it possible to deliver effective results to users while simultaneously protecting data and achieving high analytical accuracy.

[0276] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0277] Step 1:

[0278] The terminal uses sensors of smartphones or wearable devices to acquire data on voice, facial expressions, and body movements in real time. This data is at the stage before being compressed as raw data. Specifically, the input includes the voice picked up by the microphone, the expressions captured by the camera, and the body movements measured by the acceleration sensor. Since it is difficult to send these data to the server simultaneously, they are once compressed within the terminal to improve communication efficiency. As output, compressed data is obtained.

[0279] Step 2:

[0280] The terminal sends the compressed data to the server via the Internet. To ensure the security of data transmission, the data is encrypted. The main input here is the compressed data of voice, facial expressions, and body movements compressed in Step 1. As output, a secure transmission state to the server is maintained.

[0281] Step 3:

[0282] The server performs preprocessing on the received data, such as noise removal and data normalization. The input here is the encrypted compressed data sent from the terminal. Specific operations include filtering the background noise of voice data, correcting the color tone for face data, and smoothing the motion data. Through these processes, clean data suitable for analysis is obtained. The output is the preprocessed data.

[0283] Step 4:

[0284] The server performs multidimensional analysis using the preprocessed data. In this analysis, a generative AI model is utilized. From the voice data, emotions and keywords are extracted using natural language processing technology, and machine vision technology is applied to the face data and motion data for analysis. The input is the preprocessed data obtained in Step 3, and the output is the results of the emotional state and action prediction. Thereby, the server can predict the potential emotions and actions of the other party.

[0285] Step 5:

[0286] Based on the analysis results, the server generates the optimal strategies and suggestions for the user. This process is carried out through the prompt text generated by the generation AI model. The input is the analysis results obtained in Step 4, and the output is text data including specific suggestions and strategies. Thereby, the server constructs measures for user support.

[0287] Step 6:

[0288] The user visually obtains the strategies generated by the server through the interface of the device. Since the optimal strategy is displayed, the user can adjust the response in real time based on this and promote negotiation and communication advantageously. The input is the proposal and strategy data from the server, and the output is the user's decision-making and the actual actions based on it. Thereby, the user realizes an appropriate response according to the situation.

[0289] (Application Example 1)

[0290] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0291] In a modern home environment, communication among family members is not always efficient. Also, it is not easy to determine individual emotional and health states and take appropriate actions accordingly. Furthermore, in modern times when the protection of personal information is being increasingly emphasized, it is required to protect privacy while accurately analyzing data.

[0292] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following respective means.

[0293] In this invention, the server includes means for preprocessing acquired data and performing multi-dimensional analysis to predict emotional states and behaviors; means for analyzing emotional states from changes in voice and facial expressions in a home autonomous device and providing corresponding response methods; and means for making suggestions for health management purposes. This makes it possible to facilitate communication within the home, provide appropriate advice and health management suggestions to each individual, while ensuring the privacy of information.

[0294] A "portable information terminal" is an information processing device that is portable by the user and has the function of collecting and transmitting various types of data.

[0295] "Audio data" is a form of information that includes human speech and intonation, and is a digital signal used to analyze emotions and intentions.

[0296] "Facial expression data" refers to information that digitally records the characteristic movements and expressions of an individual's face, and is used to estimate emotions.

[0297] "Body movement data" refers to digital information that represents the movements and postures of the human body, and is used to analyze intentions and states.

[0298] "Multidimensional analysis" is a method that comprehensively analyzes multiple types of data to extract correlations and hidden patterns.

[0299] "Home autonomous devices" are devices designed to operate within the home and automatically perform specific tasks.

[0300] "Health management suggestions" refer to information that provides advice and action guidelines tailored to an individual's health condition.

[0301] "Privacy protection" refers to technical and organizational measures to prevent the leakage or unauthorized access of personal information.

[0302] The system implementing this invention is built around a portable information terminal. The terminal uses built-in sensors to acquire voice, facial expression, and body movement data in real time. The data acquired from this terminal is transmitted to a server in the cloud via a communication module, where data preprocessing is performed.

[0303] The server uses natural language processing techniques to process audio data, and this process can utilize speech recognition software such as Google Cloud Speech-to-Text. This technology is used to extract emotions and keywords from the audio. Simultaneously, facial expression data is analyzed using machine vision technologies such as Amazon Rekognition. Body movement data is analyzed using motion analysis technologies such as the OpenPose library. Through these multi-faceted analyses, the emotional state and behavior of the other party are predicted.

[0304] The server uses a generative AI model to generate appropriate response strategies based on these analysis results. The generated strategies are visually presented on the displays of mobile devices and autonomous devices through a user-friendly interface. In situations where voice guidance is used, information is provided to the user audibly.

[0305] Furthermore, the server encrypts data and uses the SSL / TLS protocol for access control. This makes the system secure from a privacy perspective.

[0306] As a concrete example, consider a scenario where a home-use autonomous device analyzes the emotional state of family members during dinner each day and provides appropriate advice. If the voice data detects that someone is feeling tired, suggestions for relieving fatigue will be offered. This can facilitate smoother communication within the family and contribute to improving individual health.

[0307] As an example of a prompt sentence, "Analyze the emotion from the following voice data and generate strategies that can be proposed in a safe manner. The voice-based emotion data shows a normal level of anger and a moderate level of interest. Provide advice on how to perform the following steps." can be considered.

[0308] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0309] Step 1:

[0310] The terminal uses the built-in microphone and camera to acquire data on voice, facial expressions, and body movements in real time. The input at this step is the user's voice and visual data. The acquired data is temporarily stored in the terminal in digital form.

[0311] Step 2:

[0312] The terminal compresses the acquired data and transmits it to the server using an encryption protocol. The input is the data acquired in Step 1, and the output is a compressed and encrypted data stream. This data is sent to the server via a secure communication channel.

[0313] Step 3:

[0314] The server performs noise removal and normalization processing on the received data. The input is the compressed data received from the terminal, and the output is data in a clean state suitable for analysis. Signal processing software is used for noise removal to improve the quality of the data.

[0315] Step 4:

[0316] The server analyzes the voice data using natural language processing technology and extracts emotions and keywords. The input is the voice data, and the output is the extracted emotion information and keywords. A voice recognition engine performs these extractions, and a generative AI model supports the data analysis.

[0317] Step 5:

[0318] The server uses machine vision technology to analyze facial expression data and body movement data. The inputs are facial expression data and body movement data, and the outputs are the analyzed emotional state and predicted movement. Computer vision algorithms process the data and perform movement estimation.

[0319] Step 6:

[0320] The server uses a generated AI model to generate an appropriate response strategy based on the analysis results. The input in this step is the analysis results from steps 4 and 5, and the output is the response strategy. The generated strategy is instructed to the AI ​​model using prompt statements.

[0321] Step 7:

[0322] The server sends the generated response strategy to the terminal and presents it to the user visually or audibly. The input is the generated response strategy, and the output is the information provided to the user through the terminal's display or audio functions. The user can choose an action based on the presented information.

[0323] Step 8:

[0324] The server continuously implements encryption and access control to protect data privacy throughout the entire process. Inputs are the data at each step, and outputs are securely protected data. Appropriate encryption is performed using the SSL / TLS protocol.

[0325] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0326] This invention is a system that analyzes voice, facial expressions, and body movement data collected from a mobile information terminal in a multi-faceted manner to recognize the user's emotions in real time and generate and present appropriate strategies according to the situation. This system incorporates an emotion engine, enabling a more accurate understanding of the emotional state of the user and their negotiating partner.

[0327] First, the device continuously collects voice, facial expression, and movement data using a smartphone or wearable device. Voice data is acquired through a microphone, and facial expression data is captured by a camera. Movement data is acquired in real time using a gyroscope and accelerometer.

[0328] Next, the server receives this data and performs preprocessing such as noise reduction and standardization. The preprocessed data is then subjected to multi-dimensional analysis by a generative AI equipped with an emotion engine. Here, natural language processing techniques are applied to the audio data to extract emotion keywords. Computer vision techniques are used for facial expression data to assign emotion labels. Motion data is interpreted by a motion analysis algorithm.

[0329] The emotion engine uses this data to recognize and predict the user's emotional state and the behavioral tendencies of the negotiating partner. For example, in business negotiations, if the other party is feeling anxious, the engine immediately reflects data indicating that emotion and presents a strategy to proceed with the negotiation calmly. The user receives the strategy visually through the device interface and can take appropriate countermeasures.

[0330] Furthermore, the system receives user feedback and continuously improves the accuracy of its emotion engine's analysis. As data is accumulated on the results of users acting according to the system's suggestions, the accuracy of subsequent suggestions improves.

[0331] This system ensures user peace of mind by strictly considering privacy protection, encrypting data, and implementing access control. This provides an effective and secure form of the invention.

[0332] The following describes the processing flow.

[0333] Step 1:

[0334] The device collects audio data using the microphone of a smartphone or wearable device, records facial expressions with a camera, and captures body movements through an accelerometer and gyroscope. The data is temporarily stored in local storage.

[0335] Step 2:

[0336] The device compresses the collected voice, facial expression, and motion data and transmits it to the server using an available communication network (such as Wi-Fi or mobile data).

[0337] Step 3:

[0338] The server receives data sent from the terminal and performs preprocessing such as noise reduction, standardization, and outlier removal to improve its quality.

[0339] Step 4:

[0340] The server analyzes pre-processed data using a generative AI equipped with an emotion engine. For audio data, natural language processing (NLP) is used to analyze emotional keywords and tones, and for facial expression data, computer vision technology is used to classify emotional labels. Motion data undergoes historical analysis using a motion analysis algorithm.

[0341] Step 5:

[0342] The server models the current emotional state of the user and their negotiating partner based on the results analyzed by the emotion engine, and uses this information to predict future actions.

[0343] Step 6:

[0344] The server generates effective strategies for the user based on recognized emotional states and behavioral predictions. These strategies are designed taking negotiation techniques and communication approaches into consideration.

[0345] Step 7:

[0346] The device receives strategies generated from the server and presents them to the user visually using an intuitive user interface. The user can then adjust their actions in real time based on this information.

[0347] Step 8:

[0348] The server encrypts user data and implements strict access control. Privacy is protected, and only users with legitimate permission can view the data.

[0349] (Example 2)

[0350] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0351] In today's diverse communication landscape, quickly and accurately understanding a person's emotional state and behavioral tendencies is crucial, but conventional technologies have struggled to achieve this in real time. Furthermore, there is a need for methods that efficiently utilize feedback to improve analysis accuracy while ensuring the privacy of acquired data.

[0352] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0353] In this invention, the server includes means for acquiring voice data, facial expression data, and motion data from information and communication devices in real time; means for preprocessing the acquired data and performing multifaceted analysis to recognize and predict emotions and behavioral tendencies; and means for collecting user feedback to improve the accuracy of the analysis. This makes it possible to grasp the emotional state of the other party in real time and propose appropriate communication strategies.

[0354] "Information and communication equipment" is a general term for electronic devices that collect data such as the user's voice, facial expressions, and movements in real time.

[0355] "Voice data" refers to data that includes the user's voice characteristics, collected through sensors such as microphones.

[0356] "Facial expression data" refers to data that shows the state of the user's face as captured by a camera.

[0357] "Motion data" refers to data about the user's body movements acquired using gyroscopes and accelerometers.

[0358] "Preprocessing" refers to the process of preparing data for analysis, such as removing noise and standardizing it, to make it suitable for analysis.

[0359] "Multifaceted analysis" is a method that integrates and analyzes multiple data sources to extract more advanced information.

[0360] "Emotion recognition" is the process of identifying a user's emotional state from collected data.

[0361] "Strategy generation" is the process of designing the optimal communication strategy for users based on analyzed emotions and behavioral tendencies.

[0362] "Feedback" refers to the collection of data based on information and results obtained from users, in order to improve the system's performance and suggested features.

[0363] Encryption is a technology that transforms information to protect it from unauthorized access.

[0364] "Access control" is a management method that restricts access to data and functions within a system, allowing only authorized users to use them.

[0365] This invention is a system that uses information and communication equipment to collect and analyze data on the user's voice, facial expressions, and movements in real time, thereby accurately recognizing the user's emotional state and suggesting appropriate countermeasures. This system is mainly composed of a terminal and a server working together.

[0366] The terminal consists of information and communication devices such as smartphones and wearable devices, and acquires the user's voice in real time using a microphone, facial expressions with a camera, and movements with a gyroscope and accelerometer. This acquired data is then transmitted to a server.

[0367] The server processes the received data. First, it performs preprocessing such as noise reduction on the data to prepare it for analysis. Next, it applies natural language understanding techniques to the audio data using an emotion engine to extract emotion keywords. For facial expression data, it applies image processing techniques to identify emotion labels. For motion data, it uses a motion analysis algorithm to analyze the user's behavioral tendencies in detail. Based on this, a generative AI model constructs an appropriate strategy and presents it to the user.

[0368] Users review strategies presented by the server via their terminals and apply them to their actual communication. For example, if the other party in a negotiation appears anxious, the system might suggest a strategy such as, "It would be best to continue explaining in a calm tone." This allows users to conduct negotiations more smoothly.

[0369] In terms of security, data encryption and access control are thoroughly implemented throughout the entire system, enabling safe operation while protecting privacy.

[0370] A concrete example is a business meeting where understanding the emotions of others and facilitating smooth conversation based on those emotions is required. An example of a prompt for a generative AI model would be, "Assuming the user is participating in the meeting, please suggest the next topic to discuss based on the participants' emotions."

[0371] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0372] Step 1:

[0373] The device acquires data on the user's voice, facial expressions, and movements in real time. Specifically, it records voice with a microphone, captures facial expressions as video with a camera, and detects movements with a gyroscope and accelerometer. Since this input data contains noise, preprocessing is required.

[0374] Step 2:

[0375] The server preprocesses the audio, facial expression, and motion data received from the terminal. It applies a noise reduction filter to improve the clarity of the audio data by removing irrelevant data. Facial expression data is standardized using a video conversion algorithm, and motion data is normalized based on a baseline. The output of this step is clean data ready for analysis.

[0376] Step 3:

[0377] The server performs a multifaceted analysis of the pre-processed data. Natural language understanding techniques are applied to audio data to extract emotional keywords. Image processing algorithms are used to detect emotional labels in facial expression data. Motion data is analyzed using a motion analysis model to evaluate user behavioral tendencies. The output obtained through these analyses provides detailed information about the user's emotional state and behavior.

[0378] Step 4:

[0379] The server utilizes an emotion engine to generate the optimal strategy for the user based on the analysis results. The generating AI model designs countermeasures tailored to the situation and emotions, creating a concrete plan to propose to the user. This output includes clear action guidelines to present to the user.

[0380] Step 5:

[0381] Users receive strategies provided by the server via their devices and utilize them in real-world situations. For example, they might adjust their speaking style and gestures during a meeting according to a suggested strategy. The results and impressions obtained from this use are sent to the system as feedback and used for future analysis.

[0382] (Application Example 2)

[0383] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0384] In modern society, managing stress within the family and providing effective communication methods are crucial. However, conventional technologies have difficulty accurately understanding users' emotional states and responding appropriately based on that understanding.

[0385] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0386] In this invention, the server includes means for continuously acquiring voice information, facial expression information, and body movement information from a portable information device; means for preprocessing the acquired information and performing multi-dimensional analysis to predict emotional state and behavior; and means for playing audio media according to the user's emotional state and suggesting refreshing activities. This enables real-time understanding of emotional state within the home and appropriate responses based on that understanding.

[0387] A "portable information device" is a portable electronic device used to continuously acquire voice information, facial expression information, and body movement information.

[0388] "Audio information" refers to data of human voices and sounds acquired through a microphone.

[0389] "Facial expression information" refers to data about a person's facial expressions captured by a camera.

[0390] "Body movement information" refers to data about a person's movements acquired via gyroscopes and accelerometers.

[0391] "Multidimensional analysis" is an analytical technique that analyzes acquired information from various perspectives to predict emotional states and behaviors.

[0392] A "visual presentation mechanism" is an interface designed to display the generated strategy in an easy-to-understand manner for the user.

[0393] "Playing audio media" means playing appropriate audio content through an output device based on the user's emotional state.

[0394] "Personal information protection" refers to security measures that encrypt and properly manage information for the purpose of protecting privacy.

[0395] "Machine vision technology" is a technology that uses computers to analyze image data to understand people's facial expressions and scenes.

[0396] "Natural language processing technology" is a technique that analyzes language data to extract emotions and keywords from audio information.

[0397] The system implementing this invention has the function of continuously acquiring voice information, facial expression information, and body movement information using a smartphone or wearable device. By collecting data using a microphone, camera, gyroscope, and accelerometer built into the terminal, it becomes possible to acquire information in real time.

[0398] The server receives the acquired data, performs preprocessing such as noise reduction and standardization, and then performs multidimensional analysis using a generative AI model equipped with an emotion engine. In this analysis, natural language processing techniques are applied to the audio data to extract emotions and related keywords, and machine vision techniques are used for facial expression data. Furthermore, motion data is interpreted by a motion analysis algorithm. Specific technologies used include deep learning models used in natural language processing and image recognition algorithms used in machine vision techniques.

[0399] Based on the generated emotional state, the server generates the optimal strategy for the user and presents it through a visual interface. This allows users to receive information and suggestions tailored to their emotional state in real time. Furthermore, to support refreshing activities, audio media can be played; for example, relaxing music can be played when the user is feeling stressed, improving their comfort level.

[0400] Furthermore, the system is designed to protect personal information by encrypting all personal data and implementing thorough access controls, ensuring that users can use it with peace of mind while protecting their privacy.

[0401] As a concrete example, if the system detects stress during a casual conversation, it will suggest activities to help the user relax. This feature aims to accurately understand the user's emotional state and provide a better living environment.

[0402] An example of a prompt to input into the generating AI model is: "Based on voice data and facial expression data, please analyze the current emotional state and suggest appropriate relaxation techniques. If stress or anxiety is present, please also provide practical solutions."

[0403] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0404] Step 1:

[0405] The device acquires audio information, facial expression information, and body movement information using a microphone, camera, gyroscope, and accelerometer. Input is real-time audio, image, and motion data, and output is the raw data for each. This allows the device to prepare a variety of data.

[0406] Step 2:

[0407] The server receives raw data sent from the terminal. The input consists of raw data of voice, facial expressions, and body movements, which are preprocessed. The output is noise-removed and normalized data. The server improves the quality of the data.

[0408] Step 3:

[0409] The server runs a generative AI model to analyze pre-processed data. The input is pre-processed audio, facial expression, and body movement data, and the output is emotional state and behavioral prediction derived from the analysis. The server analyzes the data from multiple perspectives to understand the emotional state.

[0410] Step 4:

[0411] The server generates the optimal strategy for the user based on the analysis results. The input is emotional state and behavioral prediction, and the output is a specific strategic proposal. The server devises appropriate countermeasures depending on the situation.

[0412] Step 5:

[0413] Users review the generated strategies through a visual interface on their devices. The input is the strategy proposal, and the output is visualized information. Users receive the information visually and make decisions about their actions.

[0414] Step 6:

[0415] The server plays audio media based on the user's emotional state. The input is emotional state data, and the output is the playback of audio media. The server selects appropriate audio content to help alleviate the user's stress.

[0416] Step 7:

[0417] The server encrypts all personal data and implements appropriate access controls. Inputs consist of all collected and analyzed data, while output is securely protected data. The server protects data privacy.

[0418] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0419] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0420] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0421] [Third Embodiment]

[0422] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0423] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0424] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0425] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0426] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0427] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0428] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0429] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0430] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0431] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0432] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0433] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0434] This invention is a system that uses a mobile information terminal to collect and analyze data on voice, facial expressions, and body movements in real time, thereby presenting the user with effective strategies based on the other party's emotional state and predicted behavior. This allows the user to gain a greater advantage in negotiations and interpersonal situations.

[0435] First, the device acquires data from sensors in smartphones or wearable devices. This data is compressed and then sent to the server. The server receives the data, performs preprocessing such as noise reduction and normalization, and then uses generative AI to perform multi-dimensional analysis. In the analysis, speech recognition technology is used to extract emotions from speech, facial recognition technology is used to analyze facial expression data, and motion analysis technology is used to analyze motion data.

[0436] As a result, the other party's emotional state and future actions are predicted, and strategies are generated to support the user's decision-making. The generated strategies are presented visually through a user-friendly interface, making it easy for users to understand and adopt them.

[0437] For example, in business negotiations, the server can analyze the other party's tone of voice and facial movements to instantly predict whether they have doubts, and then offer appropriate suggestions to gain an advantage in the negotiation. Similarly, in job interviews, it can suggest strategies to put a nervous job applicant at ease. This allows users to take the most appropriate action for each situation.

[0438] Furthermore, the server encrypts data and controls access to ensure the privacy of information. Thus, the present invention achieves both data protection and analytical accuracy while directly influencing the actual actions of users.

[0439] The following describes the processing flow.

[0440] Step 1:

[0441] The device uses the microphone, camera, and sensors of a smartphone or wearable device to collect voice data, facial expression data, and body movement data of the user and the other party in real time. The data is temporarily stored locally and transmitted to a server via an available communication network.

[0442] Step 2:

[0443] The server receives data sent from the terminal and performs preprocessing to improve data quality. This preprocessing includes noise reduction, data standardization, and removal of outliers.

[0444] Step 3:

[0445] The server uses a generation AI to perform multimodal analysis on pre-processed data. For audio data, natural language processing techniques are used to extract emotional keywords, and for facial expression data, computer vision techniques are used to analyze reactions. For body movement data, movement patterns are recognized and their intentions are inferred.

[0446] Step 4:

[0447] The server predicts the other party's emotional state and behavioral tendencies based on the analysis results. For this prediction, machine learning algorithms are used to learn from similar past cases and estimate the likely next occurrence.

[0448] Step 5:

[0449] The server generates strategies to offer the user based on predicted emotional states and behavioral tendencies. These proposed strategies cover a wide range of areas, including negotiation techniques and communication tactics.

[0450] Step 6:

[0451] The terminal receives strategies generated from the server and presents them visually to the user. The user interface is designed to be intuitive and easy to use, allowing the user to quickly understand and execute the presented strategies.

[0452] Step 7:

[0453] The server encrypts data to ensure its security and protects privacy. User data is accessible only to authorized personnel through access control.

[0454] (Example 1)

[0455] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0456] In modern society, communication in negotiations and interpersonal relationships is highly valued, but accurately understanding the emotional state of others remains a challenge. Furthermore, there is a lack of means to efficiently predict the emotions and actions of others while ensuring privacy protection, and to provide users with appropriate strategies.

[0457] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0458] In this invention, the server includes means for instantly acquiring voice signals, facial expression information, and physical information from an information acquisition device; means for pre-processing the acquired information and performing multidimensional analysis to predict emotional states and behaviors; and means for generating appropriate strategies for the user based on the analysis results. This makes it possible to predict the emotions and behaviors of others in real time with high accuracy and to quickly provide useful strategies to the user.

[0459] An "information acquisition device" is a device that instantly collects voice signals, facial expression information, and physical information.

[0460] An "audio signal" is data in the form of sound, including the tone and volume of the other person's voice, as well as linguistic elements.

[0461] "Facial expression information" refers to data used to analyze emotions based on the movements and changes in a person's face.

[0462] "Physical information" refers to data used for analysis based on the opponent's physical movements and changes in posture.

[0463] "Multidimensional analysis" is a technique that integrates and analyzes different types of data to predict emotional states and behaviors.

[0464] "Encryption" is a technology that converts data into a format that cannot be viewed by third parties in order to protect the confidentiality and security of the data.

[0465] "Access control" is a method of ensuring data security by granting or restricting access to information.

[0466] "Natural language processing technology" is a computational technique used to analyze human emotions and intentions from speech and text.

[0467] "Machine vision technology" is a technology that uses cameras and sensors to analyze visual information and recognize objects and facial expressions.

[0468] A "strategy" refers to a set of strategies or suggestions designed to support users in taking appropriate action based on the analysis results.

[0469] "Immediately" means that the process from information acquisition to analysis and proposal of strategies is carried out in real time without any delay.

[0470] One embodiment of this invention is a system that uses a portable information terminal to acquire voice, facial expression, and body movement data in real time, and provides the user with useful strategies based on that data. This allows the user to gain an advantage in negotiation and interpersonal relationship scenarios.

[0471] The devices include smartphones and wearable devices, which collect data through microphones, cameras, accelerometers, etc. Voice signals, facial expression information, and body information are compressed by the devices before being sent to the server. The server receives this data and performs preprocessing such as noise reduction and data normalization. Technologies used include speech recognition, computer vision, and motion analysis, and a generative AI model combines these technologies to analyze the data from multiple perspectives.

[0472] Specifically, the server extracts emotions and keywords from voice data using natural language processing technology. Furthermore, it applies machine vision technology to analyze facial expressions and uses motion analysis technology to predict movement patterns. This allows the user to be presented with visual strategies using cursors and icons, assisting them in selecting the best course of action in each situation.

[0473] As a concrete example, in a business negotiation scenario, the server analyzes the other party's tone of voice and facial movements to instantly determine if they have any doubts or concerns. Based on these results, a prompt message such as "Analyze the facial expressions and vocal characteristics of the negotiating party when they have doubts, and suggest a strategy to propose" is input into the AI ​​model, allowing the negotiator to make the most appropriate proposal and gain an advantage.

[0474] Furthermore, to protect user privacy, data is encrypted and access is strictly controlled. This makes it possible to deliver effective results to users while simultaneously protecting data and achieving high analytical accuracy.

[0475] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0476] Step 1:

[0477] The device uses sensors from smartphones and wearable devices to acquire voice, facial expression, and body movement data in real time. This data is in its raw, uncompressed state. Specifically, the input consists of voice picked up by the microphone, facial expressions captured by the camera, and body movements measured by the accelerometer. Because it is difficult to send this data to a server simultaneously, it is compressed within the device to improve communication efficiency. The output is the compressed data.

[0478] Step 2:

[0479] The device sends the compressed data to the server via the internet. To ensure the security of data transmission, the data is encrypted. The main inputs here are the compressed audio, facial expression, and body movement data compressed in step 1. The output maintains a secure transmission state to the server.

[0480] Step 3:

[0481] The server performs preprocessing on the received data, such as noise reduction and data normalization. The input here is encrypted, compressed data sent from the terminal. Specifically, background noise is filtered from audio data, color correction is performed on facial data, and motion data is smoothed. These processes result in clean data suitable for analysis. The output is the preprocessed data.

[0482] Step 4:

[0483] The server performs multidimensional analysis using pre-processed data. This analysis utilizes a generative AI model to extract emotions and keywords from voice data using natural language processing techniques, and applies machine vision techniques to facial and motion data for analysis. The input is the pre-processed data obtained in step 3, and the output is the result of emotional state and behavioral predictions. This allows the server to predict the other party's potential emotions and behaviors.

[0484] Step 5:

[0485] The server generates optimal strategies and suggestions for the user based on the analysis results. This process is carried out through prompts generated by the generative AI model. The input is the analysis results obtained in step 4, and the output is text data containing specific suggestions and strategies. This allows the server to construct strategies to support the user.

[0486] Step 6:

[0487] Users visually retrieve strategies generated by the server through their device interface. The optimal strategy is displayed, allowing users to adjust their responses in real time and gain an advantage in negotiations and communication. Input is the server's suggestions and strategy data, while output is the user's decision and the resulting actions. This enables users to take appropriate responses based on the situation.

[0488] (Application Example 1)

[0489] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0490] In modern family environments, communication among family members is not always efficient. Furthermore, it is not easy to assess individual emotional and health conditions and respond appropriately. Moreover, in today's world where the protection of personal information is increasingly important, accurate data analysis while simultaneously protecting privacy is required.

[0491] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0492] In this invention, the server includes means for preprocessing acquired data and performing multi-dimensional analysis to predict emotional states and behaviors; means for analyzing emotional states from changes in voice and facial expressions in a home autonomous device and providing corresponding response methods; and means for making suggestions for health management purposes. This makes it possible to facilitate communication within the home, provide appropriate advice and health management suggestions to each individual, while ensuring the privacy of information.

[0493] A "portable information terminal" is an information processing device that is portable by the user and has the function of collecting and transmitting various types of data.

[0494] "Audio data" is a form of information that includes human speech and intonation, and is a digital signal used to analyze emotions and intentions.

[0495] "Facial expression data" refers to information that digitally records the characteristic movements and expressions of an individual's face, and is used to estimate emotions.

[0496] "Body movement data" refers to digital information that represents the movements and postures of the human body, and is used to analyze intentions and states.

[0497] "Multidimensional analysis" is a method that comprehensively analyzes multiple types of data to extract correlations and hidden patterns.

[0498] "Home autonomous devices" are devices designed to operate within the home and automatically perform specific tasks.

[0499] "Health management suggestions" refer to information that provides advice and action guidelines tailored to an individual's health condition.

[0500] "Privacy protection" refers to technical and organizational measures to prevent the leakage or unauthorized access of personal information.

[0501] The system implementing this invention is built around a portable information terminal. The terminal uses built-in sensors to acquire voice, facial expression, and body movement data in real time. The data acquired from this terminal is transmitted to a server in the cloud via a communication module, where data preprocessing is performed.

[0502] The server uses natural language processing techniques to process audio data, and this process can utilize speech recognition software such as Google Cloud Speech-to-Text. This technology is used to extract emotions and keywords from the audio. Simultaneously, facial expression data is analyzed using machine vision technologies such as Amazon Rekognition. Body movement data is analyzed using motion analysis technologies such as the OpenPose library. Through these multi-faceted analyses, the emotional state and behavior of the other party are predicted.

[0503] The server uses a generative AI model to generate appropriate response strategies based on these analysis results. The generated strategies are visually presented on the displays of mobile devices and autonomous devices through a user-friendly interface. In situations where voice guidance is used, information is provided to the user audibly.

[0504] Furthermore, the server encrypts data and uses the SSL / TLS protocol for access control. This makes the system secure from a privacy perspective.

[0505] As a concrete example, consider a scenario where a home-use autonomous device analyzes the emotional state of family members during dinner each day and provides appropriate advice. If the voice data detects that someone is feeling tired, suggestions for relieving fatigue will be offered. This can facilitate smoother communication within the family and contribute to improving individual health.

[0506] An example of a prompt might be: "Analyze the emotions from the following audio data and generate a safe strategy to suggest. The emotion data based on the audio indicates a normal level of anger and moderate interest. Provide advice on what to do next."

[0507] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0508] Step 1:

[0509] The device uses its built-in microphone and camera to acquire voice, facial expression, and body movement data in real time. The input in this step consists of the user's voice and visual data. The acquired data is temporarily stored digitally within the device.

[0510] Step 2:

[0511] The terminal compresses the acquired data and sends it to the server using an encryption protocol. The input is the data acquired in step 1, and the output is a compressed and encrypted data stream. This data is sent to the server via a secure communication channel.

[0512] Step 3:

[0513] The server performs noise reduction and normalization processing on the received data. The input is compressed data received from the terminal, and the output is clean data suitable for analysis. Signal processing software is used for noise reduction to improve data quality.

[0514] Step 4:

[0515] The server uses natural language processing techniques to analyze audio data and extract emotions and keywords. The input is audio data, and the output is extracted emotion information and keywords. A speech recognition engine performs these extractions, and a generative AI model supports the data analysis.

[0516] Step 5:

[0517] The server uses machine vision technology to analyze facial expression data and body movement data. The inputs are facial expression data and body movement data, and the outputs are the analyzed emotional state and predicted movement. Computer vision algorithms process the data and perform movement estimation.

[0518] Step 6:

[0519] The server uses a generated AI model to generate an appropriate response strategy based on the analysis results. The input in this step is the analysis results from steps 4 and 5, and the output is the response strategy. The generated strategy is instructed to the AI ​​model using prompt statements.

[0520] Step 7:

[0521] The server sends the generated response strategy to the terminal and presents it to the user visually or audibly. The input is the generated response strategy, and the output is the information provided to the user through the terminal's display or audio functions. The user can choose an action based on the presented information.

[0522] Step 8:

[0523] The server continuously implements encryption and access control to protect data privacy throughout the entire process. Inputs are the data at each step, and outputs are securely protected data. Appropriate encryption is performed using the SSL / TLS protocol.

[0524] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0525] This invention is a system that analyzes voice, facial expressions, and body movement data collected from a mobile information terminal in a multi-faceted manner to recognize the user's emotions in real time and generate and present appropriate strategies according to the situation. This system incorporates an emotion engine, enabling a more accurate understanding of the emotional state of the user and their negotiating partner.

[0526] First, the device continuously collects voice, facial expression, and movement data using a smartphone or wearable device. Voice data is acquired through a microphone, and facial expression data is captured by a camera. Movement data is acquired in real time using a gyroscope and accelerometer.

[0527] Next, the server receives this data and performs preprocessing such as noise reduction and standardization. The preprocessed data is then subjected to multi-dimensional analysis by a generative AI equipped with an emotion engine. Here, natural language processing techniques are applied to the audio data to extract emotion keywords. Computer vision techniques are used for facial expression data to assign emotion labels. Motion data is interpreted by a motion analysis algorithm.

[0528] The emotion engine uses this data to recognize and predict the user's emotional state and the behavioral tendencies of the negotiating partner. For example, in business negotiations, if the other party is feeling anxious, the engine immediately reflects data indicating that emotion and presents a strategy to proceed with the negotiation calmly. The user receives the strategy visually through the device interface and can take appropriate countermeasures.

[0529] Furthermore, the system receives user feedback and continuously improves the accuracy of its emotion engine's analysis. As data is accumulated on the results of users acting according to the system's suggestions, the accuracy of subsequent suggestions improves.

[0530] This system ensures user peace of mind by strictly considering privacy protection, encrypting data, and implementing access control. This provides an effective and secure form of the invention.

[0531] The following describes the processing flow.

[0532] Step 1:

[0533] The device collects audio data using the microphone of a smartphone or wearable device, records facial expressions with a camera, and captures body movements through an accelerometer and gyroscope. The data is temporarily stored in local storage.

[0534] Step 2:

[0535] The device compresses the collected voice, facial expression, and motion data and transmits it to the server using an available communication network (such as Wi-Fi or mobile data).

[0536] Step 3:

[0537] The server receives data sent from the terminal and performs preprocessing such as noise reduction, standardization, and outlier removal to improve its quality.

[0538] Step 4:

[0539] The server analyzes pre-processed data using a generative AI equipped with an emotion engine. For audio data, natural language processing (NLP) is used to analyze emotional keywords and tones, and for facial expression data, computer vision technology is used to classify emotional labels. Motion data undergoes historical analysis using a motion analysis algorithm.

[0540] Step 5:

[0541] The server models the current emotional state of the user and their negotiating partner based on the results analyzed by the emotion engine, and uses this information to predict future actions.

[0542] Step 6:

[0543] The server generates effective strategies for the user based on recognized emotional states and behavioral predictions. These strategies are designed taking negotiation techniques and communication approaches into consideration.

[0544] Step 7:

[0545] The device receives strategies generated from the server and presents them to the user visually using an intuitive user interface. The user can then adjust their actions in real time based on this information.

[0546] Step 8:

[0547] The server encrypts user data and implements strict access control. Privacy is protected, and only users with legitimate permission can view the data.

[0548] (Example 2)

[0549] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0550] In today's diverse communication landscape, quickly and accurately understanding a person's emotional state and behavioral tendencies is crucial, but conventional technologies have struggled to achieve this in real time. Furthermore, there is a need for methods that efficiently utilize feedback to improve analysis accuracy while ensuring the privacy of acquired data.

[0551] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0552] In this invention, the server includes means for acquiring voice data, facial expression data, and motion data from information and communication devices in real time; means for preprocessing the acquired data and performing multifaceted analysis to recognize and predict emotions and behavioral tendencies; and means for collecting user feedback to improve the accuracy of the analysis. This makes it possible to grasp the emotional state of the other party in real time and propose appropriate communication strategies.

[0553] "Information and communication equipment" is a general term for electronic devices that collect data such as the user's voice, facial expressions, and movements in real time.

[0554] "Voice data" refers to data that includes the user's voice characteristics, collected through sensors such as microphones.

[0555] "Facial expression data" refers to data that shows the state of the user's face as captured by a camera.

[0556] "Motion data" refers to data about the user's body movements acquired using gyroscopes and accelerometers.

[0557] "Preprocessing" refers to the process of preparing data for analysis, such as removing noise and standardizing it, to make it suitable for analysis.

[0558] "Multifaceted analysis" is a method that integrates and analyzes multiple data sources to extract more advanced information.

[0559] "Emotion recognition" is the process of identifying a user's emotional state from collected data.

[0560] "Strategy generation" is the process of designing the optimal communication strategy for users based on analyzed emotions and behavioral tendencies.

[0561] "Feedback" refers to the collection of data based on information and results obtained from users, in order to improve the system's performance and suggested features.

[0562] Encryption is a technology that transforms information to protect it from unauthorized access.

[0563] "Access control" is a management method that restricts access to data and functions within a system, allowing only authorized users to use them.

[0564] This invention is a system that uses information and communication equipment to collect and analyze data on the user's voice, facial expressions, and movements in real time, thereby accurately recognizing the user's emotional state and suggesting appropriate countermeasures. This system is mainly composed of a terminal and a server working together.

[0565] The terminal consists of information and communication devices such as smartphones and wearable devices, and acquires the user's voice in real time using a microphone, facial expressions with a camera, and movements with a gyroscope and accelerometer. This acquired data is then transmitted to a server.

[0566] The server processes the received data. First, it performs preprocessing such as noise reduction on the data to prepare it for analysis. Next, it applies natural language understanding techniques to the audio data using an emotion engine to extract emotion keywords. For facial expression data, it applies image processing techniques to identify emotion labels. For motion data, it uses a motion analysis algorithm to analyze the user's behavioral tendencies in detail. Based on this, a generative AI model constructs an appropriate strategy and presents it to the user.

[0567] Users review strategies presented by the server via their terminals and apply them to their actual communication. For example, if the other party in a negotiation appears anxious, the system might suggest a strategy such as, "It would be best to continue explaining in a calm tone." This allows users to conduct negotiations more smoothly.

[0568] In terms of security, data encryption and access control are thoroughly implemented throughout the entire system, enabling safe operation while protecting privacy.

[0569] A concrete example is a business meeting where understanding the emotions of others and facilitating smooth conversation based on those emotions is required. An example of a prompt for a generative AI model would be, "Assuming the user is participating in the meeting, please suggest the next topic to discuss based on the participants' emotions."

[0570] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0571] Step 1:

[0572] The device acquires data on the user's voice, facial expressions, and movements in real time. Specifically, it records voice with a microphone, captures facial expressions as video with a camera, and detects movements with a gyroscope and accelerometer. Since this input data contains noise, preprocessing is required.

[0573] Step 2:

[0574] The server preprocesses the audio, facial expression, and motion data received from the terminal. It applies a noise reduction filter to improve the clarity of the audio data by removing irrelevant data. Facial expression data is standardized using a video conversion algorithm, and motion data is normalized based on a baseline. The output of this step is clean data ready for analysis.

[0575] Step 3:

[0576] The server performs a multifaceted analysis of the pre-processed data. Natural language understanding techniques are applied to audio data to extract emotional keywords. Image processing algorithms are used to detect emotional labels in facial expression data. Motion data is analyzed using a motion analysis model to evaluate user behavioral tendencies. The output obtained through these analyses provides detailed information about the user's emotional state and behavior.

[0577] Step 4:

[0578] The server utilizes an emotion engine to generate the optimal strategy for the user based on the analysis results. The generating AI model designs countermeasures tailored to the situation and emotions, creating a concrete plan to propose to the user. This output includes clear action guidelines to present to the user.

[0579] Step 5:

[0580] Users receive strategies provided by the server via their devices and utilize them in real-world situations. For example, they might adjust their speaking style and gestures during a meeting according to a suggested strategy. The results and impressions obtained from this use are sent to the system as feedback and used for future analysis.

[0581] (Application Example 2)

[0582] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0583] In modern society, managing stress within the family and providing effective communication methods are crucial. However, conventional technologies have difficulty accurately understanding users' emotional states and responding appropriately based on that understanding.

[0584] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0585] In this invention, the server includes means for continuously acquiring voice information, facial expression information, and body movement information from a portable information device; means for preprocessing the acquired information and performing multi-dimensional analysis to predict emotional state and behavior; and means for playing audio media according to the user's emotional state and suggesting refreshing activities. This enables real-time understanding of emotional state within the home and appropriate responses based on that understanding.

[0586] A "portable information device" is a portable electronic device used to continuously acquire voice information, facial expression information, and body movement information.

[0587] "Audio information" refers to data of human voices and sounds acquired through a microphone.

[0588] "Facial expression information" refers to data about a person's facial expressions captured by a camera.

[0589] "Body movement information" refers to data about a person's movements acquired via gyroscopes and accelerometers.

[0590] "Multidimensional analysis" is an analytical technique that analyzes acquired information from various perspectives to predict emotional states and behaviors.

[0591] A "visual presentation mechanism" is an interface designed to display the generated strategy in an easy-to-understand manner for the user.

[0592] "Playing audio media" means playing appropriate audio content through an output device based on the user's emotional state.

[0593] "Personal information protection" refers to security measures that encrypt and properly manage information for the purpose of protecting privacy.

[0594] "Machine vision technology" is a technology that uses computers to analyze image data to understand people's facial expressions and scenes.

[0595] "Natural language processing technology" is a technique that analyzes language data to extract emotions and keywords from audio information.

[0596] The system implementing this invention has the function of continuously acquiring voice information, facial expression information, and body movement information using a smartphone or wearable device. By collecting data using a microphone, camera, gyroscope, and accelerometer built into the terminal, it becomes possible to acquire information in real time.

[0597] The server receives the acquired data, performs preprocessing such as noise reduction and standardization, and then performs multidimensional analysis using a generative AI model equipped with an emotion engine. In this analysis, natural language processing techniques are applied to the audio data to extract emotions and related keywords, and machine vision techniques are used for facial expression data. Furthermore, motion data is interpreted by a motion analysis algorithm. Specific technologies used include deep learning models used in natural language processing and image recognition algorithms used in machine vision techniques.

[0598] Based on the generated emotional state, the server generates the optimal strategy for the user and presents it through a visual interface. This allows users to receive information and suggestions tailored to their emotional state in real time. Furthermore, to support refreshing activities, audio media can be played; for example, relaxing music can be played when the user is feeling stressed, improving their comfort level.

[0599] Furthermore, the system is designed to protect personal information by encrypting all personal data and implementing thorough access controls, ensuring that users can use it with peace of mind while protecting their privacy.

[0600] As a concrete example, if the system detects stress during a casual conversation, it will suggest activities to help the user relax. This feature aims to accurately understand the user's emotional state and provide a better living environment.

[0601] An example of a prompt to input into the generating AI model is: "Based on voice data and facial expression data, please analyze the current emotional state and suggest appropriate relaxation techniques. If stress or anxiety is present, please also provide practical solutions."

[0602] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0603] Step 1:

[0604] The device acquires audio information, facial expression information, and body movement information using a microphone, camera, gyroscope, and accelerometer. Input is real-time audio, image, and motion data, and output is the raw data for each. This allows the device to prepare a variety of data.

[0605] Step 2:

[0606] The server receives raw data sent from the terminal. The input consists of raw data of voice, facial expressions, and body movements, which are preprocessed. The output is noise-removed and normalized data. The server improves the quality of the data.

[0607] Step 3:

[0608] The server runs a generative AI model to analyze pre-processed data. The input is pre-processed audio, facial expression, and body movement data, and the output is emotional state and behavioral prediction derived from the analysis. The server analyzes the data from multiple perspectives to understand the emotional state.

[0609] Step 4:

[0610] The server generates the optimal strategy for the user based on the analysis results. The input is emotional state and behavioral prediction, and the output is a specific strategic proposal. The server devises appropriate countermeasures depending on the situation.

[0611] Step 5:

[0612] Users review the generated strategies through a visual interface on their devices. The input is the strategy proposal, and the output is visualized information. Users receive the information visually and make decisions about their actions.

[0613] Step 6:

[0614] The server plays audio media based on the user's emotional state. The input is emotional state data, and the output is the playback of audio media. The server selects appropriate audio content to help alleviate the user's stress.

[0615] Step 7:

[0616] The server encrypts all personal data and implements appropriate access controls. Inputs consist of all collected and analyzed data, while output is securely protected data. The server protects data privacy.

[0617] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0618] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0619] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0620] [Fourth Embodiment]

[0621] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0622] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0623] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0624] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0625] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0626] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0627] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0628] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0629] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0630] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0631] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0632] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0633] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0634] This invention is a system that uses a mobile information terminal to collect and analyze data on voice, facial expressions, and body movements in real time, thereby presenting the user with effective strategies based on the other party's emotional state and predicted behavior. This allows the user to gain a greater advantage in negotiations and interpersonal situations.

[0635] First, the device acquires data from sensors in smartphones or wearable devices. This data is compressed and then sent to the server. The server receives the data, performs preprocessing such as noise reduction and normalization, and then uses generative AI to perform multi-dimensional analysis. In the analysis, speech recognition technology is used to extract emotions from speech, facial recognition technology is used to analyze facial expression data, and motion analysis technology is used to analyze motion data.

[0636] As a result, the other party's emotional state and future actions are predicted, and strategies are generated to support the user's decision-making. The generated strategies are presented visually through a user-friendly interface, making it easy for users to understand and adopt them.

[0637] For example, in business negotiations, the server can analyze the other party's tone of voice and facial movements to instantly predict whether they have doubts, and then offer appropriate suggestions to gain an advantage in the negotiation. Similarly, in job interviews, it can suggest strategies to put a nervous job applicant at ease. This allows users to take the most appropriate action for each situation.

[0638] Furthermore, the server encrypts data and controls access to ensure the privacy of information. Thus, the present invention achieves both data protection and analytical accuracy while directly influencing the actual actions of users.

[0639] The following describes the processing flow.

[0640] Step 1:

[0641] The device uses the microphone, camera, and sensors of a smartphone or wearable device to collect voice data, facial expression data, and body movement data of the user and the other party in real time. The data is temporarily stored locally and transmitted to a server via an available communication network.

[0642] Step 2:

[0643] The server receives data sent from the terminal and performs preprocessing to improve data quality. This preprocessing includes noise reduction, data standardization, and removal of outliers.

[0644] Step 3:

[0645] The server uses a generation AI to perform multimodal analysis on pre-processed data. For audio data, natural language processing techniques are used to extract emotional keywords, and for facial expression data, computer vision techniques are used to analyze reactions. For body movement data, movement patterns are recognized and their intentions are inferred.

[0646] Step 4:

[0647] The server predicts the other party's emotional state and behavioral tendencies based on the analysis results. For this prediction, machine learning algorithms are used to learn from similar past cases and estimate the likely next occurrence.

[0648] Step 5:

[0649] The server generates strategies to offer the user based on predicted emotional states and behavioral tendencies. These proposed strategies cover a wide range of areas, including negotiation techniques and communication tactics.

[0650] Step 6:

[0651] The terminal receives strategies generated from the server and presents them visually to the user. The user interface is designed to be intuitive and easy to use, allowing the user to quickly understand and execute the presented strategies.

[0652] Step 7:

[0653] The server encrypts data to ensure its security and protects privacy. User data is accessible only to authorized personnel through access control.

[0654] (Example 1)

[0655] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0656] In modern society, communication in negotiations and interpersonal relationships is highly valued, but accurately understanding the emotional state of others remains a challenge. Furthermore, there is a lack of means to efficiently predict the emotions and actions of others while ensuring privacy protection, and to provide users with appropriate strategies.

[0657] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0658] In this invention, the server includes means for instantly acquiring voice signals, facial expression information, and physical information from an information acquisition device; means for pre-processing the acquired information and performing multidimensional analysis to predict emotional states and behaviors; and means for generating appropriate strategies for the user based on the analysis results. This makes it possible to predict the emotions and behaviors of others in real time with high accuracy and to quickly provide useful strategies to the user.

[0659] An "information acquisition device" is a device that instantly collects voice signals, facial expression information, and physical information.

[0660] An "audio signal" is data in the form of sound, including the tone and volume of the other person's voice, as well as linguistic elements.

[0661] "Facial expression information" refers to data used to analyze emotions based on the movements and changes in a person's face.

[0662] "Physical information" refers to data used for analysis based on the opponent's physical movements and changes in posture.

[0663] "Multidimensional analysis" is a technique that integrates and analyzes different types of data to predict emotional states and behaviors.

[0664] "Encryption" is a technology that converts data into a format that cannot be viewed by third parties in order to protect the confidentiality and security of the data.

[0665] "Access control" is a method of ensuring data security by granting or restricting access to information.

[0666] "Natural language processing technology" is a computational technique used to analyze human emotions and intentions from speech and text.

[0667] "Machine vision technology" is a technology that uses cameras and sensors to analyze visual information and recognize objects and facial expressions.

[0668] A "strategy" refers to a set of strategies or suggestions designed to support users in taking appropriate action based on the analysis results.

[0669] "Immediately" means that the process from information acquisition to analysis and proposal of strategies is carried out in real time without any delay.

[0670] One embodiment of this invention is a system that uses a portable information terminal to acquire voice, facial expression, and body movement data in real time, and provides the user with useful strategies based on that data. This allows the user to gain an advantage in negotiation and interpersonal relationship scenarios.

[0671] The devices include smartphones and wearable devices, which collect data through microphones, cameras, accelerometers, etc. Voice signals, facial expression information, and body information are compressed by the devices before being sent to the server. The server receives this data and performs preprocessing such as noise reduction and data normalization. Technologies used include speech recognition, computer vision, and motion analysis, and a generative AI model combines these technologies to analyze the data from multiple perspectives.

[0672] Specifically, the server extracts emotions and keywords from voice data using natural language processing technology. Furthermore, it applies machine vision technology to analyze facial expressions and uses motion analysis technology to predict movement patterns. This allows the user to be presented with visual strategies using cursors and icons, assisting them in selecting the best course of action in each situation.

[0673] As a concrete example, in a business negotiation scenario, the server analyzes the other party's tone of voice and facial movements to instantly determine if they have any doubts or concerns. Based on these results, a prompt message such as "Analyze the facial expressions and vocal characteristics of the negotiating party when they have doubts, and suggest a strategy to propose" is input into the AI ​​model, allowing the negotiator to make the most appropriate proposal and gain an advantage.

[0674] Furthermore, to protect user privacy, data is encrypted and access is strictly controlled. This makes it possible to deliver effective results to users while simultaneously protecting data and achieving high analytical accuracy.

[0675] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0676] Step 1:

[0677] The device uses sensors from smartphones and wearable devices to acquire voice, facial expression, and body movement data in real time. This data is in its raw, uncompressed state. Specifically, the input consists of voice picked up by the microphone, facial expressions captured by the camera, and body movements measured by the accelerometer. Because it is difficult to send this data to a server simultaneously, it is compressed within the device to improve communication efficiency. The output is the compressed data.

[0678] Step 2:

[0679] The device sends the compressed data to the server via the internet. To ensure the security of data transmission, the data is encrypted. The main inputs here are the compressed audio, facial expression, and body movement data compressed in step 1. The output maintains a secure transmission state to the server.

[0680] Step 3:

[0681] The server performs preprocessing on the received data, such as noise reduction and data normalization. The input here is encrypted, compressed data sent from the terminal. Specifically, background noise is filtered from audio data, color correction is performed on facial data, and motion data is smoothed. These processes result in clean data suitable for analysis. The output is the preprocessed data.

[0682] Step 4:

[0683] The server performs multidimensional analysis using pre-processed data. This analysis utilizes a generative AI model to extract emotions and keywords from voice data using natural language processing techniques, and applies machine vision techniques to facial and motion data for analysis. The input is the pre-processed data obtained in step 3, and the output is the result of emotional state and behavioral predictions. This allows the server to predict the other party's potential emotions and behaviors.

[0684] Step 5:

[0685] The server generates optimal strategies and suggestions for the user based on the analysis results. This process is carried out through prompts generated by the generative AI model. The input is the analysis results obtained in step 4, and the output is text data containing specific suggestions and strategies. This allows the server to construct strategies to support the user.

[0686] Step 6:

[0687] Users visually retrieve strategies generated by the server through their device interface. The optimal strategy is displayed, allowing users to adjust their responses in real time and gain an advantage in negotiations and communication. Input is the server's suggestions and strategy data, while output is the user's decision and the resulting actions. This enables users to take appropriate responses based on the situation.

[0688] (Application Example 1)

[0689] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0690] In modern family environments, communication among family members is not always efficient. Furthermore, it is not easy to assess individual emotional and health conditions and respond appropriately. Moreover, in today's world where the protection of personal information is increasingly important, accurate data analysis while simultaneously protecting privacy is required.

[0691] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0692] In this invention, the server includes means for preprocessing acquired data and performing multi-dimensional analysis to predict emotional states and behaviors; means for analyzing emotional states from changes in voice and facial expressions in a home autonomous device and providing corresponding response methods; and means for making suggestions for health management purposes. This makes it possible to facilitate communication within the home, provide appropriate advice and health management suggestions to each individual, while ensuring the privacy of information.

[0693] A "portable information terminal" is an information processing device that is portable by the user and has the function of collecting and transmitting various types of data.

[0694] "Audio data" is a form of information that includes human speech and intonation, and is a digital signal used to analyze emotions and intentions.

[0695] "Facial expression data" refers to information that digitally records the characteristic movements and expressions of an individual's face, and is used to estimate emotions.

[0696] "Body movement data" refers to digital information that represents the movements and postures of the human body, and is used to analyze intentions and states.

[0697] "Multidimensional analysis" is a method that comprehensively analyzes multiple types of data to extract correlations and hidden patterns.

[0698] "Home autonomous devices" are devices designed to operate within the home and automatically perform specific tasks.

[0699] "Health management suggestions" refer to information that provides advice and action guidelines tailored to an individual's health condition.

[0700] "Privacy protection" refers to technical and organizational measures to prevent the leakage or unauthorized access of personal information.

[0701] The system implementing this invention is built around a portable information terminal. The terminal uses built-in sensors to acquire voice, facial expression, and body movement data in real time. The data acquired from this terminal is transmitted to a server in the cloud via a communication module, where data preprocessing is performed.

[0702] The server uses natural language processing techniques to process audio data, and this process can utilize speech recognition software such as Google Cloud Speech-to-Text. This technology is used to extract emotions and keywords from the audio. Simultaneously, facial expression data is analyzed using machine vision technologies such as Amazon Rekognition. Body movement data is analyzed using motion analysis technologies such as the OpenPose library. Through these multi-faceted analyses, the emotional state and behavior of the other party are predicted.

[0703] The server uses a generative AI model to generate appropriate response strategies based on these analysis results. The generated strategies are visually presented on the displays of mobile devices and autonomous devices through a user-friendly interface. In situations where voice guidance is used, information is provided to the user audibly.

[0704] Furthermore, the server encrypts data and uses the SSL / TLS protocol for access control. This makes the system secure from a privacy perspective.

[0705] As a concrete example, consider a scenario where a home-use autonomous device analyzes the emotional state of family members during dinner each day and provides appropriate advice. If the voice data detects that someone is feeling tired, suggestions for relieving fatigue will be offered. This can facilitate smoother communication within the family and contribute to improving individual health.

[0706] An example of a prompt might be: "Analyze the emotions from the following audio data and generate a safe strategy to suggest. The emotion data based on the audio indicates a normal level of anger and moderate interest. Provide advice on what to do next."

[0707] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0708] Step 1:

[0709] The device uses its built-in microphone and camera to acquire voice, facial expression, and body movement data in real time. The input in this step consists of the user's voice and visual data. The acquired data is temporarily stored digitally within the device.

[0710] Step 2:

[0711] The terminal compresses the acquired data and sends it to the server using an encryption protocol. The input is the data acquired in step 1, and the output is a compressed and encrypted data stream. This data is sent to the server via a secure communication channel.

[0712] Step 3:

[0713] The server performs noise reduction and normalization processing on the received data. The input is compressed data received from the terminal, and the output is clean data suitable for analysis. Signal processing software is used for noise reduction to improve data quality.

[0714] Step 4:

[0715] The server uses natural language processing techniques to analyze audio data and extract emotions and keywords. The input is audio data, and the output is extracted emotion information and keywords. A speech recognition engine performs these extractions, and a generative AI model supports the data analysis.

[0716] Step 5:

[0717] The server uses machine vision technology to analyze facial expression data and body movement data. The inputs are facial expression data and body movement data, and the outputs are the analyzed emotional state and predicted movement. Computer vision algorithms process the data and perform movement estimation.

[0718] Step 6:

[0719] The server uses a generated AI model to generate an appropriate response strategy based on the analysis results. The input in this step is the analysis results from steps 4 and 5, and the output is the response strategy. The generated strategy is instructed to the AI ​​model using prompt statements.

[0720] Step 7:

[0721] The server sends the generated response strategy to the terminal and presents it to the user visually or audibly. The input is the generated response strategy, and the output is the information provided to the user through the terminal's display or audio functions. The user can choose an action based on the presented information.

[0722] Step 8:

[0723] The server continuously implements encryption and access control to protect data privacy throughout the entire process. Inputs are the data at each step, and outputs are securely protected data. Appropriate encryption is performed using the SSL / TLS protocol.

[0724] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0725] This invention is a system that analyzes voice, facial expressions, and body movement data collected from a mobile information terminal in a multi-faceted manner to recognize the user's emotions in real time and generate and present appropriate strategies according to the situation. This system incorporates an emotion engine, enabling a more accurate understanding of the emotional state of the user and their negotiating partner.

[0726] First, the device continuously collects voice, facial expression, and movement data using a smartphone or wearable device. Voice data is acquired through a microphone, and facial expression data is captured by a camera. Movement data is acquired in real time using a gyroscope and accelerometer.

[0727] Next, the server receives this data and performs preprocessing such as noise reduction and standardization. The preprocessed data is then subjected to multi-dimensional analysis by a generative AI equipped with an emotion engine. Here, natural language processing techniques are applied to the audio data to extract emotion keywords. Computer vision techniques are used for facial expression data to assign emotion labels. Motion data is interpreted by a motion analysis algorithm.

[0728] The emotion engine uses this data to recognize and predict the user's emotional state and the behavioral tendencies of the negotiating partner. For example, in business negotiations, if the other party is feeling anxious, the engine immediately reflects data indicating that emotion and presents a strategy to proceed with the negotiation calmly. The user receives the strategy visually through the device interface and can take appropriate countermeasures.

[0729] Furthermore, the system receives user feedback and continuously improves the accuracy of its emotion engine's analysis. As data is accumulated on the results of users acting according to the system's suggestions, the accuracy of subsequent suggestions improves.

[0730] This system ensures user peace of mind by strictly considering privacy protection, encrypting data, and implementing access control. This provides an effective and secure form of the invention.

[0731] The following describes the processing flow.

[0732] Step 1:

[0733] The device collects audio data using the microphone of a smartphone or wearable device, records facial expressions with a camera, and captures body movements through an accelerometer and gyroscope. The data is temporarily stored in local storage.

[0734] Step 2:

[0735] The device compresses the collected voice, facial expression, and motion data and transmits it to the server using an available communication network (such as Wi-Fi or mobile data).

[0736] Step 3:

[0737] The server receives data sent from the terminal and performs preprocessing such as noise reduction, standardization, and outlier removal to improve its quality.

[0738] Step 4:

[0739] The server analyzes pre-processed data using a generative AI equipped with an emotion engine. For audio data, natural language processing (NLP) is used to analyze emotional keywords and tones, and for facial expression data, computer vision technology is used to classify emotional labels. Motion data undergoes historical analysis using a motion analysis algorithm.

[0740] Step 5:

[0741] The server models the current emotional state of the user and their negotiating partner based on the results analyzed by the emotion engine, and uses this information to predict future actions.

[0742] Step 6:

[0743] The server generates effective strategies for the user based on recognized emotional states and behavioral predictions. These strategies are designed taking negotiation techniques and communication approaches into consideration.

[0744] Step 7:

[0745] The device receives strategies generated from the server and presents them to the user visually using an intuitive user interface. The user can then adjust their actions in real time based on this information.

[0746] Step 8:

[0747] The server encrypts user data and implements strict access control. Privacy is protected, and only users with legitimate permission can view the data.

[0748] (Example 2)

[0749] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0750] In today's diverse communication landscape, quickly and accurately understanding a person's emotional state and behavioral tendencies is crucial, but conventional technologies have struggled to achieve this in real time. Furthermore, there is a need for methods that efficiently utilize feedback to improve analysis accuracy while ensuring the privacy of acquired data.

[0751] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0752] In this invention, the server includes means for acquiring voice data, facial expression data, and motion data from information and communication devices in real time; means for preprocessing the acquired data and performing multifaceted analysis to recognize and predict emotions and behavioral tendencies; and means for collecting user feedback to improve the accuracy of the analysis. This makes it possible to grasp the emotional state of the other party in real time and propose appropriate communication strategies.

[0753] "Information and communication equipment" is a general term for electronic devices that collect data such as the user's voice, facial expressions, and movements in real time.

[0754] "Voice data" refers to data that includes the user's voice characteristics, collected through sensors such as microphones.

[0755] "Facial expression data" refers to data that shows the state of the user's face as captured by a camera.

[0756] "Motion data" refers to data about the user's body movements acquired using gyroscopes and accelerometers.

[0757] "Preprocessing" refers to the process of preparing data for analysis, such as removing noise and standardizing it, to make it suitable for analysis.

[0758] "Multifaceted analysis" is a method that integrates and analyzes multiple data sources to extract more advanced information.

[0759] "Emotion recognition" is the process of identifying a user's emotional state from collected data.

[0760] "Strategy generation" is the process of designing the optimal communication strategy for users based on analyzed emotions and behavioral tendencies.

[0761] "Feedback" refers to the collection of data based on information and results obtained from users, in order to improve the system's performance and suggested features.

[0762] Encryption is a technology that transforms information to protect it from unauthorized access.

[0763] "Access control" is a management method that restricts access to data and functions within a system, allowing only authorized users to use them.

[0764] This invention is a system that uses information and communication equipment to collect and analyze data on the user's voice, facial expressions, and movements in real time, thereby accurately recognizing the user's emotional state and suggesting appropriate countermeasures. This system is mainly composed of a terminal and a server working together.

[0765] The terminal consists of information and communication devices such as smartphones and wearable devices, and acquires the user's voice in real time using a microphone, facial expressions with a camera, and movements with a gyroscope and accelerometer. This acquired data is then transmitted to a server.

[0766] The server processes the received data. First, it performs preprocessing such as noise reduction on the data to prepare it for analysis. Next, it applies natural language understanding techniques to the audio data using an emotion engine to extract emotion keywords. For facial expression data, it applies image processing techniques to identify emotion labels. For motion data, it uses a motion analysis algorithm to analyze the user's behavioral tendencies in detail. Based on this, a generative AI model constructs an appropriate strategy and presents it to the user.

[0767] Users review strategies presented by the server via their terminals and apply them to their actual communication. For example, if the other party in a negotiation appears anxious, the system might suggest a strategy such as, "It would be best to continue explaining in a calm tone." This allows users to conduct negotiations more smoothly.

[0768] In terms of security, data encryption and access control are thoroughly implemented throughout the entire system, enabling safe operation while protecting privacy.

[0769] A concrete example is a business meeting where understanding the emotions of others and facilitating smooth conversation based on those emotions is required. An example of a prompt for a generative AI model would be, "Assuming the user is participating in the meeting, please suggest the next topic to discuss based on the participants' emotions."

[0770] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0771] Step 1:

[0772] The device acquires data on the user's voice, facial expressions, and movements in real time. Specifically, it records voice with a microphone, captures facial expressions as video with a camera, and detects movements with a gyroscope and accelerometer. Since this input data contains noise, preprocessing is required.

[0773] Step 2:

[0774] The server preprocesses the audio, facial expression, and motion data received from the terminal. It applies a noise reduction filter to improve the clarity of the audio data by removing irrelevant data. Facial expression data is standardized using a video conversion algorithm, and motion data is normalized based on a baseline. The output of this step is clean data ready for analysis.

[0775] Step 3:

[0776] The server performs a multifaceted analysis of the pre-processed data. Natural language understanding techniques are applied to audio data to extract emotional keywords. Image processing algorithms are used to detect emotional labels in facial expression data. Motion data is analyzed using a motion analysis model to evaluate user behavioral tendencies. The output obtained through these analyses provides detailed information about the user's emotional state and behavior.

[0777] Step 4:

[0778] The server utilizes an emotion engine to generate the optimal strategy for the user based on the analysis results. The generating AI model designs countermeasures tailored to the situation and emotions, creating a concrete plan to propose to the user. This output includes clear action guidelines to present to the user.

[0779] Step 5:

[0780] Users receive strategies provided by the server via their devices and utilize them in real-world situations. For example, they might adjust their speaking style and gestures during a meeting according to a suggested strategy. The results and impressions obtained from this use are sent to the system as feedback and used for future analysis.

[0781] (Application Example 2)

[0782] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0783] In modern society, managing stress within the family and providing effective communication methods are crucial. However, conventional technologies have difficulty accurately understanding users' emotional states and responding appropriately based on that understanding.

[0784] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0785] In this invention, the server includes means for continuously acquiring voice information, facial expression information, and body movement information from a portable information device; means for preprocessing the acquired information and performing multi-dimensional analysis to predict emotional state and behavior; and means for playing audio media according to the user's emotional state and suggesting refreshing activities. This enables real-time understanding of emotional state within the home and appropriate responses based on that understanding.

[0786] A "portable information device" is a portable electronic device used to continuously acquire voice information, facial expression information, and body movement information.

[0787] "Audio information" refers to data of human voices and sounds acquired through a microphone.

[0788] "Facial expression information" refers to data about a person's facial expressions captured by a camera.

[0789] "Body movement information" refers to data about a person's movements acquired via gyroscopes and accelerometers.

[0790] "Multidimensional analysis" is an analytical technique that analyzes acquired information from various perspectives to predict emotional states and behaviors.

[0791] A "visual presentation mechanism" is an interface designed to display the generated strategy in an easy-to-understand manner for the user.

[0792] "Playing audio media" means playing appropriate audio content through an output device based on the user's emotional state.

[0793] "Personal information protection" refers to security measures that encrypt and properly manage information for the purpose of protecting privacy.

[0794] "Machine vision technology" is a technology that uses computers to analyze image data to understand people's facial expressions and scenes.

[0795] "Natural language processing technology" is a technique that analyzes language data to extract emotions and keywords from audio information.

[0796] The system implementing this invention has the function of continuously acquiring voice information, facial expression information, and body movement information using a smartphone or wearable device. By collecting data using a microphone, camera, gyroscope, and accelerometer built into the terminal, it becomes possible to acquire information in real time.

[0797] The server receives the acquired data, performs preprocessing such as noise reduction and standardization, and then performs multidimensional analysis using a generative AI model equipped with an emotion engine. In this analysis, natural language processing techniques are applied to the audio data to extract emotions and related keywords, and machine vision techniques are used for facial expression data. Furthermore, motion data is interpreted by a motion analysis algorithm. Specific technologies used include deep learning models used in natural language processing and image recognition algorithms used in machine vision techniques.

[0798] Based on the generated emotional state, the server generates the optimal strategy for the user and presents it through a visual interface. This allows users to receive information and suggestions tailored to their emotional state in real time. Furthermore, to support refreshing activities, audio media can be played; for example, relaxing music can be played when the user is feeling stressed, improving their comfort level.

[0799] Furthermore, the system is designed to protect personal information by encrypting all personal data and implementing thorough access controls, ensuring that users can use it with peace of mind while protecting their privacy.

[0800] As a concrete example, if the system detects stress during a casual conversation, it will suggest activities to help the user relax. This feature aims to accurately understand the user's emotional state and provide a better living environment.

[0801] An example of a prompt to input into the generating AI model is: "Based on voice data and facial expression data, please analyze the current emotional state and suggest appropriate relaxation techniques. If stress or anxiety is present, please also provide practical solutions."

[0802] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0803] Step 1:

[0804] The device acquires audio information, facial expression information, and body movement information using a microphone, camera, gyroscope, and accelerometer. Input is real-time audio, image, and motion data, and output is the raw data for each. This allows the device to prepare a variety of data.

[0805] Step 2:

[0806] The server receives raw data sent from the terminal. The input consists of raw data of voice, facial expressions, and body movements, which are preprocessed. The output is noise-removed and normalized data. The server improves the quality of the data.

[0807] Step 3:

[0808] The server runs a generative AI model to analyze pre-processed data. The input is pre-processed audio, facial expression, and body movement data, and the output is emotional state and behavioral prediction derived from the analysis. The server analyzes the data from multiple perspectives to understand the emotional state.

[0809] Step 4:

[0810] The server generates the optimal strategy for the user based on the analysis results. The input is emotional state and behavioral prediction, and the output is a specific strategic proposal. The server devises appropriate countermeasures depending on the situation.

[0811] Step 5:

[0812] Users review the generated strategies through a visual interface on their devices. The input is the strategy proposal, and the output is visualized information. Users receive the information visually and make decisions about their actions.

[0813] Step 6:

[0814] The server plays audio media based on the user's emotional state. The input is emotional state data, and the output is the playback of audio media. The server selects appropriate audio content to help alleviate the user's stress.

[0815] Step 7:

[0816] The server encrypts all personal data and implements appropriate access controls. Inputs consist of all collected and analyzed data, while output is securely protected data. The server protects data privacy.

[0817] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0818] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0819] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0820] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0821] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0822] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0823] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0824] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0825] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0826] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0827] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0828] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0829] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0830] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0831] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0832] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0833] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0834] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0835] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0836] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0837] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0838] The following is further disclosed regarding the embodiments described above.

[0839] (Claim 1)

[0840] A means of acquiring voice data, facial expression data, and body movement data from a mobile device in real time,

[0841] A means for preprocessing acquired data and performing multidimensional analysis to predict emotional states and behaviors,

[0842] A means for generating an appropriate strategy for the user based on the aforementioned analysis results,

[0843] A means of providing an interface for visually presenting the generated strategy to the user,

[0844] Means for encrypting data and implementing access control to protect privacy,

[0845] A system that includes this.

[0846] (Claim 2)

[0847] The system according to claim 1, which extracts emotions and keywords using natural language processing technology in the analysis of audio data.

[0848] (Claim 3)

[0849] The system according to claim 1, which uses computer vision technology in the analysis of facial expression data.

[0850] "Example 1"

[0851] (Claim 1)

[0852] A means for instantly acquiring voice signals, facial expression information, and bodily information from an information acquisition device,

[0853] A means for performing multidimensional analysis to pre-process the acquired information and predict emotional states and behaviors,

[0854] A means for generating appropriate measures for users based on the aforementioned analysis results,

[0855] A means for providing a display means for visually presenting the generated strategy to the user,

[0856] Means for encrypting data and implementing access control to maintain the confidentiality of information,

[0857] A system that includes this.

[0858] (Claim 2)

[0859] The system according to claim 1, which uses natural language processing technology to extract emotions and important words in the analysis of an audio signal.

[0860] (Claim 3)

[0861] The system according to claim 1, which uses machine vision technology in the analysis of facial expression information.

[0862] "Application Example 1"

[0863] (Claim 1)

[0864] A means of acquiring voice data, facial expression data, and body movement data from a mobile device in real time,

[0865] A means for preprocessing acquired data and performing multidimensional analysis to predict emotional states and behaviors,

[0866] A means for generating an appropriate strategy for the user based on the aforementioned analysis results,

[0867] Means for providing an interface to visually or audibly present the generated strategy to the user,

[0868] In a home-use autonomous device, a means for analyzing emotional states from changes in voice and facial expressions and providing a corresponding response method,

[0869] Means of making suggestions for the purpose of health management,

[0870] Means for encrypting data and implementing access control to protect privacy,

[0871] A system that includes this.

[0872] (Claim 2)

[0873] The system according to claim 1, which extracts emotions and keywords using natural language processing technology in the analysis of audio data.

[0874] (Claim 3)

[0875] The system according to claim 1, which uses machine vision technology in the analysis of facial expression data.

[0876] "Example 2 of combining an emotion engine"

[0877] (Claim 1)

[0878] A means of acquiring voice data, facial expression data, and motion data from information and communication devices in real time,

[0879] A means for preprocessing acquired data and performing multifaceted analysis to recognize and predict emotions and behavioral tendencies,

[0880] A means for generating appropriate measures for users based on the aforementioned analysis results,

[0881] A means of providing an interface for visually presenting the generated policies to the user,

[0882] Means for encrypting data and implementing access control to protect data confidentiality,

[0883] A means of collecting user feedback to improve analysis accuracy,

[0884] A system that includes this.

[0885] (Claim 2)

[0886] The system according to claim 1, which uses natural language understanding technology to select emotions and keywords in the analysis of audio data.

[0887] (Claim 3)

[0888] The system according to claim 1, which uses image processing technology in the analysis of facial expression data.

[0889] "Application example 2 when combining with an emotional engine"

[0890] (Claim 1)

[0891] A means for continuously acquiring voice information, facial expression information, and body movement information from a portable information device,

[0892] A means for preprocessing acquired information and performing multidimensional analysis to predict emotional states and behaviors,

[0893] A means for generating an appropriate strategy for the user based on the aforementioned analysis results,

[0894] A means of providing a mechanism for visually presenting the generated strategy to the user,

[0895] A means of playing audio media according to the user's emotional state and suggesting refreshing activities,

[0896] Means to encrypt information and implement access control in order to protect personal information,

[0897] A system that includes this.

[0898] (Claim 2)

[0899] The system according to claim 1, which extracts emotions and keywords using natural language processing technology in the analysis of speech information.

[0900] (Claim 3)

[0901] The system according to claim 1, which uses machine vision technology in the analysis of facial expression information. [Explanation of Symbols]

[0902] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of acquiring voice data, facial expression data, and body movement data from a mobile device in real time, A means for preprocessing acquired data and performing multidimensional analysis to predict emotional states and behaviors, A means for generating an appropriate strategy for the user based on the aforementioned analysis results, A means of providing an interface for visually presenting the generated strategy to the user, Means for encrypting data and implementing access control to protect privacy, A system that includes this.

2. The system according to claim 1, which extracts emotions and keywords using natural language processing technology in the analysis of audio data.

3. The system according to claim 1, which uses computer vision technology in the analysis of facial expression data.

Images