system

Abstract

We provide the system. [Solution] A sensor means for collecting user voice and video data, A communication means for securely transmitting collected audio and video data, A data analysis means that analyzes the transmitted data and evaluates the user's emotions and health status, A task generation means that generates optimal tasks and learning programs for the user based on evaluation, A presentation means that transmits and displays the generated tasks and programs to the user terminal, A feedback processing mechanism that receives user feedback and incorporates it into future proposals, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern society, individual health and mental states vary daily, often directly affecting work and learning performance. Therefore, efficient task management according to each individual's state at that time is required, but the means for this have not been fully established. To solve this problem, it is necessary to develop a system that accurately grasps the user's real-time emotions and health status and provides individually optimized tasks and learning programs based on this.

Means for Solving the Problems

[0005] This invention includes sensor means for collecting user voice and video data and securely transmitting this data to a server. The server analyzes the transmitted data and evaluates the user's emotions and health status. Furthermore, based on the evaluation results, it generates tasks and learning programs optimized for the user and sends and displays these suggestions on the user's terminal. In addition, it provides a means for continuously improving the accuracy of the system by collecting user feedback and incorporating it into future task suggestions. This enables effective task management adapted to individual circumstances.

[0006] "Sensor means" is a general term for devices and functions used to collect user voice and video data in real time.

[0007] "Communication methods" is a general term for technologies and protocols used to encrypt collected data and securely transmit it to a server.

[0008] "Data analysis methods" is a general term for algorithms and processes that use received data to analyze and evaluate the user's emotions and health status.

[0009] "Task generation means" refers to the logic and algorithms used to create tasks and learning programs optimized for the user based on analysis results.

[0010] "Presentation means" refers to the general term for interfaces and technologies used to visually display generated tasks and programs on a user's terminal.

[0011] "Feedback processing means" is a general term for processes and systems used to collect user feedback information and utilize it for future proposals.

[0012] "Anonymization" is a general term for technologies that protect privacy by removing or masking information that could identify an individual.

[0013] "Variation patterns" is a general term for analytical methods and results used to identify changes in users' emotions and health based on past data. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention is a system that proposes optimized tasks and learning programs according to the user's daily mental and physical state. The system consists of a terminal that collects the user's voice and video data, and a server that analyzes this data and creates optimal suggestions.

[0036] First, the device collects data such as the user's voice and facial expressions in real time. Voice is captured through a microphone and analyzed for tone, speed of speech, and voice quality. Facial expression data is acquired through a camera and analyzed for facial movements and changes in expression. The device immediately transmits this data to a server using a secure protocol.

[0037] Next, the server performs analysis using the received data. An AI algorithm processes the data and evaluates the user's emotional and health states. It can also refer to past data to identify patterns in the user's state changes. Based on these analysis results, the server generates tasks and learning programs optimized for each individual user.

[0038] The generated suggestions are sent to the terminal and presented to the user visually. The terminal provides a user-friendly interface, designed to make it easy for the user to understand the suggested tasks. The user selects and executes tasks and learning programs based on the suggestions. The results of the execution and feedback are then sent back to the server via the terminal.

[0039] As an example, let's consider the case of a salaried worker. In the morning, while commuting to work, the device collects voice tone and facial expressions, and determines the stress level from this data. If the server detects high stress levels, it suggests lighter work tasks and a relaxation program to do during lunchtime. The device notifies the user, and the user can incorporate these suggestions into their schedule for the day.

[0040] This system allows users to efficiently complete tasks tailored to their physical and mental state on any given day, resulting in improved performance and reduced stress.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The device collects the user's voice and video data in real time. Voice data is acquired via a microphone, and the tone, speed, and pitch of the voice are analyzed. Facial data is acquired using a camera, and facial movements and changes in expression are analyzed.

[0044] Step 2:

[0045] The device quickly encrypts the collected data and sends it to the server using a secure communication protocol. This communication is designed to protect user privacy.

[0046] Step 3:

[0047] The server decodes the received data and analyzes it using an AI algorithm. It evaluates the user's emotional state and health status from their voice and facial expressions, and compares this information with past data to identify patterns of change.

[0048] Step 4:

[0049] Based on the analysis results, the server generates tasks and learning programs tailored to the user. This includes suggestions for specific tasks and activities that match the user's current emotional and physical state.

[0050] Step 5:

[0051] The server sends the generated suggestions to the terminal. The terminal receives them and presents them to the user through a user-friendly, visualized interface.

[0052] Step 6:

[0053] The user reviews suggestions via an interface on their device and selects tasks or programs to execute. The user provides feedback and inputs the results into the device.

[0054] Step 7:

[0055] The device collects user feedback and execution results, re-encrypts them, and sends them to the server. The server stores this information and uses it as foundational data to improve the accuracy of future suggestions.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] In modern life, there is a need to effectively understand users' mental and physical states and provide appropriate activities and learning content accordingly. However, conventional systems have difficulty accurately assessing users' emotions and physical states in real time, making it impossible to provide individually optimized suggestions. To solve this problem, there is a need for technology that can accurately assess the user's state and quickly provide individually customized suggestions.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes information acquisition means for acquiring the user's voice and video and analyzing tone, speech speed, facial movements, etc.; data transmission means for securely transmitting the acquired voice and video information; and information analysis means for processing the transmitted information and evaluating the user's emotions and health status. This makes it possible to individually generate and provide optimal activities and learning procedures for the user in real time.

[0061] "Information acquisition means" refers to methods for acquiring the user's voice and video and analyzing their tone, speech speed, facial movements, etc.

[0062] "Data transmission means" refers to means for securely transmitting acquired audio and video information to a server.

[0063] "Information analysis means" refers to means for processing transmitted information and evaluating the user's emotions and health status.

[0064] A "proposal generation method" is a means for creating optimal activities and learning procedures for the user based on information analysis.

[0065] "Display means" refers to means for transmitting generated activities and learning procedures to a user device for visual presentation.

[0066] A "response processing means" is a means of collecting user feedback and using it to inform future proposals.

[0067] This invention is a system that provides activities and learning procedures optimized according to the user's mental and physical state. The system mainly consists of a terminal and a server. The terminal includes hardware that collects the user's voice and video data in real time. Specifically, the terminal incorporates a high-sensitivity microphone and a high-resolution camera to acquire the user's voice data and facial expressions. For the voice data, voice analysis software is used to analyze tone, speech speed, and voice pitch. For the video data, facial recognition and expression analysis software analyzes facial movements and changes in facial expressions in real time.

[0068] The acquired data is transmitted to the server via a secure communication protocol. The server analyzes the received data using advanced AI algorithms. This AI utilizes generative AI models to assess the user's current emotions and health status, and tracks changes in their state by analyzing past data as well.

[0069] Based on the analysis results, the server generates tasks and learning programs optimized for the user's state. For example, if the server determines that the user is experiencing high stress, it can suggest relaxation exercises or light work tasks. This generated content is sent to the terminal and presented to the user visually. The terminal's user interface is designed to make the suggestions clear and easy for the user to understand.

[0070] The user selects and performs a suggested task, and their experience and feedback are sent back to the server via the terminal. This allows the system to incorporate the user's feedback into future suggestions. A possible example of a prompt message would be, "Evaluate the user's stress level based on audio and video data, and suggest an appropriate task." This system enables users to efficiently perform activities suited to their condition, allowing them to enjoy a better quality of life.

[0071] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0072] Step 1:

[0073] The device acquires the user's voice in real time using a microphone. The input is raw audio data, and the output is digital data for speech analysis. Specifically, speech analysis software is used to decompose the audio data into tone, speech speed, and voice pitch, and feature quantities for each element are extracted.

[0074] Step 2:

[0075] The device captures the user's video with a camera and analyzes their facial expressions. The input is real-time video data, and the output is the analysis results regarding changes in facial expressions and facial movements. Facial recognition software analyzes this data to identify specific emotions and facial features.

[0076] Step 3:

[0077] The terminal aggregates audio and video analysis data and sends it to the server. The input is a dataset of audio and video analysis results, and the output is transmission data converted into a secure format. Using a data transmission means, these analysis results are sent to the server via a configured secure communication protocol.

[0078] Step 4:

[0079] The server analyzes received data using an AI algorithm to assess the user's emotions and health. The input is the analyzed audio and video data, and the output is the assessment of the user's current emotions and health. It utilizes a generative AI model to calculate multiple state indicators and infer the user's psychological and physiological state.

[0080] Step 5:

[0081] The server generates optimal activity and learning programs based on the evaluation results. The input is the evaluation results of the user's emotions and health state, and the output is a proposal for personalized activity and learning programs. Using the proposal generation means, appropriate tasks are generated and sent to the user's terminal in the next stage.

[0082] Step 6:

[0083] The terminal visually presents suggestions received from the server to the user. The input is suggested data for tasks and learning programs sent from the server, and the output is content displayed in a way that is easy for the user to understand. The terminal's interface clearly displays the suggestions and assists the user in selecting and performing activities.

[0084] Step 7:

[0085] The user selects and executes a suggested activity or learning program. The input is the displayed suggested activity or learning program, and the output is the result of the user's selection and execution. The user's selections are recorded and saved on the device as data for future use.

[0086] Step 8:

[0087] The device sends user feedback and activity results to the server. Inputs are the user's choices and execution results, while outputs are feedback data that forms the basis for future suggestions. Data from the device is transmitted to the server and incorporated into future suggestions.

[0088] (Application Example 1)

[0089] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0090] In modern society, users are expected to maintain their mental and physical health in their daily lives, but continuous management is not easy. In particular, there is a lack of methods to immediately respond to the stress and changes in physical condition that occur in daily life and to suggest appropriate tasks and mitigation activities. Solving this problem is essential.

[0091] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0092] In this invention, the server includes detection means for acquiring user acoustic and image data, communication means for securely transferring the acquired acoustic and image data, and information analysis means for analyzing the transferred data and evaluating the user's emotional and physical state. This makes it possible to propose optimal work tasks and relaxation activities in real time and on an individual basis, based on the user's daily mental and physical state.

[0093] A "user" is a person who is the subject of acoustic and image data, and who receives suggestions for optimal tasks and mitigation activities from the system.

[0094] "Acoustic data" refers to information about the user's voice and sounds acquired through devices such as microphones, and is part of the analysis used to evaluate their emotional state.

[0095] "Image data" refers to visual information about a user's face and movements collected through devices such as cameras, and is part of the analysis used to evaluate their emotional state.

[0096] "Detection means" refers to devices or functions for acquiring acoustic and image data from the user.

[0097] "Communication methods" refer to the technologies and protocols used to securely transfer acquired data, and are used to ensure the safe transmission of data.

[0098] "Information analysis means" refers to algorithms and processing technologies for analyzing transmitted data and evaluating the user's emotional state and health status.

[0099] "Business tasks" refer to specific tasks or activities that should be proposed to the user, and are selected taking into consideration the user's mental and physical state.

[0100] "Relaxation activities" refer to activities and activities suggested to reduce user stress and promote the user's mental relaxation.

[0101] The system for carrying out this invention uses a terminal equipped with a microphone and a camera as a device for acquiring the user's audio and image data. The terminal collects the user's voice tone, speed, and facial expressions in real time and transmits the data to a server using a secure protocol. Security in this series of communications is ensured using a protocol such as HTTPS.

[0102] The server uses AI algorithms to process the received data. Specifically, it uses machine learning frameworks such as TENSORFLOW® and PyTorch to perform analysis to evaluate the user's emotional state and health status. This analysis makes it possible to identify patterns in changes in the user's state by comparing it with past data.

[0103] Based on the analysis results, the server generates work tasks and mitigation activities optimized for the user. These suggestions are communicated visually and audibly to the terminal, ensuring that the user can easily understand and perform them. As a result, the user performs the suggested activities and provides feedback, allowing the system to make even more personalized suggestions in the future.

[0104] For example, if the server determines that a user is experiencing high levels of fatigue or stress, it will suggest lighter work tasks or play relaxing music. At this time, the device will notify the user via voice, saying, "I will play relaxing music. Would you like to do some stretching?" and prompt the user to make a choice.

[0105] An example of an input prompt for the generating AI model is, "Based on the user's voice data and facial expression data, please suggest the most suitable relaxation activity for the day."

[0106] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0107] Step 1:

[0108] The device collects the user's audio and image data using a microphone and camera. During this process, it captures each user's voice and facial expressions as input. The collected audio tone, intonation, and changes in facial expressions are processed in real time and converted into digital data. The output is the raw data formed by this processing.

[0109] Step 2:

[0110] The device sends the collected data to the server using a secure protocol (e.g., HTTPS). The input here is the audio and image data generated in step 1. As output, unprocessed data of the user's voice and facial expressions is transferred to the server. This data transfer is performed using end-to-end encryption, ensuring privacy.

[0111] Step 3:

[0112] The server analyzes the received data using AI algorithms. The input consists of user voice and image data. The server uses machine learning frameworks such as TensorFlow and PyTorch to perform sentiment analysis and generate data that evaluates the user's mental and physical state. The output is evaluation data that indicates the user's state, including numerical values ​​and labels representing their stress level and emotional state.

[0113] Step 4:

[0114] The server generates optimal work tasks and mitigation activities for the user based on the analysis results. It uses the evaluation data generated in step 3 as input. The AI ​​model generated in this process is used to devise appropriate activities. The output is a list of specific tasks and mitigation activities to be proposed.

[0115] Step 5:

[0116] The server sends the generated suggestions to the terminal. The input is the list of tasks and activities generated in step 4. The output is instruction data for the terminal, which prepares the terminal to present the tasks to the user visually or audibly.

[0117] Step 6:

[0118] The terminal presents the received suggestions to the user. The input is suggestion data from the server. As output, the terminal actually notifies the user of the day's work tasks and relaxation activities. This notification includes specific actions that should be taken, such as suggesting playing music or stretching.

[0119] Step 7:

[0120] The user performs tasks and activities based on suggestions from the device. They refer to the suggestions from step 6 as input and reflect them in their actual actions. The results and feedback are sent back to the server via the device and used to improve the system's adaptability by influencing future suggestions. Feedback data is then generated as output.

[0121] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0122] This invention is a system that combines a user's voice and video data with an emotion engine that recognizes the user's emotions based on that data. The aim of this system is to recognize the user's emotional state in real time from the voice and video data and propose individually optimized tasks and learning programs.

[0123] Specifically, users use smartphones or wearable devices to collect voice and facial data. These devices are equipped with high-sensitivity microphones and cameras, which are used to acquire data such as the user's voice tone, facial expressions, and heart rate. The acquired data is analyzed by an emotion engine to evaluate the user's current emotional state. The emotion engine is built using an algorithm that combines multiple deep learning models, enabling it to identify emotions with high accuracy, even down to the subtle nuances of voice tone and facial expressions.

[0124] Next, the device securely transmits the analysis results to the server. Based on the received emotional state data, the server generates tasks and learning programs optimized for each individual user. At this time, the server also compares the data with past data to identify patterns in emotional fluctuations, enabling it to provide more sophisticated suggestions. The generated suggestions are sent to the device and displayed on the user's screen.

[0125] Users review the suggestions displayed on the screen and select tasks according to their mood and circumstances for the day. In this way, users can focus more effectively on work or learning. Furthermore, after completing a task, users can input feedback into their device. This feedback is sent back to the server and used to improve future task suggestions.

[0126] As a concrete example, consider a student user studying for final exams. The device detects a decrease in concentration from voice and facial expressions and suggests relaxing music and an efficient study schedule with breaks. The student tries the suggested schedule and provides feedback on how their concentration returns afterward, contributing to the improvement of the system's accuracy. Through this entire process, the user can achieve healthy and effective task execution.

[0127] The following describes the processing flow.

[0128] Step 1:

[0129] The device collects the user's voice and video data in real time. It uses the microphone to capture voice tone and speed, and the camera to capture changes in facial expressions.

[0130] Step 2:

[0131] The device inputs the collected data into the emotion engine. The emotion engine analyzes the audio and video data and uses a deep learning model to identify emotional states.

[0132] Step 3:

[0133] The device sends the emotion engine's recognition results to the server via a secure protocol. This data includes the user's specific emotional state and related information.

[0134] Step 4:

[0135] The server analyzes the received data and compares it with past data to analyze patterns of emotional fluctuations. This analysis allows us to understand the user's emotional tendencies.

[0136] Step 5:

[0137] The server generates optimal tasks and learning programs for the user based on their emotional state. The generated suggestions take into account the user's current mental and physical condition.

[0138] Step 6:

[0139] The server sends the tasks and programs it generates to the terminal. The terminal receives them and displays them to the user in a visually easy-to-understand format.

[0140] Step 7:

[0141] The user reviews the tasks and programs presented on their device and selects which ones to execute. The selected tasks are then reflected in the schedule.

[0142] Step 8:

[0143] After completing a task, the user enters feedback into the device. This feedback includes the results of the task and changes in their emotions.

[0144] Step 9:

[0145] The device sends feedback information to the server, which records it in a database. This data is then used as foundational information to make future task suggestions more accurate.

[0146] (Example 2)

[0147] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0148] In modern society, it is crucial to quickly and accurately understand an individual's emotional state and provide guidance for optimal behavior and learning based on that understanding. However, conventional systems lack the accuracy to analyze emotional fluctuations in real time. Furthermore, there is a challenge in effectively utilizing user feedback to improve the system.

[0149] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0150] In this invention, the server includes an inference means for analyzing the user's emotional state in real time, an analysis means for comparing and analyzing it with past data, and a generation means for generating individually optimized action and learning plans. This makes it possible to monitor the user's emotional state with high accuracy and provide individually optimized action and learning guidelines.

[0151] "Receiving means" refers to a device or software function for acquiring audio and video data from the user.

[0152] "Preprocessing means" refers to a device or software function for performing processing to reduce noise in acquired data and improve the accuracy of analysis.

[0153] "Inference means" refers to an algorithm or software function for analyzing a user's emotional state in real time based on pre-processed data.

[0154] "Transmission means" refers to a communication function or device for securely transferring analysis results to a server.

[0155] "Analysis method" refers to a function that compares past data with current sentiment data on a server to identify fluctuation patterns.

[0156] "Generative means" refers to algorithms or software functions for creating individually optimized action and learning plans.

[0157] "Presentation means" refers to a function that transmits the generated plan to the user's display device and presents it in a visually easy-to-understand format.

[0158] A "feedback processing mechanism" is a function that collects evaluations from users and uses them to improve the system or make suggestions for future updates.

[0159] To implement this invention, the user uses a terminal such as a smartphone or wearable device. These terminals are equipped with a high-sensitivity microphone and camera and have the ability to acquire audio and video data. The terminals may be integrated with a sensor system using a device platform such as Arduino or Raspberry Pi.

[0160] The terminal preprocesses audio data using digital signal processing technology to remove noise. Video data is broken down frame by frame to extract the user's facial features. The hardware used at this stage includes a microphone and camera module, while the software used includes audio processing libraries and image processing libraries.

[0161] The collected data is input from the device to an emotion engine for real-time analysis of emotional states. This emotion engine utilizes machine learning frameworks such as TensorFlow and PyTorch, and operates by combining multiple deep learning models. The server uses the SSL / TLS protocol to send and receive data to securely protect the received data.

[0162] The results analyzed by the emotion engine are sent to the server and compared with past data. The server uses data analysis techniques with Python and R to identify the user's emotional fluctuation patterns and generate tasks and learning programs optimized for the user. At this time, it utilizes prompt statements that the generating AI model can operate on to generate instructions in the form of, for example, "Based on my current concentration and stress level, please suggest ways to improve my study efficiency."

[0163] The generated tasks are sent to the terminal and presented on the user's screen. The terminal uses a user interface to visually display the suggestions in an easy-to-understand manner. The user performs the displayed tasks and inputs the results as feedback into the terminal. This feedback is then sent back to the server and used to improve the accuracy of future suggestions.

[0164] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0165] Step 1:

[0166] The device acquires the user's voice and video data in real time. The device's high-sensitivity microphone captures the user's voice, and the camera records facial expressions. The input is raw audio and video data, and the output is digital audio data and video frames in a noisy format.

[0167] Step 2:

[0168] The terminal preprocesses the acquired audio and video data. Specifically, noise is removed from the audio data through digital signal processing, and facial features are enhanced from the video data through image filtering. The input is the audio and video data from step 1, and the output is the digital data with noise removed and enhanced.

[0169] Step 3:

[0170] The device sends pre-processed data to the emotion engine. Based on the transmitted data, the emotion engine uses a deep learning model to infer the user's emotions. This process includes speech tone analysis and facial expression recognition. The input is the pre-processed data, and the output is the inferred emotion label and confidence level.

[0171] Step 4:

[0172] The device securely transmits the inferred sentiment data to the server. Here, the data is encrypted using the HTTPS protocol. The input is the sentiment label and confidence level, and the output is the securely transmitted data.

[0173] Step 5:

[0174] The server compares the received sentiment data with historical data. It uses a data analysis algorithm to identify patterns in sentiment fluctuations. The input is new sentiment data and historical records, and the output is the identified fluctuation patterns.

[0175] Step 6:

[0176] The server generates individually optimized tasks and learning programs based on the fluctuation patterns. A generative AI model is used here to create prompts. The input is the pattern of emotional fluctuations, and the output is the proposed task or program.

[0177] Step 7:

[0178] The server sends the generated tasks and programs to the terminal. The terminal receives them and displays them on the user's screen. The input is the details of the task or program, and the output is a visually represented proposal.

[0179] Step 8:

[0180] The user evaluates the suggestions displayed on the screen and enters feedback into the terminal. The terminal sends this feedback to the server, which is used to improve the system. The input is the user's evaluation, and the output is the feedback data sent to the server.

[0181] (Application Example 2)

[0182] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0183] In recent years, managing stress and anxiety within the family has become increasingly important, but there is a lack of systems that provide optimal care in real time according to individual circumstances. Furthermore, there is a need for skills to accurately perceive emotional changes, but the current problem is the lack of such skills. This makes it difficult to provide appropriate support for the mental health of families.

[0184] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0185] In this invention, the server includes recording means for collecting audio and video data to track the user's emotions in real time, data processing means for decomposing the transmitted data and analyzing the user's emotions and health status, and program creation means for generating a care plan and suggestions optimized for the user based on the analysis results. This enables appropriate care and suggestions tailored to the user's individual emotional state.

[0186] "Recording means" refers to a mechanism for collecting user audio and video data.

[0187] A "communication mechanism" is a means for securely transmitting collected audio and video data.

[0188] A "data processing system" is a mechanism for analyzing transmitted data and evaluating the user's emotions and health status.

[0189] The "program creation method" is a system for generating care plans and proposals optimized for the user based on the analysis results.

[0190] "Display means" refers to a method for transmitting and displaying the generated care plan and proposals on the user's terminal.

[0191] A "response processing mechanism" is a system for receiving feedback from users and using it to improve future proposals.

[0192] To implement this invention, a robot is required to operate the emotional care system within the home. This robot is equipped with highly sensitive sensors to collect and record the user's voice and video data in real time. Furthermore, to protect the user's privacy, the data is securely transmitted to a server via a communication mechanism.

[0193] The server has a data processing system that analyzes the received data, and this uses the deep learning framework TensorFlow. This data processing system analyzes voice tone, facial expressions, heart rate, etc., to evaluate the user's emotions and health status in real time.

[0194] Based on the evaluation results, the program creation method generates the optimal care plan and suggestions for the user. These suggestions are generated by an AI model that creates prompt statements based on a large amount of historical data.

[0195] The generated plans and suggestions are quickly transmitted to the user's terminal via a display device. The robot uses speech synthesis and a display to show this information to the user and assist them in taking the suggested actions.

[0196] For example, when the family is gathered in the living room after dinner, the robot might sense from the mother's voice and facial expressions that she is a little tired. In this case, it could make a suggestion such as, "Why don't you try to relax a bit tonight and take a long, leisurely bath?"

[0197] The hardware used includes high-sensitivity sensors such as microphones and cameras. The software utilizes a deep learning analysis platform based on TensorFlow, and HTTPS is used as the secure communication protocol.

[0198] An example of a prompt is, "Analyze the user's voice tone and facial expressions to generate appropriate stress care suggestions." This prompt allows the AI ​​to create the optimal care plan.

[0199] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0200] Step 1:

[0201] The device collects the user's voice and video data. Specifically, it records data in real time using a high-sensitivity sensor. The input consists of voice tone and facial expressions, and the output is recorded data containing these.

[0202] Step 2:

[0203] The terminal securely transmits collected audio and video data to the server via a communication mechanism. The input is recorded data, and the output is secure data that reaches the server.

[0204] Step 3:

[0205] The server analyzes the received audio and video data using data processing tools. The input is secure data, and the output is an analysis result indicating the user's emotions and health status. A deep learning model using TensorFlow processes this data.

[0206] Step 4:

[0207] The server uses program creation tools based on the analysis results to generate the optimal care plan and suggestions for the user. The input is the analysis results, and the output is the plan proposed to the user. Here, the generating AI model creates the plan using prompt statements.

[0208] Step 5:

[0209] The server transmits and presents the generated care plan and proposals to the terminal via a display device. The input is the care plan, and the output is the information displayed on the terminal.

[0210] Step 6:

[0211] The user reviews the proposed plan and enters feedback on their device. The input is the user's selected feedback, and the output is data used to improve future proposals.

[0212] Step 7:

[0213] The terminal sends user feedback data to the server. The input is user feedback, and the output is data provided to the server. The server uses this data to continuously improve the accuracy of the system.

[0214] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0215] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0216] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0217] [Second Embodiment]

[0218] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0219] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0220] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0221] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0222] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0223] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0224] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0225] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0226] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0227] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0228] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0229] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0230] This invention is a system that proposes optimized tasks and learning programs according to the user's daily mental and physical state. The system consists of a terminal that collects the user's voice and video data, and a server that analyzes this data and creates optimal suggestions.

[0231] First, the device collects data such as the user's voice and facial expressions in real time. Voice is captured through a microphone and analyzed for tone, speed of speech, and voice quality. Facial expression data is acquired through a camera and analyzed for facial movements and changes in expression. The device immediately transmits this data to a server using a secure protocol.

[0232] Next, the server performs analysis using the received data. An AI algorithm processes the data and evaluates the user's emotional and health states. It can also refer to past data to identify patterns in the user's state changes. Based on these analysis results, the server generates tasks and learning programs optimized for each individual user.

[0233] The generated suggestions are sent to the terminal and presented to the user visually. The terminal provides a user-friendly interface, designed to make it easy for the user to understand the suggested tasks. The user selects and executes tasks and learning programs based on the suggestions. The results of the execution and feedback are then sent back to the server via the terminal.

[0234] As an example, let's consider the case of a salaried worker. In the morning, while commuting to work, the device collects voice tone and facial expressions, and determines the stress level from this data. If the server detects high stress levels, it suggests lighter work tasks and a relaxation program to do during lunchtime. The device notifies the user, and the user can incorporate these suggestions into their schedule for the day.

[0235] This system allows users to efficiently complete tasks tailored to their physical and mental state on any given day, resulting in improved performance and reduced stress.

[0236] The following describes the processing flow.

[0237] Step 1:

[0238] The device collects the user's voice and video data in real time. Voice data is acquired via a microphone, and the tone, speed, and pitch of the voice are analyzed. Facial data is acquired using a camera, and facial movements and changes in expression are analyzed.

[0239] Step 2:

[0240] The device quickly encrypts the collected data and sends it to the server using a secure communication protocol. This communication is designed to protect user privacy.

[0241] Step 3:

[0242] The server decodes the received data and analyzes it using an AI algorithm. It evaluates the user's emotional state and health status from their voice and facial expressions, and compares this information with past data to identify patterns of change.

[0243] Step 4:

[0244] Based on the analysis results, the server generates tasks and learning programs tailored to the user. This includes suggestions for specific tasks and activities that match the user's current emotional and physical state.

[0245] Step 5:

[0246] The server sends the generated suggestions to the terminal. The terminal receives them and presents them to the user through a user-friendly, visualized interface.

[0247] Step 6:

[0248] The user reviews suggestions via an interface on their device and selects tasks or programs to execute. The user provides feedback and inputs the results into the device.

[0249] Step 7:

[0250] The device collects user feedback and execution results, re-encrypts them, and sends them to the server. The server stores this information and uses it as foundational data to improve the accuracy of future suggestions.

[0251] (Example 1)

[0252] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0253] In modern life, there is a need to effectively understand users' mental and physical states and provide appropriate activities and learning content accordingly. However, conventional systems have difficulty accurately assessing users' emotions and physical states in real time, making it impossible to provide individually optimized suggestions. To solve this problem, there is a need for technology that can accurately assess the user's state and quickly provide individually customized suggestions.

[0254] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0255] In this invention, the server includes information acquisition means for acquiring the user's voice and video and analyzing tone, speech speed, facial movements, etc.; data transmission means for securely transmitting the acquired voice and video information; and information analysis means for processing the transmitted information and evaluating the user's emotions and health status. This makes it possible to individually generate and provide optimal activities and learning procedures for the user in real time.

[0256] "Information acquisition means" refers to methods for acquiring the user's voice and video and analyzing their tone, speech speed, facial movements, etc.

[0257] "Data transmission means" refers to means for securely transmitting acquired audio and video information to a server.

[0258] "Information analysis means" refers to means for processing transmitted information and evaluating the user's emotions and health status.

[0259] A "proposal generation method" is a means for creating optimal activities and learning procedures for the user based on information analysis.

[0260] "Display means" refers to means for transmitting generated activities and learning procedures to a user device for visual presentation.

[0261] A "response processing means" is a means of collecting user feedback and using it to inform future proposals.

[0262] This invention is a system that provides activities and learning procedures optimized according to the user's mental and physical state. The system mainly consists of a terminal and a server. The terminal includes hardware that collects the user's voice and video data in real time. Specifically, the terminal incorporates a high-sensitivity microphone and a high-resolution camera to acquire the user's voice data and facial expressions. For the voice data, voice analysis software is used to analyze tone, speech speed, and voice pitch. For the video data, facial recognition and expression analysis software analyzes facial movements and changes in facial expressions in real time.

[0263] The acquired data is transmitted to the server via a secure communication protocol. The server analyzes the received data using advanced AI algorithms. This AI utilizes generative AI models to assess the user's current emotions and health status, and tracks changes in their state by analyzing past data as well.

[0264] Based on the analysis results, the server generates tasks and learning programs optimized for the user's state. For example, if the server determines that the user is experiencing high stress, it can suggest relaxation exercises or light work tasks. This generated content is sent to the terminal and presented to the user visually. The terminal's user interface is designed to make the suggestions clear and easy for the user to understand.

[0265] The user selects and performs a suggested task, and their experience and feedback are sent back to the server via the terminal. This allows the system to incorporate the user's feedback into future suggestions. A possible example of a prompt message would be, "Evaluate the user's stress level based on audio and video data, and suggest an appropriate task." This system enables users to efficiently perform activities suited to their condition, allowing them to enjoy a better quality of life.

[0266] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0267] Step 1:

[0268] The device acquires the user's voice in real time using a microphone. The input is raw audio data, and the output is digital data for speech analysis. Specifically, speech analysis software is used to decompose the audio data into tone, speech speed, and voice pitch, and feature quantities for each element are extracted.

[0269] Step 2:

[0270] The device captures the user's video with a camera and analyzes their facial expressions. The input is real-time video data, and the output is the analysis results regarding changes in facial expressions and facial movements. Facial recognition software analyzes this data to identify specific emotions and facial features.

[0271] Step 3:

[0272] The terminal aggregates audio and video analysis data and sends it to the server. The input is a dataset of audio and video analysis results, and the output is transmission data converted into a secure format. Using a data transmission means, these analysis results are sent to the server via a configured secure communication protocol.

[0273] Step 4:

[0274] The server analyzes received data using an AI algorithm to assess the user's emotions and health. The input is the analyzed audio and video data, and the output is the assessment of the user's current emotions and health. It utilizes a generative AI model to calculate multiple state indicators and infer the user's psychological and physiological state.

[0275] Step 5:

[0276] The server generates optimal activity and learning programs based on the evaluation results. The input is the evaluation results of the user's emotions and health state, and the output is a proposal for personalized activity and learning programs. Using the proposal generation means, appropriate tasks are generated and sent to the user's terminal in the next stage.

[0277] Step 6:

[0278] The terminal visually presents suggestions received from the server to the user. The input is suggested data for tasks and learning programs sent from the server, and the output is content displayed in a way that is easy for the user to understand. The terminal's interface clearly displays the suggestions and assists the user in selecting and performing activities.

[0279] Step 7:

[0280] The user selects and executes a suggested activity or learning program. The input is the displayed suggested activity or learning program, and the output is the result of the user's selection and execution. The user's selections are recorded and saved on the device as data for future use.

[0281] Step 8:

[0282] The terminal sends the user's feedback and the results of activities to the server. The input is the user's selection and execution results, and the output is the feedback data that serves as the basis for the next proposal. The data from the terminal is transmitted to the server and reflected in future proposals.

[0283] (Application Example 1)

[0284] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0285] In modern society, users are required to maintain their mental and physical health in their daily lives, but its continuous management is not easy. In particular, there is a lack of a method for immediately responding to the stress and changes in physical condition that occur in daily life and proposing appropriate tasks and relaxation activities. It is required to solve this problem.

[0286] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0287] In this invention, the server includes a detection means for acquiring the user's acoustic data and image data, a communication means for securely transferring the acquired acoustic data and image data, and an information analysis means for analyzing the transferred data and evaluating the user's emotional state and health state. Thereby, it becomes possible to propose optimal business tasks and relaxation activities in real time and individually based on the user's daily mental state and health state.

[0288] The "user" is the person who is the subject of the acoustic data and image data, and is the human who receives the proposal of the optimal task and relaxation activity by the system.

[0289] The "acoustic data" is information regarding the user's voice and sound acquired through a device such as a microphone, and is a part of the analysis target for evaluating the emotional state.

[0290] "Image data" refers to visual information about a user's face and movements collected through devices such as cameras, and is part of the analysis used to evaluate their emotional state.

[0291] "Detection means" refers to devices or functions for acquiring acoustic and image data from the user.

[0292] "Communication methods" refer to the technologies and protocols used to securely transfer acquired data, and are used to ensure the safe transmission of data.

[0293] "Information analysis means" refers to algorithms and processing technologies for analyzing transmitted data and evaluating the user's emotional state and health status.

[0294] "Business tasks" refer to specific tasks or activities that should be proposed to the user, and are selected taking into consideration the user's mental and physical state.

[0295] "Relaxation activities" refer to activities and activities suggested to reduce user stress and promote the user's mental relaxation.

[0296] The system for carrying out this invention uses a terminal equipped with a microphone and a camera as a device for acquiring the user's audio and image data. The terminal collects the user's voice tone, speed, and facial expressions in real time and transmits the data to a server using a secure protocol. Security in this series of communications is ensured using a protocol such as HTTPS.

[0297] The server uses AI algorithms to process the received data. Specifically, it uses machine learning frameworks such as TensorFlow and PyTorch to perform analysis to evaluate the user's emotional state and health status. This analysis makes it possible to identify patterns in changes in the user's state by comparing it with past data.

[0298] Based on the analysis results, the server generates work tasks and mitigation activities optimized for the user. These suggestions are communicated visually and audibly to the terminal, ensuring that the user can easily understand and perform them. As a result, the user performs the suggested activities and provides feedback, allowing the system to make even more personalized suggestions in the future.

[0299] For example, if the server determines that a user is experiencing high levels of fatigue or stress, it will suggest lighter work tasks or play relaxing music. At this time, the device will notify the user via voice, saying, "I will play relaxing music. Would you like to do some stretching?" and prompt the user to make a choice.

[0300] An example of an input prompt for the generating AI model is, "Based on the user's voice data and facial expression data, please suggest the most suitable relaxation activity for the day."

[0301] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0302] Step 1:

[0303] The device collects the user's audio and image data using a microphone and camera. During this process, it captures each user's voice and facial expressions as input. The collected audio tone, intonation, and changes in facial expressions are processed in real time and converted into digital data. The output is the raw data formed by this processing.

[0304] Step 2:

[0305] The terminal sends the collected data to the server using a secure protocol (e.g., HTTPS). The input here is the voice and image data generated in Step 1. As output, the raw data of the user's voice and expression is transferred to the server. This data transfer is performed using end-to-end encryption to protect privacy.

[0306] Step 3:

[0307] The server analyzes the received data using AI algorithms. The input is the user's voice and image data. The server performs sentiment analysis using machine learning frameworks such as TensorFlow or PyTorch to generate data for evaluating the user's mental state and health status. The output is evaluation data indicating the user's state, including numerical values or labels representing the person's stress level and emotional state.

[0308] Step 4:

[0309] Based on the analysis results, the server generates the most suitable work tasks and relaxation activities for the user. The evaluation data generated in Step 3 is used as the input. An AI model generated in this process is used to devise appropriate activities. The output is a list of specific tasks and relaxation activities to be proposed.

[0310] Step 5:

[0311] The server sends the generated proposals to the terminal. The input is the list of tasks and activities generated in Step 4. The output is instruction data for the terminal, which enables the terminal to prepare to present the tasks visually or audibly to the user.

[0312] Step 6:

[0313] The terminal presents the received suggestions to the user. The input is suggestion data from the server. As output, the terminal actually notifies the user of the day's work tasks and relaxation activities. This notification includes specific actions that should be taken, such as suggesting playing music or stretching.

[0314] Step 7:

[0315] The user performs tasks and activities based on suggestions from the device. They refer to the suggestions from step 6 as input and reflect them in their actual actions. The results and feedback are sent back to the server via the device and used to improve the system's adaptability by influencing future suggestions. Feedback data is then generated as output.

[0316] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0317] This invention is a system that combines a user's voice and video data with an emotion engine that recognizes the user's emotions based on that data. The aim of this system is to recognize the user's emotional state in real time from the voice and video data and propose individually optimized tasks and learning programs.

[0318] Specifically, users use smartphones or wearable devices to collect voice and facial data. These devices are equipped with high-sensitivity microphones and cameras, which are used to acquire data such as the user's voice tone, facial expressions, and heart rate. The acquired data is analyzed by an emotion engine to evaluate the user's current emotional state. The emotion engine is built using an algorithm that combines multiple deep learning models, enabling it to identify emotions with high accuracy, even down to the subtle nuances of voice tone and facial expressions.

[0319] Next, the device securely transmits the analysis results to the server. Based on the received emotional state data, the server generates tasks and learning programs optimized for each individual user. At this time, the server also compares the data with past data to identify patterns in emotional fluctuations, enabling it to provide more sophisticated suggestions. The generated suggestions are sent to the device and displayed on the user's screen.

[0320] Users review the suggestions displayed on the screen and select tasks according to their mood and circumstances for the day. In this way, users can focus more effectively on work or learning. Furthermore, after completing a task, users can input feedback into their device. This feedback is sent back to the server and used to improve future task suggestions.

[0321] As a concrete example, consider a student user studying for final exams. The device detects a decrease in concentration from voice and facial expressions and suggests relaxing music and an efficient study schedule with breaks. The student tries the suggested schedule and provides feedback on how their concentration returns afterward, contributing to the improvement of the system's accuracy. Through this entire process, the user can achieve healthy and effective task execution.

[0322] The following describes the processing flow.

[0323] Step 1:

[0324] The device collects the user's voice and video data in real time. It uses the microphone to capture voice tone and speed, and the camera to capture changes in facial expressions.

[0325] Step 2:

[0326] The device inputs the collected data into the emotion engine. The emotion engine analyzes the audio and video data and uses a deep learning model to identify emotional states.

[0327] Step 3:

[0328] The device sends the emotion engine's recognition results to the server via a secure protocol. This data includes the user's specific emotional state and related information.

[0329] Step 4:

[0330] The server analyzes the received data and compares it with past data to analyze patterns of emotional fluctuations. This analysis allows us to understand the user's emotional tendencies.

[0331] Step 5:

[0332] The server generates optimal tasks and learning programs for the user based on their emotional state. The generated suggestions take into account the user's current mental and physical condition.

[0333] Step 6:

[0334] The server sends the tasks and programs it generates to the terminal. The terminal receives them and displays them to the user in a visually easy-to-understand format.

[0335] Step 7:

[0336] The user reviews the tasks and programs presented on their device and selects which ones to execute. The selected tasks are then reflected in the schedule.

[0337] Step 8:

[0338] After completing a task, the user enters feedback into the device. This feedback includes the results of the task and changes in their emotions.

[0339] Step 9:

[0340] The device sends feedback information to the server, which records it in a database. This data is then used as foundational information to make future task suggestions more accurate.

[0341] (Example 2)

[0342] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0343] In modern society, it is crucial to quickly and accurately understand an individual's emotional state and provide guidance for optimal behavior and learning based on that understanding. However, conventional systems lack the accuracy to analyze emotional fluctuations in real time. Furthermore, there is a challenge in effectively utilizing user feedback to improve the system.

[0344] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0345] In this invention, the server includes an inference means for analyzing the user's emotional state in real time, an analysis means for comparing and analyzing it with past data, and a generation means for generating individually optimized action and learning plans. This makes it possible to monitor the user's emotional state with high accuracy and provide individually optimized action and learning guidelines.

[0346] "Receiving means" refers to a device or software function for acquiring audio and video data from the user.

[0347] "Preprocessing means" refers to a device or software function for performing processing to reduce noise in acquired data and improve the accuracy of analysis.

[0348] "Inference means" refers to an algorithm or software function for analyzing a user's emotional state in real time based on pre-processed data.

[0349] "Transmission means" refers to a communication function or device for securely transferring analysis results to a server.

[0350] "Analysis method" refers to a function that compares past data with current sentiment data on a server to identify fluctuation patterns.

[0351] "Generative means" refers to algorithms or software functions for creating individually optimized action and learning plans.

[0352] "Presentation means" refers to a function that transmits the generated plan to the user's display device and presents it in a visually easy-to-understand format.

[0353] A "feedback processing mechanism" is a function that collects evaluations from users and uses them to improve the system or make suggestions for future updates.

[0354] To implement this invention, the user uses a terminal such as a smartphone or wearable device. These terminals are equipped with a high-sensitivity microphone and camera and have the ability to acquire audio and video data. The terminals may be integrated with a sensor system using a device platform such as Arduino or Raspberry Pi.

[0355] The terminal preprocesses audio data using digital signal processing technology to remove noise. Video data is broken down frame by frame to extract the user's facial features. The hardware used at this stage includes a microphone and camera module, while the software used includes audio processing libraries and image processing libraries.

[0356] The collected data is input from the device to an emotion engine for real-time analysis of emotional states. This emotion engine utilizes machine learning frameworks such as TensorFlow and PyTorch, and operates by combining multiple deep learning models. The server uses the SSL / TLS protocol to send and receive data to securely protect the received data.

[0357] The results analyzed by the emotion engine are sent to the server and compared with past data. The server uses data analysis techniques with Python and R to identify the user's emotional fluctuation patterns and generate tasks and learning programs optimized for the user. At this time, it utilizes prompt statements that the generating AI model can operate on to generate instructions in the form of, for example, "Based on my current concentration and stress level, please suggest ways to improve my study efficiency."

[0358] The generated tasks are sent to the terminal and presented on the user's screen. The terminal uses a user interface to visually display the suggestions in an easy-to-understand manner. The user performs the displayed tasks and inputs the results as feedback into the terminal. This feedback is then sent back to the server and used to improve the accuracy of future suggestions.

[0359] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0360] Step 1:

[0361] The device acquires the user's voice and video data in real time. The device's high-sensitivity microphone captures the user's voice, and the camera records facial expressions. The input is raw audio and video data, and the output is digital audio data and video frames in a noisy format.

[0362] Step 2:

[0363] The terminal preprocesses the acquired audio and video data. Specifically, noise is removed from the audio data through digital signal processing, and facial features are enhanced from the video data through image filtering. The input is the audio and video data from step 1, and the output is the digital data with noise removed and enhanced.

[0364] Step 3:

[0365] The device sends pre-processed data to the emotion engine. Based on the transmitted data, the emotion engine uses a deep learning model to infer the user's emotions. This process includes speech tone analysis and facial expression recognition. The input is the pre-processed data, and the output is the inferred emotion label and confidence level.

[0366] Step 4:

[0367] The device securely transmits the inferred sentiment data to the server. Here, the data is encrypted using the HTTPS protocol. The input is the sentiment label and confidence level, and the output is the securely transmitted data.

[0368] Step 5:

[0369] The server compares the received sentiment data with historical data. It uses a data analysis algorithm to identify patterns in sentiment fluctuations. The input is new sentiment data and historical records, and the output is the identified fluctuation patterns.

[0370] Step 6:

[0371] The server generates individually optimized tasks and learning programs based on the fluctuation patterns. A generative AI model is used here to create prompts. The input is the pattern of emotional fluctuations, and the output is the proposed task or program.

[0372] Step 7:

[0373] The server sends the generated tasks and programs to the terminal. The terminal receives them and displays them on the user's screen. The input is the details of the task or program, and the output is a visually represented proposal.

[0374] Step 8:

[0375] The user evaluates the suggestions displayed on the screen and enters feedback into the terminal. The terminal sends this feedback to the server, which is used to improve the system. The input is the user's evaluation, and the output is the feedback data sent to the server.

[0376] (Application Example 2)

[0377] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0378] In recent years, managing stress and anxiety within the family has become increasingly important, but there is a lack of systems that provide optimal care in real time according to individual circumstances. Furthermore, there is a need for skills to accurately perceive emotional changes, but the current problem is the lack of such skills. This makes it difficult to provide appropriate support for the mental health of families.

[0379] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0380] In this invention, the server includes recording means for collecting audio and video data to track the user's emotions in real time, data processing means for decomposing the transmitted data and analyzing the user's emotions and health status, and program creation means for generating a care plan and suggestions optimized for the user based on the analysis results. This enables appropriate care and suggestions tailored to the user's individual emotional state.

[0381] "Recording means" refers to a mechanism for collecting user audio and video data.

[0382] A "communication mechanism" is a means for securely transmitting collected audio and video data.

[0383] A "data processing system" is a mechanism for analyzing transmitted data and evaluating the user's emotions and health status.

[0384] The "program creation method" is a system for generating care plans and proposals optimized for the user based on the analysis results.

[0385] "Display means" refers to a method for transmitting and displaying the generated care plan and proposals on the user's terminal.

[0386] A "response processing mechanism" is a system for receiving feedback from users and using it to improve future proposals.

[0387] To implement this invention, a robot is required to operate the emotional care system within the home. This robot is equipped with highly sensitive sensors to collect and record the user's voice and video data in real time. Furthermore, to protect the user's privacy, the data is securely transmitted to a server via a communication mechanism.

[0388] The server has a data processing system that analyzes the received data, and this uses the deep learning framework TensorFlow. This data processing system analyzes voice tone, facial expressions, heart rate, etc., to evaluate the user's emotions and health status in real time.

[0389] Based on the evaluation results, the program creation method generates the optimal care plan and suggestions for the user. These suggestions are generated by an AI model that creates prompt statements based on a large amount of historical data.

[0390] The generated plans and suggestions are quickly transmitted to the user's terminal via a display device. The robot uses speech synthesis and a display to show this information to the user and assist them in taking the suggested actions.

[0391] For example, when the family is gathered in the living room after dinner, the robot might sense from the mother's voice and facial expressions that she is a little tired. In this case, it could make a suggestion such as, "Why don't you try to relax a bit tonight and take a long, leisurely bath?"

[0392] The hardware used includes high-sensitivity sensors such as microphones and cameras. The software utilizes a deep learning analysis platform based on TensorFlow, and HTTPS is used as the secure communication protocol.

[0393] An example of a prompt is, "Analyze the user's voice tone and facial expressions to generate appropriate stress care suggestions." This prompt allows the AI ​​to create the optimal care plan.

[0394] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0395] Step 1:

[0396] The device collects the user's voice and video data. Specifically, it records data in real time using a high-sensitivity sensor. The input consists of voice tone and facial expressions, and the output is recorded data containing these.

[0397] Step 2:

[0398] The terminal securely transmits collected audio and video data to the server via a communication mechanism. The input is recorded data, and the output is secure data that reaches the server.

[0399] Step 3:

[0400] The server analyzes the received audio and video data using data processing tools. The input is secure data, and the output is an analysis result indicating the user's emotions and health status. A deep learning model using TensorFlow processes this data.

[0401] Step 4:

[0402] The server uses program creation tools based on the analysis results to generate the optimal care plan and suggestions for the user. The input is the analysis results, and the output is the plan proposed to the user. Here, the generating AI model creates the plan using prompt statements.

[0403] Step 5:

[0404] The server transmits and presents the generated care plan and proposals to the terminal via a display device. The input is the care plan, and the output is the information displayed on the terminal.

[0405] Step 6:

[0406] The user reviews the proposed plan and enters feedback on their device. The input is the user's selected feedback, and the output is data used to improve future proposals.

[0407] Step 7:

[0408] The terminal sends user feedback data to the server. The input is user feedback, and the output is data provided to the server. The server uses this data to continuously improve the accuracy of the system.

[0409] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0410] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0411] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0412] [Third Embodiment]

[0413] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0414] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0415] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0416] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0417] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0418] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0419] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0420] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0421] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0422] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0423] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0424] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0425] This invention is a system that proposes optimized tasks and learning programs according to the user's daily mental and physical state. The system consists of a terminal that collects the user's voice and video data, and a server that analyzes this data and creates optimal suggestions.

[0426] First, the device collects data such as the user's voice and facial expressions in real time. Voice is captured through a microphone and analyzed for tone, speed of speech, and voice quality. Facial expression data is acquired through a camera and analyzed for facial movements and changes in expression. The device immediately transmits this data to a server using a secure protocol.

[0427] Next, the server performs analysis using the received data. An AI algorithm processes the data and evaluates the user's emotional and health states. It can also refer to past data to identify patterns in the user's state changes. Based on these analysis results, the server generates tasks and learning programs optimized for each individual user.

[0428] The generated suggestions are sent to the terminal and presented to the user visually. The terminal provides a user-friendly interface, designed to make it easy for the user to understand the suggested tasks. The user selects and executes tasks and learning programs based on the suggestions. The results of the execution and feedback are then sent back to the server via the terminal.

[0429] As an example, let's consider the case of a salaried worker. In the morning, while commuting to work, the device collects voice tone and facial expressions, and determines the stress level from this data. If the server detects high stress levels, it suggests lighter work tasks and a relaxation program to do during lunchtime. The device notifies the user, and the user can incorporate these suggestions into their schedule for the day.

[0430] This system allows users to efficiently complete tasks tailored to their physical and mental state on any given day, resulting in improved performance and reduced stress.

[0431] The following describes the processing flow.

[0432] Step 1:

[0433] The device collects the user's voice and video data in real time. Voice data is acquired via a microphone, and the tone, speed, and pitch of the voice are analyzed. Facial data is acquired using a camera, and facial movements and changes in expression are analyzed.

[0434] Step 2:

[0435] The device quickly encrypts the collected data and sends it to the server using a secure communication protocol. This communication is designed to protect user privacy.

[0436] Step 3:

[0437] The server decodes the received data and analyzes it using an AI algorithm. It evaluates the user's emotional state and health status from their voice and facial expressions, and compares this information with past data to identify patterns of change.

[0438] Step 4:

[0439] Based on the analysis results, the server generates tasks and learning programs tailored to the user. This includes suggestions for specific tasks and activities that match the user's current emotional and physical state.

[0440] Step 5:

[0441] The server sends the generated suggestions to the terminal. The terminal receives them and presents them to the user through a user-friendly, visualized interface.

[0442] Step 6:

[0443] The user reviews suggestions via an interface on their device and selects tasks or programs to execute. The user provides feedback and inputs the results into the device.

[0444] Step 7:

[0445] The device collects user feedback and execution results, re-encrypts them, and sends them to the server. The server stores this information and uses it as foundational data to improve the accuracy of future suggestions.

[0446] (Example 1)

[0447] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0448] In modern life, there is a need to effectively understand users' mental and physical states and provide appropriate activities and learning content accordingly. However, conventional systems have difficulty accurately assessing users' emotions and physical states in real time, making it impossible to provide individually optimized suggestions. To solve this problem, there is a need for technology that can accurately assess the user's state and quickly provide individually customized suggestions.

[0449] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0450] In this invention, the server includes information acquisition means for acquiring the user's voice and video and analyzing tone, speech speed, facial movements, etc.; data transmission means for securely transmitting the acquired voice and video information; and information analysis means for processing the transmitted information and evaluating the user's emotions and health status. This makes it possible to individually generate and provide optimal activities and learning procedures for the user in real time.

[0451] "Information acquisition means" refers to methods for acquiring the user's voice and video and analyzing their tone, speech speed, facial movements, etc.

[0452] "Data transmission means" refers to means for securely transmitting acquired audio and video information to a server.

[0453] "Information analysis means" refers to means for processing transmitted information and evaluating the user's emotions and health status.

[0454] A "proposal generation method" is a means for creating optimal activities and learning procedures for the user based on information analysis.

[0455] "Display means" refers to means for transmitting generated activities and learning procedures to a user device for visual presentation.

[0456] A "response processing means" is a means of collecting user feedback and using it to inform future proposals.

[0457] This invention is a system that provides activities and learning procedures optimized according to the user's mental and physical state. The system mainly consists of a terminal and a server. The terminal includes hardware that collects the user's voice and video data in real time. Specifically, the terminal incorporates a high-sensitivity microphone and a high-resolution camera to acquire the user's voice data and facial expressions. For the voice data, voice analysis software is used to analyze tone, speech speed, and voice pitch. For the video data, facial recognition and expression analysis software analyzes facial movements and changes in facial expressions in real time.

[0458] The acquired data is transmitted to the server via a secure communication protocol. The server analyzes the received data using advanced AI algorithms. This AI utilizes generative AI models to assess the user's current emotions and health status, and tracks changes in their state by analyzing past data as well.

[0459] Based on the analysis results, the server generates tasks and learning programs optimized for the user's state. For example, if the server determines that the user is experiencing high stress, it can suggest relaxation exercises or light work tasks. This generated content is sent to the terminal and presented to the user visually. The terminal's user interface is designed to make the suggestions clear and easy for the user to understand.

[0460] The user selects and performs a suggested task, and their experience and feedback are sent back to the server via the terminal. This allows the system to incorporate the user's feedback into future suggestions. A possible example of a prompt message would be, "Evaluate the user's stress level based on audio and video data, and suggest an appropriate task." This system enables users to efficiently perform activities suited to their condition, allowing them to enjoy a better quality of life.

[0461] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0462] Step 1:

[0463] The device acquires the user's voice in real time using a microphone. The input is raw audio data, and the output is digital data for speech analysis. Specifically, speech analysis software is used to decompose the audio data into tone, speech speed, and voice pitch, and feature quantities for each element are extracted.

[0464] Step 2:

[0465] The device captures the user's video with a camera and analyzes their facial expressions. The input is real-time video data, and the output is the analysis results regarding changes in facial expressions and facial movements. Facial recognition software analyzes this data to identify specific emotions and facial features.

[0466] Step 3:

[0467] The terminal aggregates audio and video analysis data and sends it to the server. The input is a dataset of audio and video analysis results, and the output is transmission data converted into a secure format. Using a data transmission means, these analysis results are sent to the server via a configured secure communication protocol.

[0468] Step 4:

[0469] The server analyzes received data using an AI algorithm to assess the user's emotions and health. The input is the analyzed audio and video data, and the output is the assessment of the user's current emotions and health. It utilizes a generative AI model to calculate multiple state indicators and infer the user's psychological and physiological state.

[0470] Step 5:

[0471] The server generates optimal activity and learning programs based on the evaluation results. The input is the evaluation results of the user's emotions and health state, and the output is a proposal for personalized activity and learning programs. Using the proposal generation means, appropriate tasks are generated and sent to the user's terminal in the next stage.

[0472] Step 6:

[0473] The terminal visually presents suggestions received from the server to the user. The input is suggested data for tasks and learning programs sent from the server, and the output is content displayed in a way that is easy for the user to understand. The terminal's interface clearly displays the suggestions and assists the user in selecting and performing activities.

[0474] Step 7:

[0475] The user selects and executes a suggested activity or learning program. The input is the displayed suggested activity or learning program, and the output is the result of the user's selection and execution. The user's selections are recorded and saved on the device as data for future use.

[0476] Step 8:

[0477] The device sends user feedback and activity results to the server. Inputs are the user's choices and execution results, while outputs are feedback data that forms the basis for future suggestions. Data from the device is transmitted to the server and incorporated into future suggestions.

[0478] (Application Example 1)

[0479] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0480] In modern society, users are expected to maintain their mental and physical health in their daily lives, but continuous management is not easy. In particular, there is a lack of methods to immediately respond to the stress and changes in physical condition that occur in daily life and to suggest appropriate tasks and mitigation activities. Solving this problem is essential.

[0481] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0482] In this invention, the server includes detection means for acquiring user acoustic and image data, communication means for securely transferring the acquired acoustic and image data, and information analysis means for analyzing the transferred data and evaluating the user's emotional and physical state. This makes it possible to propose optimal work tasks and relaxation activities in real time and on an individual basis, based on the user's daily mental and physical state.

[0483] A "user" is a person who is the subject of acoustic and image data, and who receives suggestions for optimal tasks and mitigation activities from the system.

[0484] "Acoustic data" refers to information about the user's voice and sounds acquired through devices such as microphones, and is part of the analysis used to evaluate their emotional state.

[0485] "Image data" refers to visual information about a user's face and movements collected through devices such as cameras, and is part of the analysis used to evaluate their emotional state.

[0486] "Detection means" refers to devices or functions for acquiring acoustic and image data from the user.

[0487] "Communication methods" refer to the technologies and protocols used to securely transfer acquired data, and are used to ensure the safe transmission of data.

[0488] "Information analysis means" refers to algorithms and processing technologies for analyzing transmitted data and evaluating the user's emotional state and health status.

[0489] "Business tasks" refer to specific tasks or activities that should be proposed to the user, and are selected taking into consideration the user's mental and physical state.

[0490] "Relaxation activities" refer to activities and activities suggested to reduce user stress and promote the user's mental relaxation.

[0491] The system for carrying out this invention uses a terminal equipped with a microphone and a camera as a device for acquiring the user's audio and image data. The terminal collects the user's voice tone, speed, and facial expressions in real time and transmits the data to a server using a secure protocol. Security in this series of communications is ensured using a protocol such as HTTPS.

[0492] The server uses AI algorithms to process the received data. Specifically, it uses machine learning frameworks such as TensorFlow and PyTorch to perform analysis to evaluate the user's emotional state and health status. This analysis makes it possible to identify patterns in changes in the user's state by comparing it with past data.

[0493] Based on the analysis results, the server generates work tasks and mitigation activities optimized for the user. These suggestions are communicated visually and audibly to the terminal, ensuring that the user can easily understand and perform them. As a result, the user performs the suggested activities and provides feedback, allowing the system to make even more personalized suggestions in the future.

[0494] For example, if the server determines that a user is experiencing high levels of fatigue or stress, it will suggest lighter work tasks or play relaxing music. At this time, the device will notify the user via voice, saying, "I will play relaxing music. Would you like to do some stretching?" and prompt the user to make a choice.

[0495] An example of an input prompt for the generating AI model is, "Based on the user's voice data and facial expression data, please suggest the most suitable relaxation activity for the day."

[0496] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0497] Step 1:

[0498] The device collects the user's audio and image data using a microphone and camera. During this process, it captures each user's voice and facial expressions as input. The collected audio tone, intonation, and changes in facial expressions are processed in real time and converted into digital data. The output is the raw data formed by this processing.

[0499] Step 2:

[0500] The device sends the collected data to the server using a secure protocol (e.g., HTTPS). The input here is the audio and image data generated in step 1. As output, unprocessed data of the user's voice and facial expressions is transferred to the server. This data transfer is performed using end-to-end encryption, ensuring privacy.

[0501] Step 3:

[0502] The server analyzes the received data using AI algorithms. The input consists of user voice and image data. The server uses machine learning frameworks such as TensorFlow and PyTorch to perform sentiment analysis and generate data that evaluates the user's mental and physical state. The output is evaluation data that indicates the user's state, including numerical values ​​and labels representing their stress level and emotional state.

[0503] Step 4:

[0504] The server generates optimal work tasks and mitigation activities for the user based on the analysis results. It uses the evaluation data generated in step 3 as input. The AI ​​model generated in this process is used to devise appropriate activities. The output is a list of specific tasks and mitigation activities to be proposed.

[0505] Step 5:

[0506] The server sends the generated suggestions to the terminal. The input is the list of tasks and activities generated in step 4. The output is instruction data for the terminal, which prepares the terminal to present the tasks to the user visually or audibly.

[0507] Step 6:

[0508] The terminal presents the received suggestions to the user. The input is suggestion data from the server. As output, the terminal actually notifies the user of the day's work tasks and relaxation activities. This notification includes specific actions that should be taken, such as suggesting playing music or stretching.

[0509] Step 7:

[0510] The user performs tasks and activities based on suggestions from the device. They refer to the suggestions from step 6 as input and reflect them in their actual actions. The results and feedback are sent back to the server via the device and used to improve the system's adaptability by influencing future suggestions. Feedback data is then generated as output.

[0511] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0512] This invention is a system that combines a user's voice and video data with an emotion engine that recognizes the user's emotions based on that data. The aim of this system is to recognize the user's emotional state in real time from the voice and video data and propose individually optimized tasks and learning programs.

[0513] Specifically, users use smartphones or wearable devices to collect voice and facial data. These devices are equipped with high-sensitivity microphones and cameras, which are used to acquire data such as the user's voice tone, facial expressions, and heart rate. The acquired data is analyzed by an emotion engine to evaluate the user's current emotional state. The emotion engine is built using an algorithm that combines multiple deep learning models, enabling it to identify emotions with high accuracy, even down to the subtle nuances of voice tone and facial expressions.

[0514] Next, the device securely transmits the analysis results to the server. Based on the received emotional state data, the server generates tasks and learning programs optimized for each individual user. At this time, the server also compares the data with past data to identify patterns in emotional fluctuations, enabling it to provide more sophisticated suggestions. The generated suggestions are sent to the device and displayed on the user's screen.

[0515] Users review the suggestions displayed on the screen and select tasks according to their mood and circumstances for the day. In this way, users can focus more effectively on work or learning. Furthermore, after completing a task, users can input feedback into their device. This feedback is sent back to the server and used to improve future task suggestions.

[0516] As a concrete example, consider a student user studying for final exams. The device detects a decrease in concentration from voice and facial expressions and suggests relaxing music and an efficient study schedule with breaks. The student tries the suggested schedule and provides feedback on how their concentration returns afterward, contributing to the improvement of the system's accuracy. Through this entire process, the user can achieve healthy and effective task execution.

[0517] The following describes the processing flow.

[0518] Step 1:

[0519] The device collects the user's voice and video data in real time. It uses the microphone to capture voice tone and speed, and the camera to capture changes in facial expressions.

[0520] Step 2:

[0521] The device inputs the collected data into the emotion engine. The emotion engine analyzes the audio and video data and uses a deep learning model to identify emotional states.

[0522] Step 3:

[0523] The device sends the emotion engine's recognition results to the server via a secure protocol. This data includes the user's specific emotional state and related information.

[0524] Step 4:

[0525] The server analyzes the received data and compares it with past data to analyze patterns of emotional fluctuations. This analysis allows us to understand the user's emotional tendencies.

[0526] Step 5:

[0527] The server generates optimal tasks and learning programs for the user based on their emotional state. The generated suggestions take into account the user's current mental and physical condition.

[0528] Step 6:

[0529] The server sends the tasks and programs it generates to the terminal. The terminal receives them and displays them to the user in a visually easy-to-understand format.

[0530] Step 7:

[0531] The user reviews the tasks and programs presented on their device and selects which ones to execute. The selected tasks are then reflected in the schedule.

[0532] Step 8:

[0533] After completing a task, the user enters feedback into the device. This feedback includes the results of the task and changes in their emotions.

[0534] Step 9:

[0535] The device sends feedback information to the server, which records it in a database. This data is then used as foundational information to make future task suggestions more accurate.

[0536] (Example 2)

[0537] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0538] In modern society, it is crucial to quickly and accurately understand an individual's emotional state and provide guidance for optimal behavior and learning based on that understanding. However, conventional systems lack the accuracy to analyze emotional fluctuations in real time. Furthermore, there is a challenge in effectively utilizing user feedback to improve the system.

[0539] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0540] In this invention, the server includes an inference means for analyzing the user's emotional state in real time, an analysis means for comparing and analyzing it with past data, and a generation means for generating individually optimized action and learning plans. This makes it possible to monitor the user's emotional state with high accuracy and provide individually optimized action and learning guidelines.

[0541] "Receiving means" refers to a device or software function for acquiring audio and video data from the user.

[0542] "Preprocessing means" refers to a device or software function for performing processing to reduce noise in acquired data and improve the accuracy of analysis.

[0543] "Inference means" refers to an algorithm or software function for analyzing a user's emotional state in real time based on pre-processed data.

[0544] "Transmission means" refers to a communication function or device for securely transferring analysis results to a server.

[0545] "Analysis method" refers to a function that compares past data with current sentiment data on a server to identify fluctuation patterns.

[0546] "Generative means" refers to algorithms or software functions for creating individually optimized action and learning plans.

[0547] "Presentation means" refers to a function that transmits the generated plan to the user's display device and presents it in a visually easy-to-understand format.

[0548] A "feedback processing mechanism" is a function that collects evaluations from users and uses them to improve the system or make suggestions for future updates.

[0549] To implement this invention, the user uses a terminal such as a smartphone or wearable device. These terminals are equipped with a high-sensitivity microphone and camera and have the ability to acquire audio and video data. The terminals may be integrated with a sensor system using a device platform such as Arduino or Raspberry Pi.

[0550] The terminal preprocesses audio data using digital signal processing technology to remove noise. Video data is broken down frame by frame to extract the user's facial features. The hardware used at this stage includes a microphone and camera module, while the software used includes audio processing libraries and image processing libraries.

[0551] The collected data is input from the device to an emotion engine for real-time analysis of emotional states. This emotion engine utilizes machine learning frameworks such as TensorFlow and PyTorch, and operates by combining multiple deep learning models. The server uses the SSL / TLS protocol to send and receive data to securely protect the received data.

[0552] The results analyzed by the emotion engine are sent to the server and compared with past data. The server uses data analysis techniques with Python and R to identify the user's emotional fluctuation patterns and generate tasks and learning programs optimized for the user. At this time, it utilizes prompt statements that the generating AI model can operate on to generate instructions in the form of, for example, "Based on my current concentration and stress level, please suggest ways to improve my study efficiency."

[0553] The generated tasks are sent to the terminal and presented on the user's screen. The terminal uses a user interface to visually display the suggestions in an easy-to-understand manner. The user performs the displayed tasks and inputs the results as feedback into the terminal. This feedback is then sent back to the server and used to improve the accuracy of future suggestions.

[0554] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0555] Step 1:

[0556] The device acquires the user's voice and video data in real time. The device's high-sensitivity microphone captures the user's voice, and the camera records facial expressions. The input is raw audio and video data, and the output is digital audio data and video frames in a noisy format.

[0557] Step 2:

[0558] The terminal preprocesses the acquired audio and video data. Specifically, noise is removed from the audio data through digital signal processing, and facial features are enhanced from the video data through image filtering. The input is the audio and video data from step 1, and the output is the digital data with noise removed and enhanced.

[0559] Step 3:

[0560] The device sends pre-processed data to the emotion engine. Based on the transmitted data, the emotion engine uses a deep learning model to infer the user's emotions. This process includes speech tone analysis and facial expression recognition. The input is the pre-processed data, and the output is the inferred emotion label and confidence level.

[0561] Step 4:

[0562] The device securely transmits the inferred sentiment data to the server. Here, the data is encrypted using the HTTPS protocol. The input is the sentiment label and confidence level, and the output is the securely transmitted data.

[0563] Step 5:

[0564] The server compares the received sentiment data with historical data. It uses a data analysis algorithm to identify patterns in sentiment fluctuations. The input is new sentiment data and historical records, and the output is the identified fluctuation patterns.

[0565] Step 6:

[0566] The server generates individually optimized tasks and learning programs based on the fluctuation patterns. A generative AI model is used here to create prompts. The input is the pattern of emotional fluctuations, and the output is the proposed task or program.

[0567] Step 7:

[0568] The server sends the generated tasks and programs to the terminal. The terminal receives them and displays them on the user's screen. The input is the details of the task or program, and the output is a visually represented proposal.

[0569] Step 8:

[0570] The user evaluates the suggestions displayed on the screen and enters feedback into the terminal. The terminal sends this feedback to the server, which is used to improve the system. The input is the user's evaluation, and the output is the feedback data sent to the server.

[0571] (Application Example 2)

[0572] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0573] In recent years, managing stress and anxiety within the family has become increasingly important, but there is a lack of systems that provide optimal care in real time according to individual circumstances. Furthermore, there is a need for skills to accurately perceive emotional changes, but the current problem is the lack of such skills. This makes it difficult to provide appropriate support for the mental health of families.

[0574] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0575] In this invention, the server includes recording means for collecting audio and video data to track the user's emotions in real time, data processing means for decomposing the transmitted data and analyzing the user's emotions and health status, and program creation means for generating a care plan and suggestions optimized for the user based on the analysis results. This enables appropriate care and suggestions tailored to the user's individual emotional state.

[0576] "Recording means" refers to a mechanism for collecting user audio and video data.

[0577] A "communication mechanism" is a means for securely transmitting collected audio and video data.

[0578] A "data processing system" is a mechanism for analyzing transmitted data and evaluating the user's emotions and health status.

[0579] The "program creation method" is a system for generating care plans and proposals optimized for the user based on the analysis results.

[0580] "Display means" refers to a method for transmitting and displaying the generated care plan and proposals on the user's terminal.

[0581] A "response processing mechanism" is a system for receiving feedback from users and using it to improve future proposals.

[0582] To implement this invention, a robot is required to operate the emotional care system within the home. This robot is equipped with highly sensitive sensors to collect and record the user's voice and video data in real time. Furthermore, to protect the user's privacy, the data is securely transmitted to a server via a communication mechanism.

[0583] The server has a data processing system that analyzes the received data, and this uses the deep learning framework TensorFlow. This data processing system analyzes voice tone, facial expressions, heart rate, etc., to evaluate the user's emotions and health status in real time.

[0584] Based on the evaluation results, the program creation method generates the optimal care plan and suggestions for the user. These suggestions are generated by an AI model that creates prompt statements based on a large amount of historical data.

[0585] The generated plans and suggestions are quickly transmitted to the user's terminal via a display device. The robot uses speech synthesis and a display to show this information to the user and assist them in taking the suggested actions.

[0586] For example, when the family is gathered in the living room after dinner, the robot might sense from the mother's voice and facial expressions that she is a little tired. In this case, it could make a suggestion such as, "Why don't you try to relax a bit tonight and take a long, leisurely bath?"

[0587] The hardware used includes high-sensitivity sensors such as microphones and cameras. The software utilizes a deep learning analysis platform based on TensorFlow, and HTTPS is used as the secure communication protocol.

[0588] An example of a prompt is, "Analyze the user's voice tone and facial expressions to generate appropriate stress care suggestions." This prompt allows the AI ​​to create the optimal care plan.

[0589] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0590] Step 1:

[0591] The device collects the user's voice and video data. Specifically, it records data in real time using a high-sensitivity sensor. The input consists of voice tone and facial expressions, and the output is recorded data containing these.

[0592] Step 2:

[0593] The terminal securely transmits collected audio and video data to the server via a communication mechanism. The input is recorded data, and the output is secure data that reaches the server.

[0594] Step 3:

[0595] The server analyzes the received audio and video data using data processing tools. The input is secure data, and the output is an analysis result indicating the user's emotions and health status. A deep learning model using TensorFlow processes this data.

[0596] Step 4:

[0597] The server uses program creation tools based on the analysis results to generate the optimal care plan and suggestions for the user. The input is the analysis results, and the output is the plan proposed to the user. Here, the generating AI model creates the plan using prompt statements.

[0598] Step 5:

[0599] The server transmits and presents the generated care plan and proposals to the terminal via a display device. The input is the care plan, and the output is the information displayed on the terminal.

[0600] Step 6:

[0601] The user reviews the proposed plan and enters feedback on their device. The input is the user's selected feedback, and the output is data used to improve future proposals.

[0602] Step 7:

[0603] The terminal sends user feedback data to the server. The input is user feedback, and the output is data provided to the server. The server uses this data to continuously improve the accuracy of the system.

[0604] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0605] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0606] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0607] [Fourth Embodiment]

[0608] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0609] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0610] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0611] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0612] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0613] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0614] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0615] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0616] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0617] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0618] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0619] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0620] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0621] This invention is a system that proposes optimized tasks and learning programs according to the user's daily mental and physical state. The system consists of a terminal that collects the user's voice and video data, and a server that analyzes this data and creates optimal suggestions.

[0622] First, the device collects data such as the user's voice and facial expressions in real time. Voice is captured through a microphone and analyzed for tone, speed of speech, and voice quality. Facial expression data is acquired through a camera and analyzed for facial movements and changes in expression. The device immediately transmits this data to a server using a secure protocol.

[0623] Next, the server performs analysis using the received data. An AI algorithm processes the data and evaluates the user's emotional and health states. It can also refer to past data to identify patterns in the user's state changes. Based on these analysis results, the server generates tasks and learning programs optimized for each individual user.

[0624] The generated suggestions are sent to the terminal and presented to the user visually. The terminal provides a user-friendly interface, designed to make it easy for the user to understand the suggested tasks. The user selects and executes tasks and learning programs based on the suggestions. The results of the execution and feedback are then sent back to the server via the terminal.

[0625] As an example, let's consider the case of a salaried worker. In the morning, while commuting to work, the device collects voice tone and facial expressions, and determines the stress level from this data. If the server detects high stress levels, it suggests lighter work tasks and a relaxation program to do during lunchtime. The device notifies the user, and the user can incorporate these suggestions into their schedule for the day.

[0626] This system allows users to efficiently complete tasks tailored to their physical and mental state on any given day, resulting in improved performance and reduced stress.

[0627] The following describes the processing flow.

[0628] Step 1:

[0629] The device collects the user's voice and video data in real time. Voice data is acquired via a microphone, and the tone, speed, and pitch of the voice are analyzed. Facial data is acquired using a camera, and facial movements and changes in expression are analyzed.

[0630] Step 2:

[0631] The device quickly encrypts the collected data and sends it to the server using a secure communication protocol. This communication is designed to protect user privacy.

[0632] Step 3:

[0633] The server decodes the received data and analyzes it using an AI algorithm. It evaluates the user's emotional state and health status from their voice and facial expressions, and compares this information with past data to identify patterns of change.

[0634] Step 4:

[0635] Based on the analysis results, the server generates tasks and learning programs tailored to the user. This includes suggestions for specific tasks and activities that match the user's current emotional and physical state.

[0636] Step 5:

[0637] The server sends the generated suggestions to the terminal. The terminal receives them and presents them to the user through a user-friendly, visualized interface.

[0638] Step 6:

[0639] The user reviews suggestions via an interface on their device and selects tasks or programs to execute. The user provides feedback and inputs the results into the device.

[0640] Step 7:

[0641] The device collects user feedback and execution results, re-encrypts them, and sends them to the server. The server stores this information and uses it as foundational data to improve the accuracy of future suggestions.

[0642] (Example 1)

[0643] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0644] In modern life, there is a need to effectively understand users' mental and physical states and provide appropriate activities and learning content accordingly. However, conventional systems have difficulty accurately assessing users' emotions and physical states in real time, making it impossible to provide individually optimized suggestions. To solve this problem, there is a need for technology that can accurately assess the user's state and quickly provide individually customized suggestions.

[0645] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0646] In this invention, the server includes information acquisition means for acquiring the user's voice and video and analyzing tone, speech speed, facial movements, etc.; data transmission means for securely transmitting the acquired voice and video information; and information analysis means for processing the transmitted information and evaluating the user's emotions and health status. This makes it possible to individually generate and provide optimal activities and learning procedures for the user in real time.

[0647] "Information acquisition means" refers to methods for acquiring the user's voice and video and analyzing their tone, speech speed, facial movements, etc.

[0648] "Data transmission means" refers to means for securely transmitting acquired audio and video information to a server.

[0649] "Information analysis means" refers to means for processing transmitted information and evaluating the user's emotions and health status.

[0650] A "proposal generation method" is a means for creating optimal activities and learning procedures for the user based on information analysis.

[0651] "Display means" refers to means for transmitting generated activities and learning procedures to a user device for visual presentation.

[0652] A "response processing means" is a means of collecting user feedback and using it to inform future proposals.

[0653] This invention is a system that provides activities and learning procedures optimized according to the user's mental and physical state. The system mainly consists of a terminal and a server. The terminal includes hardware that collects the user's voice and video data in real time. Specifically, the terminal incorporates a high-sensitivity microphone and a high-resolution camera to acquire the user's voice data and facial expressions. For the voice data, voice analysis software is used to analyze tone, speech speed, and voice pitch. For the video data, facial recognition and expression analysis software analyzes facial movements and changes in facial expressions in real time.

[0654] The acquired data is transmitted to the server via a secure communication protocol. The server analyzes the received data using advanced AI algorithms. This AI utilizes generative AI models to assess the user's current emotions and health status, and tracks changes in their state by analyzing past data as well.

[0655] Based on the analysis results, the server generates tasks and learning programs optimized for the user's state. For example, if the server determines that the user is experiencing high stress, it can suggest relaxation exercises or light work tasks. This generated content is sent to the terminal and presented to the user visually. The terminal's user interface is designed to make the suggestions clear and easy for the user to understand.

[0656] The user selects and performs a suggested task, and their experience and feedback are sent back to the server via the terminal. This allows the system to incorporate the user's feedback into future suggestions. A possible example of a prompt message would be, "Evaluate the user's stress level based on audio and video data, and suggest an appropriate task." This system enables users to efficiently perform activities suited to their condition, allowing them to enjoy a better quality of life.

[0657] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0658] Step 1:

[0659] The device acquires the user's voice in real time using a microphone. The input is raw audio data, and the output is digital data for speech analysis. Specifically, speech analysis software is used to decompose the audio data into tone, speech speed, and voice pitch, and feature quantities for each element are extracted.

[0660] Step 2:

[0661] The device captures the user's video with a camera and analyzes their facial expressions. The input is real-time video data, and the output is the analysis results regarding changes in facial expressions and facial movements. Facial recognition software analyzes this data to identify specific emotions and facial features.

[0662] Step 3:

[0663] The terminal aggregates audio and video analysis data and sends it to the server. The input is a dataset of audio and video analysis results, and the output is transmission data converted into a secure format. Using a data transmission means, these analysis results are sent to the server via a configured secure communication protocol.

[0664] Step 4:

[0665] The server analyzes received data using an AI algorithm to assess the user's emotions and health. The input is the analyzed audio and video data, and the output is the assessment of the user's current emotions and health. It utilizes a generative AI model to calculate multiple state indicators and infer the user's psychological and physiological state.

[0666] Step 5:

[0667] The server generates optimal activity and learning programs based on the evaluation results. The input is the evaluation results of the user's emotions and health state, and the output is a proposal for personalized activity and learning programs. Using the proposal generation means, appropriate tasks are generated and sent to the user's terminal in the next stage.

[0668] Step 6:

[0669] The terminal visually presents suggestions received from the server to the user. The input is suggested data for tasks and learning programs sent from the server, and the output is content displayed in a way that is easy for the user to understand. The terminal's interface clearly displays the suggestions and assists the user in selecting and performing activities.

[0670] Step 7:

[0671] The user selects and executes a suggested activity or learning program. The input is the displayed suggested activity or learning program, and the output is the result of the user's selection and execution. The user's selections are recorded and stored on the device as data for future use.

[0672] Step 8:

[0673] The device sends user feedback and activity results to the server. Inputs are the user's choices and execution results, while outputs are feedback data that forms the basis for future suggestions. Data from the device is transmitted to the server and incorporated into future suggestions.

[0674] (Application Example 1)

[0675] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0676] In modern society, users are expected to maintain their mental and physical health in their daily lives, but continuous management is not easy. In particular, there is a lack of methods to immediately respond to the stress and changes in physical condition that occur in daily life and to suggest appropriate tasks and mitigation activities. Solving this problem is essential.

[0677] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0678] In this invention, the server includes detection means for acquiring user acoustic and image data, communication means for securely transferring the acquired acoustic and image data, and information analysis means for analyzing the transferred data and evaluating the user's emotional and physical state. This makes it possible to propose optimal work tasks and relaxation activities in real time and on an individual basis, based on the user's daily mental and physical state.

[0679] A "user" is a person who is the subject of acoustic and image data, and who receives suggestions for optimal tasks and mitigation activities from the system.

[0680] "Acoustic data" refers to information about the user's voice and sounds acquired through devices such as microphones, and is part of the analysis used to evaluate their emotional state.

[0681] "Image data" refers to visual information about a user's face and movements collected through devices such as cameras, and is part of the analysis used to evaluate their emotional state.

[0682] "Detection means" refers to devices or functions for acquiring acoustic and image data from the user.

[0683] "Communication methods" refer to the technologies and protocols used to securely transfer acquired data, and are used to ensure the safe transmission of data.

[0684] "Information analysis means" refers to algorithms and processing technologies for analyzing transmitted data and evaluating the user's emotional state and health status.

[0685] "Business tasks" refer to specific tasks or activities that should be proposed to the user, and are selected taking into consideration the user's mental and physical state.

[0686] "Relaxation activities" refer to activities and activities suggested to reduce user stress and promote the user's mental relaxation.

[0687] The system for carrying out this invention uses a terminal equipped with a microphone and a camera as a device for acquiring the user's audio and image data. The terminal collects the user's voice tone, speed, and facial expressions in real time and transmits the data to a server using a secure protocol. Security in this series of communications is ensured using a protocol such as HTTPS.

[0688] The server uses AI algorithms to process the received data. Specifically, it uses machine learning frameworks such as TensorFlow and PyTorch to perform analysis to evaluate the user's emotional state and health status. This analysis makes it possible to identify patterns in changes in the user's state by comparing it with past data.

[0689] Based on the analysis results, the server generates work tasks and mitigation activities optimized for the user. These suggestions are communicated visually and audibly to the terminal, ensuring that the user can easily understand and perform them. As a result, the user performs the suggested activities and provides feedback, allowing the system to make even more personalized suggestions in the future.

[0690] For example, if the server determines that a user is experiencing high levels of fatigue or stress, it will suggest lighter work tasks or play relaxing music. At this time, the device will notify the user via voice, saying, "I will play relaxing music. Would you like to do some stretching?" and prompt the user to make a choice.

[0691] An example of an input prompt for the generating AI model is, "Based on the user's voice data and facial expression data, please suggest the most suitable relaxation activity for the day."

[0692] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0693] Step 1:

[0694] The device collects the user's audio and image data using a microphone and camera. During this process, it captures each user's voice and facial expressions as input. The collected audio tone, intonation, and changes in facial expressions are processed in real time and converted into digital data. The output is the raw data formed by this processing.

[0695] Step 2:

[0696] The device sends the collected data to the server using a secure protocol (e.g., HTTPS). The input here is the audio and image data generated in step 1. As output, unprocessed data of the user's voice and facial expressions is transferred to the server. This data transfer is performed using end-to-end encryption, ensuring privacy.

[0697] Step 3:

[0698] The server analyzes the received data using AI algorithms. The input consists of user voice and image data. The server uses machine learning frameworks such as TensorFlow and PyTorch to perform sentiment analysis and generate data that evaluates the user's mental and physical state. The output is evaluation data that indicates the user's state, including numerical values ​​and labels representing their stress level and emotional state.

[0699] Step 4:

[0700] The server generates optimal work tasks and mitigation activities for the user based on the analysis results. It uses the evaluation data generated in step 3 as input. The AI ​​model generated in this process is used to devise appropriate activities. The output is a list of specific tasks and mitigation activities to be proposed.

[0701] Step 5:

[0702] The server sends the generated suggestions to the terminal. The input is the list of tasks and activities generated in step 4. The output is instruction data for the terminal, which prepares the terminal to present the tasks to the user visually or audibly.

[0703] Step 6:

[0704] The terminal presents the received suggestions to the user. The input is suggestion data from the server. As output, the terminal actually notifies the user of the day's work tasks and relaxation activities. This notification includes specific actions that should be taken, such as suggesting playing music or stretching.

[0705] Step 7:

[0706] The user performs tasks and activities based on suggestions from the device. They refer to the suggestions from step 6 as input and reflect them in their actual actions. The results and feedback are sent back to the server via the device and used to improve the system's adaptability by influencing future suggestions. Feedback data is then generated as output.

[0707] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0708] This invention is a system that combines a user's voice and video data with an emotion engine that recognizes the user's emotions based on that data. The aim of this system is to recognize the user's emotional state in real time from the voice and video data and propose individually optimized tasks and learning programs.

[0709] Specifically, users use smartphones or wearable devices to collect voice and facial data. These devices are equipped with high-sensitivity microphones and cameras, which are used to acquire data such as the user's voice tone, facial expressions, and heart rate. The acquired data is analyzed by an emotion engine to evaluate the user's current emotional state. The emotion engine is built using an algorithm that combines multiple deep learning models, enabling it to identify emotions with high accuracy, even down to the subtle nuances of voice tone and facial expressions.

[0710] Next, the device securely transmits the analysis results to the server. Based on the received emotional state data, the server generates tasks and learning programs optimized for each individual user. At this time, the server also compares the data with past data to identify patterns in emotional fluctuations, enabling it to provide more sophisticated suggestions. The generated suggestions are sent to the device and displayed on the user's screen.

[0711] Users review the suggestions displayed on the screen and select tasks according to their mood and circumstances for the day. In this way, users can focus more effectively on work or learning. Furthermore, after completing a task, users can input feedback into their device. This feedback is sent back to the server and used to improve future task suggestions.

[0712] As a concrete example, consider a student user studying for final exams. The device detects a decrease in concentration from voice and facial expressions and suggests relaxing music and an efficient study schedule with breaks. The student tries the suggested schedule and provides feedback on how their concentration returns afterward, contributing to the improvement of the system's accuracy. Through this entire process, the user can achieve healthy and effective task execution.

[0713] The following describes the processing flow.

[0714] Step 1:

[0715] The device collects the user's voice and video data in real time. It uses the microphone to capture voice tone and speed, and the camera to capture changes in facial expressions.

[0716] Step 2:

[0717] The device inputs the collected data into the emotion engine. The emotion engine analyzes the audio and video data and uses a deep learning model to identify emotional states.

[0718] Step 3:

[0719] The device sends the emotion engine's recognition results to the server via a secure protocol. This data includes the user's specific emotional state and related information.

[0720] Step 4:

[0721] The server analyzes the received data and compares it with past data to analyze patterns of emotional fluctuations. This analysis allows us to understand the user's emotional tendencies.

[0722] Step 5:

[0723] The server generates optimal tasks and learning programs for the user based on their emotional state. The generated suggestions take into account the user's current mental and physical condition.

[0724] Step 6:

[0725] The server sends the tasks and programs it generates to the terminal. The terminal receives them and displays them to the user in a visually easy-to-understand format.

[0726] Step 7:

[0727] The user reviews the tasks and programs presented on their device and selects which ones to execute. The selected tasks are then reflected in the schedule.

[0728] Step 8:

[0729] After completing a task, the user enters feedback into the device. This feedback includes the results of the task and changes in their emotions.

[0730] Step 9:

[0731] The device sends feedback information to the server, which records it in a database. This data is then used as foundational information to make future task suggestions more accurate.

[0732] (Example 2)

[0733] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0734] In modern society, it is crucial to quickly and accurately understand an individual's emotional state and provide guidance for optimal behavior and learning based on that understanding. However, conventional systems lack the accuracy to analyze emotional fluctuations in real time. Furthermore, there is a challenge in effectively utilizing user feedback to improve the system.

[0735] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0736] In this invention, the server includes an inference means for analyzing the user's emotional state in real time, an analysis means for comparing and analyzing it with past data, and a generation means for generating individually optimized action and learning plans. This makes it possible to monitor the user's emotional state with high accuracy and provide individually optimized action and learning guidelines.

[0737] "Receiving means" refers to a device or software function for acquiring audio and video data from the user.

[0738] "Preprocessing means" refers to a device or software function for performing processing to reduce noise in acquired data and improve the accuracy of analysis.

[0739] "Inference means" refers to an algorithm or software function for analyzing a user's emotional state in real time based on pre-processed data.

[0740] "Transmission means" refers to a communication function or device for securely transferring analysis results to a server.

[0741] "Analysis method" refers to a function that compares past data with current sentiment data on a server to identify fluctuation patterns.

[0742] "Generative means" refers to algorithms or software functions for creating individually optimized action and learning plans.

[0743] "Presentation means" refers to a function that transmits the generated plan to the user's display device and presents it in a visually easy-to-understand format.

[0744] A "feedback processing mechanism" is a function that collects evaluations from users and uses them to improve the system or make suggestions for future updates.

[0745] To implement this invention, the user uses a terminal such as a smartphone or wearable device. These terminals are equipped with a high-sensitivity microphone and camera and have the ability to acquire audio and video data. The terminals may be integrated with a sensor system using a device platform such as Arduino or Raspberry Pi.

[0746] The terminal preprocesses audio data using digital signal processing technology to remove noise. Video data is broken down frame by frame to extract the user's facial features. The hardware used at this stage includes a microphone and camera module, while the software used includes audio processing libraries and image processing libraries.

[0747] The collected data is input from the device to an emotion engine for real-time analysis of emotional states. This emotion engine utilizes machine learning frameworks such as TensorFlow and PyTorch, and operates by combining multiple deep learning models. The server uses the SSL / TLS protocol to send and receive data to securely protect the received data.

[0748] The results analyzed by the emotion engine are sent to the server and compared with past data. The server uses data analysis techniques with Python and R to identify the user's emotional fluctuation patterns and generate tasks and learning programs optimized for the user. At this time, it utilizes prompt statements that the generating AI model can operate on to generate instructions in the form of, for example, "Based on my current concentration and stress level, please suggest ways to improve my study efficiency."

[0749] The generated tasks are sent to the terminal and presented on the user's screen. The terminal uses a user interface to visually display the suggestions in an easy-to-understand manner. The user performs the displayed tasks and inputs the results as feedback into the terminal. This feedback is then sent back to the server and used to improve the accuracy of future suggestions.

[0750] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0751] Step 1:

[0752] The device acquires the user's voice and video data in real time. The device's high-sensitivity microphone captures the user's voice, and the camera records facial expressions. The input is raw audio and video data, and the output is digital audio data and video frames in a noisy format.

[0753] Step 2:

[0754] The terminal preprocesses the acquired audio and video data. Specifically, noise is removed from the audio data through digital signal processing, and facial features are enhanced from the video data through image filtering. The input is the audio and video data from step 1, and the output is the digital data with noise removed and enhanced.

[0755] Step 3:

[0756] The device sends pre-processed data to the emotion engine. Based on the transmitted data, the emotion engine uses a deep learning model to infer the user's emotions. This process includes speech tone analysis and facial expression recognition. The input is the pre-processed data, and the output is the inferred emotion label and confidence level.

[0757] Step 4:

[0758] The device securely transmits the inferred sentiment data to the server. Here, the data is encrypted using the HTTPS protocol. The input is the sentiment label and confidence level, and the output is the securely transmitted data.

[0759] Step 5:

[0760] The server compares the received sentiment data with historical data. It uses a data analysis algorithm to identify patterns in sentiment fluctuations. The input is new sentiment data and historical records, and the output is the identified fluctuation patterns.

[0761] Step 6:

[0762] The server generates individually optimized tasks and learning programs based on the fluctuation patterns. A generative AI model is used here to create prompts. The input is the pattern of emotional fluctuations, and the output is the proposed task or program.

[0763] Step 7:

[0764] The server sends the generated tasks and programs to the terminal. The terminal receives them and displays them on the user's screen. The input is the details of the task or program, and the output is a visually represented proposal.

[0765] Step 8:

[0766] The user evaluates the suggestions displayed on the screen and enters feedback into the terminal. The terminal sends this feedback to the server, which is used to improve the system. The input is the user's evaluation, and the output is the feedback data sent to the server.

[0767] (Application Example 2)

[0768] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0769] In recent years, managing stress and anxiety within the family has become increasingly important, but there is a lack of systems that provide optimal care in real time according to individual circumstances. Furthermore, there is a need for skills to accurately perceive emotional changes, but the current problem is the lack of such skills. This makes it difficult to provide appropriate support for the mental health of families.

[0770] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0771] In this invention, the server includes recording means for collecting audio and video data to track the user's emotions in real time, data processing means for decomposing the transmitted data and analyzing the user's emotions and health status, and program creation means for generating a care plan and suggestions optimized for the user based on the analysis results. This enables appropriate care and suggestions tailored to the user's individual emotional state.

[0772] "Recording means" refers to a mechanism for collecting user audio and video data.

[0773] A "communication mechanism" is a means for securely transmitting collected audio and video data.

[0774] A "data processing system" is a mechanism for analyzing transmitted data and evaluating the user's emotions and health status.

[0775] The "program creation method" is a system for generating care plans and proposals optimized for the user based on the analysis results.

[0776] "Display means" refers to a method for transmitting and displaying the generated care plan and proposals on the user's terminal.

[0777] A "response processing mechanism" is a system for receiving feedback from users and using it to improve future proposals.

[0778] To implement this invention, a robot is required to operate the emotional care system within the home. This robot is equipped with highly sensitive sensors to collect and record the user's voice and video data in real time. Furthermore, to protect the user's privacy, the data is securely transmitted to a server via a communication mechanism.

[0779] The server has a data processing system that analyzes the received data, and this uses the deep learning framework TensorFlow. This data processing system analyzes voice tone, facial expressions, heart rate, etc., to evaluate the user's emotions and health status in real time.

[0780] Based on the evaluation results, the program creation method generates the optimal care plan and suggestions for the user. These suggestions are generated by an AI model that creates prompt statements based on a large amount of historical data.

[0781] The generated plans and suggestions are quickly transmitted to the user's terminal via a display device. The robot uses speech synthesis and a display to show this information to the user and assist them in taking the suggested actions.

[0782] For example, when the family is gathered in the living room after dinner, the robot might sense from the mother's voice and facial expressions that she is a little tired. In this case, it could make a suggestion such as, "Why don't you try to relax a bit tonight and take a long, leisurely bath?"

[0783] The hardware used includes high-sensitivity sensors such as microphones and cameras. The software utilizes a deep learning analysis platform based on TensorFlow, and HTTPS is used as the secure communication protocol.

[0784] An example of a prompt is, "Analyze the user's voice tone and facial expressions to generate appropriate stress care suggestions." This prompt allows the AI ​​to create the optimal care plan.

[0785] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0786] Step 1:

[0787] The device collects the user's voice and video data. Specifically, it records data in real time using a high-sensitivity sensor. The input consists of voice tone and facial expressions, and the output is recorded data containing these.

[0788] Step 2:

[0789] The terminal securely transmits collected audio and video data to the server via a communication mechanism. The input is recorded data, and the output is secure data that reaches the server.

[0790] Step 3:

[0791] The server analyzes the received audio and video data using data processing tools. The input is secure data, and the output is an analysis result indicating the user's emotions and health status. A deep learning model using TensorFlow processes this data.

[0792] Step 4:

[0793] The server uses program creation tools based on the analysis results to generate the optimal care plan and suggestions for the user. The input is the analysis results, and the output is the plan proposed to the user. Here, the generating AI model creates the plan using prompt statements.

[0794] Step 5:

[0795] The server transmits and presents the generated care plan and proposals to the terminal via a display device. The input is the care plan, and the output is the information displayed on the terminal.

[0796] Step 6:

[0797] The user reviews the proposed plan and enters feedback on their device. The input is the user's selected feedback, and the output is data used to improve future proposals.

[0798] Step 7:

[0799] The terminal sends user feedback data to the server. The input is user feedback, and the output is data provided to the server. The server uses this data to continuously improve the accuracy of the system.

[0800] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0801] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0802] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0803] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0804] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0805] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0806] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0807] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0808] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0809] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0810] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0811] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0812] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0813] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0814] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0815] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0816] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0817] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0818] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0819] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0820] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0821] The following is further disclosed regarding the embodiments described above.

[0822] (Claim 1)

[0823] A sensor means for collecting user voice and video data,

[0824] A communication means for securely transmitting collected audio and video data,

[0825] A data analysis means that analyzes the transmitted data and evaluates the user's emotions and health status,

[0826] A task generation means that generates optimal tasks and learning programs for the user based on evaluation,

[0827] A presentation means that transmits and displays the generated tasks and programs to the user terminal,

[0828] A feedback processing mechanism that receives user feedback and incorporates it into future proposals,

[0829] A system that includes this.

[0830] (Claim 2)

[0831] The system according to claim 1, further comprising a function for anonymizing and transmitting data collected from users.

[0832] (Claim 3)

[0833] The system according to claim 1, comprising a function to analyze the fluctuation patterns of the user's emotions and health status by comparing them with past data.

[0834] "Example 1"

[0835] (Claim 1)

[0836] Information acquisition means that acquires the user's voice and video and analyzes tone, speech speed, facial movements, etc.

[0837] A data transmission means for securely transmitting acquired audio and video information,

[0838] Information analysis means for processing transmitted information and evaluating the user's emotions and health status,

[0839] A proposal generation means that creates optimal activities and learning procedures for the user based on information analysis,

[0840] A display means that transmits and displays the generated activities and learning procedures on the user's device,

[0841] A response processing method for collecting user feedback and using it for future proposals,

[0842] A system that includes this.

[0843] (Claim 2)

[0844] The system according to claim 1, further comprising a function for transmitting information collected from users in an unidentifiable format.

[0845] (Claim 3)

[0846] The system according to claim 1, comprising a function to analyze the user's emotional and health trends by comparing them with past information.

[0847] "Application Example 1"

[0848] (Claim 1)

[0849] A detection means for acquiring user audio data and image data,

[0850] A communication means for securely transferring acquired acoustic and image data,

[0851] Information analysis means for analyzing transferred data and evaluating the user's emotional state and health state,

[0852] A task generation means that generates optimal work tasks and mitigation activities for the user based on evaluation,

[0853] A presentation means for transmitting the generated tasks and activities to an autonomous device and displaying them visually or audibly,

[0854] A response processing mechanism that receives user responses and incorporates them into future proposals,

[0855] A system that includes this.

[0856] (Claim 2)

[0857] The system according to claim 1, which includes a function to anonymize and transfer data obtained from a user.

[0858] (Claim 3)

[0859] The system according to claim 1, comprising a function to analyze the fluctuation patterns of the user's emotional state and health state by comparing them with past data.

[0860] "Example 2 of combining an emotion engine"

[0861] (Claim 1)

[0862] A receiving means for acquiring user audio and video data,

[0863] A preprocessing means for reducing noise and improving accuracy from acquired data,

[0864] An inference method that uses pre-processed data to analyze the user's emotional state in real time,

[0865] A means of securely transferring analysis results to a server,

[0866] An analytical method that compares and analyzes historical data and current sentiment data on a server to identify fluctuation patterns,

[0867] A generation means for generating individually optimized action and learning plans,

[0868] A presentation means that transmits the generated plan to the user's display device and visualizes it,

[0869] A feedback processing method that collects user evaluations and uses them for improvement,

[0870] A system that includes this.

[0871] (Claim 2)

[0872] The system according to claim 1, comprising a function for anonymizing and processing data received from a user.

[0873] (Claim 3)

[0874] The system according to claim 1, comprising a function to predict trends in the user's emotional state based on an analysis of past data.

[0875] "Application example 2 when combining with an emotional engine"

[0876] (Claim 1)

[0877] A recording means for collecting audio and video data to track user emotions in real time,

[0878] A communication mechanism for securely transmitting collected audio and video data,

[0879] A data processing means that decomposes the transmitted data and analyzes the user's emotions and health status,

[0880] A program creation method that generates user-optimized care plans and proposals based on the analysis results,

[0881] A display means for transmitting and displaying the generated care plan and proposals on the user's terminal,

[0882] A response processing mechanism that receives responses from users and uses them to improve future proposals,

[0883] A system that includes this.

[0884] (Claim 2)

[0885] The system according to claim 1, comprising a function for transmitting data collected from users in a manner that prevents the identification of individuals.

[0886] (Claim 3)

[0887] The system according to claim 1, comprising a function for analyzing the fluctuation patterns of the user's emotions and health status by comparing them with past data. [Explanation of Symbols]

[0888] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A sensor means for collecting user voice and video data, A communication means for securely transmitting collected audio and video data, A data analysis means that analyzes the transmitted data and evaluates the user's emotions and health status, A task generation means that generates optimal tasks and learning programs for the user based on evaluation, A presentation means that transmits and displays the generated tasks and programs to the user terminal, A feedback processing mechanism that receives user feedback and incorporates it into future proposals, A system that includes this.

2. The system according to claim 1, which includes a function for anonymizing and transmitting data collected from users.

3. The system according to claim 1, further comprising a function to analyze the fluctuation patterns of the user's emotions and health status by comparing them with past data.

Images