system

Abstract

Provide a system. 【Solution means】 Means for analyzing input information from a user using natural language processing technology and determining emotions; Means for generating advice suitable for the user's situation based on the determined emotions; Means for accumulating information on the user's emotions and actions and monitoring trends; Means for automatically sending additional advice to the user as needed; Means for anonymizing information to ensure the anonymity of the user; Means for analyzing daily conversations and evaluating the emotional state in real time; Means for displaying content that promotes meditation and relaxation on a visual device based on the emotional state; A system including the above.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern society, access to mental health support for maintaining mental health is often restricted by time, geographical constraints, and concerns about privacy. Also, it is difficult to provide individually customized support, and it is not easy for users to manage their own emotions and behaviors and continuously improve them. Against this background, there is an increasing need for a mental health support system that is available 24 hours a day, 365 days a year and can be used with confidence by users.

Means for Solving the Problems

[0005] This invention provides a means for analyzing user input data using natural language processing technology and determining emotions. It also includes a means for generating advice appropriate to the user's situation based on the determined emotions, thereby realizing personalized support. Furthermore, it has a means for accumulating user emotion and behavior data and monitoring trends, automatically sending additional advice when necessary. This system includes anonymization measures to protect user privacy, providing a secure and reliable environment for use.

[0006] "Natural language processing" is a technology that enables computers to understand, analyze, and generate human language.

[0007] "User input data" refers to data provided by the user in text or audio format.

[0008] "Means for determining emotions" refers to algorithms and processes that analyze user input data to identify emotional states.

[0009] "Means of generating advice" refers to a function that creates advice and information tailored to the user's situation based on their determined emotions.

[0010] "Means for accumulating emotional and behavioral data" refers to processes and functions for recording and saving a user's emotional state and behavioral history.

[0011] "Means of monitoring trends" refers to a function that analyzes accumulated data to continuously track and evaluate changes in emotions and behavior.

[0012] "Means for automatically sending additional advice" refers to a function that automatically sends additional advice to the user as needed, based on the monitored data.

[0013] "Anonymization methods for protecting privacy" refer to the process of protecting users' personal information and transforming data into a format that does not link it to a specific individual. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), etc.

[0018] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] The system of this invention is a mechanism that analyzes the user's emotions and provides personalized advice for the purpose of supporting mental health.

[0036] The device receives text and voice data entered by the user. This initial data includes the process of converting speech to text on the device if speech recognition is required. The received data is then sent to the server.

[0037] The server analyzes the received text data using natural language processing (NLP) techniques to determine the user's emotions. This analysis process involves analyzing the structure of the language to identify keywords and phrases that indicate emotion, thereby estimating the user's emotional state.

[0038] Once the assessment is complete, the server generates personalized advice based on the results. This advice may include mental health resources, relaxation techniques, or behavioral recommendations that address the emotional state. The server can also refer to accumulated historical data to provide more accurate advice depending on the situation.

[0039] The generated advice is sent to the user's device, which then displays it to the user. The user receives the advice through their device and can record a diary of their feelings and actions.

[0040] This diary feature helps users track their emotional changes and deepen their self-understanding. The server uses these records to monitor long-term emotional and behavioral trends and automatically sends additional advice if there are signs of the user's condition deteriorating.

[0041] Furthermore, the system handles all data anonymized to protect user privacy. This allows users to continue using the system with peace of mind.

[0042] In this way, we aim to create a system that supports the improvement of users' mental health and provides appropriate support at all times.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The device receives text or voice input from the user. In the case of voice input, the device uses speech recognition technology to convert the voice data into text data.

[0046] Step 2:

[0047] The terminal sends the converted or unconverted text data to the server. During this process, a communication protocol is used to ensure secure data transfer.

[0048] Step 3:

[0049] The server processes the received text data using a natural language processing (NLP) engine. This involves grammatical analysis and keyword detection to determine the user's sentiment.

[0050] Step 4:

[0051] Based on the determined emotions, the server retrieves appropriate advice and mental health resources from the database to generate an appropriate response for the user's situation.

[0052] Step 5:

[0053] The server sends the generated advice to the terminal. The terminal displays the received advice to the user and provides notifications as needed.

[0054] Step 6:

[0055] Users can record their responses to advice provided via the device, as well as their daily emotional changes, through the diary function.

[0056] Step 7:

[0057] The server stores user diaries and past emotional data, and monitors long-term trends. Based on this data, it automatically generates and sends additional advice as needed when the user's emotional state changes.

[0058] Step 8:

[0059] All data is anonymized and managed and stored on the server in a way that protects privacy. This allows users to continue using the system with peace of mind.

[0060] (Example 1)

[0061] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0062] In modern society, while individuals need to cope with stress and emotional fluctuations, they often lack easy access to individualized mental health support. Furthermore, concerns about privacy lead to caution regarding the handling of personal data, making it difficult to receive support with confidence. Under these circumstances, a system is needed that allows individuals to understand their own emotional and behavioral changes and receive appropriate support.

[0063] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0064] In this invention, the server includes means for analyzing user input information using natural language processing technology and determining emotions, means for creating personalized advice using a generative AI model, and means for de-identifying information to maintain user anonymity. This enables users to receive appropriate mental health support while protecting their privacy.

[0065] "Natural language processing technology" refers to the technology used in information processing devices to analyze natural language data and perform information extraction and semantic understanding.

[0066] A "means for determining emotions" refers to a device or program that analyzes received data and identifies the user's emotional state based on language, actions, etc.

[0067] A "generative AI model" is an artificial intelligence model used to generate the optimal output based on input data, and is particularly applied to language generation and providing advice.

[0068] "Means for creating personalized advice" refers to a device or program that automatically provides the most appropriate advice according to the user's specific circumstances and emotions.

[0069] "Speech recognition means" refers to a technology or device that collects speech data and converts it into text data.

[0070] "Means of de-identifying information" refers to techniques or processes that remove or transform specific identifiers from information so that the data cannot be used as personally identifiable information.

[0071] "Means for monitoring trends" refers to a device or program for continuously observing and recording changes and trends in data over time.

[0072] The system for implementing this invention analyzes user emotions based on user input and provides appropriate advice. It mainly consists of a terminal and a server, each performing a different role.

[0073] First, the user inputs text or audio data related to their emotions through the device. In the case of audio data, the device uses speech recognition technology to convert the input into text data. This conversion is generally performed using "speech recognition software," such as "Google Cloud Speech-to-Text."

[0074] The terminal then sends the converted text data to a server via the network. The server analyzes this data using natural language processing technology to determine the user's emotions. This process utilizes "natural language understanding software" and "text analysis tools" as "natural language processing systems." An example is "IBM Watson® Natural Language Understanding."

[0075] Based on the analysis, the server uses a generative AI model to create personalized advice. This generative AI model includes a "natural language generation engine," and models such as "OpenAI® GPT-3®" are applied. The generated advice includes specific suggestions related to the user's emotional state, offering relaxation techniques and advice on daily actions.

[0076] For example, if a user enters "I've been feeling more stressed at work lately," the server will analyze their emotions based on this data and generate advice such as "Take regular short breaks to refresh yourself." This generation is based on the following prompt:

[0077] "The user has recently reported feeling stressed. Please provide appropriate mental health support advice."

[0078] This allows users to receive immediate advice through their devices and gain concrete methods for managing their own mental health.

[0079] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0080] Step 1:

[0081] The user inputs text or audio data expressing their emotions into the device. For example, the input might be a statement like "I've been feeling tired lately." The device converts this audio input into text data using speech recognition technology. The speech recognition system used is called a "speech recognition module." At this stage, by processing the audio data into text data, the device obtains emotional information in text format as output.

[0082] Step 2:

[0083] The terminal sends the converted text data to the server. This involves the process of the terminal sending the text data and data communication taking place. As a result of this operation, data related to the user's emotions is output as analysis data to the server.

[0084] Step 3:

[0085] The server analyzes the received text data using natural language processing techniques. The server uses an "emotion analysis module" to extract keywords indicating specific emotions within the text. If the input data is "I've been feeling tired lately," the analysis detects emotions such as "fatigue" and "stress." At this point, emotional information is obtained as output through text analysis.

[0086] Step 4:

[0087] The server generates personalized advice using a generative AI model based on the analysis results. This generation process utilizes an "advice generation module" to create advice tailored to the emotional state. For example, it might generate advice such as, "It is recommended that you take a break." The input is emotional information, and the generated advice is the output.

[0088] Step 5:

[0089] The server sends the generated advice to the terminal. Data communication takes place here to deliver the advice to the user. This process outputs the personalized advice as display data on the terminal.

[0090] Step 6:

[0091] The terminal displays advice received from the server to the user. The terminal uses a "display module" to visually present the generated advice to the user. For example, the terminal screen might display "Try taking a short break to refresh yourself." This allows the user to receive the advice.

[0092] Step 7:

[0093] Users review the advice they receive and record changes in their emotions and behaviors in the diary function. Here, users record their daily emotional state and accumulate it as self-management data. Advice and personal emotional information are taken as input, and the recorded self-management data is obtained as output.

[0094] Step 8:

[0095] The server analyzes user diary data to monitor long-term emotional and behavioral trends. This process uses a "trend analysis module" for anomaly detection and trend analysis. The input is user diary data, and the analysis results output trend information. This prepares the server to send additional, situation-specific advice.

[0096] (Application Example 1)

[0097] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0098] In modern society, there is a growing need to appropriately assess individual emotional states and respond instantly. However, conventional systems have limitations in the accuracy of emotional assessment, and it has been difficult to provide personalized advice while protecting user privacy. In particular, there is a lack of real-time emotional analysis and direct support using visual devices.

[0099] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0100] In this invention, the server includes means for analyzing user input information using natural language processing technology to determine emotions, means for generating advice appropriate to the user's situation based on the determined emotions, and means for accumulating information on the user's emotions and behaviors and monitoring trends. This makes it possible to provide appropriate advice in real time according to the user's emotional state and to display content that promotes meditation and relaxation via a visual device.

[0101] "Natural language processing technology" is a technology that enables machines to understand and analyze human language, and it involves processing text and audio data to extract information.

[0102] "Users" refer to individuals who use this system or device and who provide information about their emotions and behaviors.

[0103] "Input information" refers to data in text or audio format provided by the user, and forms the basis for sentiment analysis.

[0104] "Determining emotions" refers to the act of identifying a user's psychological state at a given time based on the information they input.

[0105] "Generating advice" is the act of processing data based on identified emotions to provide users with useful information and action recommendations.

[0106] "Accumulating information" refers to the act of saving records of users' past emotions and behaviors and accumulating them in a database for later analysis and monitoring.

[0107] "Monitoring trends" is a process of analyzing long-term changes and patterns based on accumulated information and evaluating the user's condition.

[0108] "Ensuring anonymity" refers to the act of processing data in a way that prevents it from being identified by an individual, in order to protect the user's personal information.

[0109] "Visual devices" refer to equipment used to display information, and in this context, they are hardware used to provide content to users.

[0110] "Displaying content" refers to the act of providing information or exercises to users visually using visual devices.

[0111] The system for implementing this invention consists of a mechanism that applies natural language processing technology to provide individualized support for the mental health of users. The system mainly consists of two main components: a visual device worn by the user and a server that supports it in the backend.

[0112] The server converts the user's voice data into text via speech recognition software. This software includes speech recognition technologies such as Google Speech-to-Text. The converted text data is then analyzed using natural language processing libraries (e.g., SpaCy or NLTK) to understand the user's emotional state. Based on this analysis, the server generates personalized advice. The generated content is displayed on a visual device, such as smart glasses. This content could include meditation guidance or suggestions for relaxation exercises.

[0113] Visual devices play a role in presenting information to users. Through gentle visual content, users can engage in relaxation and meditation, which is expected to improve their mental state. This allows users to receive immediate mental health support in various aspects of their daily lives.

[0114] As a concrete example, if a user walking in a beautiful park says, "I've been feeling stressed lately," the system could analyze the results and play a video of natural sounds on smart glasses to promote relaxation. In this process, a generative AI model is used, and an example of a prompt message would be, "Tell us about the stress you've been feeling lately. Based on that, we'll give you the best advice."

[0115] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0116] Step 1:

[0117] When a user speaks, "I've been feeling stressed lately," the device captures the audio using its built-in microphone. The input is audio data. The device uses speech recognition software (e.g., Google Speech-to-Text) to convert this audio data into text data. The converted text is then sent to the server.

[0118] Step 2:

[0119] The server receives text data. The input is converted text. The server uses a natural language processing library (e.g., SpaCy or NLTK) to analyze the text data. Through this analysis, the server determines the user's emotional state. Specifically, it identifies keywords and phrases in the text that represent emotions and determines the level of stress.

[0120] Step 3:

[0121] The server generates appropriate advice and content based on the determined emotions. The input is the result of the emotion assessment, and the output is personalized advice for the user. By utilizing a generative AI model and using the prompt "Tell us about the stress you've been feeling lately. We will provide you with the best advice based on that," the generated advice is optimized.

[0122] Step 4:

[0123] The generated advice and content are transmitted to a visual device. The terminal, specifically smart glasses, receives this data and displays it to the user. The output is content that promotes meditation and relaxation. This allows the user to receive visual guidance to enhance relaxation.

[0124] Step 5:

[0125] While the user is experiencing the presented content, the server records the user's feedback and uses it to provide advice and generate content for future use. The input is the user's feedback, and this information is stored on the server and used to improve future analysis.

[0126] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0127] The present invention's system analyzes diverse input data from users to recognize emotions and provide mental health support. The system combines natural language processing technology and an emotion engine to perform more accurate emotion determination.

[0128] The device receives text, voice, or video data from the user and sends it to the server. For voice data, the device includes a process of converting the data to text using speech recognition technology. For video data, it also has the capability to extract facial expression data using facial recognition technology.

[0129] The server analyzes incoming data using both natural language processing (NLP) and an emotion engine. NLP analyzes key keywords and phrases, while the emotion engine analyzes facial expressions and voice tone to improve the accuracy of emotion recognition. It also uses time-series data to track and monitor changes in the user's emotions and analyze emotional trends from a long-term perspective.

[0130] After analysis, the server automatically generates optimal advice based on the determined emotions. This advice includes actionable and specific mental health strategies for the user. This advice is sent to the device and displayed to the user as a notification.

[0131] Furthermore, users can use the device's diary function to record changes in their emotions and behavior. The recorded data is then sent back to the server for aggregation. During this process, the data is always anonymized, ensuring that user privacy is fully protected.

[0132] As an example, if a user inputs "I've been experiencing increased work stress lately," the system combines language analysis and facial expression analysis to determine the level of stress and provides advice on specific relaxation techniques and stress management methods. This advice is continuously evaluated and adjusted as needed.

[0133] In this way, the system of the present invention realizes a form that supports and contributes to the improvement of the user's mental health.

[0134] The following describes the processing flow.

[0135] Step 1:

[0136] Users input information into the device via text, voice, or video. For voice input and video, the device uses speech recognition to convert the audio to text, and extracts facial expression data from video.

[0137] Step 2:

[0138] The device sends the converted text data and extracted facial expression data to the server. This transmission uses a secure communication protocol to protect the user's information.

[0139] Step 3:

[0140] The server processes the received text data using a natural language processing (NLP) engine, analyzing keywords and phrases to predict emotions. Simultaneously, it uses an emotion engine to analyze voice tone and facial expression data to determine emotions in more detail.

[0141] Step 4:

[0142] The server integrates text analysis results with voice and facial expression analysis results to determine the user's emotional state. This integration process improves the accuracy of emotion recognition.

[0143] Step 5:

[0144] The server retrieves or generates appropriate advice from the database based on the identified emotions. For example, if it determines that the user is experiencing high stress levels, it will suggest relaxation techniques.

[0145] Step 6:

[0146] The server sends the generated advice to the terminal, and the terminal displays the advice to the user in the form of a notification or message.

[0147] Step 7:

[0148] Users record feedback on advice provided through their devices and daily changes in their emotions using the diary function. This encourages self-reflection and is used for future analysis.

[0149] Step 8:

[0150] The server collects user feedback and diary data and monitors it over time. The data is anonymized to understand sentiment trends and use them for future analysis.

[0151] Step 9:

[0152] Based on long-term analysis, the server automatically provides additional advice and necessary interventions in response to changes in the user's emotions, thereby improving the user's mental health.

[0153] (Example 2)

[0154] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0155] In modern society, systems for providing individual mental health care are limited. In particular, accurately recognizing users' emotions and providing appropriate advice is crucial, but current technology makes it difficult to monitor and respond to changes in users' emotions over the long term. Therefore, a new system is needed to effectively support the management and improvement of individuals' mental states.

[0156] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0157] In this invention, the server includes means for analyzing user input information using natural language processing technology to determine emotions, means for generating suggestions appropriate to the user's situation based on the determined emotions, and means for accumulating information on the user's emotions and behavior and monitoring trends. This enables accurate recognition of the user's emotions, continuous monitoring, and appropriate mental health care tailored to individual situations.

[0158] "Natural language processing technology" is a technology that enables computers to understand and interpret human language, and is used to analyze user input.

[0159] "Means for determining emotions" refers to technology that identifies a user's emotions from input data and analyzes their state.

[0160] A "means for generating suggestions" is a mechanism for automatically creating appropriate advice and action guidelines based on the user's emotional state.

[0161] "Means of accumulating information and monitoring trends" refers to methods for collecting user sentiment and behavioral data and tracking changes in that data over a long period of time.

[0162] "Means of de-identifying information to protect anonymity" refers to processes that remove personal information from collected data in order to protect user privacy.

[0163] "Speech recognition means" refers to a technology that can convert speech data into text data and is used to analyze the user's voice input.

[0164] "Methods for analyzing facial expressions and vocal characteristics to improve the accuracy of emotion recognition" refer to technologies that analyze the user's visual and auditory characteristics to more accurately evaluate emotions.

[0165] This invention relates to a system that analyzes a user's emotions and provides mental health support. The system mainly consists of terminals and servers, each playing a specific role.

[0166] The device receives input from the user as text, voice, or video data. For voice data, the device converts it into text data using speech recognition technology. Specifically, general-purpose speech recognition software can be used. For video data, facial recognition technology is used to extract facial expression data. For this purpose, image processing libraries can be used, for example.

[0167] The server receives data transmitted from the terminal and analyzes it using natural language processing technology. During this process, natural language processing software is used to analyze key keywords and phrases from the text data. Furthermore, an emotion engine analyzes facial expressions and vocal characteristics to improve the accuracy of emotion recognition. Existing emotion analysis tools could be used for this purpose. The server also stores the received information and monitors changes in the user's emotions and behavior over a long period.

[0168] Based on the analysis results, the server generates personalized suggestions for the user. These suggestions include actionable mental health strategies and are provided to the user as notifications via the terminal. This allows the user to receive specific advice based on their current condition.

[0169] Furthermore, users can use a diary function through their device to record changes in their emotions and behavior. This data is anonymized and sent to a server to ensure privacy, and used for further analysis.

[0170] For example, if a user inputs "I've been experiencing increased work stress lately," the system uses language analysis and voice / facial expression analysis to determine the level of stress. Based on the results, the server generates advice on relaxation techniques and stress management methods. This advice is then notified to the terminal and provided to the user.

[0171] An example of a prompt for a generating AI model is: "The user has input that 'work-related stress has been increasing recently.' Based on this information, please suggest specific stress management methods."

[0172] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0173] Step 1:

[0174] The user provides emotional input data using the device. This input can be in text, audio, or video format. The device receives this data and begins processing it according to its format.

[0175] Step 2:

[0176] When the device receives audio data, it converts it into text using speech recognition technology. Specifically, it analyzes the sound data and outputs it as text data. For example, it recognizes the audio "I am tired" and outputs it as text. This converted text is then input to the next processing step.

[0177] Step 3:

[0178] In the case of video data, the terminal uses facial recognition technology to extract facial expression data. The process involves analyzing specific facial features from the video data, converting them into numerical data, and outputting it. This numerical data is sent to a server and used as input for emotion analysis.

[0179] Step 4:

[0180] The server analyzes the text data sent from the terminal using natural language processing technology. Specifically, it extracts keywords and phrases from the text and analyzes their meaning. For example, the keyword "tired" might be extracted and output as information for sentiment analysis.

[0181] Step 5:

[0182] The server analyzes facial and voice feature data using an emotion engine to recognize overall emotions. It quantifies the analysis results and outputs a specific emotional state. This output serves as the basis for deriving user-appropriate suggestions in subsequent processing steps.

[0183] Step 6:

[0184] The server generates personalized suggestions based on the analyzed emotional data. Using a generative AI model, it might suggest relaxation techniques, for example. These suggestions are sent to the user's device in text format.

[0185] Step 7:

[0186] Users check the suggestions they receive as notifications on their devices and take action as needed. They also use the diary function to record their emotions and actions on their devices. This record is sent to the server to be used for future analysis.

[0187] Step 8:

[0188] The device anonymizes the recorded data and sends it to the server. The server stores this data for future analysis and updates a database that monitors user sentiment. The stored data is continuously analyzed.

[0189] (Application Example 2)

[0190] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0191] In modern society, people's lives are extremely busy, and mental stress is constantly increasing. At the same time, opportunities to receive individualized mental health support are limited. Therefore, there is a need for technology that allows users to easily access mental health support in their daily lives while ensuring their privacy.

[0192] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0193] In this invention, the server includes means for analyzing user input data using natural language processing technology to determine emotions, means for generating advice appropriate to the user's situation based on the determined emotions, and means for interacting with the user and suggesting actions corresponding to those emotions in real time. This makes it possible for users to easily receive accurate mental health support in their daily lives.

[0194] "Natural language processing technology" refers to the technology that enables computers to understand, analyze, and generate human language.

[0195] "Means for determining emotions" refers to a method or device for identifying a user's emotional state based on data obtained from the user.

[0196] "Means for generating advice" refers to a method or apparatus that provides specific, actionable advice to a user based on their determined emotional state.

[0197] "Means of monitoring trends" refers to methods or devices for tracking user sentiment and behavioral data over the long term and analyzing changes over time.

[0198] "A means of suggesting actions that respond to emotions in real time" refers to a method or device for immediately responding to a user's current emotional state and recommending appropriate actions.

[0199] "Voice and vision sensors" are devices used to acquire the user's voice and video, and their role is to collect data.

[0200] "Means of anonymizing data" refers to methods or devices that make data unlinkable to a specific individual in order to conceal a user's personal information and protect their privacy.

[0201] To implement this invention, it is first necessary to incorporate an emotion recognition system into a household robot. The robot will be equipped with a high-resolution camera and microphone to collect audio and video in real time.

[0202] The server receives this data and uses natural language processing technology to analyze the user's voice data. Software such as Google's Dialogflow and Microsoft's Emotion API are used here. Using these, the robot performs voice recognition and emotion analysis to determine the user's emotional state.

[0203] Next, based on the determined emotional data, the server generates appropriate advice for the user. For example, if the user is stressed, it might suggest relaxation music or recommend short relaxation exercises.

[0204] Furthermore, the data collected by voice and visual sensors is anonymized so that personal information cannot be identified, enabling long-term data storage and analysis.

[0205] For example, if a user says, "I'm tired today," the robot can suggest, "How about taking a deep breath and relaxing?" Furthermore, it will offer a feature that tracks the user's daily emotional changes and allows for the identification of long-term mental health trends.

[0206] An example of a prompt would be, "When the user says 'I'm tired today' in a stressed voice, generate relaxation and encouragement advice." Through this prompt, the generating AI model will provide a more personalized response.

[0207] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0208] Step 1:

[0209] The device collects the user's voice and video data in real time using a high-resolution camera and microphone, and transmits this data to a server. The input is the user's voice and video, and the output is the transmission of data to the server.

[0210] Step 2:

[0211] The server converts the received audio data into text data using natural language processing technology. Speech recognition is performed using Google's Dialogflow. The input is audio data sent from the device, and the output is the text data obtained by converting the audio.

[0212] Step 3:

[0213] The server analyzes the converted text data and uses an emotion engine to determine the user's emotional state. It uses Microsoft's Emotion API for emotion identification. The input is text data, and the output is data indicating the user's emotional state.

[0214] Step 4:

[0215] The server generates advice based on the determined emotional state and sends it to the terminal. Prompts are used with the generating AI model to automatically generate mental support appropriate to the user. The input is the user's emotional state, and the output is appropriate advice.

[0216] Step 5:

[0217] The user reviews the advice received through the device and takes action as needed. Specific actions include suggestions such as playing music or taking deep breaths, and the device records the user's actions. The input is the advice sent from the server, and the output is the user's actions.

[0218] Step 6:

[0219] The device continuously records the user's actions and emotional changes, anonymizes the data, and sends it to the server. The input is user action data, and the output is stored on the server as anonymized data.

[0220] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0221] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0222] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0223] [Second Embodiment]

[0224] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0225] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0226] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0227] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0228] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0229] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0230] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0231] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0232] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0233] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0234] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0235] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0236] The system of this invention is a mechanism that analyzes the user's emotions and provides personalized advice for the purpose of supporting mental health.

[0237] The device receives text and voice data entered by the user. This initial data includes the process of converting speech to text on the device if speech recognition is required. The received data is then sent to the server.

[0238] The server analyzes the received text data using natural language processing (NLP) techniques to determine the user's emotions. This analysis process involves analyzing the structure of the language to identify keywords and phrases that indicate emotion, thereby estimating the user's emotional state.

[0239] Once the assessment is complete, the server generates personalized advice based on the results. This advice may include mental health resources, relaxation techniques, or behavioral recommendations that address the emotional state. The server can also refer to accumulated historical data to provide more accurate advice depending on the situation.

[0240] The generated advice is sent to the user's device, which then displays it to the user. The user receives the advice through their device and can record a diary of their feelings and actions.

[0241] This diary feature helps users track their emotional changes and deepen their self-understanding. The server uses these records to monitor long-term emotional and behavioral trends and automatically sends additional advice if there are signs of the user's condition deteriorating.

[0242] Furthermore, the system handles all data anonymized to protect user privacy. This allows users to continue using the system with peace of mind.

[0243] In this way, we aim to create a system that supports the improvement of users' mental health and provides appropriate support at all times.

[0244] The following describes the processing flow.

[0245] Step 1:

[0246] The device receives text or voice input from the user. In the case of voice input, the device uses speech recognition technology to convert the voice data into text data.

[0247] Step 2:

[0248] The terminal sends the converted or unconverted text data to the server. During this process, a communication protocol is used to ensure secure data transfer.

[0249] Step 3:

[0250] The server processes the received text data using a natural language processing (NLP) engine. This involves grammatical analysis and keyword detection to determine the user's sentiment.

[0251] Step 4:

[0252] Based on the determined emotions, the server retrieves appropriate advice and mental health resources from the database to generate an appropriate response for the user's situation.

[0253] Step 5:

[0254] The server sends the generated advice to the terminal. The terminal displays the received advice to the user and provides notifications as needed.

[0255] Step 6:

[0256] Users can record their responses to advice provided via the device, as well as their daily emotional changes, through the diary function.

[0257] Step 7:

[0258] The server stores user diaries and past emotional data, and monitors long-term trends. Based on this data, it automatically generates and sends additional advice as needed when the user's emotional state changes.

[0259] Step 8:

[0260] All data is anonymized and managed and stored on the server in a way that protects privacy. This allows users to continue using the system with peace of mind.

[0261] (Example 1)

[0262] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0263] In modern society, while individuals need to cope with stress and emotional fluctuations, they often lack easy access to individualized mental health support. Furthermore, concerns about privacy lead to caution regarding the handling of personal data, making it difficult to receive support with confidence. Under these circumstances, a system is needed that allows individuals to understand their own emotional and behavioral changes and receive appropriate support.

[0264] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0265] In this invention, the server includes means for analyzing user input information using natural language processing technology and determining emotions, means for creating personalized advice using a generative AI model, and means for de-identifying information to maintain user anonymity. This enables users to receive appropriate mental health support while protecting their privacy.

[0266] "Natural language processing technology" refers to the technology used in information processing devices to analyze natural language data and perform information extraction and semantic understanding.

[0267] A "means for determining emotions" refers to a device or program that analyzes received data and identifies the user's emotional state based on language, actions, etc.

[0268] A "generative AI model" is an artificial intelligence model used to generate the optimal output based on input data, and is particularly applied to language generation and providing advice.

[0269] "Means for creating personalized advice" refers to a device or program that automatically provides the most appropriate advice according to the user's specific circumstances and emotions.

[0270] "Speech recognition means" refers to a technology or device that collects speech data and converts it into text data.

[0271] "Means of de-identifying information" refers to techniques or processes that remove or transform specific identifiers from information so that the data cannot be used as personally identifiable information.

[0272] "Means for monitoring trends" refers to a device or program for continuously observing and recording changes and trends in data over time.

[0273] The system for implementing this invention analyzes user emotions based on user input and provides appropriate advice. It mainly consists of a terminal and a server, each performing a different role.

[0274] First, the user inputs text or audio data related to their emotions through the device. In the case of audio data, the device uses speech recognition technology to convert the input into text data. This conversion is generally performed using "speech recognition software," such as "Google Cloud Speech-to-Text."

[0275] The terminal then sends the converted text data to a server via the network. The server analyzes this data using natural language processing technology to determine the user's emotions. This process utilizes "natural language processing systems," such as "natural language understanding software" and "text analysis tools." An example is "IBM Watson Natural Language Understanding."

[0276] Based on the analysis, the server uses a generative AI model to create personalized advice. This generative AI model includes a "natural language generation engine," and models such as "OpenAI GPT-3" are applied. The generated advice includes specific suggestions related to the user's emotional state, offering relaxation techniques and advice on daily actions.

[0277] For example, if a user enters "I've been feeling more stressed at work lately," the server will analyze their emotions based on this data and generate advice such as "Take regular short breaks to refresh yourself." This generation is based on the following prompt:

[0278] "The user has recently reported feeling stressed. Please provide appropriate mental health support advice."

[0279] This allows users to receive immediate advice through their devices and gain concrete methods for managing their own mental health.

[0280] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0281] Step 1:

[0282] The user inputs text or voice data representing their emotions into the terminal. The input here is, for example, a statement like "I've been tired lately". The terminal converts this voice input into text data using voice recognition technology. The voice recognition system used is the "voice recognition module". At this stage, by processing the voice data into text data, emotional information in text format is obtained as the output.

[0283] Step 2:

[0284] The terminal sends the converted text data to the server. Here, the process of the terminal sending the text data is included, and data communication is performed. By this operation, data regarding the user's emotions is output as the server's analysis data.

[0285] Step 3:

[0286] The server analyzes the received text data using natural language processing technology. The server uses the "emotion analysis module" to extract keywords indicating specific emotions in the text. If the input data is "I've been tired lately", emotions such as "fatigue" and "stress" are detected through analysis. Here, emotional information is obtained as the output through text analysis.

[0287] Step 4:

[0288] The server generates personalized advice using a generation AI model based on the analysis results. In this generation process, the "advice generation module" is used, and advice corresponding to the emotional state is formed. For example, advice like "It is recommended to set aside break time" is generated. The input is emotional information, and the generated advice is the output.

[0289] Step 5:

[0290] The server sends the generated advice to the terminal. Data communication takes place here to deliver the advice to the user. This process outputs the personalized advice as display data on the terminal.

[0291] Step 6:

[0292] The terminal displays advice received from the server to the user. The terminal uses a "display module" to visually present the generated advice to the user. For example, the terminal screen might display "Try taking a short break to refresh yourself." This allows the user to receive the advice.

[0293] Step 7:

[0294] Users review the advice they receive and record changes in their emotions and behaviors in the diary function. Here, users record their daily emotional state and accumulate it as self-management data. Advice and personal emotional information are taken as input, and the recorded self-management data is obtained as output.

[0295] Step 8:

[0296] The server analyzes user diary data to monitor long-term emotional and behavioral trends. This process uses a "trend analysis module" for anomaly detection and trend analysis. The input is user diary data, and the analysis results output trend information. This prepares the server to send additional, situation-specific advice.

[0297] (Application Example 1)

[0298] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0299] In modern society, there is a growing need to appropriately assess individual emotional states and respond instantly. However, conventional systems have limitations in the accuracy of emotional assessment, and it has been difficult to provide personalized advice while protecting user privacy. In particular, there is a lack of real-time emotional analysis and direct support using visual devices.

[0300] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0301] In this invention, the server includes means for analyzing user input information using natural language processing technology to determine emotions, means for generating advice appropriate to the user's situation based on the determined emotions, and means for accumulating information on the user's emotions and behaviors and monitoring trends. This makes it possible to provide appropriate advice in real time according to the user's emotional state and to display content that promotes meditation and relaxation via a visual device.

[0302] "Natural language processing technology" is a technology that enables machines to understand and analyze human language, and it involves processing text and audio data to extract information.

[0303] "Users" refer to individuals who use this system or device and who provide information about their emotions and behaviors.

[0304] "Input information" refers to data in text or audio format provided by the user, and forms the basis for sentiment analysis.

[0305] "Determining emotions" refers to the act of identifying a user's psychological state at a given time based on the information they input.

[0306] "Generating advice" is the act of processing data based on identified emotions to provide users with useful information and action recommendations.

[0307] "To accumulate information" refers to the act of storing records related to a user's past emotions and actions and aggregating them in a database for subsequent analysis and monitoring.

[0308] "To monitor trends" refers to the process of analyzing long-term changes and patterns based on the accumulated information and evaluating the user's state.

[0309] "To ensure anonymity" refers to the act of processing data so that it does not match an individual for the purpose of protecting the user's personal information.

[0310] "Visual device" refers to a device for displaying information, and here it is the hardware used to provide content to the user.

[0311] "To display content" refers to the act of visually providing information or exercises to the user using a visual device.

[0312] The system for implementing this invention is configured with a mechanism that applies natural language processing technology to individually support the user's mental health. The system is mainly composed of two main components. That is, a visual device worn by the user and a server that supports it at the backend.

[0313] The server converts the user's voice data into text via speech recognition software. The software used here includes speech recognition technologies such as Google Speech-to-Text. The converted text data is analyzed for the user's emotional state using natural language processing libraries (e.g., SpaCy or NLTK). Based on the results of this analysis, the server generates individualized advice. The generated content is displayed on the display of a visual device, such as smart glasses. Examples of the content to be displayed include meditation guidance and the presentation of exercises to promote relaxation.

[0314] Visual devices play a role in presenting information to users. Through gentle visual content, users can engage in relaxation and meditation, which is expected to improve their mental state. This allows users to receive immediate mental health support in various aspects of their daily lives.

[0315] As a concrete example, if a user walking in a beautiful park says, "I've been feeling stressed lately," the system could analyze the results and play a video of natural sounds on smart glasses to promote relaxation. In this process, a generative AI model is used, and an example of a prompt message would be, "Tell us about the stress you've been feeling lately. Based on that, we'll give you the best advice."

[0316] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0317] Step 1:

[0318] When a user speaks, "I've been feeling stressed lately," the device captures the audio using its built-in microphone. The input is audio data. The device uses speech recognition software (e.g., Google Speech-to-Text) to convert this audio data into text data. The converted text is then sent to the server.

[0319] Step 2:

[0320] The server receives text data. The input is converted text. The server uses a natural language processing library (e.g., SpaCy or NLTK) to analyze the text data. Through this analysis, the server determines the user's emotional state. Specifically, it identifies keywords and phrases in the text that represent emotions and determines the level of stress.

[0321] Step 3:

[0322] The server generates appropriate advice and content based on the determined emotions. The input is the result of the emotion assessment, and the output is personalized advice for the user. By utilizing a generative AI model and using the prompt "Tell us about the stress you've been feeling lately. We will provide you with the best advice based on that," the generated advice is optimized.

[0323] Step 4:

[0324] The generated advice and content are transmitted to a visual device. The terminal, specifically smart glasses, receives this data and displays it to the user. The output is content that promotes meditation and relaxation. This allows the user to receive visual guidance to enhance relaxation.

[0325] Step 5:

[0326] While the user is experiencing the presented content, the server records the user's feedback and uses it to provide advice and generate content for future use. The input is the user's feedback, and this information is stored on the server and used to improve future analysis.

[0327] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0328] The present invention's system analyzes diverse input data from users to recognize emotions and provide mental health support. The system combines natural language processing technology and an emotion engine to perform more accurate emotion determination.

[0329] The device receives text, voice, or video data from the user and sends it to the server. For voice data, the device includes a process of converting the data to text using speech recognition technology. For video data, it also has the capability to extract facial expression data using facial recognition technology.

[0330] The server analyzes incoming data using both natural language processing (NLP) and an emotion engine. NLP analyzes key keywords and phrases, while the emotion engine analyzes facial expressions and voice tone to improve the accuracy of emotion recognition. It also uses time-series data to track and monitor changes in the user's emotions and analyze emotional trends from a long-term perspective.

[0331] After analysis, the server automatically generates optimal advice based on the determined emotions. This advice includes actionable and specific mental health strategies for the user. This advice is sent to the device and displayed to the user as a notification.

[0332] Furthermore, users can use the device's diary function to record changes in their emotions and behavior. The recorded data is then sent back to the server for aggregation. During this process, the data is always anonymized, ensuring that user privacy is fully protected.

[0333] As an example, if a user inputs "I've been experiencing increased work stress lately," the system combines language analysis and facial expression analysis to determine the level of stress and provides advice on specific relaxation techniques and stress management methods. This advice is continuously evaluated and adjusted as needed.

[0334] In this way, the system of the present invention realizes a form that supports and contributes to the improvement of the user's mental health.

[0335] The following describes the processing flow.

[0336] Step 1:

[0337] Users input information into the device via text, voice, or video. For voice input and video, the device uses speech recognition to convert the audio to text, and extracts facial expression data from video.

[0338] Step 2:

[0339] The device sends the converted text data and extracted facial expression data to the server. This transmission uses a secure communication protocol to protect the user's information.

[0340] Step 3:

[0341] The server processes the received text data using a natural language processing (NLP) engine, analyzing keywords and phrases to predict emotions. Simultaneously, it uses an emotion engine to analyze voice tone and facial expression data to determine emotions in more detail.

[0342] Step 4:

[0343] The server integrates text analysis results with voice and facial expression analysis results to determine the user's emotional state. This integration process improves the accuracy of emotion recognition.

[0344] Step 5:

[0345] The server retrieves or generates appropriate advice from the database based on the identified emotions. For example, if it determines that the user is experiencing high stress levels, it will suggest relaxation techniques.

[0346] Step 6:

[0347] The server sends the generated advice to the terminal, and the terminal displays the advice to the user in the form of a notification or message.

[0348] Step 7:

[0349] Users record feedback on advice provided through their devices and daily changes in their emotions using the diary function. This encourages self-reflection and is used for future analysis.

[0350] Step 8:

[0351] The server collects user feedback and diary data and monitors it over time. The data is anonymized to understand sentiment trends and use them for future analysis.

[0352] Step 9:

[0353] Based on long-term analysis, the server automatically provides additional advice and necessary interventions in response to changes in the user's emotions, thereby improving the user's mental health.

[0354] (Example 2)

[0355] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0356] In modern society, systems for providing individual mental health care are limited. In particular, accurately recognizing users' emotions and providing appropriate advice is crucial, but current technology makes it difficult to monitor and respond to changes in users' emotions over the long term. Therefore, a new system is needed to effectively support the management and improvement of individuals' mental states.

[0357] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0358] In this invention, the server includes means for analyzing user input information using natural language processing technology to determine emotions, means for generating suggestions appropriate to the user's situation based on the determined emotions, and means for accumulating information on the user's emotions and behavior and monitoring trends. This enables accurate recognition of the user's emotions, continuous monitoring, and appropriate mental health care tailored to individual situations.

[0359] "Natural language processing technology" is a technology that enables computers to understand and interpret human language, and is used to analyze user input.

[0360] "Means for determining emotions" refers to technology that identifies a user's emotions from input data and analyzes their state.

[0361] A "means for generating suggestions" is a mechanism for automatically creating appropriate advice and action guidelines based on the user's emotional state.

[0362] "Means of accumulating information and monitoring trends" refers to methods for collecting user sentiment and behavioral data and tracking changes in that data over a long period of time.

[0363] "Means of de-identifying information to protect anonymity" refers to processes that remove personal information from collected data in order to protect user privacy.

[0364] "Speech recognition means" refers to a technology that can convert speech data into text data and is used to analyze the user's voice input.

[0365] "Methods for analyzing facial expressions and vocal characteristics to improve the accuracy of emotion recognition" refer to technologies that analyze the user's visual and auditory characteristics to more accurately evaluate emotions.

[0366] This invention relates to a system that analyzes a user's emotions and provides mental health support. The system mainly consists of terminals and servers, each playing a specific role.

[0367] The device receives input from the user as text, voice, or video data. For voice data, the device converts it into text data using speech recognition technology. Specifically, general-purpose speech recognition software can be used. For video data, facial recognition technology is used to extract facial expression data. For this purpose, image processing libraries can be used, for example.

[0368] The server receives data transmitted from the terminal and analyzes it using natural language processing technology. During this process, natural language processing software is used to analyze key keywords and phrases from the text data. Furthermore, an emotion engine analyzes facial expressions and vocal characteristics to improve the accuracy of emotion recognition. Existing emotion analysis tools could be used for this purpose. The server also stores the received information and monitors changes in the user's emotions and behavior over a long period.

[0369] Based on the analysis results, the server generates personalized suggestions for the user. These suggestions include actionable mental health strategies and are provided to the user as notifications via the terminal. This allows the user to receive specific advice based on their current condition.

[0370] Furthermore, users can use a diary function through their device to record changes in their emotions and behavior. This data is anonymized and sent to a server to ensure privacy, and used for further analysis.

[0371] For example, if a user inputs "I've been experiencing increased work stress lately," the system uses language analysis and voice / facial expression analysis to determine the level of stress. Based on the results, the server generates advice on relaxation techniques and stress management methods. This advice is then notified to the terminal and provided to the user.

[0372] An example of a prompt for a generating AI model is: "The user has input that 'work-related stress has been increasing recently.' Based on this information, please suggest specific stress management methods."

[0373] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0374] Step 1:

[0375] The user provides emotional input data using the device. This input can be in text, audio, or video format. The device receives this data and begins processing it according to its format.

[0376] Step 2:

[0377] When the device receives audio data, it converts it into text using speech recognition technology. Specifically, it analyzes the sound data and outputs it as text data. For example, it recognizes the audio "I am tired" and outputs it as text. This converted text is then input to the next processing step.

[0378] Step 3:

[0379] In the case of video data, the terminal uses facial recognition technology to extract facial expression data. The process involves analyzing specific facial features from the video data, converting them into numerical data, and outputting it. This numerical data is sent to a server and used as input for emotion analysis.

[0380] Step 4:

[0381] The server analyzes the text data sent from the terminal using natural language processing technology. Specifically, it extracts keywords and phrases from the text and analyzes their meaning. For example, the keyword "tired" might be extracted and output as information for sentiment analysis.

[0382] Step 5:

[0383] The server analyzes facial and voice feature data using an emotion engine to recognize overall emotions. It quantifies the analysis results and outputs a specific emotional state. This output serves as the basis for deriving user-appropriate suggestions in subsequent processing steps.

[0384] Step 6:

[0385] The server generates personalized suggestions based on the analyzed emotional data. Using a generative AI model, it might suggest relaxation techniques, for example. These suggestions are sent to the user's device in text format.

[0386] Step 7:

[0387] Users check the suggestions they receive as notifications on their devices and take action as needed. They also use the diary function to record their emotions and actions on their devices. This record is sent to the server to be used for future analysis.

[0388] Step 8:

[0389] The device anonymizes the recorded data and sends it to the server. The server stores this data for future analysis and updates a database that monitors user sentiment. The stored data is continuously analyzed.

[0390] (Application Example 2)

[0391] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0392] In modern society, people's lives are extremely busy, and mental stress is constantly increasing. At the same time, opportunities to receive individualized mental health support are limited. Therefore, there is a need for technology that allows users to easily access mental health support in their daily lives while ensuring their privacy.

[0393] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0394] In this invention, the server includes means for analyzing user input data using natural language processing technology to determine emotions, means for generating advice appropriate to the user's situation based on the determined emotions, and means for interacting with the user and suggesting actions corresponding to those emotions in real time. This makes it possible for users to easily receive accurate mental health support in their daily lives.

[0395] "Natural language processing technology" refers to the technology that enables computers to understand, analyze, and generate human language.

[0396] "Means for determining emotions" refers to a method or device for identifying a user's emotional state based on data obtained from the user.

[0397] "Means for generating advice" refers to a method or apparatus that provides specific, actionable advice to a user based on their determined emotional state.

[0398] "Means of monitoring trends" refers to methods or devices for tracking user sentiment and behavioral data over the long term and analyzing changes over time.

[0399] "A means of suggesting actions that respond to emotions in real time" refers to a method or device for immediately responding to a user's current emotional state and recommending appropriate actions.

[0400] "Voice and vision sensors" are devices used to acquire the user's voice and video, and their role is to collect data.

[0401] "Means of anonymizing data" refers to methods or devices that make data unlinkable to a specific individual in order to conceal a user's personal information and protect their privacy.

[0402] To implement this invention, it is first necessary to incorporate an emotion recognition system into a household robot. The robot will be equipped with a high-resolution camera and microphone to collect audio and video in real time.

[0403] The server receives this data and uses natural language processing technology to analyze the user's voice data. Software such as Google's Dialogflow and Microsoft's Emotion API are used here. Using these, the robot performs voice recognition and emotion analysis to determine the user's emotional state.

[0404] Next, based on the determined emotional data, the server generates appropriate advice for the user. For example, if the user is stressed, it might suggest relaxation music or recommend short relaxation exercises.

[0405] Furthermore, the data collected by voice and visual sensors is anonymized so that personal information cannot be identified, enabling long-term data storage and analysis.

[0406] For example, if a user says, "I'm tired today," the robot can suggest, "How about taking a deep breath and relaxing?" Furthermore, it will offer a feature that tracks the user's daily emotional changes and allows for the identification of long-term mental health trends.

[0407] An example of a prompt would be, "When the user says 'I'm tired today' in a stressed voice, generate relaxation and encouragement advice." Through this prompt, the generating AI model will provide a more personalized response.

[0408] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0409] Step 1:

[0410] The device collects the user's voice and video data in real time using a high-resolution camera and microphone, and transmits this data to a server. The input is the user's voice and video, and the output is the transmission of data to the server.

[0411] Step 2:

[0412] The server converts the received audio data into text data using natural language processing technology. Speech recognition is performed using Google's Dialogflow. The input is audio data sent from the device, and the output is the text data obtained by converting the audio.

[0413] Step 3:

[0414] The server analyzes the converted text data and uses an emotion engine to determine the user's emotional state. It uses Microsoft's Emotion API for emotion identification. The input is text data, and the output is data indicating the user's emotional state.

[0415] Step 4:

[0416] The server generates advice based on the determined emotional state and sends it to the terminal. Prompts are used with the generating AI model to automatically generate mental support appropriate to the user. The input is the user's emotional state, and the output is appropriate advice.

[0417] Step 5:

[0418] The user reviews the advice received through the device and takes action as needed. Specific actions include suggestions such as playing music or taking deep breaths, and the device records the user's actions. The input is the advice sent from the server, and the output is the user's actions.

[0419] Step 6:

[0420] The device continuously records the user's actions and emotional changes, anonymizes the data, and sends it to the server. The input is user action data, and the output is stored on the server as anonymized data.

[0421] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0422] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0423] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0424] [Third Embodiment]

[0425] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0426] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0427] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0428] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0429] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0430] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0431] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0432] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0433] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0434] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0435] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0436] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0437] The system of this invention is a mechanism that analyzes the user's emotions and provides personalized advice for the purpose of supporting mental health.

[0438] The device receives text and voice data entered by the user. This initial data includes the process of converting speech to text on the device if speech recognition is required. The received data is then sent to the server.

[0439] The server analyzes the received text data using natural language processing (NLP) techniques to determine the user's emotions. This analysis process involves analyzing the structure of the language to identify keywords and phrases that indicate emotion, thereby estimating the user's emotional state.

[0440] Once the assessment is complete, the server generates personalized advice based on the results. This advice may include mental health resources, relaxation techniques, or behavioral recommendations that address the emotional state. The server can also refer to accumulated historical data to provide more accurate advice depending on the situation.

[0441] The generated advice is sent to the user's device, which then displays it to the user. The user receives the advice through their device and can record a diary of their feelings and actions.

[0442] This diary feature helps users track their emotional changes and deepen their self-understanding. The server uses these records to monitor long-term emotional and behavioral trends and automatically sends additional advice if there are signs of the user's condition deteriorating.

[0443] Furthermore, the system handles all data anonymized to protect user privacy. This allows users to continue using the system with peace of mind.

[0444] In this way, we aim to create a system that supports the improvement of users' mental health and provides appropriate support at all times.

[0445] The following describes the processing flow.

[0446] Step 1:

[0447] The device receives text or voice input from the user. In the case of voice input, the device uses speech recognition technology to convert the voice data into text data.

[0448] Step 2:

[0449] The terminal sends the converted or unconverted text data to the server. During this process, a communication protocol is used to ensure secure data transfer.

[0450] Step 3:

[0451] The server processes the received text data using a natural language processing (NLP) engine. This involves grammatical analysis and keyword detection to determine the user's sentiment.

[0452] Step 4:

[0453] Based on the determined emotions, the server retrieves appropriate advice and mental health resources from the database to generate an appropriate response for the user's situation.

[0454] Step 5:

[0455] The server sends the generated advice to the terminal. The terminal displays the received advice to the user and provides notifications as needed.

[0456] Step 6:

[0457] Users can record their responses to advice provided via the device, as well as their daily emotional changes, through the diary function.

[0458] Step 7:

[0459] The server stores user diaries and past emotional data, and monitors long-term trends. Based on this data, it automatically generates and sends additional advice as needed when the user's emotional state changes.

[0460] Step 8:

[0461] All data is anonymized and managed and stored on the server in a way that protects privacy. This allows users to continue using the system with peace of mind.

[0462] (Example 1)

[0463] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0464] In modern society, while individuals need to cope with stress and emotional fluctuations, they often lack easy access to individualized mental health support. Furthermore, concerns about privacy lead to caution regarding the handling of personal data, making it difficult to receive support with confidence. Under these circumstances, a system is needed that allows individuals to understand their own emotional and behavioral changes and receive appropriate support.

[0465] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0466] In this invention, the server includes means for analyzing user input information using natural language processing technology and determining emotions, means for creating personalized advice using a generative AI model, and means for de-identifying information to maintain user anonymity. This enables users to receive appropriate mental health support while protecting their privacy.

[0467] "Natural language processing technology" refers to the technology used in information processing devices to analyze natural language data and perform information extraction and semantic understanding.

[0468] A "means for determining emotions" refers to a device or program that analyzes received data and identifies the user's emotional state based on language, actions, etc.

[0469] A "generative AI model" is an artificial intelligence model used to generate the optimal output based on input data, and is particularly applied to language generation and providing advice.

[0470] "Means for creating personalized advice" refers to a device or program that automatically provides the most appropriate advice according to the user's specific circumstances and emotions.

[0471] "Speech recognition means" refers to a technology or device that collects speech data and converts it into text data.

[0472] "Means of de-identifying information" refers to techniques or processes that remove or transform specific identifiers from information so that the data cannot be used as personally identifiable information.

[0473] "Means for monitoring trends" refers to a device or program for continuously observing and recording changes and trends in data over time.

[0474] The system for implementing this invention analyzes user emotions based on user input and provides appropriate advice. It mainly consists of a terminal and a server, each performing a different role.

[0475] First, the user inputs text or audio data related to their emotions through the device. In the case of audio data, the device uses speech recognition technology to convert the input into text data. This conversion is generally performed using "speech recognition software," such as "Google Cloud Speech-to-Text."

[0476] The terminal then sends the converted text data to a server via the network. The server analyzes this data using natural language processing technology to determine the user's emotions. This process utilizes "natural language processing systems," such as "natural language understanding software" and "text analysis tools." An example is "IBM Watson Natural Language Understanding."

[0477] Based on the analysis, the server uses a generative AI model to create personalized advice. This generative AI model includes a "natural language generation engine," and models such as "OpenAI GPT-3" are applied. The generated advice includes specific suggestions related to the user's emotional state, offering relaxation techniques and advice on daily actions.

[0478] For example, if a user enters "I've been feeling more stressed at work lately," the server will analyze their emotions based on this data and generate advice such as "Take regular short breaks to refresh yourself." This generation is based on the following prompt:

[0479] "The user has recently reported feeling stressed. Please provide appropriate mental health support advice."

[0480] This allows users to receive immediate advice through their devices and gain concrete methods for managing their own mental health.

[0481] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0482] Step 1:

[0483] The user inputs text or audio data expressing their emotions into the device. For example, the input might be a statement like "I've been feeling tired lately." The device converts this audio input into text data using speech recognition technology. The speech recognition system used is called a "speech recognition module." At this stage, by processing the audio data into text data, the device obtains emotional information in text format as output.

[0484] Step 2:

[0485] The terminal sends the converted text data to the server. This involves the process of the terminal sending the text data and data communication taking place. As a result of this operation, data related to the user's emotions is output as analysis data to the server.

[0486] Step 3:

[0487] The server analyzes the received text data using natural language processing techniques. The server uses an "emotion analysis module" to extract keywords indicating specific emotions within the text. If the input data is "I've been feeling tired lately," the analysis detects emotions such as "fatigue" and "stress." At this point, emotional information is obtained as output through text analysis.

[0488] Step 4:

[0489] The server generates personalized advice using a generative AI model based on the analysis results. This generation process utilizes an "advice generation module" to create advice tailored to the emotional state. For example, it might generate advice such as, "It is recommended that you take a break." The input is emotional information, and the generated advice is the output.

[0490] Step 5:

[0491] The server sends the generated advice to the terminal. Data communication takes place here to deliver the advice to the user. This process outputs the personalized advice as display data on the terminal.

[0492] Step 6:

[0493] The terminal displays advice received from the server to the user. The terminal uses a "display module" to visually present the generated advice to the user. For example, the terminal screen might display "Try taking a short break to refresh yourself." This allows the user to receive the advice.

[0494] Step 7:

[0495] Users review the advice they receive and record changes in their emotions and behaviors in the diary function. Here, users record their daily emotional state and accumulate it as self-management data. Advice and personal emotional information are taken as input, and the recorded self-management data is obtained as output.

[0496] Step 8:

[0497] The server analyzes user diary data to monitor long-term emotional and behavioral trends. This process uses a "trend analysis module" for anomaly detection and trend analysis. The input is user diary data, and the analysis results output trend information. This prepares the server to send additional, situation-specific advice.

[0498] (Application Example 1)

[0499] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0500] In modern society, there is a growing need to appropriately assess individual emotional states and respond instantly. However, conventional systems have limitations in the accuracy of emotional assessment, and it has been difficult to provide personalized advice while protecting user privacy. In particular, there is a lack of real-time emotional analysis and direct support using visual devices.

[0501] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0502] In this invention, the server includes means for analyzing user input information using natural language processing technology to determine emotions, means for generating advice appropriate to the user's situation based on the determined emotions, and means for accumulating information on the user's emotions and behaviors and monitoring trends. This makes it possible to provide appropriate advice in real time according to the user's emotional state and to display content that promotes meditation and relaxation via a visual device.

[0503] "Natural language processing technology" is a technology that enables machines to understand and analyze human language, and it involves processing text and audio data to extract information.

[0504] "Users" refer to individuals who use this system or device and who provide information about their emotions and behaviors.

[0505] "Input information" refers to data in text or audio format provided by the user, and forms the basis for sentiment analysis.

[0506] "Determining emotions" refers to the act of identifying a user's psychological state at a given time based on the information they input.

[0507] "Generating advice" is the act of processing data based on identified emotions to provide users with useful information and action recommendations.

[0508] "Accumulating information" refers to the act of saving records of users' past emotions and behaviors and accumulating them in a database for later analysis and monitoring.

[0509] "Monitoring trends" is a process of analyzing long-term changes and patterns based on accumulated information and evaluating the user's condition.

[0510] "Ensuring anonymity" refers to the act of processing data in a way that prevents it from being identified by an individual, in order to protect the user's personal information.

[0511] "Visual devices" refer to equipment used to display information, and in this context, they are hardware used to provide content to users.

[0512] "Displaying content" refers to the act of providing information or exercises to users visually using visual devices.

[0513] The system for implementing this invention consists of a mechanism that applies natural language processing technology to provide individualized support for the mental health of users. The system mainly consists of two main components: a visual device worn by the user and a server that supports it in the backend.

[0514] The server converts the user's voice data into text via speech recognition software. This software includes speech recognition technologies such as Google Speech-to-Text. The converted text data is then analyzed using natural language processing libraries (e.g., SpaCy or NLTK) to understand the user's emotional state. Based on this analysis, the server generates personalized advice. The generated content is displayed on a visual device, such as smart glasses. This content could include meditation guidance or suggestions for relaxation exercises.

[0515] Visual devices play a role in presenting information to users. Through gentle visual content, users can engage in relaxation and meditation, which is expected to improve their mental state. This allows users to receive immediate mental health support in various aspects of their daily lives.

[0516] As a concrete example, if a user walking in a beautiful park says, "I've been feeling stressed lately," the system could analyze the results and play a video of natural sounds on smart glasses to promote relaxation. In this process, a generative AI model is used, and an example of a prompt message would be, "Tell us about the stress you've been feeling lately. Based on that, we'll give you the best advice."

[0517] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0518] Step 1:

[0519] When a user speaks, "I've been feeling stressed lately," the device captures the audio using its built-in microphone. The input is audio data. The device uses speech recognition software (e.g., Google Speech-to-Text) to convert this audio data into text data. The converted text is then sent to the server.

[0520] Step 2:

[0521] The server receives text data. The input is converted text. The server uses a natural language processing library (e.g., SpaCy or NLTK) to analyze the text data. Through this analysis, the server determines the user's emotional state. Specifically, it identifies keywords and phrases in the text that represent emotions and determines the level of stress.

[0522] Step 3:

[0523] The server generates appropriate advice and content based on the determined emotions. The input is the result of the emotion assessment, and the output is personalized advice for the user. By utilizing a generative AI model and using the prompt "Tell us about the stress you've been feeling lately. We will provide you with the best advice based on that," the generated advice is optimized.

[0524] Step 4:

[0525] The generated advice and content are transmitted to a visual device. The terminal, specifically smart glasses, receives this data and displays it to the user. The output is content that promotes meditation and relaxation. This allows the user to receive visual guidance to enhance relaxation.

[0526] Step 5:

[0527] While the user is experiencing the presented content, the server records the user's feedback and uses it to provide advice and generate content for future use. The input is the user's feedback, and this information is stored on the server and used to improve future analysis.

[0528] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0529] The present invention's system analyzes diverse input data from users to recognize emotions and provide mental health support. The system combines natural language processing technology and an emotion engine to perform more accurate emotion determination.

[0530] The device receives text, voice, or video data from the user and sends it to the server. For voice data, the device includes a process of converting the data to text using speech recognition technology. For video data, it also has the capability to extract facial expression data using facial recognition technology.

[0531] The server analyzes incoming data using both natural language processing (NLP) and an emotion engine. NLP analyzes key keywords and phrases, while the emotion engine analyzes facial expressions and voice tone to improve the accuracy of emotion recognition. It also uses time-series data to track and monitor changes in the user's emotions and analyze emotional trends from a long-term perspective.

[0532] After analysis, the server automatically generates optimal advice based on the determined emotions. This advice includes actionable and specific mental health strategies for the user. This advice is sent to the device and displayed to the user as a notification.

[0533] Furthermore, users can use the device's diary function to record changes in their emotions and behavior. The recorded data is then sent back to the server for aggregation. During this process, the data is always anonymized, ensuring that user privacy is fully protected.

[0534] As an example, if a user inputs "I've been experiencing increased work stress lately," the system combines language analysis and facial expression analysis to determine the level of stress and provides advice on specific relaxation techniques and stress management methods. This advice is continuously evaluated and adjusted as needed.

[0535] In this way, the system of the present invention realizes a form that supports and contributes to the improvement of the user's mental health.

[0536] The following describes the processing flow.

[0537] Step 1:

[0538] Users input information into the device via text, voice, or video. For voice input and video, the device uses speech recognition to convert the audio to text, and extracts facial expression data from video.

[0539] Step 2:

[0540] The device sends the converted text data and extracted facial expression data to the server. This transmission uses a secure communication protocol to protect the user's information.

[0541] Step 3:

[0542] The server processes the received text data using a natural language processing (NLP) engine, analyzing keywords and phrases to predict emotions. Simultaneously, it uses an emotion engine to analyze voice tone and facial expression data to determine emotions in more detail.

[0543] Step 4:

[0544] The server integrates text analysis results with voice and facial expression analysis results to determine the user's emotional state. This integration process improves the accuracy of emotion recognition.

[0545] Step 5:

[0546] The server retrieves or generates appropriate advice from the database based on the identified emotions. For example, if it determines that the user is experiencing high stress levels, it will suggest relaxation techniques.

[0547] Step 6:

[0548] The server sends the generated advice to the terminal, and the terminal displays the advice to the user in the form of a notification or message.

[0549] Step 7:

[0550] Users record feedback on advice provided through their devices and daily changes in their emotions using the diary function. This encourages self-reflection and is used for future analysis.

[0551] Step 8:

[0552] The server collects user feedback and diary data and monitors it over time. The data is anonymized to understand sentiment trends and use them for future analysis.

[0553] Step 9:

[0554] Based on long-term analysis, the server automatically provides additional advice and necessary interventions in response to changes in the user's emotions, thereby improving the user's mental health.

[0555] (Example 2)

[0556] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0557] In modern society, systems for providing individual mental health care are limited. In particular, accurately recognizing users' emotions and providing appropriate advice is crucial, but current technology makes it difficult to monitor and respond to changes in users' emotions over the long term. Therefore, a new system is needed to effectively support the management and improvement of individuals' mental states.

[0558] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0559] In this invention, the server includes means for analyzing user input information using natural language processing technology to determine emotions, means for generating suggestions appropriate to the user's situation based on the determined emotions, and means for accumulating information on the user's emotions and behavior and monitoring trends. This enables accurate recognition of the user's emotions, continuous monitoring, and appropriate mental health care tailored to individual situations.

[0560] "Natural language processing technology" is a technology that enables computers to understand and interpret human language, and is used to analyze user input.

[0561] "Means for determining emotions" refers to technology that identifies a user's emotions from input data and analyzes their state.

[0562] A "means for generating suggestions" is a mechanism for automatically creating appropriate advice and action guidelines based on the user's emotional state.

[0563] "Means of accumulating information and monitoring trends" refers to methods for collecting user sentiment and behavioral data and tracking changes in that data over a long period of time.

[0564] "Means of de-identifying information to protect anonymity" refers to processes that remove personal information from collected data in order to protect user privacy.

[0565] "Speech recognition means" refers to a technology that can convert speech data into text data and is used to analyze the user's voice input.

[0566] "Methods for analyzing facial expressions and vocal characteristics to improve the accuracy of emotion recognition" refer to technologies that analyze the user's visual and auditory characteristics to more accurately evaluate emotions.

[0567] This invention relates to a system that analyzes a user's emotions and provides mental health support. The system mainly consists of terminals and servers, each playing a specific role.

[0568] The device receives input from the user as text, voice, or video data. For voice data, the device converts it into text data using speech recognition technology. Specifically, general-purpose speech recognition software can be used. For video data, facial recognition technology is used to extract facial expression data. For this purpose, image processing libraries can be used, for example.

[0569] The server receives data transmitted from the terminal and analyzes it using natural language processing technology. During this process, natural language processing software is used to analyze key keywords and phrases from the text data. Furthermore, an emotion engine analyzes facial expressions and vocal characteristics to improve the accuracy of emotion recognition. Existing emotion analysis tools could be used for this purpose. The server also stores the received information and monitors changes in the user's emotions and behavior over a long period.

[0570] Based on the analysis results, the server generates personalized suggestions for the user. These suggestions include actionable mental health strategies and are provided to the user as notifications via the terminal. This allows the user to receive specific advice based on their current condition.

[0571] Furthermore, users can use a diary function through their device to record changes in their emotions and behavior. This data is anonymized and sent to a server to ensure privacy, and used for further analysis.

[0572] For example, if a user inputs "I've been experiencing increased work stress lately," the system uses language analysis and voice / facial expression analysis to determine the level of stress. Based on the results, the server generates advice on relaxation techniques and stress management methods. This advice is then notified to the terminal and provided to the user.

[0573] An example of a prompt for a generating AI model is: "The user has input that 'work-related stress has been increasing recently.' Based on this information, please suggest specific stress management methods."

[0574] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0575] Step 1:

[0576] The user provides emotional input data using the device. This input can be in text, audio, or video format. The device receives this data and begins processing it according to its format.

[0577] Step 2:

[0578] When the device receives audio data, it converts it into text using speech recognition technology. Specifically, it analyzes the sound data and outputs it as text data. For example, it recognizes the audio "I am tired" and outputs it as text. This converted text is then input to the next processing step.

[0579] Step 3:

[0580] In the case of video data, the terminal uses facial recognition technology to extract facial expression data. The process involves analyzing specific facial features from the video data, converting them into numerical data, and outputting it. This numerical data is sent to a server and used as input for emotion analysis.

[0581] Step 4:

[0582] The server analyzes the text data sent from the terminal using natural language processing technology. Specifically, it extracts keywords and phrases from the text and analyzes their meaning. For example, the keyword "tired" might be extracted and output as information for sentiment analysis.

[0583] Step 5:

[0584] The server analyzes facial and voice feature data using an emotion engine to recognize overall emotions. It quantifies the analysis results and outputs a specific emotional state. This output serves as the basis for deriving user-appropriate suggestions in subsequent processing steps.

[0585] Step 6:

[0586] The server generates personalized suggestions based on the analyzed emotional data. Using a generative AI model, it might suggest relaxation techniques, for example. These suggestions are sent to the user's device in text format.

[0587] Step 7:

[0588] Users check the suggestions they receive as notifications on their devices and take action as needed. They also use the diary function to record their emotions and actions on their devices. This record is sent to the server to be used for future analysis.

[0589] Step 8:

[0590] The device anonymizes the recorded data and sends it to the server. The server stores this data for future analysis and updates a database that monitors user sentiment. The stored data is continuously analyzed.

[0591] (Application Example 2)

[0592] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0593] In modern society, people's lives are extremely busy, and mental stress is constantly increasing. At the same time, opportunities to receive individualized mental health support are limited. Therefore, there is a need for technology that allows users to easily access mental health support in their daily lives while ensuring their privacy.

[0594] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0595] In this invention, the server includes means for analyzing user input data using natural language processing technology to determine emotions, means for generating advice appropriate to the user's situation based on the determined emotions, and means for interacting with the user and suggesting actions corresponding to those emotions in real time. This makes it possible for users to easily receive accurate mental health support in their daily lives.

[0596] "Natural language processing technology" refers to the technology that enables computers to understand, analyze, and generate human language.

[0597] "Means for determining emotions" refers to a method or device for identifying a user's emotional state based on data obtained from the user.

[0598] "Means for generating advice" refers to a method or apparatus that provides specific, actionable advice to a user based on their determined emotional state.

[0599] "Means of monitoring trends" refers to methods or devices for tracking user sentiment and behavioral data over the long term and analyzing changes over time.

[0600] "A means of suggesting actions that respond to emotions in real time" refers to a method or device for immediately responding to a user's current emotional state and recommending appropriate actions.

[0601] "Voice and vision sensors" are devices used to acquire the user's voice and video, and their role is to collect data.

[0602] "Means of anonymizing data" refers to methods or devices that make data unlinkable to a specific individual in order to conceal a user's personal information and protect their privacy.

[0603] To implement this invention, it is first necessary to incorporate an emotion recognition system into a household robot. The robot will be equipped with a high-resolution camera and microphone to collect audio and video in real time.

[0604] The server receives this data and uses natural language processing technology to analyze the user's voice data. Software such as Google's Dialogflow and Microsoft's Emotion API are used here. Using these, the robot performs voice recognition and emotion analysis to determine the user's emotional state.

[0605] Next, based on the determined emotional data, the server generates appropriate advice for the user. For example, if the user is stressed, it might suggest relaxation music or recommend short relaxation exercises.

[0606] Furthermore, the data collected by voice and visual sensors is anonymized so that personal information cannot be identified, enabling long-term data storage and analysis.

[0607] For example, if a user says, "I'm tired today," the robot can suggest, "How about taking a deep breath and relaxing?" Furthermore, it will offer a feature that tracks the user's daily emotional changes and allows for the identification of long-term mental health trends.

[0608] An example of a prompt would be, "When the user says 'I'm tired today' in a stressed voice, generate relaxation and encouragement advice." Through this prompt, the generating AI model will provide a more personalized response.

[0609] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0610] Step 1:

[0611] The device collects the user's voice and video data in real time using a high-resolution camera and microphone, and transmits this data to a server. The input is the user's voice and video, and the output is the transmission of data to the server.

[0612] Step 2:

[0613] The server converts the received audio data into text data using natural language processing technology. Speech recognition is performed using Google's Dialogflow. The input is audio data sent from the device, and the output is the text data obtained by converting the audio.

[0614] Step 3:

[0615] The server analyzes the converted text data and uses an emotion engine to determine the user's emotional state. It uses Microsoft's Emotion API for emotion identification. The input is text data, and the output is data indicating the user's emotional state.

[0616] Step 4:

[0617] The server generates advice based on the determined emotional state and sends it to the terminal. Prompts are used with the generating AI model to automatically generate mental support appropriate to the user. The input is the user's emotional state, and the output is appropriate advice.

[0618] Step 5:

[0619] The user reviews the advice received through the device and takes action as needed. Specific actions include suggestions such as playing music or taking deep breaths, and the device records the user's actions. The input is the advice sent from the server, and the output is the user's actions.

[0620] Step 6:

[0621] The device continuously records the user's actions and emotional changes, anonymizes the data, and sends it to the server. The input is user action data, and the output is stored on the server as anonymized data.

[0622] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0623] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0624] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0625] [Fourth Embodiment]

[0626] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0627] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0628] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0629] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0630] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0631] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0632] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0633] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0634] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0635] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0636] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0637] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0638] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0639] The system of this invention is a mechanism that analyzes the user's emotions and provides personalized advice for the purpose of supporting mental health.

[0640] The device receives text and voice data entered by the user. This initial data includes the process of converting speech to text on the device if speech recognition is required. The received data is then sent to the server.

[0641] The server analyzes the received text data using natural language processing (NLP) techniques to determine the user's emotions. This analysis process involves analyzing the structure of the language to identify keywords and phrases that indicate emotion, thereby estimating the user's emotional state.

[0642] Once the assessment is complete, the server generates personalized advice based on the results. This advice may include mental health resources, relaxation techniques, or behavioral recommendations that address the emotional state. The server can also refer to accumulated historical data to provide more accurate advice depending on the situation.

[0643] The generated advice is sent to the user's device, which then displays it to the user. The user receives the advice through their device and can record a diary of their feelings and actions.

[0644] This diary feature helps users track their emotional changes and deepen their self-understanding. The server uses these records to monitor long-term emotional and behavioral trends and automatically sends additional advice if there are signs of the user's condition deteriorating.

[0645] Furthermore, the system handles all data anonymized to protect user privacy. This allows users to continue using the system with peace of mind.

[0646] In this way, we aim to create a system that supports the improvement of users' mental health and provides appropriate support at all times.

[0647] The following describes the processing flow.

[0648] Step 1:

[0649] The device receives text or voice input from the user. In the case of voice input, the device uses speech recognition technology to convert the voice data into text data.

[0650] Step 2:

[0651] The terminal sends the converted or unconverted text data to the server. During this process, a communication protocol is used to ensure secure data transfer.

[0652] Step 3:

[0653] The server processes the received text data using a natural language processing (NLP) engine. This involves grammatical analysis and keyword detection to determine the user's sentiment.

[0654] Step 4:

[0655] Based on the determined emotions, the server retrieves appropriate advice and mental health resources from the database to generate an appropriate response for the user's situation.

[0656] Step 5:

[0657] The server sends the generated advice to the terminal. The terminal displays the received advice to the user and provides notifications as needed.

[0658] Step 6:

[0659] Users can record their responses to advice provided via the device, as well as their daily emotional changes, through the diary function.

[0660] Step 7:

[0661] The server stores user diaries and past sentiment data, and monitors long-term trends. Based on this data, it automatically generates and sends additional advice as needed when the user's emotional state changes.

[0662] Step 8:

[0663] All data is anonymized and managed and stored on the server in a way that protects privacy. This allows users to continue using the system with peace of mind.

[0664] (Example 1)

[0665] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0666] In modern society, while individuals need to cope with stress and emotional fluctuations, they often lack easy access to individualized mental health support. Furthermore, concerns about privacy lead to caution regarding the handling of personal data, making it difficult to receive support with confidence. Under these circumstances, a system is needed that allows individuals to understand their own emotional and behavioral changes and receive appropriate support.

[0667] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0668] In this invention, the server includes means for analyzing user input information using natural language processing technology and determining emotions, means for creating personalized advice using a generative AI model, and means for de-identifying information to maintain user anonymity. This enables users to receive appropriate mental health support while protecting their privacy.

[0669] "Natural language processing technology" refers to the technology used in information processing devices to analyze natural language data and perform information extraction and semantic understanding.

[0670] A "means for determining emotions" refers to a device or program that analyzes received data and identifies the user's emotional state based on language, actions, etc.

[0671] A "generative AI model" is an artificial intelligence model used to generate the optimal output based on input data, and is particularly applied to language generation and providing advice.

[0672] "Means for creating personalized advice" refers to a device or program that automatically provides the most appropriate advice according to the user's specific circumstances and emotions.

[0673] "Speech recognition means" refers to a technology or device that collects speech data and converts it into text data.

[0674] "Means of de-identifying information" refers to techniques or processes that remove or transform specific identifiers from information so that the data cannot be used as personally identifiable information.

[0675] "Means for monitoring trends" refers to a device or program for continuously observing and recording changes and trends in data over time.

[0676] The system for implementing this invention analyzes user emotions based on user input and provides appropriate advice. It mainly consists of a terminal and a server, each performing a different role.

[0677] First, the user inputs text or audio data related to their emotions through the device. In the case of audio data, the device uses speech recognition technology to convert the input into text data. This conversion is generally performed using "speech recognition software," such as "Google Cloud Speech-to-Text."

[0678] The terminal then sends the converted text data to a server via the network. The server analyzes this data using natural language processing technology to determine the user's emotions. This process utilizes "natural language processing systems," such as "natural language understanding software" and "text analysis tools." An example is "IBM Watson Natural Language Understanding."

[0679] Based on the analysis, the server uses a generative AI model to create personalized advice. This generative AI model includes a "natural language generation engine," and models such as "OpenAI GPT-3" are applied. The generated advice includes specific suggestions related to the user's emotional state, offering relaxation techniques and advice on daily actions.

[0680] For example, if a user enters "I've been feeling more stressed at work lately," the server will analyze their emotions based on this data and generate advice such as "Take regular short breaks to refresh yourself." This generation is based on the following prompt:

[0681] "The user has recently reported feeling stressed. Please provide appropriate mental health support advice."

[0682] This allows users to receive immediate advice through their devices and gain concrete methods for managing their own mental health.

[0683] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0684] Step 1:

[0685] The user inputs text or audio data expressing their emotions into the device. For example, the input might be a statement like "I've been feeling tired lately." The device converts this audio input into text data using speech recognition technology. The speech recognition system used is called a "speech recognition module." At this stage, by processing the audio data into text data, the device obtains emotional information in text format as output.

[0686] Step 2:

[0687] The terminal sends the converted text data to the server. This involves the process of the terminal sending the text data and data communication taking place. As a result of this operation, data related to the user's emotions is output as analysis data to the server.

[0688] Step 3:

[0689] The server analyzes the received text data using natural language processing techniques. The server uses an "emotion analysis module" to extract keywords indicating specific emotions within the text. If the input data is "I've been feeling tired lately," the analysis detects emotions such as "fatigue" and "stress." At this point, emotional information is obtained as output through text analysis.

[0690] Step 4:

[0691] The server generates personalized advice using a generative AI model based on the analysis results. This generation process utilizes an "advice generation module" to create advice tailored to the emotional state. For example, it might generate advice such as, "It is recommended that you take a break." The input is emotional information, and the generated advice is the output.

[0692] Step 5:

[0693] The server sends the generated advice to the terminal. Data communication takes place here to deliver the advice to the user. This process outputs the personalized advice as display data on the terminal.

[0694] Step 6:

[0695] The terminal displays advice received from the server to the user. The terminal uses a "display module" to visually present the generated advice to the user. For example, the terminal screen might display "Try taking a short break to refresh yourself." This allows the user to receive the advice.

[0696] Step 7:

[0697] Users review the advice they receive and record changes in their emotions and behaviors in the diary function. Here, users record their daily emotional state and accumulate it as self-management data. Advice and personal emotional information are taken as input, and the recorded self-management data is obtained as output.

[0698] Step 8:

[0699] The server analyzes user diary data to monitor long-term emotional and behavioral trends. This process uses a "trend analysis module" for anomaly detection and trend analysis. The input is user diary data, and the analysis results output trend information. This prepares the server to send additional, situation-specific advice.

[0700] (Application Example 1)

[0701] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0702] In modern society, there is a growing need to appropriately assess individual emotional states and respond instantly. However, conventional systems have limitations in the accuracy of emotional assessment, and it has been difficult to provide personalized advice while protecting user privacy. In particular, there is a lack of real-time emotional analysis and direct support using visual devices.

[0703] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0704] In this invention, the server includes means for analyzing user input information using natural language processing technology to determine emotions, means for generating advice appropriate to the user's situation based on the determined emotions, and means for accumulating information on the user's emotions and behaviors and monitoring trends. This makes it possible to provide appropriate advice in real time according to the user's emotional state and to display content that promotes meditation and relaxation via a visual device.

[0705] "Natural language processing technology" is a technology that enables machines to understand and analyze human language, and it involves processing text and audio data to extract information.

[0706] "Users" refer to individuals who use this system or device and who provide information about their emotions and behaviors.

[0707] "Input information" refers to data in text or audio format provided by the user, and forms the basis for sentiment analysis.

[0708] "Determining emotions" refers to the act of identifying a user's psychological state at a given time based on the information they input.

[0709] "Generating advice" is the act of processing data based on identified emotions to provide users with useful information and action recommendations.

[0710] "Accumulating information" refers to the act of saving records of users' past emotions and behaviors and accumulating them in a database for later analysis and monitoring.

[0711] "Monitoring trends" is a process of analyzing long-term changes and patterns based on accumulated information and evaluating the user's condition.

[0712] "Ensuring anonymity" refers to the act of processing data in a way that prevents it from being identified by an individual, in order to protect the user's personal information.

[0713] "Visual devices" refer to equipment used to display information, and in this context, they are hardware used to provide content to users.

[0714] "Displaying content" refers to the act of providing information or exercises to users visually using visual devices.

[0715] The system for implementing this invention consists of a mechanism that applies natural language processing technology to provide individualized support for the mental health of users. The system mainly consists of two main components: a visual device worn by the user and a server that supports it in the backend.

[0716] The server converts the user's voice data into text via speech recognition software. This software includes speech recognition technologies such as Google Speech-to-Text. The converted text data is then analyzed using natural language processing libraries (e.g., SpaCy or NLTK) to understand the user's emotional state. Based on this analysis, the server generates personalized advice. The generated content is displayed on a visual device, such as smart glasses. This content could include meditation guidance or suggestions for relaxation exercises.

[0717] Visual devices play a role in presenting information to users. Through gentle visual content, users can engage in relaxation and meditation, which is expected to improve their mental state. This allows users to receive immediate mental health support in various aspects of their daily lives.

[0718] As a concrete example, if a user walking in a beautiful park says, "I've been feeling stressed lately," the system could analyze the results and play a video of natural sounds on smart glasses to promote relaxation. In this process, a generative AI model is used, and an example of a prompt message would be, "Tell us about the stress you've been feeling lately. Based on that, we'll give you the best advice."

[0719] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0720] Step 1:

[0721] When a user speaks, "I've been feeling stressed lately," the device captures the audio using its built-in microphone. The input is audio data. The device uses speech recognition software (e.g., Google Speech-to-Text) to convert this audio data into text data. The converted text is then sent to the server.

[0722] Step 2:

[0723] The server receives text data. The input is converted text. The server uses a natural language processing library (e.g., SpaCy or NLTK) to analyze the text data. Through this analysis, the server determines the user's emotional state. Specifically, it identifies keywords and phrases in the text that represent emotions and determines the level of stress.

[0724] Step 3:

[0725] The server generates appropriate advice and content based on the determined emotions. The input is the result of the emotion assessment, and the output is personalized advice for the user. By utilizing a generative AI model and using the prompt "Tell us about the stress you've been feeling lately. We will provide you with the best advice based on that," the generated advice is optimized.

[0726] Step 4:

[0727] The generated advice and content are transmitted to a visual device. The terminal, specifically smart glasses, receives this data and displays it to the user. The output is content that promotes meditation and relaxation. This allows the user to receive visual guidance to enhance relaxation.

[0728] Step 5:

[0729] While the user is experiencing the presented content, the server records the user's feedback and uses it to provide advice and generate content for future use. The input is the user's feedback, and this information is stored on the server and used to improve future analysis.

[0730] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0731] The present invention's system analyzes diverse input data from users to recognize emotions and provide mental health support. The system combines natural language processing technology and an emotion engine to perform more accurate emotion determination.

[0732] The device receives text, voice, or video data from the user and sends it to the server. For voice data, the device includes a process of converting the data to text using speech recognition technology. For video data, it also has the capability to extract facial expression data using facial recognition technology.

[0733] The server analyzes incoming data using both natural language processing (NLP) and an emotion engine. NLP analyzes key keywords and phrases, while the emotion engine analyzes facial expressions and voice tone to improve the accuracy of emotion recognition. It also uses time-series data to track and monitor changes in the user's emotions and analyze emotional trends from a long-term perspective.

[0734] After analysis, the server automatically generates optimal advice based on the determined emotions. This advice includes actionable and specific mental health strategies for the user. This advice is sent to the device and displayed to the user as a notification.

[0735] Furthermore, users can use the device's diary function to record changes in their emotions and behavior. The recorded data is then sent back to the server for aggregation. During this process, the data is always anonymized, ensuring that user privacy is fully protected.

[0736] As an example, if a user inputs "I've been experiencing increased work stress lately," the system combines language analysis and facial expression analysis to determine the level of stress and provides advice on specific relaxation techniques and stress management methods. This advice is continuously evaluated and adjusted as needed.

[0737] In this way, the system of the present invention realizes a form that supports and contributes to the improvement of the user's mental health.

[0738] The following describes the processing flow.

[0739] Step 1:

[0740] Users input information into the device via text, voice, or video. For voice input and video, the device uses speech recognition to convert the audio to text, and extracts facial expression data from video.

[0741] Step 2:

[0742] The device sends the converted text data and extracted facial expression data to the server. This transmission uses a secure communication protocol to protect the user's information.

[0743] Step 3:

[0744] The server processes the received text data using a natural language processing (NLP) engine, analyzing keywords and phrases to predict emotions. Simultaneously, it uses an emotion engine to analyze voice tone and facial expression data to determine emotions in more detail.

[0745] Step 4:

[0746] The server integrates text analysis results with voice and facial expression analysis results to determine the user's emotional state. This integration process improves the accuracy of emotion recognition.

[0747] Step 5:

[0748] The server retrieves or generates appropriate advice from the database based on the identified emotions. For example, if it determines that the user is experiencing high stress levels, it will suggest relaxation techniques.

[0749] Step 6:

[0750] The server sends the generated advice to the terminal, and the terminal displays the advice to the user in the form of a notification or message.

[0751] Step 7:

[0752] Users record feedback on advice provided through their devices and daily changes in their emotions using the diary function. This encourages self-reflection and is used for future analysis.

[0753] Step 8:

[0754] The server collects user feedback and diary data and monitors it over time. The data is anonymized to understand sentiment trends and use them for future analysis.

[0755] Step 9:

[0756] Based on long-term analysis, the server automatically provides additional advice and necessary interventions in response to changes in the user's emotions, thereby improving the user's mental health.

[0757] (Example 2)

[0758] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0759] In modern society, systems for providing individual mental health care are limited. In particular, accurately recognizing users' emotions and providing appropriate advice is crucial, but current technology makes it difficult to monitor and respond to changes in users' emotions over the long term. Therefore, a new system is needed to effectively support the management and improvement of individuals' mental states.

[0760] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0761] In this invention, the server includes means for analyzing user input information using natural language processing technology to determine emotions, means for generating suggestions appropriate to the user's situation based on the determined emotions, and means for accumulating information on the user's emotions and behavior and monitoring trends. This enables accurate recognition of the user's emotions, continuous monitoring, and appropriate mental health care tailored to individual situations.

[0762] "Natural language processing technology" is a technology that enables computers to understand and interpret human language, and is used to analyze user input.

[0763] "Means for determining emotions" refers to technology that identifies a user's emotions from input data and analyzes their state.

[0764] A "means for generating suggestions" is a mechanism for automatically creating appropriate advice and action guidelines based on the user's emotional state.

[0765] "Means of accumulating information and monitoring trends" refers to methods for collecting user sentiment and behavioral data and tracking changes in that data over a long period of time.

[0766] "Means of de-identifying information to protect anonymity" refers to processes that remove personal information from collected data in order to protect user privacy.

[0767] "Speech recognition means" refers to a technology that can convert speech data into text data and is used to analyze the user's voice input.

[0768] "Methods for analyzing facial expressions and vocal characteristics to improve the accuracy of emotion recognition" refer to technologies that analyze the user's visual and auditory characteristics to more accurately evaluate emotions.

[0769] This invention relates to a system that analyzes a user's emotions and provides mental health support. The system mainly consists of terminals and servers, each playing a specific role.

[0770] The device receives input from the user as text, voice, or video data. For voice data, the device converts it into text data using speech recognition technology. Specifically, general-purpose speech recognition software can be used. For video data, facial recognition technology is used to extract facial expression data. For this purpose, image processing libraries can be used, for example.

[0771] The server receives data transmitted from the terminal and analyzes it using natural language processing technology. During this process, natural language processing software is used to analyze key keywords and phrases from the text data. Furthermore, an emotion engine analyzes facial expressions and vocal characteristics to improve the accuracy of emotion recognition. Existing emotion analysis tools could be used for this purpose. The server also stores the received information and monitors changes in the user's emotions and behavior over a long period.

[0772] Based on the analysis results, the server generates personalized suggestions for the user. These suggestions include actionable mental health strategies and are provided to the user as notifications via the terminal. This allows the user to receive specific advice based on their current condition.

[0773] Furthermore, users can use a diary function through their device to record changes in their emotions and behavior. This data is anonymized and sent to a server to ensure privacy, and used for further analysis.

[0774] For example, if a user inputs "I've been experiencing increased work stress lately," the system uses language analysis and voice / facial expression analysis to determine the level of stress. Based on the results, the server generates advice on relaxation techniques and stress management methods. This advice is then notified to the terminal and provided to the user.

[0775] An example of a prompt for a generating AI model is: "The user has input that 'work-related stress has been increasing recently.' Based on this information, please suggest specific stress management methods."

[0776] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0777] Step 1:

[0778] The user provides emotional input data using the device. This input can be in text, audio, or video format. The device receives this data and begins processing it according to its format.

[0779] Step 2:

[0780] When the device receives audio data, it converts it into text using speech recognition technology. Specifically, it analyzes the sound data and outputs it as text data. For example, it recognizes the audio "I am tired" and outputs it as text. This converted text is then input to the next processing step.

[0781] Step 3:

[0782] In the case of video data, the terminal uses facial recognition technology to extract facial expression data. The process involves analyzing specific facial features from the video data, converting them into numerical data, and outputting it. This numerical data is sent to a server and used as input for emotion analysis.

[0783] Step 4:

[0784] The server analyzes the text data sent from the terminal using natural language processing technology. Specifically, it extracts keywords and phrases from the text and analyzes their meaning. For example, the keyword "tired" might be extracted and output as information for sentiment analysis.

[0785] Step 5:

[0786] The server analyzes facial and voice feature data using an emotion engine to recognize overall emotions. It quantifies the analysis results and outputs a specific emotional state. This output serves as the basis for deriving user-appropriate suggestions in subsequent processing steps.

[0787] Step 6:

[0788] The server generates personalized suggestions based on the analyzed emotional data. Using a generative AI model, it might suggest relaxation techniques, for example. These suggestions are sent to the user's device in text format.

[0789] Step 7:

[0790] Users check the suggestions they receive as notifications on their devices and take action as needed. They also use the diary function to record their emotions and actions on their devices. This record is sent to the server to be used for future analysis.

[0791] Step 8:

[0792] The device anonymizes the recorded data and sends it to the server. The server stores this data for future analysis and updates a database that monitors user sentiment. The stored data is continuously analyzed.

[0793] (Application Example 2)

[0794] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0795] In modern society, people's lives are extremely busy, and mental stress is constantly increasing. At the same time, opportunities to receive individualized mental health support are limited. Therefore, there is a need for technology that allows users to easily access mental health support in their daily lives while ensuring their privacy.

[0796] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0797] In this invention, the server includes means for analyzing user input data using natural language processing technology to determine emotions, means for generating advice appropriate to the user's situation based on the determined emotions, and means for interacting with the user and suggesting actions corresponding to those emotions in real time. This makes it possible for users to easily receive accurate mental health support in their daily lives.

[0798] "Natural language processing technology" refers to the technology that enables computers to understand, analyze, and generate human language.

[0799] "Means for determining emotions" refers to a method or device for identifying a user's emotional state based on data obtained from the user.

[0800] "Means for generating advice" refers to a method or apparatus that provides specific, actionable advice to a user based on their determined emotional state.

[0801] "Means of monitoring trends" refers to methods or devices for tracking user sentiment and behavioral data over the long term and analyzing changes over time.

[0802] "A means of suggesting actions that respond to emotions in real time" refers to a method or device for immediately responding to a user's current emotional state and recommending appropriate actions.

[0803] "Voice and vision sensors" are devices used to acquire the user's voice and video, and their role is to collect data.

[0804] "Means of anonymizing data" refers to methods or devices that make data unlinkable to a specific individual in order to conceal a user's personal information and protect their privacy.

[0805] To implement this invention, it is first necessary to incorporate an emotion recognition system into a household robot. The robot will be equipped with a high-resolution camera and microphone to collect audio and video in real time.

[0806] The server receives this data and uses natural language processing technology to analyze the user's voice data. Software such as Google's Dialogflow and Microsoft's Emotion API are used here. Using these, the robot performs voice recognition and emotion analysis to determine the user's emotional state.

[0807] Next, based on the determined emotional data, the server generates appropriate advice for the user. For example, if the user is stressed, it might suggest relaxation music or recommend short relaxation exercises.

[0808] Furthermore, the data collected by voice and visual sensors is anonymized so that personal information cannot be identified, enabling long-term data storage and analysis.

[0809] For example, if a user says, "I'm tired today," the robot can suggest, "How about taking a deep breath and relaxing?" Furthermore, it will offer a feature that tracks the user's daily emotional changes and allows for the identification of long-term mental health trends.

[0810] An example of a prompt would be, "When the user says 'I'm tired today' in a stressed voice, generate relaxation and encouragement advice." Through this prompt, the generating AI model will provide a more personalized response.

[0811] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0812] Step 1:

[0813] The device collects the user's voice and video data in real time using a high-resolution camera and microphone, and transmits this data to a server. The input is the user's voice and video, and the output is the transmission of data to the server.

[0814] Step 2:

[0815] The server converts the received audio data into text data using natural language processing technology. Speech recognition is performed using Google's Dialogflow. The input is audio data sent from the device, and the output is the text data obtained by converting the audio.

[0816] Step 3:

[0817] The server analyzes the converted text data and uses an emotion engine to determine the user's emotional state. It uses Microsoft's Emotion API for emotion identification. The input is text data, and the output is data indicating the user's emotional state.

[0818] Step 4:

[0819] The server generates advice based on the determined emotional state and sends it to the terminal. Prompts are used with the generating AI model to automatically generate mental support appropriate to the user. The input is the user's emotional state, and the output is appropriate advice.

[0820] Step 5:

[0821] The user reviews the advice received through the device and takes action as needed. Specific actions include suggestions such as playing music or taking deep breaths, and the device records the user's actions. The input is the advice sent from the server, and the output is the user's actions.

[0822] Step 6:

[0823] The device continuously records the user's actions and emotional changes, anonymizes the data, and sends it to the server. The input is user action data, and the output is stored on the server as anonymized data.

[0824] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0825] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0826] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0827] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0828] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0829] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0830] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0831] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0832] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0833] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0834] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0835] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0836] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0837] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0838] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0839] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0840] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0841] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0842] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0843] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0844] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0845] The following is further disclosed regarding the embodiments described above.

[0846] (Claim 1)

[0847] A method for analyzing user input data using natural language processing technology and determining emotions,

[0848] A means of generating advice appropriate to the user's situation based on the determined emotions,

[0849] A means of accumulating data on users' emotions and behavior and monitoring trends,

[0850] A means of automatically sending additional advice to the user as needed,

[0851] Means of anonymizing data to protect user privacy,

[0852] A system that includes this.

[0853] (Claim 2)

[0854] The system according to claim 1, which provides a diary function for users to record changes in their emotions and behavior, and allows users to check their own trends while protecting their privacy.

[0855] (Claim 3)

[0856] The system according to claim 1, characterized in that it uses speech recognition means to convert speech data into text data based on user input.

[0857] "Example 1"

[0858] (Claim 1)

[0859] A means of analyzing user input information using natural language processing technology to determine emotions,

[0860] A means of generating advice appropriate to the user's situation based on the determined emotions,

[0861] A means of accumulating information on users' emotions and behavior and monitoring its trends,

[0862] A means of automatically sending additional advice to users as needed,

[0863] Means of de-identifying information in order to maintain user anonymity,

[0864] A speech recognition means for converting audio data into text data,

[0865] A means of creating personalized advice using a generative AI model,

[0866] A system that includes this.

[0867] (Claim 2)

[0868] The system according to claim 1, which provides a diary function for users to record changes in their emotions and behavior, and allows users to check their own progress while maintaining anonymity.

[0869] (Claim 3)

[0870] The system according to claim 1, comprising means for improving the accuracy of advice by referring to past data.

[0871] "Application Example 1"

[0872] (Claim 1)

[0873] A means of analyzing user input information using natural language processing technology to determine emotions,

[0874] A means of generating advice appropriate to the user's situation based on the determined emotions,

[0875] A means of accumulating information on users' emotions and behavior and monitoring trends,

[0876] A means of automatically sending additional advice to users as needed,

[0877] Means to de-identify information in order to ensure user anonymity,

[0878] A method for analyzing everyday conversations and evaluating emotional states in real time,

[0879] A means of displaying content that promotes meditation or relaxation on a visual device based on emotional state,

[0880] A system that includes this.

[0881] (Claim 2)

[0882] The system according to claim 1, which provides a function for users to record changes in their own emotions and behaviors, and enables users to check their own tendencies while ensuring anonymity.

[0883] (Claim 3)

[0884] The system according to claim 1, characterized by using speech recognition means to convert speech information into text information based on user input.

[0885] "Example 2 of combining an emotion engine"

[0886] (Claim 1)

[0887] A means of analyzing user input information using natural language processing technology to determine emotions,

[0888] A means for generating suggestions appropriate to the user's situation based on the determined emotions,

[0889] A means of accumulating information on users' emotions and behavior and monitoring trends,

[0890] A means of automatically sending additional suggestions to the user as needed,

[0891] Means of de-identifying information to protect user anonymity,

[0892] A method for analyzing facial expressions and vocal characteristics to improve the accuracy of emotion recognition,

[0893] A system that includes this.

[0894] (Claim 2)

[0895] The system according to claim 1, which provides a diary function for users to record changes in their emotions and behavior, and allows users to check their own trends while protecting anonymity.

[0896] (Claim 3)

[0897] The system according to claim 1, characterized in that it uses speech recognition means to convert speech information into text information based on user input.

[0898] "Application example 2 when combining with an emotional engine"

[0899] (Claim 1)

[0900] A method for analyzing user input data using natural language processing technology and determining emotions,

[0901] A means of generating advice appropriate to the user's situation based on the determined emotions,

[0902] A means of accumulating data on users' emotions and behavior and monitoring trends,

[0903] A means of interacting with users and suggesting actions that respond to their emotions in real time,

[0904] A means of collecting user information using voice and visual sensors,

[0905] Means of anonymizing data to protect user privacy,

[0906] A system that includes this.

[0907] (Claim 2)

[0908] The system according to claim 1, which provides a diary function for users to record changes in their emotions and behavior, and allows users to check their own trends while protecting their privacy.

[0909] (Claim 3)

[0910] The system according to claim 1, characterized in that it uses speech recognition means to convert speech data into text data based on user input. [Explanation of symbols]

[0911] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of analyzing user input information using natural language processing technology to determine emotions, A means of generating advice appropriate to the user's situation based on the determined emotions, A means of accumulating information on users' emotions and behavior and monitoring trends, A means of automatically sending additional advice to users as needed, Means to de-identify information in order to ensure user anonymity, A method for analyzing everyday conversations and evaluating emotional states in real time, A means of displaying content that promotes meditation or relaxation on a visual device based on emotional state, A system that includes this.

2. The system according to claim 1, which provides a function for users to record changes in their own emotions and behaviors, and enables users to check their own tendencies while ensuring anonymity.

3. The system according to claim 1, characterized in that it uses speech recognition means that converts speech information into text information based on user input.

Images