system

Abstract

We provide the system. [Solution] A method for analyzing users' emotional states and behavioral patterns using natural language processing technology, A means for generating personalized advice based on the analyzed emotional state and behavioral patterns, Means for providing the generated advice to the user, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern society, the number of people suffering from stress, anxiety, and depressive symptoms continues to increase. Many people have problems related to mental health, but conventional counseling means may be difficult to use due to time constraints and privacy issues. In addition, lack of self-awareness and knowledge / skills of mental health care are also problems. For these reasons, there is a need for easy access to mental health support tailored to individual needs.

Means for Solving the Problems

[0005] This invention provides a system that analyzes a user's emotional state and behavioral patterns in real time using natural language processing technology and generates personalized advice based on the results. Furthermore, it collects user feedback and improves the accuracy of the advice based on that feedback. It also provides means to monitor the user's emotional changes and behavioral pattern trends and make appropriate decisions when early intervention is needed, thereby realizing continuous mental health support.

[0006] "Natural language processing technology" is a technology that enables computers to understand and process human language, and it involves analyzing input in the form of text or speech.

[0007] "Emotional state" refers to the user's psychological state and includes emotions such as joy, anger, sadness, and fear.

[0008] "Behavioral patterns" refer to the tendencies and patterns of actions that users repeatedly perform in their daily lives or in specific situations.

[0009] "Personalized advice" refers to suggestions and instructions that are customized to the specific circumstances and needs of each individual user.

[0010] "Feedback" refers to information that users convey to the system regarding their reactions and evaluations of the support they received.

[0011] A "trend" refers to the changes or tendencies observed in users' emotions and behavioral patterns over time.

[0012] "Intervention" refers to appropriate responses or measures taken before a particular problem or situation worsens. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] Shows an emotion map to which multiple emotions are mapped. [Figure 10] Shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, a labeled communication I / F (Interface) is an interface that includes a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] In the system of the present invention, the user accesses a mental health counseling service through a terminal. The terminal receives the user's input in text, audio, or video format. In the case of audio or video, the terminal uses speech recognition technology to convert it into text.

[0035] Subsequently, the device uses natural language processing (NLP) technology to analyze the received input. This analysis identifies the user's emotional state and behavioral patterns, generating data to send to the server. The server receives this data, continuously monitors emotional trends and behavioral patterns, and stores the results.

[0036] Based on the analysis results, the server generates personalized advice tailored to the user. For example, if a user enters "I can't sleep because of work stress," the server can suggest relaxation techniques for stress management. This advice is then provided to the user by the device.

[0037] When a user receives advice and provides feedback, the device sends this information to the server. The server uses the feedback to improve the accuracy of the advice.

[0038] Furthermore, the server meticulously monitors the user's emotional changes and behavioral patterns, and determines early intervention as needed. In this way, the system can provide the user with sustained mental health support and reduce stress and anxiety in their daily life.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] Users access mental health counseling services using their devices and input their feelings and situations in text, voice, or video format.

[0042] Step 2:

[0043] The device receives input from the user and applies speech recognition technology to convert speech to text as needed.

[0044] Step 3:

[0045] The device uses a natural language processing (NLP) engine to analyze text input. This analysis extracts the user's emotional state and behavioral patterns.

[0046] Step 4:

[0047] The device sends the analysis results to the server, and data based on emotional state and behavioral patterns is stored on the server.

[0048] Step 5:

[0049] The server uses stored data to monitor and continuously analyze user sentiment trends and behavioral patterns.

[0050] Step 6:

[0051] The server generates personalized advice tailored to the user based on monitoring results. This advice includes suggestions for stress management and relaxation techniques.

[0052] Step 7:

[0053] The server sends the generated advice to the terminal, and the terminal then presents the content to the user.

[0054] Step 8:

[0055] The user enters feedback on the advice provided. The device collects this feedback and sends it to the server.

[0056] Step 9:

[0057] The server analyzes user feedback and uses it to improve the accuracy of its advice. This feedback is then used for future monitoring and advice generation.

[0058] Step 10:

[0059] The server continuously monitors the user's emotions and behavioral patterns, preparing to intervene early as needed. This process allows the server to provide consistent mental health support to the user.

[0060] (Example 1)

[0061] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0062] In modern society, the impact of stress and anxiety on mental health is increasing, and there is a need for mental health support optimized for individual users to address this. However, conventional technologies have difficulty accurately understanding users' emotional states and behavioral patterns and providing individually optimized advice based on them, and there is a lack of mechanisms to continuously improve using feedback. As a result, there is a challenge in that the effectiveness of user mental health support is limited.

[0063] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0064] In this invention, the server includes means for analyzing the user's emotional state and behavioral patterns using natural language processing technology, means for using speech recognition technology to convert the user's voice or video input into text, and means for improving the data analysis and advice generation process using a generative AI model. This makes it possible to provide highly accurate and personalized advice based on the user's input, and to continuously improve it by incorporating feedback.

[0065] "Emotional state" refers to the psychological and emotional situation a user is experiencing at a given time, and includes specific emotions such as stress, happiness, and anxiety.

[0066] "Behavioral patterns" refer to characteristic tendencies in a user's behavior and reactions, and include their daily actions and decision-making patterns.

[0067] "Natural language processing technology" refers to technologies that enable computers to understand, interpret, and generate human language, and includes methods for analyzing time-series data.

[0068] "Means of generating advice" refers to methods of creating specific suggestions and instructions tailored to users' needs, based on their emotional state and behavioral patterns.

[0069] "Speech recognition technology" refers to the technology that converts speech into text, allowing speech input to be treated as text data.

[0070] A "generative AI model" refers to a model that uses artificial intelligence to automatically learn specific patterns and rules from data, and then makes decisions and predictions based on that learning.

[0071] "Personalized advice" refers to guidance and advice that are customized according to the individual user's characteristics and needs.

[0072] "Means of collecting feedback" refers to methods of recording and saving information and responses provided by users, and using that information to improve the system.

[0073] In this invention, a user can access a mental health counseling service through a terminal. First, the terminal receives input from the user. The input can be in text, voice, or video format. In the case of voice or video, the terminal uses speech recognition technology to convert the input to text. General-purpose speech recognition software is used for this conversion.

[0074] The terminal then uses natural language processing (NLP) techniques to analyze the user's text data. This process is performed to identify emotional states and behavioral patterns from the information entered by the user. General-purpose software capable of natural language processing is used for this analysis.

[0075] The data obtained through analysis is sent to a server. The server receives this data and continuously monitors emotional tendencies and behavioral patterns. The server uses a generative AI model to generate personalized advice based on trends.

[0076] For example, if a user enters "I've been experiencing a lot of stress at work lately and haven't been getting enough sleep," the server can generate relaxation advice such as "Try deep breathing exercises" or "Set aside time to relax at a set time." This advice is then delivered to the user via their device.

[0077] After a user implements the advice, they input feedback about its effectiveness into a terminal, which then sends this feedback to a server. The server analyzes this feedback to improve the accuracy of future advice. In this way, the system of the present invention aims to provide continuous mental health support tailored to each individual user.

[0078] An example of a prompt could be, "If a user has data indicating a long-term increase in stress, what advice would be appropriate?" This would allow the AI ​​model to provide more accurate advice.

[0079] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0080] Step 1:

[0081] The user accesses the mental health counseling service through a device and enters the necessary information in text, voice, or video format. The entered data relates to the user's psychological state and current concerns. The output of this step is the user's input data.

[0082] Step 2:

[0083] The terminal converts incoming audio or video data into text using speech recognition technology. This process utilizes speech recognition software to convert audio signals into character data. The input is audio or video data, and the output is text data.

[0084] Step 3:

[0085] The device analyzes text data using natural language processing (NLP) techniques. It identifies the user's emotional state and behavioral patterns from the input text data. Specifically, it uses an algorithm to identify emotions from text and generates emotion tags. The output of this step is the analyzed emotional state and behavioral patterns.

[0086] Step 4:

[0087] The terminal sends data, including analysis results, to the server. This data includes emotional states, behavioral patterns, and related metadata. The input is the analysis result data, and the output is the data sent to the server.

[0088] Step 5:

[0089] The server monitors user sentiment trends based on the received data. The server compares current data with historical data stored in the database to analyze changes in user sentiment and behavioral patterns. This analysis allows for understanding how the user's psychological state has changed over a long period. The output of this step is the sentiment trend analysis result.

[0090] Step 6:

[0091] The server uses a generative AI model to generate personalized advice suitable for the user. Based on the sentiment trend analysis results and the user's current psychological state, the generative AI model proposes the optimal advice. The input for this step is the sentiment trend analysis results, and the output is the generated advice.

[0092] Step 7:

[0093] The server sends the generated advice to the terminal, which then provides it to the user. The terminal can display the advice on the screen or explain it verbally. The input is the generated advice, and the output is the advice provided to the user.

[0094] Step 8:

[0095] The user tries out the provided advice and inputs the results and their impressions as feedback into the device. This input is the user's feedback data.

[0096] Step 9:

[0097] The device collects user feedback and sends it to the server. The server analyzes the feedback and uses it to improve future advice. This process allows the generative AI model to utilize the feedback to generate more accurate advice. The input is the feedback data, and the output is the data sent to the server.

[0098] (Application Example 1)

[0099] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0100] There is a need to provide efficient and sustainable support to reduce the stress and anxiety users experience in their daily lives and to improve their mental health. Furthermore, a lack of personalized approaches to address users' individual emotional states is a challenge. Additionally, there is a need to realize more accessible and practical mental health support by utilizing robots within the home.

[0101] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0102] In this invention, the server includes means for analyzing the user's emotional state and behavioral patterns using natural language processing technology, means for generating personalized advice based on the analyzed emotional state and behavioral patterns, means for providing the generated advice to the user through a robotic device, and means for guiding stress-reducing activities through dialogue with the user. This enables continuous and personalized mental health support using a robot within the home.

[0103] A "user" refers to an individual who utilizes mental health services through a system or robotic device.

[0104] "Emotional state" refers to information that indicates a user's psychological situation or mood, and is the subject of analysis using natural language processing technology.

[0105] "Behavioral patterns" refer to tendencies related to users' behavior and habits, and these are also analyzed using natural language processing technology.

[0106] "Natural language processing technology" refers to the technology that enables computers to understand and process human language.

[0107] "Advice" refers to individual suggestions or recommendations that the system generates based on the user's emotional state and behavioral patterns.

[0108] A "robot device" refers to a mechanical device placed in a home that provides advice through interaction with the user.

[0109] "Dialogue" refers to communication between a user and a robotic device, primarily conducted through voice or text.

[0110] "Stress reduction activities" refer to specific actions and suggestions aimed at alleviating users' stress and anxiety.

[0111] The system for implementing this invention begins with the user providing emotional states and behavioral patterns to a robotic device via voice or text. The robotic device converts this information into text using speech recognition software. Specifically, it uses the Google® Cloud Speech-to-Text API to convert voice input into text data.

[0112] The device analyzes the converted text data using natural language processing (NLP) technology, specifically an NLP model built with PyTorch, to understand the user's emotional state and behavioral patterns. This analysis identifies the types of stress and anxiety the user is currently experiencing and monitors their emotional trends.

[0113] The server receives the analyzed information and uses libraries such as Pandas and NumPy for data analysis. Based on the generated sentiment data, it generates personalized advice tailored to the user through continuous monitoring. The generated advice is delivered to the user through a robotic device. Through interaction with the user, it becomes possible to guide specific activities for stress reduction.

[0114] As a concrete example, consider a scenario where a user says, "I've been feeling exhausted lately from working remotely." The robotic device analyzes this emotional state and suggests and plays relaxing music. It also supports stress reduction by suggesting stretching exercises to do together.

[0115] For example, prompting the user with questions like, "How are you feeling today? Is there anything in particular that's stressing you out?" can help gain a more detailed understanding of their emotional state.

[0116] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0117] Step 1:

[0118] The user speaks to the robotic device, providing input regarding their emotional state and behavioral patterns. This includes the user's voice.

[0119] Step 2:

[0120] The robotic device uses the Google Cloud Speech-to-Text API to convert speech input into text data. The input for this process is the user's voice, and the output is text data. Specifically, it analyzes the speech signal and generates the corresponding text.

[0121] Step 3:

[0122] The device receives text data and performs natural language processing using a generative AI model built with PyTorch. It analyzes the user's emotions and extracts behavioral patterns. The input to this process is text data, and the output is the user's emotional state and behavioral patterns. Specifically, it performs operations to classify information using morphological analysis and sentiment analysis.

[0123] Step 4:

[0124] The server receives the analysis results and uses Pandas and NumPy to analyze data trends and patterns. The input to this process is emotional states and behavioral patterns, and the output is the analyzed emotional trends and suggested actions. Specifically, it tracks changes in emotions through statistical analysis of the data.

[0125] Step 5:

[0126] The server generates personalized advice tailored to the user based on the obtained sentiment trends. The input to this process is the sentiment trends, and the output is personalized advice. Specifically, it creates advice by combining prompt statements based on the analysis results.

[0127] Step 6:

[0128] The robotic device provides the user with generated advice and initiates a dialogue. The input to this process is personalized advice, and the output is feedback to the user. Specifically, it performs actions to explain and convey advice to the user in appropriate language.

[0129] Step 7:

[0130] The user provides feedback on the advice, and this information is sent to the server. The input to this process is the user's feedback, and the output is feedback data. Specifically, it records the user's input and passes it to the server.

[0131] Step 8:

[0132] The server strives to improve the accuracy of its advice based on the collected feedback. The input to this process is feedback data, and the output is improved advice logic. Specifically, it analyzes the feedback and introduces new patterns for generating advice.

[0133] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0134] The system of this invention achieves more accurate counseling by incorporating an emotion engine into the process of analyzing the user's emotional state and behavioral patterns. Users access the system through a terminal and begin using the system by making inputs in text, voice, or video format.

[0135] The device acquires input from the user, and converts voice and video input into text using speech recognition technology. This text data is sent to the emotion engine, which then performs analysis. Specifically, the emotion engine uses NLP (Neuro-Linguistic Programming) techniques to identify emotional indicators in the text, and also analyzes voice tone and context. This makes it possible to recognize detailed emotional states, including the intensity and type of emotion.

[0136] The device sends the results of the emotion engine's analysis to the server, which uses this data to continuously monitor the user's emotional trends and behavioral patterns. When the server generates personalized advice based on the stored data, it takes into account the detailed emotional data from the emotion engine. In this way, more accurate advice is provided for the problems the user is facing.

[0137] For example, if a user enters "I've been feeling irritable lately and can't concentrate on anything," the emotion engine identifies the emotion "irritability" and its intensity. Based on this information, the server generates stress management advice and suggests relaxation techniques.

[0138] When a user inputs feedback on the advice they received into their device, the device sends this feedback to the server, which uses it to improve the accuracy of the advice. The emotion engine also uses the feedback to further improve the accuracy of its emotion analysis. Ultimately, the server closely monitors the user's emotions and behavioral patterns, providing sustainable and flexible mental health support by intervening early as needed.

[0139] The following describes the processing flow.

[0140] Step 1:

[0141] Users connect to a mental health counseling system via their device and input their feelings and current situation through text, voice, or video.

[0142] Step 2:

[0143] The device receives voice and video input from the user and converts it into text format using speech recognition technology.

[0144] Step 3:

[0145] The device sends the generated text data to the emotion engine, which uses natural language processing techniques to analyze the emotional state. It identifies the type of emotion (e.g., joy, sadness, anger) and its intensity.

[0146] Step 4:

[0147] The emotion engine analyzes the emotion data and sends it to the server. The server stores the received data and monitors the user's emotion trends and behavioral patterns.

[0148] Step 5:

[0149] Based on accumulated data, the server generates personalized advice tailored to the user's current emotional state. Detailed emotional information from the emotion engine is used to generate the advice.

[0150] Step 6:

[0151] The server sends the generated advice to the terminal, and the terminal presents this advice to the user.

[0152] Step 7:

[0153] The system responds to the advice the user receives, inputting its effects and opinions as feedback into the device.

[0154] Step 8:

[0155] The device collects user feedback and sends it to the server. The server analyzes this feedback to improve the accuracy of the advice provided.

[0156] Step 9:

[0157] The server uses the feedback to improve the accuracy of the emotion engine's analysis and incorporates it into subsequent analyses.

[0158] Step 10:

[0159] The system provides continuous mental health support by having the server continuously monitor users' emotions and behavioral patterns in detail and intervening quickly as needed.

[0160] (Example 2)

[0161] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0162] In modern society, there is a need to accurately understand individuals' emotional states and behavioral patterns and provide appropriate mental health support based on that understanding. However, current systems face challenges in accurately analyzing users' diverse emotions and behavioral patterns and continuously providing personalized advice. Furthermore, it is difficult to fully utilize user feedback to improve the accuracy of advice. In addition, there is a lack of sufficient mechanisms to grasp emotional and behavioral trends early and provide appropriate intervention.

[0163] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0164] In this invention, the server includes means for acquiring information from the user and converting the content in audio or video format into text information; means for using the text information to utilize information processing technology to analyze the emotional state and behavioral patterns; and means for generating personalized advice based on the analyzed emotional state and behavioral patterns. This makes it possible to accurately analyze the emotional state based on information provided by the user in various formats, grasp behavioral trends, and provide personalized care.

[0165] "Information" refers to data obtained from users in the form of text, audio, or video.

[0166] "Textual information" refers to data expressed in characters, obtained by converting the content of audio or video formats.

[0167] "Information processing technology" refers to the technologies and methods used to analyze digital data and extract information for specific purposes.

[0168] "Emotional state" refers to elements that indicate an individual's internal mental state, such as the type and intensity of their psychological emotions.

[0169] "Behavioral patterns" refer to tendencies and recurring behavioral patterns related to user actions.

[0170] "Personalized advice" refers to recommendations and suggestions that are tailored specifically to a user, based on their particular emotional state or behavioral patterns.

[0171] "Intervention" refers to appropriate responses and measures taken as needed, based on continuous monitoring of the user's emotions and behavioral patterns.

[0172] The system of this invention analyzes the user's emotional state and behavioral patterns and provides personalized advice. The system starts operating when the user accesses the system through a terminal and inputs information in text, voice, or video format. The terminal converts the voice or video information from the user into text information using speech recognition technology. As for specific software, for example, voice service technology can be used for speech recognition.

[0173] The converted text information is sent from the device to the emotion engine. The emotion engine uses natural language processing technology to analyze the text information. This makes it possible to determine detailed emotional states and their intensity, taking into account not only the content of the text but also the tone of voice and context. Emotion analysis can be achieved by utilizing natural language processing technology, such as text analysis services.

[0174] The analyzed emotional data is sent from the device to the server. The server continuously monitors the user's emotional trends and behavioral patterns based on the received data. The server uses this data to generate personalized advice based on detailed analysis. Advice based on an understanding of the emotional state includes suggestions for relaxation methods and stress management techniques.

[0175] For example, when a user enters "I've been feeling irritable lately and can't concentrate on anything," the emotion engine detects the emotion "irritability" and its intensity, and the server uses this information to generate stress management advice, such as suggestions for meditation or relaxation music.

[0176] Furthermore, when a user enters feedback on this advice into their device, the feedback is sent to the server. The server uses the collected feedback to improve the quality of the advice, and the emotion engine further enhances its analytical accuracy. The server also closely monitors the user's emotions and behavioral patterns, and is equipped to intervene early as needed. This enables the system to provide users with sustainable and flexible mental health support.

[0177] Examples of prompts for a generative AI model:

[0178] "If a user were to record their recent emotional state and seek stress management advice, please describe in detail how they would input the information and what kind of advice they would receive."

[0179] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0180] Step 1:

[0181] Users access the system through a terminal and provide input in text, voice, or video format. If the input is in voice or video format, the terminal uses speech recognition technology to convert the input into text. The input data is then sent to the server in text format. Specifically, the system analyzes the voice and video data, converts it into text data, and prepares it as text information for sentiment analysis.

[0182] Step 2:

[0183] The emotion engine receives text data sent from the device and analyzes the emotional state using natural language processing techniques. Through this analysis, it identifies keywords indicating emotions and their intensity within the text. For example, the expression "I'm irritated" might extract the emotion category "anger," and its intensity is numerically evaluated. The output is a list of the emotion types and their intensity.

[0184] Step 3:

[0185] The device sends the results of the emotion engine's analysis to the server. The server uses this data to monitor the user's emotional trends and behavioral patterns. Specifically, it compares the current data with the user's past data and tracks emotional patterns over time to understand changes and trends. This forms the basis for personalized advice.

[0186] Step 4:

[0187] Based on the received emotional data, the server uses a generative AI model to generate personalized advice tailored to the user. For example, if a high stress level is detected, the server will suggest relaxation techniques. Data processing involves inputting the emotional analysis results into the model to generate output optimized for the user.

[0188] Step 5:

[0189] The user receives advice generated via their device. The user then inputs feedback on the advice into the device, such as "The advice was very helpful." This feedback is sent to the server for data improvement in the next phase.

[0190] Step 6:

[0191] The server uses feedback to improve the accuracy of the emotion engine and advice generation. This allows the emotion recognition algorithm to be continuously improved. Furthermore, the server actively incorporates user experience and uses it as data to optimize the content of advice for future interactions.

[0192] (Application Example 2)

[0193] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0194] In modern society, the importance of individual mental health is increasing, but the means to provide appropriate mental care are limited. Conventional technologies make it difficult to grasp the detailed emotional state of users, hindering the rapid delivery of personalized mental care. Therefore, there is a need for a system that accurately analyzes an individual's emotional state and provides appropriate advice based on that analysis.

[0195] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0196] In this invention, the server includes means for converting the user's emotional state into text using speech recognition technology and analyzing it using natural language processing technology; means for performing emotional analysis based on the analyzed emotional state and generating personalized information; and means for presenting the information generated by the consumer autonomous machine to the user and providing mental care. This makes it possible to accurately grasp the user's emotional state and provide individually optimized mental care.

[0197] "Converting user emotional states into text using speech recognition technology" refers to the process of receiving voice input and converting its content into textual information.

[0198] "Analysis using natural language processing technology" refers to techniques for analyzing text-based data and extracting meaning and intent from it.

[0199] "Performing sentiment analysis and generating personalized information" is a process of understanding the emotional nuances of users and creating data and suggestions that are optimal for each individual user.

[0200] "Providing users with information generated by consumer-grade autonomous machines to offer mental health support" refers to the process of providing users with advice and support generated by machines to support their mental well-being.

[0201] To implement this invention, the server first receives audio data from the user's terminal. The terminal uses the "speech_recognition" library to convert this audio data into text using high-performance speech recognition technology. This converted text is then sent to the server.

[0202] The server uses natural language processing (NLP) techniques to analyze the text and gain a detailed understanding of the user's emotional state. At this stage, it uses the emotion analysis model "emotion-english-distilroberta-base" from the "transformers" library to identify emotions.

[0203] Based on identified emotions, the server utilizes a generative AI model to generate mental care advice optimized for the user. This generated advice is then delivered to the user by a robot installed as a consumer-grade autonomous machine, providing mental support as needed.

[0204] For example, if a user says via voice input, "Lately, I've been feeling tired and lacking energy," the system analyzes the emotion of "fatigue," and the server suggests relaxation methods. Through this process, the system provides specific support tailored to the user's emotions.

[0205] An example of a prompt message for a generative AI model would be, "The user is feeling tired, so please generate specific examples of relaxation techniques."

[0206] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0207] Step 1:

[0208] The user provides voice input to the device. The device uses a high-performance microphone to capture the user's voice and acquires the input audio data. At this stage, the output is raw audio data.

[0209] Step 2:

[0210] On the device, the acquired audio data is converted into text using the "speech_recognition" library. Using audio data as input, text is generated using speech recognition technology. The output of this process is text data that reflects the user's words.

[0211] Step 3:

[0212] The terminal sends the generated text data to the server via the internet. The server receives the text data and prepares for the next processing step. In this step, the text data that arrives at the server becomes the output.

[0213] Step 4:

[0214] The server analyzes the received text data using natural language processing techniques based on the "transformers" library. It applies an emotion analysis model, including "emotion-english-distilroberta-base," to identify emotions from the input text. The output of this step is an analysis result indicating the user's emotional state.

[0215] Step 5:

[0216] The server uses a generative AI model based on the acquired emotional state to generate optimal advice for the user. The generative AI model takes the prompt "Please suggest advice based on the emotions the user is feeling" as input and outputs specific advice to provide to the user.

[0217] Step 6:

[0218] The robot presents the user with advice received from the server. It uses its display and speaker to communicate the advice and support the user's mental well-being. The output in this step is the mental health information provided to the user.

[0219] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0220] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0221] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0222] [Second Embodiment]

[0223] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0224] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0225] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0226] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0227] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0228] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0229] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0230] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0231] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0232] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0233] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0234] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0235] In the system of the present invention, the user accesses a mental health counseling service through a terminal. The terminal receives the user's input in text, audio, or video format. In the case of audio or video, the terminal uses speech recognition technology to convert it into text.

[0236] Subsequently, the device uses natural language processing (NLP) technology to analyze the received input. This analysis identifies the user's emotional state and behavioral patterns, generating data to send to the server. The server receives this data, continuously monitors emotional trends and behavioral patterns, and stores the results.

[0237] Based on the analysis results, the server generates personalized advice tailored to the user. For example, if a user enters "I can't sleep because of work stress," the server can suggest relaxation techniques for stress management. This advice is then provided to the user by the device.

[0238] When a user receives advice and provides feedback, the device sends this information to the server. The server uses the feedback to improve the accuracy of the advice.

[0239] Furthermore, the server meticulously monitors the user's emotional changes and behavioral patterns, and determines early intervention as needed. In this way, the system can provide the user with sustained mental health support and reduce stress and anxiety in their daily life.

[0240] The following describes the processing flow.

[0241] Step 1:

[0242] Users access mental health counseling services using their devices and input their feelings and situations in text, voice, or video format.

[0243] Step 2:

[0244] The device receives input from the user and applies speech recognition technology to convert speech to text as needed.

[0245] Step 3:

[0246] The device uses a natural language processing (NLP) engine to analyze text input. This analysis extracts the user's emotional state and behavioral patterns.

[0247] Step 4:

[0248] The device sends the analysis results to the server, and data based on emotional state and behavioral patterns is stored on the server.

[0249] Step 5:

[0250] The server uses stored data to monitor and continuously analyze user sentiment trends and behavioral patterns.

[0251] Step 6:

[0252] The server generates personalized advice tailored to the user based on monitoring results. This advice includes suggestions for stress management and relaxation techniques.

[0253] Step 7:

[0254] The server sends the generated advice to the terminal, and the terminal then presents the content to the user.

[0255] Step 8:

[0256] The user enters feedback on the advice provided. The device collects this feedback and sends it to the server.

[0257] Step 9:

[0258] The server analyzes user feedback and uses it to improve the accuracy of its advice. This feedback is then used for future monitoring and advice generation.

[0259] Step 10:

[0260] The server continuously monitors the user's emotions and behavioral patterns, preparing to intervene early as needed. This process allows the server to provide consistent mental health support to the user.

[0261] (Example 1)

[0262] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0263] In modern society, the impact of stress and anxiety on mental health is increasing, and there is a need for mental health support optimized for individual users to address this. However, conventional technologies have difficulty accurately understanding users' emotional states and behavioral patterns and providing individually optimized advice based on them, and there is a lack of mechanisms to continuously improve using feedback. As a result, there is a challenge in that the effectiveness of user mental health support is limited.

[0264] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0265] In this invention, the server includes means for analyzing the user's emotional state and behavioral patterns using natural language processing technology, means for using speech recognition technology to convert the user's voice or video input into text, and means for improving the data analysis and advice generation process using a generative AI model. This makes it possible to provide highly accurate and personalized advice based on the user's input, and to continuously improve it by incorporating feedback.

[0266] "Emotional state" refers to the psychological and emotional situation a user is experiencing at a given time, and includes specific emotions such as stress, happiness, and anxiety.

[0267] "Behavioral patterns" refer to characteristic tendencies in a user's behavior and reactions, and include their daily actions and decision-making patterns.

[0268] "Natural language processing technology" refers to technologies that enable computers to understand, interpret, and generate human language, and includes methods for analyzing time-series data.

[0269] "Means of generating advice" refers to methods of creating specific suggestions and instructions tailored to users' needs, based on their emotional state and behavioral patterns.

[0270] "Speech recognition technology" refers to the technology that converts speech into text, allowing speech input to be treated as text data.

[0271] A "generative AI model" refers to a model that uses artificial intelligence to automatically learn specific patterns and rules from data, and then makes decisions and predictions based on that learning.

[0272] "Personalized advice" refers to guidance and advice that are customized according to the individual user's characteristics and needs.

[0273] "Means of collecting feedback" refers to methods of recording and saving information and responses provided by users, and using that information to improve the system.

[0274] In this invention, a user can access a mental health counseling service through a terminal. First, the terminal receives input from the user. The input can be in text, voice, or video format. In the case of voice or video, the terminal uses speech recognition technology to convert the input to text. General-purpose speech recognition software is used for this conversion.

[0275] The terminal then uses natural language processing (NLP) techniques to analyze the user's text data. This process is performed to identify emotional states and behavioral patterns from the information entered by the user. General-purpose software capable of natural language processing is used for this analysis.

[0276] The data obtained through analysis is sent to a server. The server receives this data and continuously monitors emotional tendencies and behavioral patterns. The server uses a generative AI model to generate personalized advice based on trends.

[0277] For example, if a user enters "I've been experiencing a lot of stress at work lately and haven't been getting enough sleep," the server can generate relaxation advice such as "Try deep breathing exercises" or "Set aside time to relax at a set time." This advice is then delivered to the user via their device.

[0278] After a user implements the advice, they input feedback about its effectiveness into a terminal, which then sends this feedback to a server. The server analyzes this feedback to improve the accuracy of future advice. In this way, the system of the present invention aims to provide continuous mental health support tailored to each individual user.

[0279] An example of a prompt could be, "If a user has data indicating a long-term increase in stress, what advice would be appropriate?" This would allow the AI ​​model to provide more accurate advice.

[0280] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0281] Step 1:

[0282] The user accesses the mental health counseling service through a device and enters the necessary information in text, voice, or video format. The entered data relates to the user's psychological state and current concerns. The output of this step is the user's input data.

[0283] Step 2:

[0284] The terminal converts the input voice or video data into text using speech recognition technology. For this process, speech recognition software for converting voice signals into character data is used. The input is voice or video data, and the output is text data.

[0285] Step 3:

[0286] The terminal analyzes the text data using natural language processing (NLP) technology. From the input text data, the user's emotional state and behavior patterns are identified. Specifically, an algorithm for identifying emotions from the text is used to generate emotion tags. The output of this step is the analyzed emotional state and behavior patterns.

[0287] Step 4:

[0288] The terminal sends the data including the analysis results to the server. This data includes the emotional state, behavior patterns, and related metadata. The input is the analysis result data, and the output is the data to be sent to the server.

[0289] Step 5:

[0290] The server monitors the user's emotional trends based on the received data. The server compares the current data with the past data stored in the database and analyzes the changes in the user's emotions and trends in behavior patterns. Through this analysis, it is possible to grasp how the user's psychological state has changed over a long period. The output of this step is the emotional trend analysis result.

[0291] Step 6:

[0292] The server generates personalized advice suitable for the user using a generative AI model. Based on the emotional trend analysis result and the user's current psychological state, the generative AI model proposes the optimal advice. The input of this step is the emotional trend analysis result, and the output is the generated advice.

[0293] Step 7:

[0294] The server sends the generated advice to the terminal, which then provides it to the user. The terminal can display the advice on the screen or explain it verbally. The input is the generated advice, and the output is the advice provided to the user.

[0295] Step 8:

[0296] The user tries out the provided advice and inputs the results and their impressions as feedback into the device. This input is the user's feedback data.

[0297] Step 9:

[0298] The device collects user feedback and sends it to the server. The server analyzes the feedback and uses it to improve future advice. This process allows the generative AI model to utilize the feedback to generate more accurate advice. The input is the feedback data, and the output is the data sent to the server.

[0299] (Application Example 1)

[0300] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0301] There is a need to provide efficient and sustainable support to reduce the stress and anxiety users experience in their daily lives and to improve their mental health. Furthermore, a lack of personalized approaches to address users' individual emotional states is a challenge. Additionally, there is a need to realize more accessible and practical mental health support by utilizing robots within the home.

[0302] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0303] In this invention, the server includes means for analyzing the emotional state and behavior pattern from the user using natural language processing technology, means for generating personalized advice based on the analyzed emotional state and behavior pattern, means for providing the generated advice to the user through a robot device, and means for inducing activities for stress reduction through interaction with the user. Thereby, continuous and personalized mental health support using a robot within a household becomes possible.

[0304] "User" refers to an individual who uses mental health services through a system or a robot device.

[0305] "Emotional state" is information indicating the psychological situation or mood of the user, and is the object to be analyzed using natural language processing technology.

[0306] "Behavior pattern" refers to the tendency regarding the behavior and habits of the user, and this is also analyzed using natural language processing technology.

[0307] "Natural language processing technology" refers to the technology for a computer to understand and process human language.

[0308] "Advice" refers to individual suggestions or recommendations created by the system based on the emotional state and behavior pattern of the user.

[0309] "Robot device" means a mechanical device arranged within a household and providing advice through interaction with the user.

[0310] "Interaction" refers to the communication conducted between the user and the robot device, mainly through voice or text.

[0311] "Stress reduction activities" refer to specific actions and suggestions aimed at alleviating users' stress and anxiety.

[0312] The system for implementing this invention begins with the user providing emotional states and behavioral patterns to the robot device through voice or text. The robot device converts this information into text using speech recognition software. Specifically, it uses the Google Cloud Speech-to-Text API to convert voice input into text data.

[0313] The device analyzes the converted text data using natural language processing (NLP) technology, specifically an NLP model built with PyTorch, to understand the user's emotional state and behavioral patterns. This analysis identifies the types of stress and anxiety the user is currently experiencing and monitors their emotional trends.

[0314] The server receives the analyzed information and uses libraries such as Pandas and NumPy for data analysis. Based on the generated sentiment data, it generates personalized advice tailored to the user through continuous monitoring. The generated advice is delivered to the user through a robotic device. Through interaction with the user, it becomes possible to guide specific activities for stress reduction.

[0315] As a concrete example, consider a scenario where a user says, "I've been feeling exhausted lately from working remotely." The robotic device analyzes this emotional state and suggests and plays relaxing music. It also supports stress reduction by suggesting stretching exercises to do together.

[0316] For example, prompting the user with questions like, "How are you feeling today? Is there anything in particular that's stressing you out?" can help gain a more detailed understanding of their emotional state.

[0317] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0318] Step 1:

[0319] The user speaks to the robotic device, providing input regarding their emotional state and behavioral patterns. This includes the user's voice.

[0320] Step 2:

[0321] The robotic device uses the Google Cloud Speech-to-Text API to convert speech input into text data. The input for this process is the user's voice, and the output is text data. Specifically, it analyzes the speech signal and generates the corresponding text.

[0322] Step 3:

[0323] The device receives text data and performs natural language processing using a generative AI model built with PyTorch. It analyzes the user's emotions and extracts behavioral patterns. The input to this process is text data, and the output is the user's emotional state and behavioral patterns. Specifically, it performs operations to classify information using morphological analysis and sentiment analysis.

[0324] Step 4:

[0325] The server receives the analysis results and uses Pandas and NumPy to analyze data trends and patterns. The input to this process is emotional states and behavioral patterns, and the output is the analyzed emotional trends and suggested actions. Specifically, it tracks changes in emotions through statistical analysis of the data.

[0326] Step 5:

[0327] The server generates personalized advice tailored to the user based on the obtained sentiment trends. The input to this process is the sentiment trends, and the output is personalized advice. Specifically, it creates advice by combining prompt statements based on the analysis results.

[0328] Step 6:

[0329] The robotic device provides the user with generated advice and initiates a dialogue. The input to this process is personalized advice, and the output is feedback to the user. Specifically, it performs actions to explain and convey advice to the user in appropriate language.

[0330] Step 7:

[0331] The user provides feedback on the advice, and this information is sent to the server. The input to this process is the user's feedback, and the output is feedback data. Specifically, it records the user's input and passes it to the server.

[0332] Step 8:

[0333] The server strives to improve the accuracy of its advice based on the collected feedback. The input to this process is feedback data, and the output is improved advice logic. Specifically, it analyzes the feedback and introduces new patterns for generating advice.

[0334] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0335] The system of this invention achieves more accurate counseling by incorporating an emotion engine into the process of analyzing the user's emotional state and behavioral patterns. Users access the system through a terminal and begin using the system by making inputs in text, voice, or video format.

[0336] The device acquires input from the user, and converts voice and video input into text using speech recognition technology. This text data is sent to the emotion engine, which then performs analysis. Specifically, the emotion engine uses NLP (Neuro-Linguistic Programming) techniques to identify emotional indicators in the text, and also analyzes voice tone and context. This makes it possible to recognize detailed emotional states, including the intensity and type of emotion.

[0337] The device sends the results of the emotion engine's analysis to the server, which uses this data to continuously monitor the user's emotional trends and behavioral patterns. When the server generates personalized advice based on the stored data, it takes into account the detailed emotional data from the emotion engine. In this way, more accurate advice is provided for the problems the user is facing.

[0338] For example, if a user enters "I've been feeling irritable lately and can't concentrate on anything," the emotion engine identifies the emotion "irritability" and its intensity. Based on this information, the server generates stress management advice and suggests relaxation techniques.

[0339] When a user inputs feedback on the advice they received into their device, the device sends this feedback to the server, which uses it to improve the accuracy of the advice. The emotion engine also uses the feedback to further improve the accuracy of its emotion analysis. Ultimately, the server closely monitors the user's emotions and behavioral patterns, providing sustainable and flexible mental health support by intervening early as needed.

[0340] The following describes the processing flow.

[0341] Step 1:

[0342] Users connect to a mental health counseling system via their device and input their feelings and current situation through text, voice, or video.

[0343] Step 2:

[0344] The device receives voice and video input from the user and converts it into text format using speech recognition technology.

[0345] Step 3:

[0346] The device sends the generated text data to the emotion engine, which uses natural language processing techniques to analyze the emotional state. It identifies the type of emotion (e.g., joy, sadness, anger) and its intensity.

[0347] Step 4:

[0348] The emotion engine analyzes the emotion data and sends it to the server. The server stores the received data and monitors the user's emotion trends and behavioral patterns.

[0349] Step 5:

[0350] Based on accumulated data, the server generates personalized advice tailored to the user's current emotional state. Detailed emotional information from the emotion engine is used to generate the advice.

[0351] Step 6:

[0352] The server sends the generated advice to the terminal, and the terminal presents this advice to the user.

[0353] Step 7:

[0354] The system responds to the advice the user receives, inputting its effects and opinions as feedback into the device.

[0355] Step 8:

[0356] The device collects user feedback and sends it to the server. The server analyzes this feedback to improve the accuracy of the advice provided.

[0357] Step 9:

[0358] The server uses the feedback to improve the accuracy of the emotion engine's analysis and incorporates it into subsequent analyses.

[0359] Step 10:

[0360] The system provides continuous mental health support by having the server continuously monitor users' emotions and behavioral patterns in detail and intervening quickly as needed.

[0361] (Example 2)

[0362] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0363] In modern society, there is a need to accurately understand individuals' emotional states and behavioral patterns and provide appropriate mental health support based on that understanding. However, current systems face challenges in accurately analyzing users' diverse emotions and behavioral patterns and continuously providing personalized advice. Furthermore, it is difficult to fully utilize user feedback to improve the accuracy of advice. In addition, there is a lack of sufficient mechanisms to grasp emotional and behavioral trends early and provide appropriate intervention.

[0364] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0365] In this invention, the server includes means for acquiring information from the user and converting the content in audio or video format into text information; means for using the text information to utilize information processing technology to analyze the emotional state and behavioral patterns; and means for generating personalized advice based on the analyzed emotional state and behavioral patterns. This makes it possible to accurately analyze the emotional state based on information provided by the user in various formats, grasp behavioral trends, and provide personalized care.

[0366] "Information" refers to data obtained from users in the form of text, audio, or video.

[0367] "Textual information" refers to data expressed in characters, obtained by converting the content of audio or video formats.

[0368] "Information processing technology" refers to the technologies and methods used to analyze digital data and extract information for specific purposes.

[0369] "Emotional state" refers to elements that indicate an individual's internal mental state, such as the type and intensity of their psychological emotions.

[0370] "Behavioral patterns" refer to tendencies and recurring behavioral patterns related to user actions.

[0371] "Personalized advice" refers to recommendations and suggestions that are tailored specifically to a user, based on their particular emotional state or behavioral patterns.

[0372] "Intervention" refers to appropriate responses and measures taken as needed, based on continuous monitoring of the user's emotions and behavioral patterns.

[0373] The system of this invention analyzes the user's emotional state and behavioral patterns and provides personalized advice. The system starts operating when the user accesses the system through a terminal and inputs information in text, voice, or video format. The terminal converts the voice or video information from the user into text information using speech recognition technology. As for specific software, for example, voice service technology can be used for speech recognition.

[0374] The converted text information is sent from the device to the emotion engine. The emotion engine uses natural language processing technology to analyze the text information. This makes it possible to determine detailed emotional states and their intensity, taking into account not only the content of the text but also the tone of voice and context. Emotion analysis can be achieved by utilizing natural language processing technology, such as text analysis services.

[0375] The analyzed emotional data is sent from the device to the server. The server continuously monitors the user's emotional trends and behavioral patterns based on the received data. The server uses this data to generate personalized advice based on detailed analysis. Advice based on an understanding of the emotional state includes suggestions for relaxation methods and stress management techniques.

[0376] For example, when a user enters "I've been feeling irritable lately and can't concentrate on anything," the emotion engine detects the emotion "irritability" and its intensity, and the server uses this information to generate stress management advice, such as suggestions for meditation or relaxation music.

[0377] Furthermore, when a user enters feedback on this advice into their device, the feedback is sent to the server. The server uses the collected feedback to improve the quality of the advice, and the emotion engine further enhances its analytical accuracy. The server also closely monitors the user's emotions and behavioral patterns, and is equipped to intervene early as needed. This enables the system to provide users with sustainable and flexible mental health support.

[0378] Examples of prompts for a generative AI model:

[0379] "If a user were to record their recent emotional state and seek stress management advice, please describe in detail how they would input the information and what kind of advice they would receive."

[0380] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0381] Step 1:

[0382] Users access the system through a terminal and provide input in text, voice, or video format. If the input is in voice or video format, the terminal uses speech recognition technology to convert the input into text. The input data is then sent to the server in text format. Specifically, the system analyzes the voice and video data, converts it into text data, and prepares it as text information for sentiment analysis.

[0383] Step 2:

[0384] The emotion engine receives text data sent from the device and analyzes the emotional state using natural language processing techniques. Through this analysis, it identifies keywords indicating emotions and their intensity within the text. For example, the expression "I'm irritated" might extract the emotion category "anger," and its intensity is numerically evaluated. The output is a list of the emotion types and their intensity.

[0385] Step 3:

[0386] The device sends the results of the emotion engine's analysis to the server. The server uses this data to monitor the user's emotional trends and behavioral patterns. Specifically, it compares the current data with the user's past data and tracks emotional patterns over time to understand changes and trends. This forms the basis for personalized advice.

[0387] Step 4:

[0388] Based on the received emotional data, the server uses a generative AI model to generate personalized advice tailored to the user. For example, if a high stress level is detected, the server will suggest relaxation techniques. Data processing involves inputting the emotional analysis results into the model to generate output optimized for the user.

[0389] Step 5:

[0390] The user receives advice generated via their device. The user then inputs feedback on the advice into the device, such as "The advice was very helpful." This feedback is sent to the server for data improvement in the next phase.

[0391] Step 6:

[0392] The server uses feedback to improve the accuracy of the emotion engine and advice generation. This allows the emotion recognition algorithm to be continuously improved. Furthermore, the server actively incorporates user experience and uses it as data to optimize the content of advice for future interactions.

[0393] (Application Example 2)

[0394] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0395] In modern society, the importance of individual mental health is increasing, but the means to provide appropriate mental care are limited. Conventional technologies make it difficult to grasp the detailed emotional state of users, hindering the rapid delivery of personalized mental care. Therefore, there is a need for a system that accurately analyzes an individual's emotional state and provides appropriate advice based on that analysis.

[0396] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0397] In this invention, the server includes means for converting the user's emotional state into text using speech recognition technology and analyzing it using natural language processing technology; means for performing emotional analysis based on the analyzed emotional state and generating personalized information; and means for presenting the information generated by the consumer autonomous machine to the user and providing mental care. This makes it possible to accurately grasp the user's emotional state and provide individually optimized mental care.

[0398] "Converting user emotional states into text using speech recognition technology" refers to the process of receiving voice input and converting its content into textual information.

[0399] "Analysis using natural language processing technology" refers to techniques for analyzing text-based data and extracting meaning and intent from it.

[0400] "Performing sentiment analysis and generating personalized information" is a process of understanding the emotional nuances of users and creating data and suggestions that are optimal for each individual user.

[0401] "Providing users with information generated by consumer-grade autonomous machines to offer mental health support" refers to the process of providing users with advice and support generated by machines to support their mental well-being.

[0402] To implement this invention, the server first receives audio data from the user's terminal. The terminal uses the "speech_recognition" library to convert this audio data into text using high-performance speech recognition technology. This converted text is then sent to the server.

[0403] The server uses natural language processing (NLP) techniques to analyze the text and gain a detailed understanding of the user's emotional state. At this stage, it uses the emotion analysis model "emotion-english-distilroberta-base" from the "transformers" library to identify emotions.

[0404] Based on identified emotions, the server utilizes a generative AI model to generate mental care advice optimized for the user. This generated advice is then delivered to the user by a robot installed as a consumer-grade autonomous machine, providing mental support as needed.

[0405] For example, if a user says via voice input, "Lately, I've been feeling tired and lacking energy," the system analyzes the emotion of "fatigue," and the server suggests relaxation methods. Through this process, the system provides specific support tailored to the user's emotions.

[0406] An example of a prompt message for a generative AI model would be, "The user is feeling tired, so please generate specific examples of relaxation techniques."

[0407] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0408] Step 1:

[0409] The user provides voice input to the device. The device uses a high-performance microphone to capture the user's voice and acquires the input audio data. At this stage, the output is raw audio data.

[0410] Step 2:

[0411] On the device, the acquired audio data is converted into text using the "speech_recognition" library. Using audio data as input, text is generated using speech recognition technology. The output of this process is text data that reflects the user's words.

[0412] Step 3:

[0413] The terminal sends the generated text data to the server via the internet. The server receives the text data and prepares for the next processing step. In this step, the text data that arrives at the server becomes the output.

[0414] Step 4:

[0415] The server analyzes the received text data using natural language processing techniques based on the "transformers" library. It applies an emotion analysis model, including "emotion-english-distilroberta-base," to identify emotions from the input text. The output of this step is an analysis result indicating the user's emotional state.

[0416] Step 5:

[0417] The server uses a generative AI model based on the acquired emotional state to generate optimal advice for the user. The generative AI model takes the prompt "Please suggest advice based on the emotions the user is feeling" as input and outputs specific advice to provide to the user.

[0418] Step 6:

[0419] The robot presents the user with advice received from the server. It uses its display and speaker to communicate the advice and support the user's mental well-being. The output in this step is the mental health information provided to the user.

[0420] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0421] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0422] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0423] [Third Embodiment]

[0424] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0425] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0426] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0427] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0428] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0429] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0430] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0431] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0432] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0433] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0434] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0435] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0436] In the system of the present invention, the user accesses a mental health counseling service through a terminal. The terminal receives the user's input in text, audio, or video format. In the case of audio or video, the terminal uses speech recognition technology to convert it into text.

[0437] Subsequently, the device uses natural language processing (NLP) technology to analyze the received input. This analysis identifies the user's emotional state and behavioral patterns, generating data to send to the server. The server receives this data, continuously monitors emotional trends and behavioral patterns, and stores the results.

[0438] Based on the analysis results, the server generates personalized advice tailored to the user. For example, if a user enters "I can't sleep because of work stress," the server can suggest relaxation techniques for stress management. This advice is then provided to the user by the device.

[0439] When a user receives advice and provides feedback, the device sends this information to the server. The server uses the feedback to improve the accuracy of the advice.

[0440] Furthermore, the server meticulously monitors the user's emotional changes and behavioral patterns, and determines early intervention as needed. In this way, the system can provide the user with sustained mental health support and reduce stress and anxiety in their daily life.

[0441] The following describes the processing flow.

[0442] Step 1:

[0443] Users access mental health counseling services using their devices and input their feelings and situations in text, voice, or video format.

[0444] Step 2:

[0445] The device receives input from the user and applies speech recognition technology to convert speech to text as needed.

[0446] Step 3:

[0447] The device uses a natural language processing (NLP) engine to analyze text input. This analysis extracts the user's emotional state and behavioral patterns.

[0448] Step 4:

[0449] The device sends the analysis results to the server, and data based on emotional state and behavioral patterns is stored on the server.

[0450] Step 5:

[0451] The server uses stored data to monitor and continuously analyze user sentiment trends and behavioral patterns.

[0452] Step 6:

[0453] The server generates personalized advice tailored to the user based on monitoring results. This advice includes suggestions for stress management and relaxation techniques.

[0454] Step 7:

[0455] The server sends the generated advice to the terminal, and the terminal then presents the content to the user.

[0456] Step 8:

[0457] The user enters feedback on the advice provided. The device collects this feedback and sends it to the server.

[0458] Step 9:

[0459] The server analyzes user feedback and uses it to improve the accuracy of its advice. This feedback is then used for future monitoring and advice generation.

[0460] Step 10:

[0461] The server continuously monitors the user's emotions and behavioral patterns, preparing to intervene early as needed. This process allows the server to provide consistent mental health support to the user.

[0462] (Example 1)

[0463] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0464] In modern society, the impact of stress and anxiety on mental health is increasing, and there is a need for mental health support optimized for individual users to address this. However, conventional technologies have difficulty accurately understanding users' emotional states and behavioral patterns and providing individually optimized advice based on them, and there is a lack of mechanisms to continuously improve using feedback. As a result, there is a challenge in that the effectiveness of user mental health support is limited.

[0465] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0466] In this invention, the server includes means for analyzing the user's emotional state and behavioral patterns using natural language processing technology, means for using speech recognition technology to convert the user's voice or video input into text, and means for improving the data analysis and advice generation process using a generative AI model. This makes it possible to provide highly accurate and personalized advice based on the user's input, and to continuously improve it by incorporating feedback.

[0467] "Emotional state" refers to the psychological and emotional situation a user is experiencing at a given time, and includes specific emotions such as stress, happiness, and anxiety.

[0468] "Behavioral patterns" refer to characteristic tendencies in a user's behavior and reactions, and include their daily actions and decision-making patterns.

[0469] "Natural language processing technology" refers to technologies that enable computers to understand, interpret, and generate human language, and includes methods for analyzing time-series data.

[0470] "Means of generating advice" refers to methods of creating specific suggestions and instructions tailored to users' needs, based on their emotional state and behavioral patterns.

[0471] "Speech recognition technology" refers to the technology that converts speech into text, allowing speech input to be treated as text data.

[0472] A "generative AI model" refers to a model that uses artificial intelligence to automatically learn specific patterns and rules from data, and then makes decisions and predictions based on that learning.

[0473] "Personalized advice" refers to guidance and advice that are customized according to the individual user's characteristics and needs.

[0474] "Means of collecting feedback" refers to methods of recording and saving information and responses provided by users, and using that information to improve the system.

[0475] In this invention, a user can access a mental health counseling service through a terminal. First, the terminal receives input from the user. The input can be in text, voice, or video format. In the case of voice or video, the terminal uses speech recognition technology to convert the input to text. General-purpose speech recognition software is used for this conversion.

[0476] The terminal then uses natural language processing (NLP) techniques to analyze the user's text data. This process is performed to identify emotional states and behavioral patterns from the information entered by the user. General-purpose software capable of natural language processing is used for this analysis.

[0477] The data obtained through analysis is sent to a server. The server receives this data and continuously monitors emotional tendencies and behavioral patterns. The server uses a generative AI model to generate personalized advice based on trends.

[0478] For example, if a user enters "I've been experiencing a lot of stress at work lately and haven't been getting enough sleep," the server can generate relaxation advice such as "Try deep breathing exercises" or "Set aside time to relax at a set time." This advice is then delivered to the user via their device.

[0479] After a user implements the advice, they input feedback about its effectiveness into a terminal, which then sends this feedback to a server. The server analyzes this feedback to improve the accuracy of future advice. In this way, the system of the present invention aims to provide continuous mental health support tailored to each individual user.

[0480] An example of a prompt could be, "If a user has data indicating a long-term increase in stress, what advice would be appropriate?" This would allow the AI ​​model to provide more accurate advice.

[0481] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0482] Step 1:

[0483] The user accesses the mental health counseling service through a device and enters the necessary information in text, voice, or video format. The entered data relates to the user's psychological state and current concerns. The output of this step is the user's input data.

[0484] Step 2:

[0485] The terminal converts incoming audio or video data into text using speech recognition technology. This process utilizes speech recognition software to convert audio signals into character data. The input is audio or video data, and the output is text data.

[0486] Step 3:

[0487] The device analyzes text data using natural language processing (NLP) techniques. It identifies the user's emotional state and behavioral patterns from the input text data. Specifically, it uses an algorithm to identify emotions from text and generates emotion tags. The output of this step is the analyzed emotional state and behavioral patterns.

[0488] Step 4:

[0489] The terminal sends data, including analysis results, to the server. This data includes emotional states, behavioral patterns, and related metadata. The input is the analysis result data, and the output is the data sent to the server.

[0490] Step 5:

[0491] The server monitors user sentiment trends based on the received data. The server compares current data with historical data stored in the database to analyze changes in user sentiment and behavioral patterns. This analysis allows for understanding how the user's psychological state has changed over a long period. The output of this step is the sentiment trend analysis result.

[0492] Step 6:

[0493] The server uses a generative AI model to generate personalized advice suitable for the user. Based on the sentiment trend analysis results and the user's current psychological state, the generative AI model proposes the optimal advice. The input for this step is the sentiment trend analysis results, and the output is the generated advice.

[0494] Step 7:

[0495] The server sends the generated advice to the terminal, which then provides it to the user. The terminal can display the advice on the screen or explain it verbally. The input is the generated advice, and the output is the advice provided to the user.

[0496] Step 8:

[0497] The user tries out the provided advice and inputs the results and their impressions as feedback into the device. This input is the user's feedback data.

[0498] Step 9:

[0499] The device collects user feedback and sends it to the server. The server analyzes the feedback and uses it to improve future advice. This process allows the generative AI model to utilize the feedback to generate more accurate advice. The input is the feedback data, and the output is the data sent to the server.

[0500] (Application Example 1)

[0501] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0502] There is a need to provide efficient and sustainable support to reduce the stress and anxiety users experience in their daily lives and to improve their mental health. Furthermore, a lack of personalized approaches to address users' individual emotional states is a challenge. Additionally, there is a need to realize more accessible and practical mental health support by utilizing robots within the home.

[0503] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0504] In this invention, the server includes means for analyzing the user's emotional state and behavioral patterns using natural language processing technology, means for generating personalized advice based on the analyzed emotional state and behavioral patterns, means for providing the generated advice to the user through a robotic device, and means for guiding stress-reducing activities through dialogue with the user. This enables continuous and personalized mental health support using a robot within the home.

[0505] A "user" refers to an individual who utilizes mental health services through a system or robotic device.

[0506] "Emotional state" refers to information that indicates a user's psychological situation or mood, and is the subject of analysis using natural language processing technology.

[0507] "Behavioral patterns" refer to tendencies related to users' behavior and habits, and these are also analyzed using natural language processing technology.

[0508] "Natural language processing technology" refers to the technology that enables computers to understand and process human language.

[0509] "Advice" refers to individual suggestions or recommendations that the system generates based on the user's emotional state and behavioral patterns.

[0510] A "robot device" refers to a mechanical device placed in a home that provides advice through interaction with the user.

[0511] "Dialogue" refers to communication between a user and a robotic device, primarily conducted through voice or text.

[0512] "Stress reduction activities" refer to specific actions and suggestions aimed at alleviating users' stress and anxiety.

[0513] The system for implementing this invention begins with the user providing emotional states and behavioral patterns to the robot device through voice or text. The robot device converts this information into text using speech recognition software. Specifically, it uses the Google Cloud Speech-to-Text API to convert voice input into text data.

[0514] The device analyzes the converted text data using natural language processing (NLP) technology, specifically an NLP model built with PyTorch, to understand the user's emotional state and behavioral patterns. This analysis identifies the types of stress and anxiety the user is currently experiencing and monitors their emotional trends.

[0515] The server receives the analyzed information and uses libraries such as Pandas and NumPy for data analysis. Based on the generated sentiment data, it generates personalized advice tailored to the user through continuous monitoring. The generated advice is delivered to the user through a robotic device. Through interaction with the user, it becomes possible to guide specific activities for stress reduction.

[0516] As a concrete example, consider a scenario where a user says, "I've been feeling exhausted lately from working remotely." The robotic device analyzes this emotional state and suggests and plays relaxing music. It also supports stress reduction by suggesting stretching exercises to do together.

[0517] For example, prompting the user with questions like, "How are you feeling today? Is there anything in particular that's stressing you out?" can help gain a more detailed understanding of their emotional state.

[0518] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0519] Step 1:

[0520] The user speaks to the robotic device, providing input regarding their emotional state and behavioral patterns. This includes the user's voice.

[0521] Step 2:

[0522] The robotic device uses the Google Cloud Speech-to-Text API to convert speech input into text data. The input for this process is the user's voice, and the output is text data. Specifically, it analyzes the speech signal and generates the corresponding text.

[0523] Step 3:

[0524] The device receives text data and performs natural language processing using a generative AI model built with PyTorch. It analyzes the user's emotions and extracts behavioral patterns. The input to this process is text data, and the output is the user's emotional state and behavioral patterns. Specifically, it performs operations to classify information using morphological analysis and sentiment analysis.

[0525] Step 4:

[0526] The server receives the analysis results and uses Pandas and NumPy to analyze data trends and patterns. The input to this process is emotional states and behavioral patterns, and the output is the analyzed emotional trends and suggested actions. Specifically, it tracks changes in emotions through statistical analysis of the data.

[0527] Step 5:

[0528] The server generates personalized advice tailored to the user based on the obtained sentiment trends. The input to this process is the sentiment trends, and the output is personalized advice. Specifically, it creates advice by combining prompt statements based on the analysis results.

[0529] Step 6:

[0530] The robotic device provides the user with generated advice and initiates a dialogue. The input to this process is personalized advice, and the output is feedback to the user. Specifically, it performs actions to explain and convey advice to the user in appropriate language.

[0531] Step 7:

[0532] The user provides feedback on the advice, and this information is sent to the server. The input to this process is the user's feedback, and the output is feedback data. Specifically, it records the user's input and passes it to the server.

[0533] Step 8:

[0534] The server strives to improve the accuracy of its advice based on the collected feedback. The input to this process is feedback data, and the output is improved advice logic. Specifically, it analyzes the feedback and introduces new patterns for generating advice.

[0535] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0536] The system of this invention achieves more accurate counseling by incorporating an emotion engine into the process of analyzing the user's emotional state and behavioral patterns. Users access the system through a terminal and begin using the system by making inputs in text, voice, or video format.

[0537] The device acquires input from the user, and converts voice and video input into text using speech recognition technology. This text data is sent to the emotion engine, which then performs analysis. Specifically, the emotion engine uses NLP (Neuro-Linguistic Programming) techniques to identify emotional indicators in the text, and also analyzes voice tone and context. This makes it possible to recognize detailed emotional states, including the intensity and type of emotion.

[0538] The device sends the results of the emotion engine's analysis to the server, which uses this data to continuously monitor the user's emotional trends and behavioral patterns. When the server generates personalized advice based on the stored data, it takes into account the detailed emotional data from the emotion engine. In this way, more accurate advice is provided for the problems the user is facing.

[0539] For example, if a user enters "I've been feeling irritable lately and can't concentrate on anything," the emotion engine identifies the emotion "irritability" and its intensity. Based on this information, the server generates stress management advice and suggests relaxation techniques.

[0540] When a user inputs feedback on the advice they received into their device, the device sends this feedback to the server, which uses it to improve the accuracy of the advice. The emotion engine also uses the feedback to further improve the accuracy of its emotion analysis. Ultimately, the server closely monitors the user's emotions and behavioral patterns, providing sustainable and flexible mental health support by intervening early as needed.

[0541] The following describes the processing flow.

[0542] Step 1:

[0543] Users connect to a mental health counseling system via their device and input their feelings and current situation through text, voice, or video.

[0544] Step 2:

[0545] The device receives voice and video input from the user and converts it into text format using speech recognition technology.

[0546] Step 3:

[0547] The device sends the generated text data to the emotion engine, which uses natural language processing techniques to analyze the emotional state. It identifies the type of emotion (e.g., joy, sadness, anger) and its intensity.

[0548] Step 4:

[0549] The emotion engine analyzes the emotion data and sends it to the server. The server stores the received data and monitors the user's emotion trends and behavioral patterns.

[0550] Step 5:

[0551] Based on accumulated data, the server generates personalized advice tailored to the user's current emotional state. Detailed emotional information from the emotion engine is used to generate the advice.

[0552] Step 6:

[0553] The server sends the generated advice to the terminal, and the terminal presents this advice to the user.

[0554] Step 7:

[0555] The system responds to the advice the user receives, inputting its effects and opinions as feedback into the device.

[0556] Step 8:

[0557] The device collects user feedback and sends it to the server. The server analyzes this feedback to improve the accuracy of the advice provided.

[0558] Step 9:

[0559] The server uses the feedback to improve the accuracy of the emotion engine's analysis and incorporates it into subsequent analyses.

[0560] Step 10:

[0561] The system provides continuous mental health support by having the server continuously monitor users' emotions and behavioral patterns in detail and intervening quickly as needed.

[0562] (Example 2)

[0563] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0564] In modern society, there is a need to accurately understand individuals' emotional states and behavioral patterns and provide appropriate mental health support based on that understanding. However, current systems face challenges in accurately analyzing users' diverse emotions and behavioral patterns and continuously providing personalized advice. Furthermore, it is difficult to fully utilize user feedback to improve the accuracy of advice. In addition, there is a lack of sufficient mechanisms to grasp emotional and behavioral trends early and provide appropriate intervention.

[0565] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0566] In this invention, the server includes means for acquiring information from the user and converting the content in audio or video format into text information; means for using the text information to utilize information processing technology to analyze the emotional state and behavioral patterns; and means for generating personalized advice based on the analyzed emotional state and behavioral patterns. This makes it possible to accurately analyze the emotional state based on information provided by the user in various formats, grasp behavioral trends, and provide personalized care.

[0567] "Information" refers to data obtained from users in the form of text, audio, or video.

[0568] "Textual information" refers to data expressed in characters, obtained by converting the content of audio or video formats.

[0569] "Information processing technology" refers to the technologies and methods used to analyze digital data and extract information for specific purposes.

[0570] "Emotional state" refers to elements that indicate an individual's internal mental state, such as the type and intensity of their psychological emotions.

[0571] "Behavioral patterns" refer to tendencies and recurring behavioral patterns related to user actions.

[0572] "Personalized advice" refers to recommendations and suggestions that are tailored specifically to a user, based on their particular emotional state or behavioral patterns.

[0573] "Intervention" refers to appropriate responses and measures taken as needed, based on continuous monitoring of the user's emotions and behavioral patterns.

[0574] The system of this invention analyzes the user's emotional state and behavioral patterns and provides personalized advice. The system starts operating when the user accesses the system through a terminal and inputs information in text, voice, or video format. The terminal converts the voice or video information from the user into text information using speech recognition technology. As for specific software, for example, voice service technology can be used for speech recognition.

[0575] The converted text information is sent from the device to the emotion engine. The emotion engine uses natural language processing technology to analyze the text information. This makes it possible to determine detailed emotional states and their intensity, taking into account not only the content of the text but also the tone of voice and context. Emotion analysis can be achieved by utilizing natural language processing technology, such as text analysis services.

[0576] The analyzed emotional data is sent from the device to the server. The server continuously monitors the user's emotional trends and behavioral patterns based on the received data. The server uses this data to generate personalized advice based on detailed analysis. Advice based on an understanding of the emotional state includes suggestions for relaxation methods and stress management techniques.

[0577] For example, when a user enters "I've been feeling irritable lately and can't concentrate on anything," the emotion engine detects the emotion "irritability" and its intensity, and the server uses this information to generate stress management advice, such as suggestions for meditation or relaxation music.

[0578] Furthermore, when a user enters feedback on this advice into their device, the feedback is sent to the server. The server uses the collected feedback to improve the quality of the advice, and the emotion engine further enhances its analytical accuracy. The server also closely monitors the user's emotions and behavioral patterns, and is equipped to intervene early as needed. This enables the system to provide users with sustainable and flexible mental health support.

[0579] Examples of prompts for a generative AI model:

[0580] "If a user were to record their recent emotional state and seek stress management advice, please describe in detail how they would input the information and what kind of advice they would receive."

[0581] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0582] Step 1:

[0583] Users access the system through a terminal and provide input in text, voice, or video format. If the input is in voice or video format, the terminal uses speech recognition technology to convert the input into text. The input data is then sent to the server in text format. Specifically, the system analyzes the voice and video data, converts it into text data, and prepares it as text information for sentiment analysis.

[0584] Step 2:

[0585] The emotion engine receives text data sent from the device and analyzes the emotional state using natural language processing techniques. Through this analysis, it identifies keywords indicating emotions and their intensity within the text. For example, the expression "I'm irritated" might extract the emotion category "anger," and its intensity is numerically evaluated. The output is a list of the emotion types and their intensity.

[0586] Step 3:

[0587] The device sends the results of the emotion engine's analysis to the server. The server uses this data to monitor the user's emotional trends and behavioral patterns. Specifically, it compares the current data with the user's past data and tracks emotional patterns over time to understand changes and trends. This forms the basis for personalized advice.

[0588] Step 4:

[0589] Based on the received emotional data, the server uses a generative AI model to generate personalized advice tailored to the user. For example, if a high stress level is detected, the server will suggest relaxation techniques. Data processing involves inputting the emotional analysis results into the model to generate output optimized for the user.

[0590] Step 5:

[0591] The user receives advice generated via their device. The user then inputs feedback on the advice into the device, such as "The advice was very helpful." This feedback is sent to the server for data improvement in the next phase.

[0592] Step 6:

[0593] The server uses feedback to improve the accuracy of the emotion engine and advice generation. This allows the emotion recognition algorithm to be continuously improved. Furthermore, the server actively incorporates user experience and uses it as data to optimize the content of advice for future interactions.

[0594] (Application Example 2)

[0595] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0596] In modern society, the importance of individual mental health is increasing, but the means to provide appropriate mental care are limited. Conventional technologies make it difficult to grasp the detailed emotional state of users, hindering the rapid delivery of personalized mental care. Therefore, there is a need for a system that accurately analyzes an individual's emotional state and provides appropriate advice based on that analysis.

[0597] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0598] In this invention, the server includes means for converting the user's emotional state into text using speech recognition technology and analyzing it using natural language processing technology; means for performing emotional analysis based on the analyzed emotional state and generating personalized information; and means for presenting the information generated by the consumer autonomous machine to the user and providing mental care. This makes it possible to accurately grasp the user's emotional state and provide individually optimized mental care.

[0599] "Converting user emotional states into text using speech recognition technology" refers to the process of receiving voice input and converting its content into textual information.

[0600] "Analysis using natural language processing technology" refers to techniques for analyzing text-based data and extracting meaning and intent from it.

[0601] "Performing sentiment analysis and generating personalized information" is a process of understanding the emotional nuances of users and creating data and suggestions that are optimal for each individual user.

[0602] "Providing users with information generated by consumer-grade autonomous machines to offer mental health support" refers to the process of providing users with advice and support generated by machines to support their mental well-being.

[0603] To implement this invention, the server first receives audio data from the user's terminal. The terminal uses the "speech_recognition" library to convert this audio data into text using high-performance speech recognition technology. This converted text is then sent to the server.

[0604] The server uses natural language processing (NLP) techniques to analyze the text and gain a detailed understanding of the user's emotional state. At this stage, it uses the emotion analysis model "emotion-english-distilroberta-base" from the "transformers" library to identify emotions.

[0605] Based on identified emotions, the server utilizes a generative AI model to generate mental care advice optimized for the user. This generated advice is then delivered to the user by a robot installed as a consumer-grade autonomous machine, providing mental support as needed.

[0606] For example, if a user says via voice input, "Lately, I've been feeling tired and lacking energy," the system analyzes the emotion of "fatigue," and the server suggests relaxation methods. Through this process, the system provides specific support tailored to the user's emotions.

[0607] An example of a prompt message for a generative AI model would be, "The user is feeling tired, so please generate specific examples of relaxation techniques."

[0608] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0609] Step 1:

[0610] The user provides voice input to the device. The device uses a high-performance microphone to capture the user's voice and acquires the input audio data. At this stage, the output is raw audio data.

[0611] Step 2:

[0612] On the device, the acquired audio data is converted into text using the "speech_recognition" library. Using audio data as input, text is generated using speech recognition technology. The output of this process is text data that reflects the user's words.

[0613] Step 3:

[0614] The terminal sends the generated text data to the server via the internet. The server receives the text data and prepares for the next processing step. In this step, the text data that arrives at the server becomes the output.

[0615] Step 4:

[0616] The server analyzes the received text data using natural language processing techniques based on the "transformers" library. It applies an emotion analysis model, including "emotion-english-distilroberta-base," to identify emotions from the input text. The output of this step is an analysis result indicating the user's emotional state.

[0617] Step 5:

[0618] The server uses a generative AI model based on the acquired emotional state to generate optimal advice for the user. The generative AI model takes the prompt "Please suggest advice based on the emotions the user is feeling" as input and outputs specific advice to provide to the user.

[0619] Step 6:

[0620] The robot presents the user with advice received from the server. It uses its display and speaker to communicate the advice and support the user's mental well-being. The output in this step is the mental health information provided to the user.

[0621] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0622] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0623] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0624] [Fourth Embodiment]

[0625] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0626] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0627] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0628] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0629] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0630] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0631] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0632] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0633] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0634] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0635] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0636] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0637] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0638] In the system of the present invention, the user accesses a mental health counseling service through a terminal. The terminal receives the user's input in text, audio, or video format. In the case of audio or video, the terminal uses speech recognition technology to convert it into text.

[0639] Subsequently, the device uses natural language processing (NLP) technology to analyze the received input. This analysis identifies the user's emotional state and behavioral patterns, generating data to send to the server. The server receives this data, continuously monitors emotional trends and behavioral patterns, and stores the results.

[0640] Based on the analysis results, the server generates personalized advice tailored to the user. For example, if a user enters "I can't sleep because of work stress," the server can suggest relaxation techniques for stress management. This advice is then provided to the user by the device.

[0641] When a user receives advice and provides feedback, the device sends this information to the server. The server uses the feedback to improve the accuracy of the advice.

[0642] Furthermore, the server meticulously monitors the user's emotional changes and behavioral patterns, and determines early intervention as needed. In this way, the system can provide the user with sustained mental health support and reduce stress and anxiety in their daily life.

[0643] The following describes the processing flow.

[0644] Step 1:

[0645] Users access mental health counseling services using their devices and input their feelings and situations in text, voice, or video format.

[0646] Step 2:

[0647] The device receives input from the user and applies speech recognition technology to convert speech to text as needed.

[0648] Step 3:

[0649] The device uses a natural language processing (NLP) engine to analyze text input. This analysis extracts the user's emotional state and behavioral patterns.

[0650] Step 4:

[0651] The device sends the analysis results to the server, and data based on emotional state and behavioral patterns is stored on the server.

[0652] Step 5:

[0653] The server uses stored data to monitor and continuously analyze user sentiment trends and behavioral patterns.

[0654] Step 6:

[0655] The server generates personalized advice tailored to the user based on monitoring results. This advice includes suggestions for stress management and relaxation techniques.

[0656] Step 7:

[0657] The server sends the generated advice to the terminal, and the terminal then presents the content to the user.

[0658] Step 8:

[0659] The user enters feedback on the advice provided. The device collects this feedback and sends it to the server.

[0660] Step 9:

[0661] The server analyzes user feedback and uses it to improve the accuracy of its advice. This feedback is then used for future monitoring and advice generation.

[0662] Step 10:

[0663] The server continuously monitors the user's emotions and behavioral patterns, preparing to intervene early as needed. This process allows the server to provide consistent mental health support to the user.

[0664] (Example 1)

[0665] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0666] In modern society, the impact of stress and anxiety on mental health is increasing, and there is a need for mental health support optimized for individual users to address this. However, conventional technologies have difficulty accurately understanding users' emotional states and behavioral patterns and providing individually optimized advice based on them, and there is a lack of mechanisms to continuously improve using feedback. As a result, there is a challenge in that the effectiveness of user mental health support is limited.

[0667] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0668] In this invention, the server includes means for analyzing the user's emotional state and behavioral patterns using natural language processing technology, means for using speech recognition technology to convert the user's voice or video input into text, and means for improving the data analysis and advice generation process using a generative AI model. This makes it possible to provide highly accurate and personalized advice based on the user's input, and to continuously improve it by incorporating feedback.

[0669] "Emotional state" refers to the psychological and emotional situation a user is experiencing at a given time, and includes specific emotions such as stress, happiness, and anxiety.

[0670] "Behavioral patterns" refer to characteristic tendencies in a user's behavior and reactions, and include their daily actions and decision-making patterns.

[0671] "Natural language processing technology" refers to technologies that enable computers to understand, interpret, and generate human language, and includes methods for analyzing time-series data.

[0672] "Means of generating advice" refers to methods of creating specific suggestions and instructions tailored to users' needs, based on their emotional state and behavioral patterns.

[0673] "Speech recognition technology" refers to the technology that converts speech into text, allowing speech input to be treated as text data.

[0674] A "generative AI model" refers to a model that uses artificial intelligence to automatically learn specific patterns and rules from data, and then makes decisions and predictions based on that learning.

[0675] "Personalized advice" refers to guidance and advice that are customized according to the individual user's characteristics and needs.

[0676] "Means of collecting feedback" refers to methods of recording and saving information and responses provided by users, and using that information to improve the system.

[0677] In this invention, a user can access a mental health counseling service through a terminal. First, the terminal receives input from the user. The input can be in text, voice, or video format. In the case of voice or video, the terminal uses speech recognition technology to convert the input to text. General-purpose speech recognition software is used for this conversion.

[0678] The terminal then uses natural language processing (NLP) techniques to analyze the user's text data. This process is performed to identify emotional states and behavioral patterns from the information entered by the user. General-purpose software capable of natural language processing is used for this analysis.

[0679] The data obtained through analysis is sent to a server. The server receives this data and continuously monitors emotional tendencies and behavioral patterns. The server uses a generative AI model to generate personalized advice based on trends.

[0680] For example, if a user enters "I've been experiencing a lot of stress at work lately and haven't been getting enough sleep," the server can generate relaxation advice such as "Try deep breathing exercises" or "Set aside time to relax at a set time." This advice is then delivered to the user via their device.

[0681] After a user implements the advice, they input feedback about its effectiveness into a terminal, which then sends this feedback to a server. The server analyzes this feedback to improve the accuracy of future advice. In this way, the system of the present invention aims to provide continuous mental health support tailored to each individual user.

[0682] An example of a prompt could be, "If a user has data indicating a long-term increase in stress, what advice would be appropriate?" This would allow the AI ​​model to provide more accurate advice.

[0683] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0684] Step 1:

[0685] The user accesses the mental health counseling service through a device and enters the necessary information in text, voice, or video format. The entered data relates to the user's psychological state and current concerns. The output of this step is the user's input data.

[0686] Step 2:

[0687] The terminal converts incoming audio or video data into text using speech recognition technology. This process utilizes speech recognition software to convert audio signals into character data. The input is audio or video data, and the output is text data.

[0688] Step 3:

[0689] The device analyzes text data using natural language processing (NLP) techniques. It identifies the user's emotional state and behavioral patterns from the input text data. Specifically, it uses an algorithm to identify emotions from text and generates emotion tags. The output of this step is the analyzed emotional state and behavioral patterns.

[0690] Step 4:

[0691] The terminal sends data, including analysis results, to the server. This data includes emotional states, behavioral patterns, and related metadata. The input is the analysis result data, and the output is the data sent to the server.

[0692] Step 5:

[0693] The server monitors user sentiment trends based on the received data. The server compares current data with historical data stored in the database to analyze changes in user sentiment and behavioral patterns. This analysis allows for understanding how the user's psychological state has changed over a long period. The output of this step is the sentiment trend analysis result.

[0694] Step 6:

[0695] The server uses a generative AI model to generate personalized advice suitable for the user. Based on the sentiment trend analysis results and the user's current psychological state, the generative AI model proposes the optimal advice. The input for this step is the sentiment trend analysis results, and the output is the generated advice.

[0696] Step 7:

[0697] The server sends the generated advice to the terminal, which then provides it to the user. The terminal can display the advice on the screen or explain it verbally. The input is the generated advice, and the output is the advice provided to the user.

[0698] Step 8:

[0699] The user tries out the provided advice and inputs the results and their impressions as feedback into the device. This input is the user's feedback data.

[0700] Step 9:

[0701] The device collects user feedback and sends it to the server. The server analyzes the feedback and uses it to improve future advice. This process allows the generative AI model to utilize the feedback to generate more accurate advice. The input is the feedback data, and the output is the data sent to the server.

[0702] (Application Example 1)

[0703] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0704] There is a need to provide efficient and sustainable support to reduce the stress and anxiety users experience in their daily lives and to improve their mental health. Furthermore, a lack of personalized approaches to address users' individual emotional states is a challenge. Additionally, there is a need to realize more accessible and practical mental health support by utilizing robots within the home.

[0705] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0706] In this invention, the server includes means for analyzing the user's emotional state and behavioral patterns using natural language processing technology, means for generating personalized advice based on the analyzed emotional state and behavioral patterns, means for providing the generated advice to the user through a robotic device, and means for guiding stress-reducing activities through dialogue with the user. This enables continuous and personalized mental health support using a robot within the home.

[0707] A "user" refers to an individual who utilizes mental health services through a system or robotic device.

[0708] "Emotional state" refers to information that indicates a user's psychological situation or mood, and is the subject of analysis using natural language processing technology.

[0709] "Behavioral patterns" refer to tendencies related to users' behavior and habits, and these are also analyzed using natural language processing technology.

[0710] "Natural language processing technology" refers to the technology that enables computers to understand and process human language.

[0711] "Advice" refers to individual suggestions or recommendations that the system generates based on the user's emotional state and behavioral patterns.

[0712] A "robot device" refers to a mechanical device placed in a home that provides advice through interaction with the user.

[0713] "Dialogue" refers to communication between a user and a robotic device, primarily conducted through voice or text.

[0714] "Stress reduction activities" refer to specific actions and suggestions aimed at alleviating users' stress and anxiety.

[0715] The system for implementing this invention begins with the user providing emotional states and behavioral patterns to the robot device through voice or text. The robot device converts this information into text using speech recognition software. Specifically, it uses the Google Cloud Speech-to-Text API to convert voice input into text data.

[0716] The device analyzes the converted text data using natural language processing (NLP) technology, specifically an NLP model built with PyTorch, to understand the user's emotional state and behavioral patterns. This analysis identifies the types of stress and anxiety the user is currently experiencing and monitors their emotional trends.

[0717] The server receives the analyzed information and uses libraries such as Pandas and NumPy for data analysis. Based on the generated sentiment data, it generates personalized advice tailored to the user through continuous monitoring. The generated advice is delivered to the user through a robotic device. Through interaction with the user, it becomes possible to guide specific activities for stress reduction.

[0718] As a concrete example, consider a scenario where a user says, "I've been feeling exhausted lately from working remotely." The robotic device analyzes this emotional state and suggests and plays relaxing music. It also supports stress reduction by suggesting stretching exercises to do together.

[0719] For example, prompting the user with questions like, "How are you feeling today? Is there anything in particular that's stressing you out?" can help gain a more detailed understanding of their emotional state.

[0720] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0721] Step 1:

[0722] The user speaks to the robotic device, providing input regarding their emotional state and behavioral patterns. This includes the user's voice.

[0723] Step 2:

[0724] The robotic device uses the Google Cloud Speech-to-Text API to convert speech input into text data. The input for this process is the user's voice, and the output is text data. Specifically, it analyzes the speech signal and generates the corresponding text.

[0725] Step 3:

[0726] The device receives text data and performs natural language processing using a generative AI model built with PyTorch. It analyzes the user's emotions and extracts behavioral patterns. The input to this process is text data, and the output is the user's emotional state and behavioral patterns. Specifically, it performs operations to classify information using morphological analysis and sentiment analysis.

[0727] Step 4:

[0728] The server receives the analysis results and uses Pandas and NumPy to analyze data trends and patterns. The input to this process is emotional states and behavioral patterns, and the output is the analyzed emotional trends and suggested actions. Specifically, it tracks changes in emotions through statistical analysis of the data.

[0729] Step 5:

[0730] The server generates personalized advice tailored to the user based on the obtained sentiment trends. The input to this process is the sentiment trends, and the output is personalized advice. Specifically, it creates advice by combining prompt statements based on the analysis results.

[0731] Step 6:

[0732] The robotic device provides the user with generated advice and initiates a dialogue. The input to this process is personalized advice, and the output is feedback to the user. Specifically, it performs actions to explain and convey advice to the user in appropriate language.

[0733] Step 7:

[0734] The user provides feedback on the advice, and this information is sent to the server. The input to this process is the user's feedback, and the output is feedback data. Specifically, it records the user's input and passes it to the server.

[0735] Step 8:

[0736] The server strives to improve the accuracy of its advice based on the collected feedback. The input to this process is feedback data, and the output is improved advice logic. Specifically, it analyzes the feedback and introduces new patterns for generating advice.

[0737] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0738] The system of this invention achieves more accurate counseling by incorporating an emotion engine into the process of analyzing the user's emotional state and behavioral patterns. Users access the system through a terminal and begin using the system by making inputs in text, voice, or video format.

[0739] The device acquires input from the user, and converts voice and video input into text using speech recognition technology. This text data is sent to the emotion engine, which then performs analysis. Specifically, the emotion engine uses NLP (Neuro-Linguistic Programming) techniques to identify emotional indicators in the text, and also analyzes voice tone and context. This makes it possible to recognize detailed emotional states, including the intensity and type of emotion.

[0740] The device sends the results of the emotion engine's analysis to the server, which uses this data to continuously monitor the user's emotional trends and behavioral patterns. When the server generates personalized advice based on the stored data, it takes into account the detailed emotional data from the emotion engine. In this way, more accurate advice is provided for the problems the user is facing.

[0741] For example, if a user enters "I've been feeling irritable lately and can't concentrate on anything," the emotion engine identifies the emotion "irritability" and its intensity. Based on this information, the server generates stress management advice and suggests relaxation techniques.

[0742] When a user inputs feedback on the advice they received into their device, the device sends this feedback to the server, which uses it to improve the accuracy of the advice. The emotion engine also uses the feedback to further improve the accuracy of its emotion analysis. Ultimately, the server closely monitors the user's emotions and behavioral patterns, providing sustainable and flexible mental health support by intervening early as needed.

[0743] The following describes the processing flow.

[0744] Step 1:

[0745] Users connect to a mental health counseling system via their device and input their feelings and current situation through text, voice, or video.

[0746] Step 2:

[0747] The device receives voice and video input from the user and converts it into text format using speech recognition technology.

[0748] Step 3:

[0749] The device sends the generated text data to the emotion engine, which uses natural language processing techniques to analyze the emotional state. It identifies the type of emotion (e.g., joy, sadness, anger) and its intensity.

[0750] Step 4:

[0751] The emotion engine analyzes the emotion data and sends it to the server. The server stores the received data and monitors the user's emotion trends and behavioral patterns.

[0752] Step 5:

[0753] Based on accumulated data, the server generates personalized advice tailored to the user's current emotional state. Detailed emotional information from the emotion engine is used to generate the advice.

[0754] Step 6:

[0755] The server sends the generated advice to the terminal, and the terminal presents this advice to the user.

[0756] Step 7:

[0757] The system responds to the advice the user receives, inputting its effects and opinions as feedback into the device.

[0758] Step 8:

[0759] The device collects user feedback and sends it to the server. The server analyzes this feedback to improve the accuracy of the advice provided.

[0760] Step 9:

[0761] The server uses the feedback to improve the accuracy of the emotion engine's analysis and incorporates it into subsequent analyses.

[0762] Step 10:

[0763] The system provides continuous mental health support by having the server continuously monitor users' emotions and behavioral patterns in detail and intervening quickly as needed.

[0764] (Example 2)

[0765] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0766] In modern society, there is a need to accurately understand individuals' emotional states and behavioral patterns and provide appropriate mental health support based on that understanding. However, current systems face challenges in accurately analyzing users' diverse emotions and behavioral patterns and continuously providing personalized advice. Furthermore, it is difficult to fully utilize user feedback to improve the accuracy of advice. In addition, there is a lack of sufficient mechanisms to grasp emotional and behavioral trends early and provide appropriate intervention.

[0767] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0768] In this invention, the server includes means for acquiring information from the user and converting the content in audio or video format into text information; means for using the text information to utilize information processing technology to analyze the emotional state and behavioral patterns; and means for generating personalized advice based on the analyzed emotional state and behavioral patterns. This makes it possible to accurately analyze the emotional state based on information provided by the user in various formats, grasp behavioral trends, and provide personalized care.

[0769] "Information" refers to data obtained from users in the form of text, audio, or video.

[0770] "Textual information" refers to data expressed in characters, obtained by converting the content of audio or video formats.

[0771] "Information processing technology" refers to the technologies and methods used to analyze digital data and extract information for specific purposes.

[0772] "Emotional state" refers to elements that indicate an individual's internal mental state, such as the type and intensity of their psychological emotions.

[0773] "Behavioral patterns" refer to tendencies and recurring behavioral patterns related to user actions.

[0774] "Personalized advice" refers to recommendations and suggestions that are tailored specifically to a user, based on their particular emotional state or behavioral patterns.

[0775] "Intervention" refers to appropriate responses and measures taken as needed, based on continuous monitoring of the user's emotions and behavioral patterns.

[0776] The system of this invention analyzes the user's emotional state and behavioral patterns and provides personalized advice. The system starts operating when the user accesses the system through a terminal and inputs information in text, voice, or video format. The terminal converts the voice or video information from the user into text information using speech recognition technology. As for specific software, for example, voice service technology can be used for speech recognition.

[0777] The converted text information is sent from the device to the emotion engine. The emotion engine uses natural language processing technology to analyze the text information. This makes it possible to determine detailed emotional states and their intensity, taking into account not only the content of the text but also the tone of voice and context. Emotion analysis can be achieved by utilizing natural language processing technology, such as text analysis services.

[0778] The analyzed emotional data is sent from the device to the server. The server continuously monitors the user's emotional trends and behavioral patterns based on the received data. The server uses this data to generate personalized advice based on detailed analysis. Advice based on an understanding of the emotional state includes suggestions for relaxation methods and stress management techniques.

[0779] For example, when a user enters "I've been feeling irritable lately and can't concentrate on anything," the emotion engine detects the emotion "irritability" and its intensity, and the server uses this information to generate stress management advice, such as suggestions for meditation or relaxation music.

[0780] Furthermore, when a user enters feedback on this advice into their device, the feedback is sent to the server. The server uses the collected feedback to improve the quality of the advice, and the emotion engine further enhances its analytical accuracy. The server also closely monitors the user's emotions and behavioral patterns, and is equipped to intervene early as needed. This enables the system to provide users with sustainable and flexible mental health support.

[0781] Examples of prompts for a generative AI model:

[0782] "If a user were to record their recent emotional state and seek stress management advice, please describe in detail how they would input the information and what kind of advice they would receive."

[0783] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0784] Step 1:

[0785] Users access the system through a terminal and provide input in text, voice, or video format. If the input is in voice or video format, the terminal uses speech recognition technology to convert the input into text. The input data is then sent to the server in text format. Specifically, the system analyzes the voice and video data, converts it into text data, and prepares it as text information for sentiment analysis.

[0786] Step 2:

[0787] The emotion engine receives text data sent from the device and analyzes the emotional state using natural language processing techniques. Through this analysis, it identifies keywords indicating emotions and their intensity within the text. For example, the expression "I'm irritated" might extract the emotion category "anger," and its intensity is numerically evaluated. The output is a list of the emotion types and their intensity.

[0788] Step 3:

[0789] The device sends the results of the emotion engine's analysis to the server. The server uses this data to monitor the user's emotional trends and behavioral patterns. Specifically, it compares the current data with the user's past data and tracks emotional patterns over time to understand changes and trends. This forms the basis for personalized advice.

[0790] Step 4:

[0791] Based on the received emotional data, the server uses a generative AI model to generate personalized advice tailored to the user. For example, if a high stress level is detected, the server will suggest relaxation techniques. Data processing involves inputting the emotional analysis results into the model to generate output optimized for the user.

[0792] Step 5:

[0793] The user receives advice generated via their device. The user then inputs feedback on the advice into the device, such as "The advice was very helpful." This feedback is sent to the server for data improvement in the next phase.

[0794] Step 6:

[0795] The server uses feedback to improve the accuracy of the emotion engine and advice generation. This allows the emotion recognition algorithm to be continuously improved. Furthermore, the server actively incorporates user experience and uses it as data to optimize the content of advice for future interactions.

[0796] (Application Example 2)

[0797] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0798] In modern society, the importance of individual mental health is increasing, but the means to provide appropriate mental care are limited. Conventional technologies make it difficult to grasp the detailed emotional state of users, hindering the rapid delivery of personalized mental care. Therefore, there is a need for a system that accurately analyzes an individual's emotional state and provides appropriate advice based on that analysis.

[0799] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0800] In this invention, the server includes means for converting the user's emotional state into text using speech recognition technology and analyzing it using natural language processing technology; means for performing emotional analysis based on the analyzed emotional state and generating personalized information; and means for presenting the information generated by the consumer autonomous machine to the user and providing mental care. This makes it possible to accurately grasp the user's emotional state and provide individually optimized mental care.

[0801] "Converting user emotional states into text using speech recognition technology" refers to the process of receiving voice input and converting its content into textual information.

[0802] "Analysis using natural language processing technology" refers to techniques for analyzing text-based data and extracting meaning and intent from it.

[0803] "Performing sentiment analysis and generating personalized information" is a process of understanding the emotional nuances of users and creating data and suggestions that are optimal for each individual user.

[0804] "Providing users with information generated by consumer-grade autonomous machines to offer mental health support" refers to the process of providing users with advice and support generated by machines to support their mental well-being.

[0805] To implement this invention, the server first receives audio data from the user's terminal. The terminal uses the "speech_recognition" library to convert this audio data into text using high-performance speech recognition technology. This converted text is then sent to the server.

[0806] The server uses natural language processing (NLP) techniques to analyze the text and gain a detailed understanding of the user's emotional state. At this stage, it uses the emotion analysis model "emotion-english-distilroberta-base" from the "transformers" library to identify emotions.

[0807] Based on identified emotions, the server utilizes a generative AI model to generate mental care advice optimized for the user. This generated advice is then delivered to the user by a robot installed as a consumer-grade autonomous machine, providing mental support as needed.

[0808] For example, if a user says via voice input, "Lately, I've been feeling tired and lacking energy," the system analyzes the emotion of "fatigue," and the server suggests relaxation methods. Through this process, the system provides specific support tailored to the user's emotions.

[0809] An example of a prompt message for a generative AI model would be, "The user is feeling tired, so please generate specific examples of relaxation techniques."

[0810] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0811] Step 1:

[0812] The user provides voice input to the device. The device uses a high-performance microphone to capture the user's voice and acquires the input audio data. At this stage, the output is raw audio data.

[0813] Step 2:

[0814] On the device, the acquired audio data is converted into text using the "speech_recognition" library. Using audio data as input, text is generated using speech recognition technology. The output of this process is text data that reflects the user's words.

[0815] Step 3:

[0816] The terminal sends the generated text data to the server via the internet. The server receives the text data and prepares for the next processing step. In this step, the text data that arrives at the server becomes the output.

[0817] Step 4:

[0818] The server analyzes the received text data using natural language processing techniques based on the "transformers" library. It applies an emotion analysis model, including "emotion-english-distilroberta-base," to identify emotions from the input text. The output of this step is an analysis result indicating the user's emotional state.

[0819] Step 5:

[0820] The server uses a generative AI model based on the acquired emotional state to generate optimal advice for the user. The generative AI model takes the prompt "Please suggest advice based on the emotions the user is feeling" as input and outputs specific advice to provide to the user.

[0821] Step 6:

[0822] The robot presents the user with advice received from the server. It uses its display and speaker to communicate the advice and support the user's mental well-being. The output in this step is the mental health information provided to the user.

[0823] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0824] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0825] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0826] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0827] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0828] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0829] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0830] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0831] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0832] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0833] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0834] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0835] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0836] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0837] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0838] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0839] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0840] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0841] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0842] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0843] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0844] The following is further disclosed regarding the embodiments described above.

[0845] (Claim 1)

[0846] A method for analyzing users' emotional states and behavioral patterns using natural language processing technology,

[0847] A means for generating personalized advice based on the analyzed emotional state and behavioral patterns,

[0848] Means for providing the generated advice to the user,

[0849] A system that includes this.

[0850] (Claim 2)

[0851] Collect user feedback on the generated advice,

[0852] The system according to claim 1, further comprising means for improving the accuracy of the personalized advice based on the feedback.

[0853] (Claim 3)

[0854] By monitoring changes in user emotions and trends in behavioral patterns,

[0855] The system according to claim 1, comprising means for determining the need for early intervention based on the trend.

[0856] "Example 1"

[0857] (Claim 1)

[0858] A method for analyzing users' emotional states and behavioral patterns using natural language processing technology,

[0859] A means for generating personalized advice based on the analyzed emotional state and behavioral patterns,

[0860] Means for providing the generated advice to the user,

[0861] Means for using speech recognition technology to convert user voice or video input into text,

[0862] A means to improve the data analysis and advice generation process using generative AI models,

[0863] A system that includes this.

[0864] (Claim 2)

[0865] Collect user feedback on the generated advice,

[0866] The system according to claim 1, further comprising means for improving the accuracy of the personalized advice based on the feedback.

[0867] (Claim 3)

[0868] By monitoring changes in user emotions and trends in behavioral patterns,

[0869] The system according to claim 1, comprising means for determining the need for early intervention based on the trend.

[0870] "Application Example 1"

[0871] (Claim 1)

[0872] A method for analyzing users' emotional states and behavioral patterns using natural language processing technology,

[0873] A means for generating personalized advice based on the analyzed emotional state and behavioral patterns,

[0874] Means for providing the generated advice to the user through a robotic device,

[0875] A means of guiding users to activities that reduce stress through dialogue,

[0876] A system that includes this.

[0877] (Claim 2)

[0878] Collect user feedback on the generated advice,

[0879] The system according to claim 1, further comprising means for improving the accuracy of the personalized advice based on the evaluation.

[0880] (Claim 3)

[0881] We monitor changes in users' emotions and behavioral patterns,

[0882] The system according to claim 1, comprising means for determining the need for early intervention based on the said trend.

[0883] "Example 2 of combining an emotion engine"

[0884] (Claim 1)

[0885] A means for obtaining information from a user and converting the content of audio or video format into text information,

[0886] A means for analyzing emotional states and behavioral patterns by utilizing information processing technology using the textual information,

[0887] A means for generating personalized advice based on analyzed emotional states and behavioral patterns,

[0888] Means for providing the generated advice to the user,

[0889] A means of continuously understanding users' emotional changes and behavioral patterns, and intervening early as needed based on that information,

[0890] A system that includes this.

[0891] (Claim 2)

[0892] We collect user feedback on the generated advice.

[0893] The system according to claim 1, comprising means for improving the accuracy of personalized advice based on the opinion.

[0894] (Claim 3)

[0895] By continuously monitoring changes in users' emotions and behavioral patterns,

[0896] The system according to claim 1, comprising means for determining necessary interventions based on information obtained from the monitoring.

[0897] "Application example 2 when combining with an emotional engine"

[0898] (Claim 1)

[0899] A method for converting the emotional state of a user into text using speech recognition technology and analyzing it using natural language processing technology,

[0900] A means for performing emotional analysis based on the analyzed emotional state and generating personalized information,

[0901] A means of providing mental care by presenting information generated by consumer-grade autonomous machines to users,

[0902] A system that includes this.

[0903] (Claim 2)

[0904] The system according to claim 1, further comprising means for collecting user responses to the generated information and improving the accuracy of the personalized information based on said responses.

[0905] (Claim 3)

[0906] The system according to claim 1, comprising means for continuously monitoring changes in the user's emotions and behavioral patterns, and determining the need for early intervention based on such changes. [Explanation of Symbols]

[0907] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A method for analyzing users' emotional states and behavioral patterns using natural language processing technology, A means for generating personalized advice based on the analyzed emotional state and behavioral patterns, Means for providing the generated advice to the user, A system that includes this.

2. Collect user feedback on the generated advice, The system according to claim 1, further comprising means for improving the accuracy of the personalized advice based on the feedback.

3. By monitoring changes in user emotions and trends in behavioral patterns, The system according to claim 1, comprising means for determining the need for early intervention based on the trend.

Images