System

Abstract

A system is provided.SOLUTION: A system, comprising: means for receiving a user input; means for performing a real-time sentiment analysis based on the received user input; means for selecting an intervention message based on a sentiment analysis result; means for providing the selected intervention message to a user; and means for recording a chat history.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Current mental health management faces challenges such as unequal access, stigma, privacy issues, lack of real-time feedback, and difficulty with early intervention. These challenges make it difficult to receive appropriate mental health care when needed. This invention aims to solve these challenges and provide preventative, affordable, and private mental health care. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including a means for receiving user input, a means for performing real-time sentiment analysis based on the received user input, a means for selecting an intervention message based on the sentiment analysis result, a means for providing the selected intervention message to the user, and a means for recording the chat history. Furthermore, by adding a means for providing access to an expert based on the sentiment analysis result and a means for tracking changes in the user's sentiment and providing intervention at an early stage, more attentive care can be achieved.

[0006] "User input" refers to information such as text or voice that a user inputs to the system via a terminal.

[0007] "Real-time sentiment analysis" is the process of analyzing user input immediately after it is received to instantly identify the current emotional state.

[0008] An "intervention message" is an advice or support message provided to the user based on the results of sentiment analysis.

[0009] A "chat history" is a record of interactions between a user and the system.

[0010] "Access to Experts" refers to the means and processes by which users can obtain expert support and advice when needed.

[0011] An "emotional change tracking means" is a method or technique for continuously monitoring a user's emotional state and identifying changes.

[0012] "Early intervention measures" are measures that provide appropriate support and advice based on the user's emotional state before the problem becomes serious. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The system of the present invention aims to provide preventative, affordable, and private mental health care for users. The program processing of this system will be explained in natural language below, along with specific examples.

[0035] System Overview

[0036] This system consists of three components: a server, a terminal, and a user. Users access the system using their terminal and input information about their mental health status. The server plays a central role in receiving this input data, analyzing it, and providing a response.

[0037] Receiving User Input

[0038] A user inputs a message into the system using a device (e.g., a smartphone or a PC). For example, the user inputs, "I'm feeling overwhelmed with work."

[0039] Sending input data

[0040] The terminal sends this input data over the network to the server, which receives the message from the user and prepares for analysis.

[0041] sentiment analysis

[0042] The server performs real-time sentiment analysis on the received user input using a sentiment analysis model to identify specific emotions (e.g., anxiety, joy, sadness, anger) from the user input.

[0043] Intervention message selection

[0044] The server selects an appropriate intervention message based on the results of the sentiment analysis, for example, if the user is identified as "anxious," it selects the intervention message "Try some relaxation exercises."

[0045] Chat history recording

[0046] The server records interactions with users as a chat history, allowing past conversation data to be accumulated and referenced later.

[0047] Sending a response message

[0048] The server transmits the selected intervention message again to the terminal via the network, and the terminal displays the received message to the user.

[0049] User Awareness

[0050] The user sees the intervention message displayed on the device and follows the advice provided: "Try some relaxation exercises" is the message they receive and they act on it.

[0051] Specific examples

[0052] For example, consider a case where a user types the message "I'm feeling overwhelmed with work." This message is sent to the server via the device. The server receives the message and performs real-time sentiment analysis, identifying the emotion "anxiety." The server then selects "Try some relaxation exercises" as an intervention message corresponding to "anxiety." This message is recorded in the chat history and sent to the device for display to the user. Finally, the user can confirm this message and actually try some relaxation exercises.

[0053] In this way, the system can provide appropriate support in real time according to the user's mental health condition.

[0054] The processing flow will be explained below.

[0055] Step 1:

[0056] A user uses a terminal to enter a message into the system. For example, the user enters, "I'm feeling overwhelmed with work."

[0057] Step 2:

[0058] The device sends the input message to the server via the network. The data is sent in JSON format.

[0059] Step 3:

[0060] The server retrieves the received user message for analysis, and stores it in an internal data structure.

[0061] Step 4:

[0062] The server loads the sentiment analysis model and performs real-time sentiment analysis based on the message. In this process, a text analysis algorithm is used to identify the user's emotional state. For example, the emotion "anxious" is identified.

[0063] Step 5:

[0064] The server uses the results of the emotion analysis to select an appropriate intervention message based on a predefined mapping of emotions and messages. For example, the message "Try some relaxation exercises" is selected as the message corresponding to "anxious."

[0065] Step 6:

[0066] The server records the current conversation as a chat history, including user input, sentiment analysis results, and selected intervening messages, and stores this data in a database.

[0067] Step 7:

[0068] The server sends the selected intervention message back to the terminal via the network. The message is again sent in JSON format.

[0069] Step 8:

[0070] The terminal displays the intervention message received from the server to the user, for example, a message saying "Try some relaxation exercises" is displayed on the user interface.

[0071] Step 9:

[0072] The user acknowledges the displayed intervention message and performs the suggested action (e.g., relaxation exercises).

[0073] This series of steps enables the system to provide users with real-time, emotion-based mental health care.

[0074] Example 1

[0075] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0076] As mental health care has become increasingly important in recent years, there is a demand for easily accessible mental health care systems. However, existing systems have issues such as the time it takes to analyze user input and provide corresponding messages, the often mechanical responses that do not adequately address individual emotional states, and the difficulty in utilizing past interaction history. Another problem is the lack of a means to detect emotional changes early and provide appropriate intervention.

[0077] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0078] In this invention, the server includes a means for transmitting user input to the server via an internet connection, a means for performing real-time emotion analysis based on the received user input, and a means for selecting an intervention message based on the emotion analysis results and recording it in a database. This makes it possible to provide a response to the user's input quickly and according to the individual emotional state. Furthermore, by tracking changes in the user's emotions based on the chat history and enabling early intervention in response to emotional changes, more effective mental health care can be achieved.

[0079] "User input" refers to a text message that a user enters into the system using a terminal.

[0080] An "Internet line" is a communication path for sending and receiving digital data, including Wi-Fi and mobile networks (4G, 5G, etc.).

[0081] "Server" refers to a central computer system that processes, analyzes, and responds to received data.

[0082] "Real-time sentiment analysis" refers to the process of instantly analyzing sentiment and extracting specific emotions based on received user input.

[0083] "Artificial intelligence model" refers to a machine learning algorithm or neural network used for data analysis or prediction, such as a natural language processing model.

[0084] A "database" refers to a system that systematically organizes and stores data so that it can be searched and used later.

[0085] An "intervention message" refers to a message containing advice or suggested actions provided to a user based on the results of sentiment analysis.

[0086] "Chat history" refers to a record of past interactions with a user, and refers to data saved in a format that can be referenced and analyzed later.

[0087] "Changes in emotion" refers to changes in the user's emotional state over time, and is generally evaluated based on the difference from the initial state.

[0088] "Intervention" refers to the action of providing appropriate mental health care or advice based on the results of emotion analysis.

[0089] The system of the present invention aims to provide preventative, affordable, and private mental health care for users. Specific embodiments of the system are described below.

[0090] This system is primarily composed of three components: a server, a terminal, and a user. Users access the system using a terminal (such as a smartphone or PC) and enter information about their own mental health status. The server plays a central role in receiving this input data, analyzing it, and providing a response.

[0091] When a user uses a device to input a message, such as "I'm feeling overwhelmed with work," this input data is sent from the device to a server via an internet connection. The internet connection used here includes communication paths such as Wi-Fi, 4G, and 5G.

[0092] The server decodes the received user input data to analyze it. To do this, it uses natural language processing libraries (e.g., NLTK or spaCy) or generative AI models (e.g., OpenAI's GPT-3). These tools are used to perform real-time sentiment analysis and extract specific emotions (e.g., anxiety, joy, sadness, anger) from the user's input.

[0093] Based on the extracted emotion, the server selects an appropriate intervention message, which is then recorded in a database such as MySQL or PostgreSQL. For example, if the emotion analysis identifies "anxiety," the system might select and record the intervention message "Try some relaxation exercises."

[0094] The server then structures the selected intervention message as an HTTP response and sends it to the terminal over the Internet. The terminal then decodes the received message and displays it on its user interface, providing appropriate advice to the user.

[0095] The user is expected to check the intervention message displayed on the device and follow the advice provided. Specifically, the message "Try some relaxation exercises" encourages the user to perform relaxation exercises.

[0096] The server also records interactions with the user as a chat history, which helps track changes in the user's emotions and is stored in a database for later reference and analysis.

[0097] As a specific example, the following prompt sentence can be input into a generative AI model to provide sentiment analysis and intervention messages:

[0098] User message: "I'm feeling overwhelmed with work."

[0099] Analyze the sentiment of the user message and provide a supportive intervention message.

[0100] Based on this prompt, the generative AI model analyzes the user's emotions and generates an appropriate intervention message.

[0101] In this way, the system can provide appropriate support in real time according to the user's mental health condition.

[0102] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0103] Step 1:

[0104] A user accesses the system using a terminal and enters a message in a text box or chat window. The entered message is text data such as "I'm feeling overwhelmed with work." This input is sent to the next step as is.

[0105] Step 2:

[0106] The terminal sends the entered text data to the server via the Internet. At this time, the data is encoded in JSON format and sent as an HTTP request. The input here is the text message entered by the user, and the output is the encoded HTTP request.

[0107] Step 3:

[0108] The server decodes the received HTTP request and extracts the user's input message. It performs a decoding process to analyze the received data. The input is the encoded HTTP request from the terminal, and the output is the user input in text format.

[0109] Step 4:

[0110] The server performs real-time sentiment analysis based on the extracted user input message. Specifically, it uses natural language processing libraries (e.g., NLTK or spaCy) and generative AI models (e.g., OpenAI's GPT-3). The input for sentiment analysis is the user's text message, and the output is the analyzed emotion label (e.g., anxiety, joy, sadness, anger).

[0111] Step 5:

[0112] The server selects an appropriate intervention message based on the results of the sentiment analysis. It selects the corresponding intervention message from among those pre-stored in a database (e.g., MySQL or PostgreSQL). The input for this task is the sentiment label, and the output is the corresponding intervention message (e.g., "Try some relaxation exercises").

[0113] Step 6:

[0114] The server records the selected intervention message as a chat history in a database. This allows data to be accumulated in a form that can be referenced later. The input here is the intervention message, and the output is the chat history recorded in the database.

[0115] Step 7:

[0116] The server re-encodes the selected intervention message and sends it to the terminal as an HTTP response. The input is the intervention message and the output is the encoded HTTP response.

[0117] Step 8:

[0118] The terminal decodes the received HTTP response and displays the intervention message in a user interface, where the input is the encoded HTTP response and the output is the intervention message to be displayed (e.g., "Try some relaxation exercises").

[0119] Step 9:

[0120] The user checks the intervention message displayed on the device and acts according to the advice provided. Specifically, the user performs relaxation exercises based on the displayed message. The input is the intervention message displayed on the device, and the output is the user's actual behavior.

[0121] In this way, the entire system works to support the user's mental health in real time.

[0122] (Application example 1)

[0123] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0124] There is a need for a system that can detect mental health issues experienced by passengers in autonomous vehicles in real time and provide appropriate support. However, existing systems have difficulty accurately analyzing passenger mental health status and are unable to provide specific intervention measures. In addition, there is a lack of a system in place that allows passengers to receive mental health care in a private space.

[0125] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0126] In this invention, the server includes means for receiving user input, means for performing real-time sentiment analysis based on the received user input, means for selecting an intervention message based on the sentiment analysis result, means for providing the selected intervention message to the user, means for recording a chat history, and means for conducting mental health checks on passengers of the autonomous vehicle while the vehicle is in operation, thereby enabling passengers to receive real-time mental health care while the vehicle is in operation.

[0127] "User input" refers to messages and emotional states that a user inputs through a terminal.

[0128] "Real-time emotion analysis" refers to the process of instantly analyzing the emotional state of a user's input and extracting specific emotions.

[0129] An "intervention message" refers to advice or instructions provided to a user based on the results of sentiment analysis.

[0130] "Chat history" refers to the data that records and saves all conversations between a user and the system.

[0131] An "autonomous vehicle" refers to a vehicle that has the ability to move autonomously without a driver performing any driving operations.

[0132] A "mental health check" refers to a series of processes for diagnosing and assessing a user's emotional and mental state.

[0133] The purpose of the system of the present invention is to provide real-time mental health care for passengers in autonomous vehicles. This system consists of three components: a server, a terminal, and a user. The program processing of this system is explained below in natural language.

[0134] Hardware and software used

[0135] Hardware: On-board computers in autonomous vehicles, passenger smartphones

[0136] Software: Python 3.8+, aiohttp (web framework), TextBlob (sentiment analysis library)

[0137] Specific processing of the system

[0138] Receiving User Input

[0139] The user inputs their emotional state into the system using a device (such as a car screen or a smartphone). For example, the user might input, "I'm tired from work today."

[0140] Sending input data

[0141] The terminal transmits this input data via the network to the in-vehicle server, which receives the message from the user and prepares for analysis.

[0142] sentiment analysis

[0143] The server performs real-time sentiment analysis on the received user input, using the TextBlob library to identify specific sentiments (e.g., positive, negative, neutral) from the user input.

[0144] Intervention message selection

[0145] The server selects an appropriate intervention message based on the results of the sentiment analysis, for example, if the sentiment is identified as "negative," it selects the intervention message "Try some relaxation exercises."

[0146] Chat history recording

[0147] The server records interactions with users as a chat history, allowing past conversation data to be accumulated and referenced later.

[0148] Sending a response message

[0149] The selected intervention message is sent again via the network to the terminal, which displays the received message to the user.

[0150] Specific examples

[0151] For example, consider a case where a passenger types a message such as "I'm tired from work today." This message is sent to the server through the terminal. The server receives the message and performs real-time sentiment analysis, identifying the sentiment as "negative." The server then selects "Try some relaxation exercises," records this as a message history, and sends it to the terminal for display to the user. The user can confirm this message and actually try some relaxation exercises.

[0152] Prompt Sentence Examples

[0153] "Passengers input their emotional state via their smartphone or on-board screen."

[0154] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0155] Step 1:

[0156] Receiving User Input

[0157] The user inputs their emotional state using a device (such as a car screen or a smartphone). For example, the user might input, "I'm tired from work today." This input data is recognized by the device in text format.

[0158] Step 2:

[0159] Sending input data

[0160] The terminal transmits user input data to the in-vehicle server via a network. The input data is text data indicating the user's emotional state and is transferred from the terminal to the server.

[0161] Step 3:

[0162] Performing sentiment analysis

[0163] The server analyzes the received user input data. Specifically, it analyzes the sentiment of the input text using the TextBlob library. For example, the text "Work today has left me exhausted" is identified as expressing a negative sentiment (negative polarity).

[0164] Step 4:

[0165] Intervention message generation

[0166] The server selects an appropriate intervention message based on the result of the sentiment analysis. For example, if the sentiment is identified as "negative," it generates an intervention message such as "Try some relaxation exercises." This message is selected from fixed messages based on the result of the sentiment analysis.

[0167] Step 5:

[0168] Chat history recording

[0169] The server records the user's interactions as a chat history. This is the process of storing the user's input data and the corresponding generated intervention messages in a database. For example, the exchange "I'm tired from work today" and "Try some exercises to relax" is recorded.

[0170] Step 6:

[0171] Sending a response message

[0172] The server transmits the generated intervention message to the terminal again via the network, for example, a message saying "Try some relaxing exercises" is sent to the terminal, which then displays it to the user.

[0173] Step 7:

[0174] User perception and behavior

[0175] The user checks the intervention message displayed on the terminal, for example, "Try some relaxation exercises," and takes action according to the advice.

[0176] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0177] The system of the present invention aims to provide preventative, affordable, and private mental health care for users. In particular, by adding an emotion engine that recognizes the user's emotions, more accurate emotion analysis and intervention can be achieved. The overall configuration and specific processing of this system are described below.

[0178] System Overview

[0179] This system consists of three components: a server, a terminal, and a user. Users access the system using their terminal and input information about their mental health status. The server plays a central role in receiving this input data, analyzing it, and providing a response. The emotion engine is a module that further enhances emotion analysis.

[0180] Receiving User Input

[0181] A user inputs a message into the system using a device (e.g., a smartphone or a PC). This input can include not only text input, but also voice input and input using facial recognition. For example, a user inputs, "I'm feeling overwhelmed with work."

[0182] Sending input data

[0183] The device sends this input data to the server via the network. The data is sent in JSON format. Audio data and image data are also sent in the same way.

[0184] Performing sentiment analysis

[0185] The server receives the received user input data and performs real-time emotion analysis using the emotion engine. The emotion engine analyzes the following elements:

[0186] Text analysis: Analyze the content of a user's messages to identify their emotional state.

[0187] Voice analysis: Analyzes the tone and pitch of a user's voice to identify their emotional state.

[0188] Facial Expression Analysis: Analyzes facial expressions from the user's facial recognition data to identify their emotional state.

[0189] For example, if a user enters the text "I'm feeling overwhelmed with work," and has a stressed facial expression, the emotion "anxious" is identified.

[0190] Intervention message selection

[0191] The server selects an appropriate intervention message based on the result of the sentiment analysis, which is chosen based on a predefined emotion-message mapping.

[0192] For example, the message selected to correspond to "anxious" is "Try some relaxation exercises."

[0193] Chat history recording

[0194] The server records the user's interactions as a chat history. The recorded data includes the user's input, the results of sentiment analysis, and selected intervention messages. This data is stored in a database.

[0195] Sending a response message

[0196] The server then sends the selected intervention message back to the terminal via the network, again in JSON format.

[0197] Displaying a response message

[0198] The terminal displays the intervention message received from the server to the user, for example, a message saying "Try some relaxation exercises" is displayed on the user interface.

[0199] User perception and behavior

[0200] The user checks the intervention message displayed on the terminal and performs the suggested action (e.g., relaxation exercises).

[0201] Specific examples

[0202] For example, consider a case where a user types the message "I'm feeling overwhelmed with work," speaking in a high-pitched voice and displaying facial expressions that indicate stress. This message and additional data are sent to the server via the device. The server performs emotion analysis using text, voice, and facial expression analysis modules, and identifies the emotion "anxious." The server then selects "Try some relaxation exercises" as an intervention message corresponding to "anxious" and records it in the chat history. The server then sends this intervention message to the device, which then displays it to the user. Finally, the user can confirm the message and actually try some relaxation exercises.

[0203] This system can respond to the user's diverse emotional states in real time and provide more personalized mental health care.

[0204] The processing flow will be explained below.

[0205] Step 1:

[0206] A user uses a terminal to enter a message into the system. For example, the user enters, "I'm feeling overwhelmed with work."

[0207] Step 2:

[0208] The device sends the input message along with voice and facial expression data to the server in JSON format or another suitable format.

[0209] Step 3:

[0210] The server takes the received user input data for analysis and stores it in an internal data structure.

[0211] Step 4:

[0212] The server loads the emotion engine and performs real-time emotion analysis using text, voice, and facial expression data. The emotion analysis is performed as follows:

[0213] The server performs text analysis to identify the emotion from the content of the input message.

[0214] The server performs voice analysis and identifies emotions from the tone and pitch of the user's voice.

[0215] The server performs facial expression analysis and identifies emotions from the user's facial recognition data.

[0216] Step 5:

[0217] The server aggregates the results of the emotion analysis and identifies an overall emotional state. For example, if the results of text, voice, and facial expression analysis all indicate "anxious," the server identifies the overall emotion as "anxious."

[0218] Step 6:

[0219] The server selects an appropriate intervention message based on the results of emotion analysis. For example, it selects "Try some relaxation exercises" as the message for "anxious" based on predefined emotion-message mapping.

[0220] Step 7:

[0221] The server records the current conversation as a chat history, including user input, sentiment analysis results, and selected intervening messages, and stores the recorded data in a database.

[0222] Step 8:

[0223] The server sends the selected intervention message back to the terminal over the network, again in JSON format or another suitable format.

[0224] Step 9:

[0225] The terminal displays the intervention message received from the server to the user, for example, a message saying "Try some relaxation exercises" is displayed on the user interface.

[0226] Step 10:

[0227] The user acknowledges the displayed intervention message and performs the suggested action (e.g., relaxation exercises).

[0228] This series of steps enables the system to more accurately grasp the user's mental health status through multifaceted emotion analysis and provide appropriate real-time mental health care.

[0229] Example 2

[0230] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0231] Mental health issues are serious in modern society, and many users are experiencing increasing stress and anxiety. However, systems that provide affordable and private mental health care are not yet widespread. Another challenge is the lack of real-time emotional analysis of users and appropriate interventions based on that analysis.

[0232] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0233] In this invention, the server includes means for receiving user input, means for transmitting the received user input to the server via a network, means for performing real-time emotion analysis based on the received user input, means for selecting an intervention message based on the emotion analysis result, means for providing the selected intervention message to the user, and means for recording the chat history, thereby enabling the user to understand their own emotional state in real time and receive appropriate mental health care.

[0234] A "means for receiving user input" is a device or software that provides an interface for a user to enter messages or data into the system.

[0235] The "means for transmitting received user input to a server over a network" is a device or software that communicates user input to a server over the Internet or other communications network.

[0236] "Means for performing real-time sentiment analysis based on received user input" refers to analytical equipment or software for analyzing received data and determining the emotional state of the user in real time.

[0237] The "means for selecting an intervention message based on the result of sentiment analysis" is a device or software that selects the most appropriate advice or message based on the result of sentiment analysis.

[0238] The "means for providing a selected intervention message to a user" refers to a device or software that displays or notifies a user of a selected intervention message.

[0239] The "means for recording chat history" is a database or storage device for storing interactions with users and analysis results.

[0240] A "means for providing access to a professional" is a device or software that provides contact and access for a user to obtain professional advice or counseling as needed.

[0241] The "means for tracking emotional changes and providing early intervention" is a device or software that monitors changes in the user's emotional state and takes prompt action if necessary.

[0242] MODE FOR CARRYING OUT THE INVENTION

[0243] The system of the present invention aims to provide preventative, affordable, and private mental health care for users. In particular, by adding an emotion engine that recognizes the user's emotions, more accurate emotion analysis and intervention can be achieved. The overall configuration and specific processing of this system are described below.

[0244] Receiving User Input

[0245] Users input messages into the system using devices such as smartphones or PCs. This input can be text input, voice input, or input using facial recognition. For example, a user might input "I'm feeling overwhelmed with work." This input is done through the interface on the device.

[0246] Sending input data

[0247] The device sends input data acquired from the user to the server in JSON format. Voice data and image data are also sent in the same way. Specifically, the API on the device generates a JSON-formatted data packet and sends it through the network interface.

[0248] Performing sentiment analysis

[0249] The server first stores the received user input data in a database, and then performs real-time emotion analysis using the emotion engine. The emotion engine consists of the following analysis modules:

[0250] Text Analysis Module: Uses a natural language processing (NLP) engine to analyze text data and identify emotional states.

[0251] Audio analysis module: Analyzes the tone and pitch of audio data.

[0252] Facial Expression Analysis Module: Analyzes facial expressions from image data by using machine learning algorithms to extract visual features and infer emotions.

[0253] For example, if a user enters the text "I'm feeling overwhelmed with work," and there is a high-pitched voice tone indicative of stress and facial recognition data, the emotion engine will identify the emotion "anxious."

[0254] Selection of intervention messages

[0255] The server selects an appropriate intervention message based on the emotion analysis results. This message is selected from a predefined emotion-message mapping table. Specifically, the message "Try some relaxation exercises" is selected as the message corresponding to the emotion "anxious."

[0256] Chat history recording

[0257] The server records user interactions as chat history in a database. The recorded data includes user input, sentiment analysis results, and selected intervention messages. Specifically, it performs appropriate escaping and executes INSERT queries in the database using techniques to prevent SQL injection.

[0258] Sending a response message

[0259] The server encodes the selected intervention message in JSON format and sends it to the terminal. Specifically, it sends the packet again using the network interface.

[0260] Displaying a response message

[0261] The terminal decodes the intervention message received from the server and displays it to the user. The user interface (UI) displays the message content. For example, the UI displays a message such as "Try some relaxation exercises."

[0262] User perception and behavior

[0263] The user checks the intervention message displayed on the device and actually performs the suggested action (e.g., relaxation exercises), thereby improving their mental health.

[0264] Specific examples

[0265] As a specific example, if a user inputs the message "I'm feeling overwhelmed with work," with a high-pitched voice tone and facial expression indicating stress, this message and additional data are sent via the device to the server. The server receives this message and sequentially invokes the text analysis module, voice analysis module, and facial expression analysis module to identify the emotion "anxious." The server then selects an intervention message, "Try some relaxation exercises," from the emotion-message mapping table and sends it to the device. The device receives this and displays it to the user. The user confirms this message, tries to do some relaxation exercises, and enters the results as feedback into the system, starting the next cycle.

[0266] Prompt Sentence Examples

[0267] Please explain the operation of your system to support users' mental health care, breaking it down into specific steps. Please explain the operation of each step in detail, paying particular attention to the specific processing involved in performing analysis and sending and receiving data.

[0268] In this way, the present invention makes it possible to provide personalized mental health care according to the user's emotional state.

[0269] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0270] Step 1:

[0271] A user inputs a message into the system using a device (e.g., a smartphone or a PC). This input can include not only text input, but also voice input and input using facial recognition. For example, a user inputs the text "I'm feeling overwhelmed with work." Voice input and facial recognition data are also processed simultaneously.

[0272] Input: User text, voice, and facial recognition data

[0273] Output: Getting input data via the terminal

[0274] Step 2:

[0275] The device converts the input data received from the user into JSON format. Specifically, it encodes text data, compresses audio data, and encodes image data. This operation is performed through the API on the device.

[0276] Input: User input data (text, voice, facial recognition data)

[0277] Output: JSON formatted data packet

[0278] Step 3:

[0279] The device sends the encoded JSON-formatted data packets through the network interface. Specifically, the device sends the data to the server using a network protocol, using a secure communication protocol such as HTTPS.

[0280] Input: JSON formatted data packet

[0281] Output: Send data to the server

[0282] Step 4:

[0283] The server decodes the received JSON-formatted user data and stores it in a database. Specifically, it executes an INSERT query using a database management system (DBMS) on the server and stores the user data in the database.

[0284] Input: User data in JSON format

[0285] Output: Data storage in database

[0286] Step 5:

[0287] The server retrieves the stored user data and performs analysis using the sentiment analysis engine. Specifically, the following analysis modules are called in sequence:

[0288] Text Analysis Module: Uses a natural language processing engine to identify emotional states from text data.

[0289] Voice Analysis Module: Analyzes the tone and pitch of voice data to identify emotional states.

[0290] Facial Expression Analysis Module: Analyzes facial expressions from image data to identify emotional states.

[0291] Input: User data retrieved from the database

[0292] Output: Emotion analysis results (emotional state)

[0293] Step 6:

[0294] The server selects an appropriate intervention message based on the results of emotion analysis. Specifically, it references a mapping table of emotions and messages to select the most appropriate message. For example, if the emotion analysis result is "anxious," the server selects the message "Try some relaxation exercises."

[0295] Input: Emotion analysis results (emotional state)

[0296] Output: Intervention message

[0297] Step 7:

[0298] The server encodes the selected intervention message in JSON format and sends it to the terminal. Specifically, the server generates packets using a network protocol and transmits the data to the terminal via secure communication such as HTTPS.

[0299] Input: Intervention message

[0300] Output: Intervention message packet in JSON format

[0301] Step 8:

[0302] The device decodes the JSON-formatted intervention message received from the server and displays it on the user interface (UI). Specifically, the device's decoding process and UI rendering engine work together to visually display the message. For example, a message such as "Try some relaxation exercises" is displayed on the UI.

[0303] Input: JSON formatted intervention message packet

[0304] Output: Intervention message displayed to the user

[0305] Step 9:

[0306] The user confirms the displayed intervention message and performs the suggested action (e.g., relaxation exercises). The user performs the action and inputs the results as feedback to the system, providing new data for the next analysis cycle.

[0307] Input: Intervention message displayed to the user

[0308] Output: User actions and feedback

[0309] (Application example 2)

[0310] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0311] In modern society, the number of individuals experiencing mental stress and anxiety is increasing, creating a demand for prompt and effective mental health care. However, many mental health care systems are expensive and lack privacy. It is necessary to solve these problems and provide a preventive mental health care system that is easy for users to use.

[0312] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving user input, means for performing real-time emotion analysis based on the received user input, means for selecting an intervention message based on the emotion analysis result, means for providing the selected intervention message to the user, means for recording the chat history, and means for recommending mental health care content based on the emotion analysis result. This makes it possible to provide appropriate intervention and mental health care content according to the user's emotional state.

[0313] "User input" refers to information provided by a system user through a terminal, and may be in the form of text, audio, image, or the like.

[0314] "Real-time" refers to processing that occurs immediately without delay, and refers to data analysis and response in real time.

[0315] "Sentiment analysis" refers to the process of analyzing user-input data to identify a user's emotional state, and includes text analysis, speech analysis, and facial recognition analysis.

[0316] An "intervention message" refers to a message of recommended actions or advice provided to a user based on the results of sentiment analysis.

[0317] "Chat history" refers to data that records the content of conversations between users and the system, and is used for later reference and analysis.

[0318] "Mental health care content" refers to information materials such as video, audio, and text that are intended to support the mental health of users.

[0319] The system of the present invention provides preventative mental health care for users and is composed of a server, a terminal, and a user. In particular, by using an emotion analysis engine, the system can analyze the user's real-time emotional state and recommend appropriate intervention messages and mental health care content.

[0320] System configuration

[0321] The system consists of the following:

[0322] Server: Uses Flask (a Python web application framework) to receive, process, and control user input data. EmotionEngine (a hypothetical emotion analysis engine library) is used for emotion analysis, and ContentRecommender (a hypothetical content recommendation library) is used to recommend mental health care content.

[0323] Terminal: A device used by a user, such as a smartphone or head-mounted display (HMD). These devices are responsible for acquiring user input (text, voice, facial recognition data, etc.) and sending it to a server.

[0324] User: An individual who uses the system. They input their emotional state using a terminal.

[0325] System Operation

[0326] 1. Receiving user input

[0327] Users input their emotional state using a smartphone or HMD. This input can be in various forms, such as text, voice, or facial recognition data. For example, when a user types a message like "I'm overwhelmed by work," their tone of voice and facial expressions, which indicate stress, are also captured.

[0328] 2. Emotion analysis by the server

[0329] The server analyzes the user input data received from the device. During this analysis, the Emotion Engine is used to identify the user's emotional state through text analysis, voice analysis, and facial recognition analysis. For example, if the user's voice tone acquired along with the text message is high and the user's facial expression indicates stress, the emotional state of "anxiety" is identified.

[0330] 3. Recommending mental health care content

[0331] Based on the results of the emotion analysis, the server uses ContentRecommender to recommend appropriate mental health care content. For example, for the emotional state of "anxiety," content such as relaxing videos and calming music is recommended. This content is sent to the user's device and displayed.

[0332] 4. Chat history recording

[0333] All user interactions are recorded by the server as chat history, which is stored in a database for later analysis and intervention.

[0334] Specific examples

[0335] For example, if a user types "I'm overwhelmed by work," and their voice tone is high and their facial expression indicates stress, this data is sent from the device to the server. The server uses the Emotion Engine to analyze the text, voice, and facial expression data to identify the emotional state of "anxiety." It then uses the Content Recommender to recommend appropriate mental health content, such as "exercise videos to help you relax." This content is then sent to the user's device and displayed.

[0336] Example prompts to input to a generative AI model:

[0337] "If a user types in 'I'm overwhelmed at work,' and has a tone of voice and facial expression that indicates stress, what kind of mental health care content would be best?"

[0338] This system enables prompt and appropriate responses to the user's emotional state, providing personalized mental health care.

[0339] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0340] Step 1:

[0341] Users input their emotional state using a smartphone or head-mounted display (HMD). This input can include text, voice, or facial recognition data. For example, a message such as "I'm overwhelmed by work," a voice with a stressful tone, or image data of a stressful facial expression can be input. This input data is sent from the device to the server in JSON format.

[0342] Step 2:

[0343] The server receives the user's input data sent from the device. This input data includes text, voice, and facial recognition data. The received data is first temporarily stored in a database. This stored data is then extracted for analysis and provided to the emotion analysis engine (EmotionEngine).

[0344] Step 3:

[0345] The server uses the Emotion Engine to analyze the received user data. This involves text analysis, voice analysis, and facial recognition analysis to identify the user's emotional state. For example, the text message "I'm overwhelmed by work," high-pitched voice, and facial expression data indicating stress are combined to identify the emotional state of "anxiety."

[0346] Step 4:

[0347] The server selects an appropriate intervention message based on the emotion analysis results. This selection is performed using a content recommendation engine (ContentRecommender). Specifically, for the emotion "anxiety," it recommends mental health care content, including exercise videos and calming music for relaxation. This recommended intervention message is generated in JSON format.

[0348] Step 5:

[0349] The server sends the selected intervention message and the emotion analysis results to the user's device in JSON format as mental health care content. This sent data includes the emotion analysis results (e.g., "anxiety") and the recommended intervention message (e.g., "exercise video for relaxation").

[0350] Step 6:

[0351] The device displays the intervention message received from the server to the user. Specifically, a message such as "Please watch an exercise video to help you relax" is displayed on the user interface. The device also plays the recommended mental health care content (e.g., a video).

[0352] Step 7:

[0353] The server records the user's interactions and changes in emotional state and saves them as a chat history, including the user's input data, emotion analysis results, and recommended intervention messages. This history data is used for future analysis and further service improvement.

[0354] Through the specific actions of each step, users can enjoy personalized mental health care. The system supports users' mental health by quickly and accurately analyzing their emotional state and providing appropriate interventions.

[0355] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0356] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0357] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0358] [Second embodiment]

[0359] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0360] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0361] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0362] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0363] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0364] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0365] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0366] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0367] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0368] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0369] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0370] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0371] The system of the present invention aims to provide preventative, affordable, and private mental health care for users. The program processing of this system will be explained in natural language below, along with specific examples.

[0372] System Overview

[0373] This system consists of three components: a server, a terminal, and a user. Users access the system using their terminal and input information about their mental health status. The server plays a central role in receiving this input data, analyzing it, and providing a response.

[0374] Receiving User Input

[0375] A user inputs a message into the system using a device (e.g., a smartphone or a PC). For example, the user inputs, "I'm feeling overwhelmed with work."

[0376] Sending input data

[0377] The terminal sends this input data over the network to the server, which receives the message from the user and prepares for analysis.

[0378] sentiment analysis

[0379] The server performs real-time sentiment analysis on the received user input using a sentiment analysis model to identify specific emotions (e.g., anxiety, joy, sadness, anger) from the user input.

[0380] Intervention message selection

[0381] The server selects an appropriate intervention message based on the results of the sentiment analysis, for example, if the user is identified as "anxious," it selects the intervention message "Try some relaxation exercises."

[0382] Chat history recording

[0383] The server records interactions with users as a chat history, allowing past conversation data to be accumulated and referenced later.

[0384] Sending a response message

[0385] The server transmits the selected intervention message again to the terminal via the network, and the terminal displays the received message to the user.

[0386] User Awareness

[0387] The user sees the intervention message displayed on the device and follows the advice provided: "Try some relaxation exercises" is the message they receive and they act on it.

[0388] Specific examples

[0389] For example, consider a case where a user types the message "I'm feeling overwhelmed with work." This message is sent to the server via the device. The server receives the message and performs real-time sentiment analysis, identifying the emotion "anxiety." The server then selects "Try some relaxation exercises" as an intervention message corresponding to "anxiety." This message is recorded in the chat history and sent to the device for display to the user. Finally, the user can confirm this message and actually try some relaxation exercises.

[0390] In this way, the system can provide appropriate support in real time according to the user's mental health condition.

[0391] The processing flow will be explained below.

[0392] Step 1:

[0393] A user uses a terminal to enter a message into the system. For example, the user enters, "I'm feeling overwhelmed with work."

[0394] Step 2:

[0395] The device sends the input message to the server via the network. The data is sent in JSON format.

[0396] Step 3:

[0397] The server retrieves the received user message for analysis, and stores it in an internal data structure.

[0398] Step 4:

[0399] The server loads the sentiment analysis model and performs real-time sentiment analysis based on the message. In this process, a text analysis algorithm is used to identify the user's emotional state. For example, the emotion "anxious" is identified.

[0400] Step 5:

[0401] The server uses the results of the emotion analysis to select an appropriate intervention message based on a predefined mapping of emotions and messages. For example, the message "Try some relaxation exercises" is selected as the message corresponding to "anxious."

[0402] Step 6:

[0403] The server records the current conversation as a chat history, including user input, sentiment analysis results, and selected intervening messages, and stores this data in a database.

[0404] Step 7:

[0405] The server sends the selected intervention message back to the terminal via the network. The message is again sent in JSON format.

[0406] Step 8:

[0407] The terminal displays the intervention message received from the server to the user, for example, a message saying "Try some relaxation exercises" is displayed on the user interface.

[0408] Step 9:

[0409] The user acknowledges the displayed intervention message and performs the suggested action (e.g., relaxation exercises).

[0410] This series of steps enables the system to provide users with real-time, emotion-based mental health care.

[0411] Example 1

[0412] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0413] As mental health care has become increasingly important in recent years, there is a demand for easily accessible mental health care systems. However, existing systems have issues such as the time it takes to analyze user input and provide corresponding messages, the often mechanical responses that do not adequately address individual emotional states, and the difficulty in utilizing past interaction history. Another problem is the lack of a means to detect emotional changes early and provide appropriate intervention.

[0414] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0415] In this invention, the server includes a means for transmitting user input to the server via an internet connection, a means for performing real-time emotion analysis based on the received user input, and a means for selecting an intervention message based on the emotion analysis results and recording it in a database. This makes it possible to provide a response to the user's input quickly and according to the individual emotional state. Furthermore, by tracking changes in the user's emotions based on the chat history and enabling early intervention in response to emotional changes, more effective mental health care can be achieved.

[0416] "User input" refers to a text message that a user enters into the system using a terminal.

[0417] An "Internet line" is a communication path for sending and receiving digital data, including Wi-Fi and mobile networks (4G, 5G, etc.).

[0418] "Server" refers to a central computer system that processes, analyzes, and responds to received data.

[0419] "Real-time sentiment analysis" refers to the process of instantly analyzing sentiment and extracting specific emotions based on received user input.

[0420] "Artificial intelligence model" refers to a machine learning algorithm or neural network used for data analysis or prediction, such as a natural language processing model.

[0421] A "database" refers to a system that systematically organizes and stores data so that it can be searched and used later.

[0422] An "intervention message" refers to a message containing advice or suggested actions provided to a user based on the results of sentiment analysis.

[0423] "Chat history" refers to a record of past interactions with a user, and refers to data saved in a format that can be referenced and analyzed later.

[0424] "Changes in emotion" refers to changes in the user's emotional state over time, and is generally evaluated based on the difference from the initial state.

[0425] "Intervention" refers to the action of providing appropriate mental health care or advice based on the results of emotion analysis.

[0426] The system of the present invention aims to provide preventative, affordable, and private mental health care for users. Specific embodiments of the system are described below.

[0427] This system is primarily composed of three components: a server, a terminal, and a user. Users access the system using a terminal (such as a smartphone or PC) and enter information about their own mental health status. The server plays a central role in receiving this input data, analyzing it, and providing a response.

[0428] When a user uses a device to input a message, such as "I'm feeling overwhelmed with work," this input data is sent from the device to a server via an internet connection. The internet connection used here includes communication paths such as Wi-Fi, 4G, and 5G.

[0429] The server decodes the received user input data to analyze it. To do this, it uses natural language processing libraries (e.g., NLTK or spaCy) or generative AI models (e.g., OpenAI's GPT-3). These tools are used to perform real-time sentiment analysis and extract specific emotions (e.g., anxiety, joy, sadness, anger) from the user's input.

[0430] Based on the extracted emotion, the server selects an appropriate intervention message, which is then recorded in a database such as MySQL or PostgreSQL. For example, if the emotion analysis identifies "anxiety," the system might select and record the intervention message "Try some relaxation exercises."

[0431] The server then structures the selected intervention message as an HTTP response and sends it to the terminal over the Internet. The terminal then decodes the received message and displays it on its user interface, providing appropriate advice to the user.

[0432] The user is expected to check the intervention message displayed on the device and follow the advice provided. Specifically, the message "Try some relaxation exercises" encourages the user to perform relaxation exercises.

[0433] The server also records interactions with the user as a chat history, which helps track changes in the user's emotions and is stored in a database for later reference and analysis.

[0434] As a specific example, the following prompt sentence can be input into a generative AI model to provide sentiment analysis and intervention messages:

[0435] User message: "I'm feeling overwhelmed with work."

[0436] Analyze the sentiment of the user message and provide a supportive intervention message.

[0437] Based on this prompt, the generative AI model analyzes the user's emotions and generates an appropriate intervention message.

[0438] In this way, the system can provide appropriate support in real time according to the user's mental health condition.

[0439] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0440] Step 1:

[0441] A user accesses the system using a terminal and enters a message in a text box or chat window. The entered message is text data such as "I'm feeling overwhelmed with work." This input is sent to the next step as is.

[0442] Step 2:

[0443] The terminal sends the entered text data to the server via the Internet. At this time, the data is encoded in JSON format and sent as an HTTP request. The input here is the text message entered by the user, and the output is the encoded HTTP request.

[0444] Step 3:

[0445] The server decodes the received HTTP request and extracts the user's input message. It performs a decoding process to analyze the received data. The input is the encoded HTTP request from the terminal, and the output is the user input in text format.

[0446] Step 4:

[0447] The server performs real-time sentiment analysis based on the extracted user input message. Specifically, it uses natural language processing libraries (e.g., NLTK or spaCy) and generative AI models (e.g., OpenAI's GPT-3). The input for sentiment analysis is the user's text message, and the output is the analyzed emotion label (e.g., anxiety, joy, sadness, anger).

[0448] Step 5:

[0449] The server selects an appropriate intervention message based on the results of the sentiment analysis. It selects the corresponding intervention message from among those pre-stored in a database (e.g., MySQL or PostgreSQL). The input for this task is the sentiment label, and the output is the corresponding intervention message (e.g., "Try some relaxation exercises").

[0450] Step 6:

[0451] The server records the selected intervention message as a chat history in a database. This allows data to be accumulated in a form that can be referenced later. The input here is the intervention message, and the output is the chat history recorded in the database.

[0452] Step 7:

[0453] The server re-encodes the selected intervention message and sends it to the terminal as an HTTP response. The input is the intervention message and the output is the encoded HTTP response.

[0454] Step 8:

[0455] The terminal decodes the received HTTP response and displays the intervention message in a user interface, where the input is the encoded HTTP response and the output is the intervention message to be displayed (e.g., "Try some relaxation exercises").

[0456] Step 9:

[0457] The user checks the intervention message displayed on the device and acts according to the advice provided. Specifically, the user performs relaxation exercises based on the displayed message. The input is the intervention message displayed on the device, and the output is the user's actual behavior.

[0458] In this way, the entire system works to support the user's mental health in real time.

[0459] (Application example 1)

[0460] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0461] There is a need for a system that can detect mental health issues experienced by passengers in autonomous vehicles in real time and provide appropriate support. However, existing systems have difficulty accurately analyzing passenger mental health status and are unable to provide specific intervention measures. In addition, there is a lack of a system in place that allows passengers to receive mental health care in a private space.

[0462] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0463] In this invention, the server includes means for receiving user input, means for performing real-time sentiment analysis based on the received user input, means for selecting an intervention message based on the sentiment analysis result, means for providing the selected intervention message to the user, means for recording a chat history, and means for conducting mental health checks on passengers of the autonomous vehicle while the vehicle is in operation, thereby enabling passengers to receive real-time mental health care while the vehicle is in operation.

[0464] "User input" refers to messages and emotional states that a user inputs through a terminal.

[0465] "Real-time emotion analysis" refers to the process of instantly analyzing the emotional state of a user's input and extracting specific emotions.

[0466] An "intervention message" refers to advice or instructions provided to a user based on the results of sentiment analysis.

[0467] "Chat history" refers to the data that records and saves all conversations between a user and the system.

[0468] An "autonomous vehicle" refers to a vehicle that has the ability to move autonomously without a driver performing any driving operations.

[0469] A "mental health check" refers to a series of processes for diagnosing and assessing a user's emotional and mental state.

[0470] The purpose of the system of the present invention is to provide real-time mental health care for passengers in autonomous vehicles. This system consists of three components: a server, a terminal, and a user. The program processing of this system is explained below in natural language.

[0471] Hardware and software used

[0472] Hardware: On-board computers in autonomous vehicles, passenger smartphones

[0473] Software: Python 3.8+, aiohttp (web framework), TextBlob (sentiment analysis library)

[0474] Specific processing of the system

[0475] Receiving User Input

[0476] The user inputs their emotional state into the system using a device (such as a car screen or a smartphone). For example, the user might input, "I'm tired from work today."

[0477] Sending input data

[0478] The terminal transmits this input data via the network to the in-vehicle server, which receives the message from the user and prepares for analysis.

[0479] sentiment analysis

[0480] The server performs real-time sentiment analysis on the received user input, using the TextBlob library to identify specific sentiments (e.g., positive, negative, neutral) from the user input.

[0481] Intervention message selection

[0482] The server selects an appropriate intervention message based on the results of the sentiment analysis, for example, if the sentiment is identified as "negative," it selects the intervention message "Try some relaxation exercises."

[0483] Chat history recording

[0484] The server records interactions with users as a chat history, allowing past conversation data to be accumulated and referenced later.

[0485] Sending a response message

[0486] The selected intervention message is sent again via the network to the terminal, which displays the received message to the user.

[0487] Specific examples

[0488] For example, consider a case where a passenger types a message such as "I'm tired from work today." This message is sent to the server through the terminal. The server receives the message and performs real-time sentiment analysis, identifying the sentiment as "negative." The server then selects "Try some relaxation exercises," records this as a message history, and sends it to the terminal for display to the user. The user can confirm this message and actually try some relaxation exercises.

[0489] Prompt Sentence Examples

[0490] "Passengers input their emotional state via their smartphone or on-board screen."

[0491] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0492] Step 1:

[0493] Receiving User Input

[0494] The user inputs their emotional state using a device (such as a car screen or a smartphone). For example, the user might input, "I'm tired from work today." This input data is recognized by the device in text format.

[0495] Step 2:

[0496] Sending input data

[0497] The terminal transmits user input data to the in-vehicle server via a network. The input data is text data indicating the user's emotional state and is transferred from the terminal to the server.

[0498] Step 3:

[0499] Performing sentiment analysis

[0500] The server analyzes the received user input data. Specifically, it analyzes the sentiment of the input text using the TextBlob library. For example, the text "Work today has left me exhausted" is identified as expressing a negative sentiment (negative polarity).

[0501] Step 4:

[0502] Intervention message generation

[0503] The server selects an appropriate intervention message based on the result of the sentiment analysis. For example, if the sentiment is identified as "negative," it generates an intervention message such as "Try some relaxation exercises." This message is selected from fixed messages based on the result of the sentiment analysis.

[0504] Step 5:

[0505] Chat history recording

[0506] The server records the user's interactions as a chat history. This is the process of storing the user's input data and the corresponding generated intervention messages in a database. For example, the exchange "I'm tired from work today" and "Try some exercises to relax" is recorded.

[0507] Step 6:

[0508] Sending a response message

[0509] The server transmits the generated intervention message to the terminal again via the network, for example, a message saying "Try some relaxing exercises" is sent to the terminal, which then displays it to the user.

[0510] Step 7:

[0511] User perception and behavior

[0512] The user checks the intervention message displayed on the terminal, for example, "Try some relaxation exercises," and takes action according to the advice.

[0513] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0514] The system of the present invention aims to provide preventative, affordable, and private mental health care for users. In particular, by adding an emotion engine that recognizes the user's emotions, more accurate emotion analysis and intervention can be achieved. The overall configuration and specific processing of this system are described below.

[0515] System Overview

[0516] This system consists of three components: a server, a terminal, and a user. Users access the system using their terminal and input information about their mental health status. The server plays a central role in receiving this input data, analyzing it, and providing a response. The emotion engine is a module that further enhances emotion analysis.

[0517] Receiving User Input

[0518] A user inputs a message into the system using a device (e.g., a smartphone or a PC). This input can include not only text input, but also voice input and input using facial recognition. For example, a user inputs, "I'm feeling overwhelmed with work."

[0519] Sending input data

[0520] The device sends this input data to the server via the network. The data is sent in JSON format. Audio data and image data are also sent in the same way.

[0521] Performing sentiment analysis

[0522] The server receives the received user input data and performs real-time emotion analysis using the emotion engine. The emotion engine analyzes the following elements:

[0523] Text analysis: Analyze the content of a user's messages to identify their emotional state.

[0524] Voice analysis: Analyzes the tone and pitch of a user's voice to identify their emotional state.

[0525] Facial Expression Analysis: Analyzes facial expressions from the user's facial recognition data to identify their emotional state.

[0526] For example, if a user enters the text "I'm feeling overwhelmed with work," and has a stressed facial expression, the emotion "anxious" is identified.

[0527] Intervention message selection

[0528] The server selects an appropriate intervention message based on the result of the sentiment analysis, which is chosen based on a predefined emotion-message mapping.

[0529] For example, the message selected to correspond to "anxious" is "Try some relaxation exercises."

[0530] Chat history recording

[0531] The server records the user's interactions as a chat history. The recorded data includes the user's input, the results of sentiment analysis, and selected intervention messages. This data is stored in a database.

[0532] Sending a response message

[0533] The server then sends the selected intervention message back to the terminal via the network, again in JSON format.

[0534] Displaying a response message

[0535] The terminal displays the intervention message received from the server to the user, for example, a message saying "Try some relaxation exercises" is displayed on the user interface.

[0536] User perception and behavior

[0537] The user checks the intervention message displayed on the terminal and performs the suggested action (e.g., relaxation exercises).

[0538] Specific examples

[0539] For example, consider a case where a user types the message "I'm feeling overwhelmed with work," speaking in a high-pitched voice and displaying facial expressions that indicate stress. This message and additional data are sent to the server via the device. The server performs emotion analysis using text, voice, and facial expression analysis modules, and identifies the emotion "anxious." The server then selects "Try some relaxation exercises" as an intervention message corresponding to "anxious" and records it in the chat history. The server then sends this intervention message to the device, which then displays it to the user. Finally, the user can confirm the message and actually try some relaxation exercises.

[0540] This system can respond to the user's diverse emotional states in real time and provide more personalized mental health care.

[0541] The processing flow will be explained below.

[0542] Step 1:

[0543] A user uses a terminal to enter a message into the system. For example, the user enters, "I'm feeling overwhelmed with work."

[0544] Step 2:

[0545] The device sends the input message along with voice and facial expression data to the server in JSON format or another suitable format.

[0546] Step 3:

[0547] The server takes the received user input data for analysis and stores it in an internal data structure.

[0548] Step 4:

[0549] The server loads the emotion engine and performs real-time emotion analysis using text, voice, and facial expression data. The emotion analysis is performed as follows:

[0550] The server performs text analysis to identify the emotion from the content of the input message.

[0551] The server performs voice analysis and identifies emotions from the tone and pitch of the user's voice.

[0552] The server performs facial expression analysis and identifies emotions from the user's facial recognition data.

[0553] Step 5:

[0554] The server aggregates the results of the emotion analysis and identifies an overall emotional state. For example, if the results of text, voice, and facial expression analysis all indicate "anxious," the server identifies the overall emotion as "anxious."

[0555] Step 6:

[0556] The server selects an appropriate intervention message based on the results of emotion analysis. For example, it selects "Try some relaxation exercises" as the message for "anxious" based on predefined emotion-message mapping.

[0557] Step 7:

[0558] The server records the current conversation as a chat history, including user input, sentiment analysis results, and selected intervening messages, and stores the recorded data in a database.

[0559] Step 8:

[0560] The server sends the selected intervention message back to the terminal over the network, again in JSON format or another suitable format.

[0561] Step 9:

[0562] The terminal displays the intervention message received from the server to the user, for example, a message saying "Try some relaxation exercises" is displayed on the user interface.

[0563] Step 10:

[0564] The user acknowledges the displayed intervention message and performs the suggested action (e.g., relaxation exercises).

[0565] This series of steps enables the system to more accurately grasp the user's mental health status through multifaceted emotion analysis and provide appropriate real-time mental health care.

[0566] Example 2

[0567] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0568] Mental health issues are serious in modern society, and many users are experiencing increasing stress and anxiety. However, systems that provide affordable and private mental health care are not yet widespread. Another challenge is the lack of real-time emotional analysis of users and appropriate interventions based on that analysis.

[0569] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0570] In this invention, the server includes means for receiving user input, means for transmitting the received user input to the server via a network, means for performing real-time emotion analysis based on the received user input, means for selecting an intervention message based on the emotion analysis result, means for providing the selected intervention message to the user, and means for recording the chat history, thereby enabling the user to understand their own emotional state in real time and receive appropriate mental health care.

[0571] A "means for receiving user input" is a device or software that provides an interface for a user to enter messages or data into the system.

[0572] The "means for transmitting received user input to a server over a network" is a device or software that communicates user input to a server over the Internet or other communications network.

[0573] "Means for performing real-time sentiment analysis based on received user input" refers to analytical equipment or software for analyzing received data and determining the emotional state of the user in real time.

[0574] The "means for selecting an intervention message based on the result of sentiment analysis" is a device or software that selects the most appropriate advice or message based on the result of sentiment analysis.

[0575] The "means for providing a selected intervention message to a user" refers to a device or software that displays or notifies a user of a selected intervention message.

[0576] The "means for recording chat history" is a database or storage device for storing interactions with users and analysis results.

[0577] A "means for providing access to a professional" is a device or software that provides contact and access for a user to obtain professional advice or counseling as needed.

[0578] The "means for tracking emotional changes and providing early intervention" is a device or software that monitors changes in the user's emotional state and takes prompt action if necessary.

[0579] MODE FOR CARRYING OUT THE INVENTION

[0580] The system of the present invention aims to provide preventative, affordable, and private mental health care for users. In particular, by adding an emotion engine that recognizes the user's emotions, more accurate emotion analysis and intervention can be achieved. The overall configuration and specific processing of this system are described below.

[0581] Receiving User Input

[0582] Users input messages into the system using devices such as smartphones or PCs. This input can be text input, voice input, or input using facial recognition. For example, a user might input "I'm feeling overwhelmed with work." This input is done through the interface on the device.

[0583] Sending input data

[0584] The device sends input data acquired from the user to the server in JSON format. Voice data and image data are also sent in the same way. Specifically, the API on the device generates a JSON-formatted data packet and sends it through the network interface.

[0585] Performing sentiment analysis

[0586] The server first stores the received user input data in a database, and then performs real-time emotion analysis using the emotion engine. The emotion engine consists of the following analysis modules:

[0587] Text Analysis Module: Uses a natural language processing (NLP) engine to analyze text data and identify emotional states.

[0588] Audio analysis module: Analyzes the tone and pitch of audio data.

[0589] Facial Expression Analysis Module: Analyzes facial expressions from image data by using machine learning algorithms to extract visual features and infer emotions.

[0590] For example, if a user enters the text "I'm feeling overwhelmed with work," and there is a high-pitched voice tone indicative of stress and facial recognition data, the emotion engine will identify the emotion "anxious."

[0591] Selection of intervention messages

[0592] The server selects an appropriate intervention message based on the emotion analysis results. This message is selected from a predefined emotion-message mapping table. Specifically, the message "Try some relaxation exercises" is selected as the message corresponding to the emotion "anxious."

[0593] Chat history recording

[0594] The server records user interactions as chat history in a database. The recorded data includes user input, sentiment analysis results, and selected intervention messages. Specifically, it performs appropriate escaping and executes INSERT queries in the database using techniques to prevent SQL injection.

[0595] Sending a response message

[0596] The server encodes the selected intervention message in JSON format and sends it to the terminal. Specifically, it sends the packet again using the network interface.

[0597] Displaying a response message

[0598] The terminal decodes the intervention message received from the server and displays it to the user. The user interface (UI) displays the message content. For example, the UI displays a message such as "Try some relaxation exercises."

[0599] User perception and behavior

[0600] The user checks the intervention message displayed on the device and actually performs the suggested action (e.g., relaxation exercises), thereby improving their mental health.

[0601] Specific examples

[0602] As a specific example, if a user inputs the message "I'm feeling overwhelmed with work," with a high-pitched voice tone and facial expression indicating stress, this message and additional data are sent via the device to the server. The server receives this message and sequentially invokes the text analysis module, voice analysis module, and facial expression analysis module to identify the emotion "anxious." The server then selects an intervention message, "Try some relaxation exercises," from the emotion-message mapping table and sends it to the device. The device receives this and displays it to the user. The user confirms this message, tries to do some relaxation exercises, and enters the results as feedback into the system, starting the next cycle.

[0603] Prompt Sentence Examples

[0604] Please explain the operation of your system to support users' mental health care, breaking it down into specific steps. Please explain the operation of each step in detail, paying particular attention to the specific processing involved in performing analysis and sending and receiving data.

[0605] In this way, the present invention makes it possible to provide personalized mental health care according to the user's emotional state.

[0606] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0607] Step 1:

[0608] A user inputs a message into the system using a device (e.g., a smartphone or a PC). This input can include not only text input, but also voice input and input using facial recognition. For example, a user inputs the text "I'm feeling overwhelmed with work." Voice input and facial recognition data are also processed simultaneously.

[0609] Input: User text, voice, and facial recognition data

[0610] Output: Getting input data via the terminal

[0611] Step 2:

[0612] The device converts the input data received from the user into JSON format. Specifically, it encodes text data, compresses audio data, and encodes image data. This operation is performed through the API on the device.

[0613] Input: User input data (text, voice, facial recognition data)

[0614] Output: JSON formatted data packet

[0615] Step 3:

[0616] The device sends the encoded JSON-formatted data packets through the network interface. Specifically, the device sends the data to the server using a network protocol, using a secure communication protocol such as HTTPS.

[0617] Input: JSON formatted data packet

[0618] Output: Send data to the server

[0619] Step 4:

[0620] The server decodes the received JSON-formatted user data and stores it in a database. Specifically, it executes an INSERT query using a database management system (DBMS) on the server and stores the user data in the database.

[0621] Input: User data in JSON format

[0622] Output: Data storage in database

[0623] Step 5:

[0624] The server retrieves the stored user data and performs analysis using the sentiment analysis engine. Specifically, the following analysis modules are called in sequence:

[0625] Text Analysis Module: Uses a natural language processing engine to identify emotional states from text data.

[0626] Voice Analysis Module: Analyzes the tone and pitch of voice data to identify emotional states.

[0627] Facial Expression Analysis Module: Analyzes facial expressions from image data to identify emotional states.

[0628] Input: User data retrieved from the database

[0629] Output: Emotion analysis results (emotional state)

[0630] Step 6:

[0631] The server selects an appropriate intervention message based on the results of emotion analysis. Specifically, it references a mapping table of emotions and messages to select the most appropriate message. For example, if the emotion analysis result is "anxious," the server selects the message "Try some relaxation exercises."

[0632] Input: Emotion analysis results (emotional state)

[0633] Output: Intervention message

[0634] Step 7:

[0635] The server encodes the selected intervention message in JSON format and sends it to the terminal. Specifically, the server generates packets using a network protocol and transmits the data to the terminal via secure communication such as HTTPS.

[0636] Input: Intervention message

[0637] Output: Intervention message packet in JSON format

[0638] Step 8:

[0639] The device decodes the JSON-formatted intervention message received from the server and displays it on the user interface (UI). Specifically, the device's decoding process and UI rendering engine work together to visually display the message. For example, a message such as "Try some relaxation exercises" is displayed on the UI.

[0640] Input: JSON formatted intervention message packet

[0641] Output: Intervention message displayed to the user

[0642] Step 9:

[0643] The user confirms the displayed intervention message and performs the suggested action (e.g., relaxation exercises). The user performs the action and inputs the results as feedback to the system, providing new data for the next analysis cycle.

[0644] Input: Intervention message displayed to the user

[0645] Output: User actions and feedback

[0646] (Application example 2)

[0647] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0648] In modern society, the number of individuals experiencing mental stress and anxiety is increasing, creating a demand for prompt and effective mental health care. However, many mental health care systems are expensive and lack privacy. It is necessary to solve these problems and provide a preventive mental health care system that is easy for users to use.

[0649] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving user input, means for performing real-time emotion analysis based on the received user input, means for selecting an intervention message based on the emotion analysis result, means for providing the selected intervention message to the user, means for recording the chat history, and means for recommending mental health care content based on the emotion analysis result. This makes it possible to provide appropriate intervention and mental health care content according to the user's emotional state.

[0650] "User input" refers to information provided by a system user through a terminal, and may be in the form of text, audio, image, or the like.

[0651] "Real-time" refers to processing that occurs immediately without delay, and refers to data analysis and response in real time.

[0652] "Sentiment analysis" refers to the process of analyzing user-input data to identify a user's emotional state, and includes text analysis, speech analysis, and facial recognition analysis.

[0653] An "intervention message" refers to a message of recommended actions or advice provided to a user based on the results of sentiment analysis.

[0654] "Chat history" refers to data that records the content of conversations between users and the system, and is used for later reference and analysis.

[0655] "Mental health care content" refers to information materials such as video, audio, and text that are intended to support the mental health of users.

[0656] The system of the present invention provides preventative mental health care for users and is composed of a server, a terminal, and a user. In particular, by using an emotion analysis engine, the system can analyze the user's real-time emotional state and recommend appropriate intervention messages and mental health care content.

[0657] System configuration

[0658] The system consists of the following:

[0659] Server: Uses Flask (a Python web application framework) to receive, process, and control user input data. EmotionEngine (a hypothetical emotion analysis engine library) is used for emotion analysis, and ContentRecommender (a hypothetical content recommendation library) is used to recommend mental health care content.

[0660] Terminal: A device used by a user, such as a smartphone or head-mounted display (HMD). These devices are responsible for acquiring user input (text, voice, facial recognition data, etc.) and sending it to a server.

[0661] User: An individual who uses the system. They input their emotional state using a terminal.

[0662] System Operation

[0663] 1. Receiving user input

[0664] Users input their emotional state using a smartphone or HMD. This input can be in various forms, such as text, voice, or facial recognition data. For example, when a user types a message like "I'm overwhelmed by work," their tone of voice and facial expressions, which indicate stress, are also captured.

[0665] 2. Emotion analysis by the server

[0666] The server analyzes the user input data received from the device. During this analysis, the Emotion Engine is used to identify the user's emotional state through text analysis, voice analysis, and facial recognition analysis. For example, if the user's voice tone acquired along with the text message is high and the user's facial expression indicates stress, the emotional state of "anxiety" is identified.

[0667] 3. Recommending mental health care content

[0668] Based on the results of the emotion analysis, the server uses ContentRecommender to recommend appropriate mental health care content. For example, for the emotional state of "anxiety," content such as relaxing videos and calming music is recommended. This content is sent to the user's device and displayed.

[0669] 4. Chat history recording

[0670] All user interactions are recorded by the server as chat history, which is stored in a database for later analysis and intervention.

[0671] Specific examples

[0672] For example, if a user types "I'm overwhelmed by work," and their voice tone is high and their facial expression indicates stress, this data is sent from the device to the server. The server uses the Emotion Engine to analyze the text, voice, and facial expression data to identify the emotional state of "anxiety." It then uses the Content Recommender to recommend appropriate mental health content, such as "exercise videos to help you relax." This content is then sent to the user's device and displayed.

[0673] Example prompts to input to a generative AI model:

[0674] "If a user types in 'I'm overwhelmed at work,' and has a tone of voice and facial expression that indicates stress, what kind of mental health care content would be best?"

[0675] This system enables prompt and appropriate responses to the user's emotional state, providing personalized mental health care.

[0676] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0677] Step 1:

[0678] Users input their emotional state using a smartphone or head-mounted display (HMD). This input can include text, voice, or facial recognition data. For example, a message such as "I'm overwhelmed by work," a voice with a stressful tone, or image data of a stressful facial expression can be input. This input data is sent from the device to the server in JSON format.

[0679] Step 2:

[0680] The server receives the user's input data sent from the device. This input data includes text, voice, and facial recognition data. The received data is first temporarily stored in a database. This stored data is then extracted for analysis and provided to the emotion analysis engine (EmotionEngine).

[0681] Step 3:

[0682] The server uses the Emotion Engine to analyze the received user data. This involves text analysis, voice analysis, and facial recognition analysis to identify the user's emotional state. For example, the text message "I'm overwhelmed by work," high-pitched voice, and facial expression data indicating stress are combined to identify the emotional state of "anxiety."

[0683] Step 4:

[0684] The server selects an appropriate intervention message based on the emotion analysis results. This selection is performed using a content recommendation engine (ContentRecommender). Specifically, for the emotion "anxiety," it recommends mental health care content, including exercise videos and calming music for relaxation. This recommended intervention message is generated in JSON format.

[0685] Step 5:

[0686] The server sends the selected intervention message and the emotion analysis results to the user's device in JSON format as mental health care content. This sent data includes the emotion analysis results (e.g., "anxiety") and the recommended intervention message (e.g., "exercise video for relaxation").

[0687] Step 6:

[0688] The device displays the intervention message received from the server to the user. Specifically, a message such as "Please watch an exercise video to help you relax" is displayed on the user interface. The device also plays the recommended mental health care content (e.g., a video).

[0689] Step 7:

[0690] The server records the user's interactions and changes in emotional state and saves them as a chat history, including the user's input data, emotion analysis results, and recommended intervention messages. This history data is used for future analysis and further service improvement.

[0691] Through the specific actions of each step, users can enjoy personalized mental health care. The system supports users' mental health by quickly and accurately analyzing their emotional state and providing appropriate interventions.

[0692] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0693] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0694] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0695] [Third embodiment]

[0696] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0697] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0698] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0699] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0700] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0701] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0702] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0703] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0704] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0705] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0706] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0707] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0708] The system of the present invention aims to provide preventative, affordable, and private mental health care for users. The program processing of this system will be explained in natural language below, along with specific examples.

[0709] System Overview

[0710] This system consists of three components: a server, a terminal, and a user. Users access the system using their terminal and input information about their mental health status. The server plays a central role in receiving this input data, analyzing it, and providing a response.

[0711] Receiving User Input

[0712] A user inputs a message into the system using a device (e.g., a smartphone or a PC). For example, the user inputs, "I'm feeling overwhelmed with work."

[0713] Sending input data

[0714] The terminal sends this input data over the network to the server, which receives the message from the user and prepares for analysis.

[0715] sentiment analysis

[0716] The server performs real-time sentiment analysis on the received user input using a sentiment analysis model to identify specific emotions (e.g., anxiety, joy, sadness, anger) from the user input.

[0717] Intervention message selection

[0718] The server selects an appropriate intervention message based on the results of the sentiment analysis, for example, if the user is identified as "anxious," it selects the intervention message "Try some relaxation exercises."

[0719] Chat history recording

[0720] The server records interactions with users as a chat history, allowing past conversation data to be accumulated and referenced later.

[0721] Sending a response message

[0722] The server transmits the selected intervention message again to the terminal via the network, and the terminal displays the received message to the user.

[0723] User Awareness

[0724] The user sees the intervention message displayed on the device and follows the advice provided: "Try some relaxation exercises" is the message they receive and they act on it.

[0725] Specific examples

[0726] For example, consider a case where a user types the message "I'm feeling overwhelmed with work." This message is sent to the server via the device. The server receives the message and performs real-time sentiment analysis, identifying the emotion "anxiety." The server then selects "Try some relaxation exercises" as an intervention message corresponding to "anxiety." This message is recorded in the chat history and sent to the device for display to the user. Finally, the user can confirm this message and actually try some relaxation exercises.

[0727] In this way, the system can provide appropriate support in real time according to the user's mental health condition.

[0728] The processing flow will be explained below.

[0729] Step 1:

[0730] A user uses a terminal to enter a message into the system. For example, the user enters, "I'm feeling overwhelmed with work."

[0731] Step 2:

[0732] The device sends the input message to the server via the network. The data is sent in JSON format.

[0733] Step 3:

[0734] The server retrieves the received user message for analysis, and stores it in an internal data structure.

[0735] Step 4:

[0736] The server loads the sentiment analysis model and performs real-time sentiment analysis based on the message. In this process, a text analysis algorithm is used to identify the user's emotional state. For example, the emotion "anxious" is identified.

[0737] Step 5:

[0738] The server uses the results of the emotion analysis to select an appropriate intervention message based on a predefined mapping of emotions and messages. For example, the message "Try some relaxation exercises" is selected as the message corresponding to "anxious."

[0739] Step 6:

[0740] The server records the current conversation as a chat history, including user input, sentiment analysis results, and selected intervening messages, and stores this data in a database.

[0741] Step 7:

[0742] The server sends the selected intervention message back to the terminal via the network. The message is again sent in JSON format.

[0743] Step 8:

[0744] The terminal displays the intervention message received from the server to the user, for example, a message saying "Try some relaxation exercises" is displayed on the user interface.

[0745] Step 9:

[0746] The user acknowledges the displayed intervention message and performs the suggested action (e.g., relaxation exercises).

[0747] This series of steps enables the system to provide users with real-time, emotion-based mental health care.

[0748] Example 1

[0749] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0750] As mental health care has become increasingly important in recent years, there is a demand for easily accessible mental health care systems. However, existing systems have issues such as the time it takes to analyze user input and provide corresponding messages, the often mechanical responses that do not adequately address individual emotional states, and the difficulty in utilizing past interaction history. Another problem is the lack of a means to detect emotional changes early and provide appropriate intervention.

[0751] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0752] In this invention, the server includes a means for transmitting user input to the server via an internet connection, a means for performing real-time emotion analysis based on the received user input, and a means for selecting an intervention message based on the emotion analysis results and recording it in a database. This makes it possible to provide a response to the user's input quickly and according to the individual emotional state. Furthermore, by tracking changes in the user's emotions based on the chat history and enabling early intervention in response to emotional changes, more effective mental health care can be achieved.

[0753] "User input" refers to a text message that a user enters into the system using a terminal.

[0754] An "Internet line" is a communication path for sending and receiving digital data, including Wi-Fi and mobile networks (4G, 5G, etc.).

[0755] "Server" refers to a central computer system that processes, analyzes, and responds to received data.

[0756] "Real-time sentiment analysis" refers to the process of instantly analyzing sentiment and extracting specific emotions based on received user input.

[0757] "Artificial intelligence model" refers to a machine learning algorithm or neural network used for data analysis or prediction, such as a natural language processing model.

[0758] A "database" refers to a system that systematically organizes and stores data so that it can be searched and used later.

[0759] An "intervention message" refers to a message containing advice or suggested actions provided to a user based on the results of sentiment analysis.

[0760] "Chat history" refers to a record of past interactions with a user, and refers to data saved in a format that can be referenced and analyzed later.

[0761] "Changes in emotion" refers to changes in the user's emotional state over time, and is generally evaluated based on the difference from the initial state.

[0762] "Intervention" refers to the action of providing appropriate mental health care or advice based on the results of emotion analysis.

[0763] The system of the present invention aims to provide preventative, affordable, and private mental health care for users. Specific embodiments of the system are described below.

[0764] This system is primarily composed of three components: a server, a terminal, and a user. Users access the system using a terminal (such as a smartphone or PC) and enter information about their own mental health status. The server plays a central role in receiving this input data, analyzing it, and providing a response.

[0765] When a user uses a device to input a message, such as "I'm feeling overwhelmed with work," this input data is sent from the device to a server via an internet connection. The internet connection used here includes communication paths such as Wi-Fi, 4G, and 5G.

[0766] The server decodes the received user input data to analyze it. To do this, it uses natural language processing libraries (e.g., NLTK or spaCy) or generative AI models (e.g., OpenAI's GPT-3). These tools are used to perform real-time sentiment analysis and extract specific emotions (e.g., anxiety, joy, sadness, anger) from the user's input.

[0767] Based on the extracted emotion, the server selects an appropriate intervention message, which is then recorded in a database such as MySQL or PostgreSQL. For example, if the emotion analysis identifies "anxiety," the system might select and record the intervention message "Try some relaxation exercises."

[0768] The server then structures the selected intervention message as an HTTP response and sends it to the terminal over the Internet. The terminal then decodes the received message and displays it on its user interface, providing appropriate advice to the user.

[0769] The user is expected to check the intervention message displayed on the device and follow the advice provided. Specifically, the message "Try some relaxation exercises" encourages the user to perform relaxation exercises.

[0770] The server also records interactions with the user as a chat history, which helps track changes in the user's emotions and is stored in a database for later reference and analysis.

[0771] As a specific example, the following prompt sentence can be input into a generative AI model to provide sentiment analysis and intervention messages:

[0772] User message: "I'm feeling overwhelmed with work."

[0773] Analyze the sentiment of the user message and provide a supportive intervention message.

[0774] Based on this prompt, the generative AI model analyzes the user's emotions and generates an appropriate intervention message.

[0775] In this way, the system can provide appropriate support in real time according to the user's mental health condition.

[0776] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0777] Step 1:

[0778] A user accesses the system using a terminal and enters a message in a text box or chat window. The entered message is text data such as "I'm feeling overwhelmed with work." This input is sent to the next step as is.

[0779] Step 2:

[0780] The terminal sends the entered text data to the server via the Internet. At this time, the data is encoded in JSON format and sent as an HTTP request. The input here is the text message entered by the user, and the output is the encoded HTTP request.

[0781] Step 3:

[0782] The server decodes the received HTTP request and extracts the user's input message. It performs a decoding process to analyze the received data. The input is the encoded HTTP request from the terminal, and the output is the user input in text format.

[0783] Step 4:

[0784] The server performs real-time sentiment analysis based on the extracted user input message. Specifically, it uses natural language processing libraries (e.g., NLTK or spaCy) and generative AI models (e.g., OpenAI's GPT-3). The input for sentiment analysis is the user's text message, and the output is the analyzed emotion label (e.g., anxiety, joy, sadness, anger).

[0785] Step 5:

[0786] The server selects an appropriate intervention message based on the results of the sentiment analysis. It selects the corresponding intervention message from among those pre-stored in a database (e.g., MySQL or PostgreSQL). The input for this task is the sentiment label, and the output is the corresponding intervention message (e.g., "Try some relaxation exercises").

[0787] Step 6:

[0788] The server records the selected intervention message as a chat history in a database. This allows data to be accumulated in a form that can be referenced later. The input here is the intervention message, and the output is the chat history recorded in the database.

[0789] Step 7:

[0790] The server re-encodes the selected intervention message and sends it to the terminal as an HTTP response. The input is the intervention message and the output is the encoded HTTP response.

[0791] Step 8:

[0792] The terminal decodes the received HTTP response and displays the intervention message in a user interface, where the input is the encoded HTTP response and the output is the intervention message to be displayed (e.g., "Try some relaxation exercises").

[0793] Step 9:

[0794] The user checks the intervention message displayed on the device and acts according to the advice provided. Specifically, the user performs relaxation exercises based on the displayed message. The input is the intervention message displayed on the device, and the output is the user's actual behavior.

[0795] In this way, the entire system works to support the user's mental health in real time.

[0796] (Application example 1)

[0797] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0798] There is a need for a system that can detect mental health issues experienced by passengers in autonomous vehicles in real time and provide appropriate support. However, existing systems have difficulty accurately analyzing passenger mental health status and are unable to provide specific intervention measures. In addition, there is a lack of a system in place that allows passengers to receive mental health care in a private space.

[0799] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0800] In this invention, the server includes means for receiving user input, means for performing real-time sentiment analysis based on the received user input, means for selecting an intervention message based on the sentiment analysis result, means for providing the selected intervention message to the user, means for recording a chat history, and means for conducting mental health checks on passengers of the autonomous vehicle while the vehicle is in operation, thereby enabling passengers to receive real-time mental health care while the vehicle is in operation.

[0801] "User input" refers to messages and emotional states that a user inputs through a terminal.

[0802] "Real-time emotion analysis" refers to the process of instantly analyzing the emotional state of a user's input and extracting specific emotions.

[0803] An "intervention message" refers to advice or instructions provided to a user based on the results of sentiment analysis.

[0804] "Chat history" refers to the data that records and saves all conversations between a user and the system.

[0805] An "autonomous vehicle" refers to a vehicle that has the ability to move autonomously without a driver performing any driving operations.

[0806] A "mental health check" refers to a series of processes for diagnosing and assessing a user's emotional and mental state.

[0807] The purpose of the system of the present invention is to provide real-time mental health care for passengers in autonomous vehicles. This system consists of three components: a server, a terminal, and a user. The program processing of this system is explained below in natural language.

[0808] Hardware and software used

[0809] Hardware: On-board computers in autonomous vehicles, passenger smartphones

[0810] Software: Python 3.8+, aiohttp (web framework), TextBlob (sentiment analysis library)

[0811] Specific processing of the system

[0812] Receiving User Input

[0813] The user inputs their emotional state into the system using a device (such as a car screen or a smartphone). For example, the user might input, "I'm tired from work today."

[0814] Sending input data

[0815] The terminal transmits this input data via the network to the in-vehicle server, which receives the message from the user and prepares for analysis.

[0816] sentiment analysis

[0817] The server performs real-time sentiment analysis on the received user input, using the TextBlob library to identify specific sentiments (e.g., positive, negative, neutral) from the user input.

[0818] Intervention message selection

[0819] The server selects an appropriate intervention message based on the results of the sentiment analysis, for example, if the sentiment is identified as "negative," it selects the intervention message "Try some relaxation exercises."

[0820] Chat history recording

[0821] The server records interactions with users as a chat history, allowing past conversation data to be accumulated and referenced later.

[0822] Sending a response message

[0823] The selected intervention message is sent again via the network to the terminal, which displays the received message to the user.

[0824] Specific examples

[0825] For example, consider a case where a passenger types a message such as "I'm tired from work today." This message is sent to the server through the terminal. The server receives the message and performs real-time sentiment analysis, identifying the sentiment as "negative." The server then selects "Try some relaxation exercises," records this as a message history, and sends it to the terminal for display to the user. The user can confirm this message and actually try some relaxation exercises.

[0826] Prompt Sentence Examples

[0827] "Passengers input their emotional state via their smartphone or on-board screen."

[0828] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0829] Step 1:

[0830] Receiving User Input

[0831] The user inputs their emotional state using a device (such as a car screen or a smartphone). For example, the user might input, "I'm tired from work today." This input data is recognized by the device in text format.

[0832] Step 2:

[0833] Sending input data

[0834] The terminal transmits user input data to the in-vehicle server via a network. The input data is text data indicating the user's emotional state and is transferred from the terminal to the server.

[0835] Step 3:

[0836] Performing sentiment analysis

[0837] The server analyzes the received user input data. Specifically, it analyzes the sentiment of the input text using the TextBlob library. For example, the text "Work today has left me exhausted" is identified as expressing a negative sentiment (negative polarity).

[0838] Step 4:

[0839] Intervention message generation

[0840] The server selects an appropriate intervention message based on the result of the sentiment analysis. For example, if the sentiment is identified as "negative," it generates an intervention message such as "Try some relaxation exercises." This message is selected from fixed messages based on the result of the sentiment analysis.

[0841] Step 5:

[0842] Chat history recording

[0843] The server records the user's interactions as a chat history. This is the process of storing the user's input data and the corresponding generated intervention messages in a database. For example, the exchange "I'm tired from work today" and "Try some exercises to relax" is recorded.

[0844] Step 6:

[0845] Sending a response message

[0846] The server transmits the generated intervention message to the terminal again via the network, for example, a message saying "Try some relaxing exercises" is sent to the terminal, which then displays it to the user.

[0847] Step 7:

[0848] User perception and behavior

[0849] The user checks the intervention message displayed on the terminal, for example, "Try some relaxation exercises," and takes action according to the advice.

[0850] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0851] The system of the present invention aims to provide preventative, affordable, and private mental health care for users. In particular, by adding an emotion engine that recognizes the user's emotions, more accurate emotion analysis and intervention can be achieved. The overall configuration and specific processing of this system are described below.

[0852] System Overview

[0853] This system consists of three components: a server, a terminal, and a user. Users access the system using their terminal and input information about their mental health status. The server plays a central role in receiving this input data, analyzing it, and providing a response. The emotion engine is a module that further enhances emotion analysis.

[0854] Receiving User Input

[0855] A user inputs a message into the system using a device (e.g., a smartphone or a PC). This input can include not only text input, but also voice input and input using facial recognition. For example, a user inputs, "I'm feeling overwhelmed with work."

[0856] Sending input data

[0857] The device sends this input data to the server via the network. The data is sent in JSON format. Audio data and image data are also sent in the same way.

[0858] Performing sentiment analysis

[0859] The server receives the received user input data and performs real-time emotion analysis using the emotion engine. The emotion engine analyzes the following elements:

[0860] Text analysis: Analyze the content of a user's messages to identify their emotional state.

[0861] Voice analysis: Analyzes the tone and pitch of a user's voice to identify their emotional state.

[0862] Facial Expression Analysis: Analyzes facial expressions from the user's facial recognition data to identify their emotional state.

[0863] For example, if a user enters the text "I'm feeling overwhelmed with work," and has a stressed facial expression, the emotion "anxious" is identified.

[0864] Intervention message selection

[0865] The server selects an appropriate intervention message based on the result of the sentiment analysis, which is chosen based on a predefined emotion-message mapping.

[0866] For example, the message selected to correspond to "anxious" is "Try some relaxation exercises."

[0867] Chat history recording

[0868] The server records the user's interactions as a chat history. The recorded data includes the user's input, the results of sentiment analysis, and selected intervention messages. This data is stored in a database.

[0869] Sending a response message

[0870] The server then sends the selected intervention message back to the terminal via the network, again in JSON format.

[0871] Displaying a response message

[0872] The terminal displays the intervention message received from the server to the user, for example, a message saying "Try some relaxation exercises" is displayed on the user interface.

[0873] User perception and behavior

[0874] The user checks the intervention message displayed on the terminal and performs the suggested action (e.g., relaxation exercises).

[0875] Specific examples

[0876] For example, consider a case where a user types the message "I'm feeling overwhelmed with work," speaking in a high-pitched voice and displaying facial expressions that indicate stress. This message and additional data are sent to the server via the device. The server performs emotion analysis using text, voice, and facial expression analysis modules, and identifies the emotion "anxious." The server then selects "Try some relaxation exercises" as an intervention message corresponding to "anxious" and records it in the chat history. The server then sends this intervention message to the device, which then displays it to the user. Finally, the user can confirm the message and actually try some relaxation exercises.

[0877] This system can respond to the user's diverse emotional states in real time and provide more personalized mental health care.

[0878] The processing flow will be explained below.

[0879] Step 1:

[0880] A user uses a terminal to enter a message into the system. For example, the user enters, "I'm feeling overwhelmed with work."

[0881] Step 2:

[0882] The device sends the input message along with voice and facial expression data to the server in JSON format or another suitable format.

[0883] Step 3:

[0884] The server takes the received user input data for analysis and stores it in an internal data structure.

[0885] Step 4:

[0886] The server loads the emotion engine and performs real-time emotion analysis using text, voice, and facial expression data. The emotion analysis is performed as follows:

[0887] The server performs text analysis to identify the emotion from the content of the input message.

[0888] The server performs voice analysis and identifies emotions from the tone and pitch of the user's voice.

[0889] The server performs facial expression analysis and identifies emotions from the user's facial recognition data.

[0890] Step 5:

[0891] The server aggregates the results of the emotion analysis and identifies an overall emotional state. For example, if the results of text, voice, and facial expression analysis all indicate "anxious," the server identifies the overall emotion as "anxious."

[0892] Step 6:

[0893] The server selects an appropriate intervention message based on the results of emotion analysis. For example, it selects "Try some relaxation exercises" as the message for "anxious" based on predefined emotion-message mapping.

[0894] Step 7:

[0895] The server records the current conversation as a chat history, including user input, sentiment analysis results, and selected intervening messages, and stores the recorded data in a database.

[0896] Step 8:

[0897] The server sends the selected intervention message back to the terminal over the network, again in JSON format or another suitable format.

[0898] Step 9:

[0899] The terminal displays the intervention message received from the server to the user, for example, a message saying "Try some relaxation exercises" is displayed on the user interface.

[0900] Step 10:

[0901] The user acknowledges the displayed intervention message and performs the suggested action (e.g., relaxation exercises).

[0902] This series of steps enables the system to more accurately grasp the user's mental health status through multifaceted emotion analysis and provide appropriate real-time mental health care.

[0903] Example 2

[0904] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0905] Mental health issues are serious in modern society, and many users are experiencing increasing stress and anxiety. However, systems that provide affordable and private mental health care are not yet widespread. Another challenge is the lack of real-time emotional analysis of users and appropriate interventions based on that analysis.

[0906] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0907] In this invention, the server includes means for receiving user input, means for transmitting the received user input to the server via a network, means for performing real-time emotion analysis based on the received user input, means for selecting an intervention message based on the emotion analysis result, means for providing the selected intervention message to the user, and means for recording the chat history, thereby enabling the user to understand their own emotional state in real time and receive appropriate mental health care.

[0908] A "means for receiving user input" is a device or software that provides an interface for a user to enter messages or data into the system.

[0909] The "means for transmitting received user input to a server over a network" is a device or software that communicates user input to a server over the Internet or other communications network.

[0910] "Means for performing real-time sentiment analysis based on received user input" refers to analytical equipment or software for analyzing received data and determining the emotional state of the user in real time.

[0911] The "means for selecting an intervention message based on the result of sentiment analysis" is a device or software that selects the most appropriate advice or message based on the result of sentiment analysis.

[0912] The "means for providing a selected intervention message to a user" refers to a device or software that displays or notifies a user of a selected intervention message.

[0913] The "means for recording chat history" is a database or storage device for storing interactions with users and analysis results.

[0914] A "means for providing access to a professional" is a device or software that provides contact and access for a user to obtain professional advice or counseling as needed.

[0915] The "means for tracking emotional changes and providing early intervention" is a device or software that monitors changes in the user's emotional state and takes prompt action if necessary.

[0916] MODE FOR CARRYING OUT THE INVENTION

[0917] The system of the present invention aims to provide preventative, affordable, and private mental health care for users. In particular, by adding an emotion engine that recognizes the user's emotions, more accurate emotion analysis and intervention can be achieved. The overall configuration and specific processing of this system are described below.

[0918] Receiving User Input

[0919] Users input messages into the system using devices such as smartphones or PCs. This input can be text input, voice input, or input using facial recognition. For example, a user might input "I'm feeling overwhelmed with work." This input is done through the interface on the device.

[0920] Sending input data

[0921] The device sends input data acquired from the user to the server in JSON format. Voice data and image data are also sent in the same way. Specifically, the API on the device generates a JSON-formatted data packet and sends it through the network interface.

[0922] Performing sentiment analysis

[0923] The server first stores the received user input data in a database, and then performs real-time emotion analysis using the emotion engine. The emotion engine consists of the following analysis modules:

[0924] Text Analysis Module: Uses a natural language processing (NLP) engine to analyze text data and identify emotional states.

[0925] Audio analysis module: Analyzes the tone and pitch of audio data.

[0926] Facial Expression Analysis Module: Analyzes facial expressions from image data by using machine learning algorithms to extract visual features and infer emotions.

[0927] For example, if a user enters the text "I'm feeling overwhelmed with work," and there is a high-pitched voice tone indicative of stress and facial recognition data, the emotion engine will identify the emotion "anxious."

[0928] Selection of intervention messages

[0929] The server selects an appropriate intervention message based on the emotion analysis results. This message is selected from a predefined emotion-message mapping table. Specifically, the message "Try some relaxation exercises" is selected as the message corresponding to the emotion "anxious."

[0930] Chat history recording

[0931] The server records user interactions as chat history in a database. The recorded data includes user input, sentiment analysis results, and selected intervention messages. Specifically, it performs appropriate escaping and executes INSERT queries in the database using techniques to prevent SQL injection.

[0932] Sending a response message

[0933] The server encodes the selected intervention message in JSON format and sends it to the terminal. Specifically, it sends the packet again using the network interface.

[0934] Displaying a response message

[0935] The terminal decodes the intervention message received from the server and displays it to the user. The user interface (UI) displays the message content. For example, the UI displays a message such as "Try some relaxation exercises."

[0936] User perception and behavior

[0937] The user checks the intervention message displayed on the device and actually performs the suggested action (e.g., relaxation exercises), thereby improving their mental health.

[0938] Specific examples

[0939] As a specific example, if a user inputs the message "I'm feeling overwhelmed with work," with a high-pitched voice tone and facial expression indicating stress, this message and additional data are sent via the device to the server. The server receives this message and sequentially invokes the text analysis module, voice analysis module, and facial expression analysis module to identify the emotion "anxious." The server then selects an intervention message, "Try some relaxation exercises," from the emotion-message mapping table and sends it to the device. The device receives this and displays it to the user. The user confirms this message, tries to do some relaxation exercises, and enters the results as feedback into the system, starting the next cycle.

[0940] Prompt Sentence Examples

[0941] Please explain the operation of your system to support users' mental health care, breaking it down into specific steps. Please explain the operation of each step in detail, paying particular attention to the specific processing involved in performing analysis and sending and receiving data.

[0942] In this way, the present invention makes it possible to provide personalized mental health care according to the user's emotional state.

[0943] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0944] Step 1:

[0945] A user inputs a message into the system using a device (e.g., a smartphone or a PC). This input can include not only text input, but also voice input and input using facial recognition. For example, a user inputs the text "I'm feeling overwhelmed with work." Voice input and facial recognition data are also processed simultaneously.

[0946] Input: User text, voice, and facial recognition data

[0947] Output: Getting input data via the terminal

[0948] Step 2:

[0949] The device converts the input data received from the user into JSON format. Specifically, it encodes text data, compresses audio data, and encodes image data. This operation is performed through the API on the device.

[0950] Input: User input data (text, voice, facial recognition data)

[0951] Output: JSON formatted data packet

[0952] Step 3:

[0953] The device sends the encoded JSON-formatted data packets through the network interface. Specifically, the device sends the data to the server using a network protocol, using a secure communication protocol such as HTTPS.

[0954] Input: JSON formatted data packet

[0955] Output: Send data to the server

[0956] Step 4:

[0957] The server decodes the received JSON-formatted user data and stores it in a database. Specifically, it executes an INSERT query using a database management system (DBMS) on the server and stores the user data in the database.

[0958] Input: User data in JSON format

[0959] Output: Data storage in database

[0960] Step 5:

[0961] The server retrieves the stored user data and performs analysis using the sentiment analysis engine. Specifically, the following analysis modules are called in sequence:

[0962] Text Analysis Module: Uses a natural language processing engine to identify emotional states from text data.

[0963] Voice Analysis Module: Analyzes the tone and pitch of voice data to identify emotional states.

[0964] Facial Expression Analysis Module: Analyzes facial expressions from image data to identify emotional states.

[0965] Input: User data retrieved from the database

[0966] Output: Emotion analysis results (emotional state)

[0967] Step 6:

[0968] The server selects an appropriate intervention message based on the results of emotion analysis. Specifically, it references a mapping table of emotions and messages to select the most appropriate message. For example, if the emotion analysis result is "anxious," the server selects the message "Try some relaxation exercises."

[0969] Input: Emotion analysis results (emotional state)

[0970] Output: Intervention message

[0971] Step 7:

[0972] The server encodes the selected intervention message in JSON format and sends it to the terminal. Specifically, the server generates packets using a network protocol and transmits the data to the terminal via secure communication such as HTTPS.

[0973] Input: Intervention message

[0974] Output: Intervention message packet in JSON format

[0975] Step 8:

[0976] The device decodes the JSON-formatted intervention message received from the server and displays it on the user interface (UI). Specifically, the device's decoding process and UI rendering engine work together to visually display the message. For example, a message such as "Try some relaxation exercises" is displayed on the UI.

[0977] Input: JSON formatted intervention message packet

[0978] Output: Intervention message displayed to the user

[0979] Step 9:

[0980] The user confirms the displayed intervention message and performs the suggested action (e.g., relaxation exercises). The user performs the action and inputs the results as feedback to the system, providing new data for the next analysis cycle.

[0981] Input: Intervention message displayed to the user

[0982] Output: User actions and feedback

[0983] (Application example 2)

[0984] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0985] In modern society, the number of individuals experiencing mental stress and anxiety is increasing, creating a demand for prompt and effective mental health care. However, many mental health care systems are expensive and lack privacy. It is necessary to solve these problems and provide a preventive mental health care system that is easy for users to use.

[0986] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving user input, means for performing real-time emotion analysis based on the received user input, means for selecting an intervention message based on the emotion analysis result, means for providing the selected intervention message to the user, means for recording the chat history, and means for recommending mental health care content based on the emotion analysis result. This makes it possible to provide appropriate intervention and mental health care content according to the user's emotional state.

[0987] "User input" refers to information provided by a system user through a terminal, and may be in the form of text, audio, image, or the like.

[0988] "Real-time" refers to processing that occurs immediately without delay, and refers to data analysis and response in real time.

[0989] "Sentiment analysis" refers to the process of analyzing user-input data to identify a user's emotional state, and includes text analysis, speech analysis, and facial recognition analysis.

[0990] An "intervention message" refers to a message of recommended actions or advice provided to a user based on the results of sentiment analysis.

[0991] "Chat history" refers to data that records the content of conversations between users and the system, and is used for later reference and analysis.

[0992] "Mental health care content" refers to information materials such as video, audio, and text that are intended to support the mental health of users.

[0993] The system of the present invention provides preventative mental health care for users and is composed of a server, a terminal, and a user. In particular, by using an emotion analysis engine, the system can analyze the user's real-time emotional state and recommend appropriate intervention messages and mental health care content.

[0994] System configuration

[0995] The system consists of the following:

[0996] Server: Uses Flask (a Python web application framework) to receive, process, and control user input data. EmotionEngine (a hypothetical emotion analysis engine library) is used for emotion analysis, and ContentRecommender (a hypothetical content recommendation library) is used to recommend mental health care content.

[0997] Terminal: A device used by a user, such as a smartphone or head-mounted display (HMD). These devices are responsible for acquiring user input (text, voice, facial recognition data, etc.) and sending it to a server.

[0998] User: An individual who uses the system. They input their emotional state using a terminal.

[0999] System Operation

[1000] 1. Receiving user input

[1001] Users input their emotional state using a smartphone or HMD. This input can be in various forms, such as text, voice, or facial recognition data. For example, when a user types a message like "I'm overwhelmed by work," their tone of voice and facial expressions, which indicate stress, are also captured.

[1002] 2. Emotion analysis by the server

[1003] The server analyzes the user input data received from the device. During this analysis, the Emotion Engine is used to identify the user's emotional state through text analysis, voice analysis, and facial recognition analysis. For example, if the user's voice tone acquired along with the text message is high and the user's facial expression indicates stress, the emotional state of "anxiety" is identified.

[1004] 3. Recommending mental health care content

[1005] Based on the results of the emotion analysis, the server uses ContentRecommender to recommend appropriate mental health care content. For example, for the emotional state of "anxiety," content such as relaxing videos and calming music is recommended. This content is sent to the user's device and displayed.

[1006] 4. Chat history recording

[1007] All user interactions are recorded by the server as chat history, which is stored in a database for later analysis and intervention.

[1008] Specific examples

[1009] For example, if a user types "I'm overwhelmed by work," and their voice tone is high and their facial expression indicates stress, this data is sent from the device to the server. The server uses the Emotion Engine to analyze the text, voice, and facial expression data to identify the emotional state of "anxiety." It then uses the Content Recommender to recommend appropriate mental health content, such as "exercise videos to help you relax." This content is then sent to the user's device and displayed.

[1010] Example prompts to input to a generative AI model:

[1011] "If a user types in 'I'm overwhelmed at work,' and has a tone of voice and facial expression that indicates stress, what kind of mental health care content would be best?"

[1012] This system enables prompt and appropriate responses to the user's emotional state, providing personalized mental health care.

[1013] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1014] Step 1:

[1015] Users input their emotional state using a smartphone or head-mounted display (HMD). This input can include text, voice, or facial recognition data. For example, a message such as "I'm overwhelmed by work," a voice with a stressful tone, or image data of a stressful facial expression can be input. This input data is sent from the device to the server in JSON format.

[1016] Step 2:

[1017] The server receives the user's input data sent from the device. This input data includes text, voice, and facial recognition data. The received data is first temporarily stored in a database. This stored data is then extracted for analysis and provided to the emotion analysis engine (EmotionEngine).

[1018] Step 3:

[1019] The server uses the Emotion Engine to analyze the received user data. This involves text analysis, voice analysis, and facial recognition analysis to identify the user's emotional state. For example, the text message "I'm overwhelmed by work," high-pitched voice, and facial expression data indicating stress are combined to identify the emotional state of "anxiety."

[1020] Step 4:

[1021] The server selects an appropriate intervention message based on the emotion analysis results. This selection is performed using a content recommendation engine (ContentRecommender). Specifically, for the emotion "anxiety," it recommends mental health care content, including exercise videos and calming music for relaxation. This recommended intervention message is generated in JSON format.

[1022] Step 5:

[1023] The server sends the selected intervention message and the emotion analysis results to the user's device in JSON format as mental health care content. This sent data includes the emotion analysis results (e.g., "anxiety") and the recommended intervention message (e.g., "exercise video for relaxation").

[1024] Step 6:

[1025] The device displays the intervention message received from the server to the user. Specifically, a message such as "Please watch an exercise video to help you relax" is displayed on the user interface. The device also plays the recommended mental health care content (e.g., a video).

[1026] Step 7:

[1027] The server records the user's interactions and changes in emotional state and saves them as a chat history, including the user's input data, emotion analysis results, and recommended intervention messages. This history data is used for future analysis and further service improvement.

[1028] Through the specific actions of each step, users can enjoy personalized mental health care. The system supports users' mental health by quickly and accurately analyzing their emotional state and providing appropriate interventions.

[1029] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1030] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1031] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1032] [Fourth embodiment]

[1033] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1034] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1035] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1036] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1037] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1038] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1039] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1040] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1041] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1042] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1043] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1044] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1045] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1046] The system of the present invention aims to provide preventative, affordable, and private mental health care for users. The program processing of this system will be explained in natural language below, along with specific examples.

[1047] System Overview

[1048] This system consists of three components: a server, a terminal, and a user. Users access the system using their terminal and input information about their mental health status. The server plays a central role in receiving this input data, analyzing it, and providing a response.

[1049] Receiving User Input

[1050] A user inputs a message into the system using a device (e.g., a smartphone or a PC). For example, the user inputs, "I'm feeling overwhelmed with work."

[1051] Sending input data

[1052] The terminal sends this input data over the network to the server, which receives the message from the user and prepares for analysis.

[1053] sentiment analysis

[1054] The server performs real-time sentiment analysis on the received user input using a sentiment analysis model to identify specific emotions (e.g., anxiety, joy, sadness, anger) from the user input.

[1055] Intervention message selection

[1056] The server selects an appropriate intervention message based on the results of the sentiment analysis, for example, if the user is identified as "anxious," it selects the intervention message "Try some relaxation exercises."

[1057] Chat history recording

[1058] The server records interactions with users as a chat history, allowing past conversation data to be accumulated and referenced later.

[1059] Sending a response message

[1060] The server transmits the selected intervention message again to the terminal via the network, and the terminal displays the received message to the user.

[1061] User Awareness

[1062] The user sees the intervention message displayed on the device and follows the advice provided: "Try some relaxation exercises" is the message they receive and they act on it.

[1063] Specific examples

[1064] For example, consider a case where a user types the message "I'm feeling overwhelmed with work." This message is sent to the server via the device. The server receives the message and performs real-time sentiment analysis, identifying the emotion "anxiety." The server then selects "Try some relaxation exercises" as an intervention message corresponding to "anxiety." This message is recorded in the chat history and sent to the device for display to the user. Finally, the user can confirm this message and actually try some relaxation exercises.

[1065] In this way, the system can provide appropriate support in real time according to the user's mental health condition.

[1066] The processing flow will be explained below.

[1067] Step 1:

[1068] A user uses a terminal to enter a message into the system. For example, the user enters, "I'm feeling overwhelmed with work."

[1069] Step 2:

[1070] The device sends the input message to the server via the network. The data is sent in JSON format.

[1071] Step 3:

[1072] The server retrieves the received user message for analysis, and stores it in an internal data structure.

[1073] Step 4:

[1074] The server loads the sentiment analysis model and performs real-time sentiment analysis based on the message. In this process, a text analysis algorithm is used to identify the user's emotional state. For example, the emotion "anxious" is identified.

[1075] Step 5:

[1076] The server uses the results of the emotion analysis to select an appropriate intervention message based on a predefined mapping of emotions and messages. For example, the message "Try some relaxation exercises" is selected as the message corresponding to "anxious."

[1077] Step 6:

[1078] The server records the current conversation as a chat history, including user input, sentiment analysis results, and selected intervening messages, and stores this data in a database.

[1079] Step 7:

[1080] The server sends the selected intervention message back to the terminal via the network. The message is again sent in JSON format.

[1081] Step 8:

[1082] The terminal displays the intervention message received from the server to the user, for example, a message saying "Try some relaxation exercises" is displayed on the user interface.

[1083] Step 9:

[1084] The user acknowledges the displayed intervention message and performs the suggested action (e.g., relaxation exercises).

[1085] This series of steps enables the system to provide users with real-time, emotion-based mental health care.

[1086] Example 1

[1087] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1088] As mental health care has become increasingly important in recent years, there is a demand for easily accessible mental health care systems. However, existing systems have issues such as the time it takes to analyze user input and provide corresponding messages, the often mechanical responses that do not adequately address individual emotional states, and the difficulty in utilizing past interaction history. Another problem is the lack of a means to detect emotional changes early and provide appropriate intervention.

[1089] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1090] In this invention, the server includes a means for transmitting user input to the server via an internet connection, a means for performing real-time emotion analysis based on the received user input, and a means for selecting an intervention message based on the emotion analysis results and recording it in a database. This makes it possible to provide a response to the user's input quickly and according to the individual emotional state. Furthermore, by tracking changes in the user's emotions based on the chat history and enabling early intervention in response to emotional changes, more effective mental health care can be achieved.

[1091] "User input" refers to a text message that a user enters into the system using a terminal.

[1092] An "Internet line" is a communication path for sending and receiving digital data, including Wi-Fi and mobile networks (4G, 5G, etc.).

[1093] "Server" refers to a central computer system that processes, analyzes, and responds to received data.

[1094] "Real-time sentiment analysis" refers to the process of instantly analyzing sentiment and extracting specific emotions based on received user input.

[1095] "Artificial intelligence model" refers to a machine learning algorithm or neural network used for data analysis or prediction, such as a natural language processing model.

[1096] A "database" refers to a system that systematically organizes and stores data so that it can be searched and used later.

[1097] An "intervention message" refers to a message containing advice or suggested actions provided to a user based on the results of sentiment analysis.

[1098] "Chat history" refers to a record of past interactions with a user, and refers to data saved in a format that can be referenced and analyzed later.

[1099] "Changes in emotion" refers to changes in the user's emotional state over time, and is generally evaluated based on the difference from the initial state.

[1100] "Intervention" refers to the action of providing appropriate mental health care or advice based on the results of emotion analysis.

[1101] The system of the present invention aims to provide preventative, affordable, and private mental health care for users. Specific embodiments of the system are described below.

[1102] This system is primarily composed of three components: a server, a terminal, and a user. Users access the system using a terminal (such as a smartphone or PC) and enter information about their own mental health status. The server plays a central role in receiving this input data, analyzing it, and providing a response.

[1103] When a user uses a device to input a message, such as "I'm feeling overwhelmed with work," this input data is sent from the device to a server via an internet connection. The internet connection used here includes communication paths such as Wi-Fi, 4G, and 5G.

[1104] The server decodes the received user input data to analyze it. To do this, it uses natural language processing libraries (e.g., NLTK or spaCy) or generative AI models (e.g., OpenAI's GPT-3). These tools are used to perform real-time sentiment analysis and extract specific emotions (e.g., anxiety, joy, sadness, anger) from the user's input.

[1105] Based on the extracted emotion, the server selects an appropriate intervention message, which is then recorded in a database such as MySQL or PostgreSQL. For example, if the emotion analysis identifies "anxiety," the system might select and record the intervention message "Try some relaxation exercises."

[1106] The server then structures the selected intervention message as an HTTP response and sends it to the terminal over the Internet. The terminal then decodes the received message and displays it on its user interface, providing appropriate advice to the user.

[1107] The user is expected to check the intervention message displayed on the device and follow the advice provided. Specifically, the message "Try some relaxation exercises" encourages the user to perform relaxation exercises.

[1108] The server also records interactions with the user as a chat history, which helps track changes in the user's emotions and is stored in a database for later reference and analysis.

[1109] As a specific example, the following prompt sentence can be input into a generative AI model to provide sentiment analysis and intervention messages:

[1110] User message: "I'm feeling overwhelmed with work."

[1111] Analyze the sentiment of the user message and provide a supportive intervention message.

[1112] Based on this prompt, the generative AI model analyzes the user's emotions and generates an appropriate intervention message.

[1113] In this way, the system can provide appropriate support in real time according to the user's mental health condition.

[1114] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1115] Step 1:

[1116] A user accesses the system using a terminal and enters a message in a text box or chat window. The entered message is text data such as "I'm feeling overwhelmed with work." This input is sent to the next step as is.

[1117] Step 2:

[1118] The terminal sends the entered text data to the server via the Internet. At this time, the data is encoded in JSON format and sent as an HTTP request. The input here is the text message entered by the user, and the output is the encoded HTTP request.

[1119] Step 3:

[1120] The server decodes the received HTTP request and extracts the user's input message. It performs a decoding process to analyze the received data. The input is the encoded HTTP request from the terminal, and the output is the user input in text format.

[1121] Step 4:

[1122] The server performs real-time sentiment analysis based on the extracted user input message. Specifically, it uses natural language processing libraries (e.g., NLTK or spaCy) and generative AI models (e.g., OpenAI's GPT-3). The input for sentiment analysis is the user's text message, and the output is the analyzed emotion label (e.g., anxiety, joy, sadness, anger).

[1123] Step 5:

[1124] The server selects an appropriate intervention message based on the results of the sentiment analysis. It selects the corresponding intervention message from among those pre-stored in a database (e.g., MySQL or PostgreSQL). The input for this task is the sentiment label, and the output is the corresponding intervention message (e.g., "Try some relaxation exercises").

[1125] Step 6:

[1126] The server records the selected intervention message as a chat history in a database. This allows data to be accumulated in a form that can be referenced later. The input here is the intervention message, and the output is the chat history recorded in the database.

[1127] Step 7:

[1128] The server re-encodes the selected intervention message and sends it to the terminal as an HTTP response. The input is the intervention message and the output is the encoded HTTP response.

[1129] Step 8:

[1130] The terminal decodes the received HTTP response and displays the intervention message in a user interface, where the input is the encoded HTTP response and the output is the intervention message to be displayed (e.g., "Try some relaxation exercises").

[1131] Step 9:

[1132] The user checks the intervention message displayed on the device and acts according to the advice provided. Specifically, the user performs relaxation exercises based on the displayed message. The input is the intervention message displayed on the device, and the output is the user's actual behavior.

[1133] In this way, the entire system works to support the user's mental health in real time.

[1134] (Application example 1)

[1135] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1136] There is a need for a system that can detect mental health issues experienced by passengers in autonomous vehicles in real time and provide appropriate support. However, existing systems have difficulty accurately analyzing passenger mental health status and are unable to provide specific intervention measures. In addition, there is a lack of a system in place that allows passengers to receive mental health care in a private space.

[1137] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1138] In this invention, the server includes means for receiving user input, means for performing real-time sentiment analysis based on the received user input, means for selecting an intervention message based on the sentiment analysis result, means for providing the selected intervention message to the user, means for recording a chat history, and means for conducting mental health checks on passengers of the autonomous vehicle while the vehicle is in operation, thereby enabling passengers to receive real-time mental health care while the vehicle is in operation.

[1139] "User input" refers to messages and emotional states that a user inputs through a terminal.

[1140] "Real-time emotion analysis" refers to the process of instantly analyzing the emotional state of a user's input and extracting specific emotions.

[1141] An "intervention message" refers to advice or instructions provided to a user based on the results of sentiment analysis.

[1142] "Chat history" refers to the data that records and saves all conversations between a user and the system.

[1143] An "autonomous vehicle" refers to a vehicle that has the ability to move autonomously without a driver performing any driving operations.

[1144] A "mental health check" refers to a series of processes for diagnosing and assessing a user's emotional and mental state.

[1145] The purpose of the system of the present invention is to provide real-time mental health care for passengers in autonomous vehicles. This system consists of three components: a server, a terminal, and a user. The program processing of this system is explained below in natural language.

[1146] Hardware and software used

[1147] Hardware: On-board computers in autonomous vehicles, passenger smartphones

[1148] Software: Python 3.8+, aiohttp (web framework), TextBlob (sentiment analysis library)

[1149] Specific processing of the system

[1150] Receiving User Input

[1151] The user inputs their emotional state into the system using a device (such as a car screen or a smartphone). For example, the user might input, "I'm tired from work today."

[1152] Sending input data

[1153] The terminal transmits this input data via the network to the in-vehicle server, which receives the message from the user and prepares for analysis.

[1154] sentiment analysis

[1155] The server performs real-time sentiment analysis on the received user input, using the TextBlob library to identify specific sentiments (e.g., positive, negative, neutral) from the user input.

[1156] Intervention message selection

[1157] The server selects an appropriate intervention message based on the results of the sentiment analysis, for example, if the sentiment is identified as "negative," it selects the intervention message "Try some relaxation exercises."

[1158] Chat history recording

[1159] The server records interactions with users as a chat history, allowing past conversation data to be accumulated and referenced later.

[1160] Sending a response message

[1161] The selected intervention message is sent again via the network to the terminal, which displays the received message to the user.

[1162] Specific examples

[1163] For example, consider a case where a passenger types a message such as "I'm tired from work today." This message is sent to the server through the terminal. The server receives the message and performs real-time sentiment analysis, identifying the sentiment as "negative." The server then selects "Try some relaxation exercises," records this as a message history, and sends it to the terminal for display to the user. The user can confirm this message and actually try some relaxation exercises.

[1164] Prompt Sentence Examples

[1165] "Passengers input their emotional state via their smartphone or on-board screen."

[1166] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1167] Step 1:

[1168] Receiving User Input

[1169] The user inputs their emotional state using a device (such as a car screen or a smartphone). For example, the user might input, "I'm tired from work today." This input data is recognized by the device in text format.

[1170] Step 2:

[1171] Sending input data

[1172] The terminal transmits user input data to the in-vehicle server via a network. The input data is text data indicating the user's emotional state and is transferred from the terminal to the server.

[1173] Step 3:

[1174] Performing sentiment analysis

[1175] The server analyzes the received user input data. Specifically, it analyzes the sentiment of the input text using the TextBlob library. For example, the text "Work today has left me exhausted" is identified as expressing a negative sentiment (negative polarity).

[1176] Step 4:

[1177] Intervention message generation

[1178] The server selects an appropriate intervention message based on the result of the sentiment analysis. For example, if the sentiment is identified as "negative," it generates an intervention message such as "Try some relaxation exercises." This message is selected from fixed messages based on the result of the sentiment analysis.

[1179] Step 5:

[1180] Chat history recording

[1181] The server records the user's interactions as a chat history. This is the process of storing the user's input data and the corresponding generated intervention messages in a database. For example, the exchange "I'm tired from work today" and "Try some exercises to relax" is recorded.

[1182] Step 6:

[1183] Sending a response message

[1184] The server transmits the generated intervention message to the terminal again via the network, for example, a message saying "Try some relaxing exercises" is sent to the terminal, which then displays it to the user.

[1185] Step 7:

[1186] User perception and behavior

[1187] The user checks the intervention message displayed on the terminal, for example, "Try some relaxation exercises," and takes action according to the advice.

[1188] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1189] The system of the present invention aims to provide preventative, affordable, and private mental health care for users. In particular, by adding an emotion engine that recognizes the user's emotions, more accurate emotion analysis and intervention can be achieved. The overall configuration and specific processing of this system are described below.

[1190] System Overview

[1191] This system consists of three components: a server, a terminal, and a user. Users access the system using their terminal and input information about their mental health status. The server plays a central role in receiving this input data, analyzing it, and providing a response. The emotion engine is a module that further enhances emotion analysis.

[1192] Receiving User Input

[1193] A user inputs a message into the system using a device (e.g., a smartphone or a PC). This input can include not only text input, but also voice input and input using facial recognition. For example, a user inputs, "I'm feeling overwhelmed with work."

[1194] Sending input data

[1195] The device sends this input data to the server via the network. The data is sent in JSON format. Audio data and image data are also sent in the same way.

[1196] Performing sentiment analysis

[1197] The server receives the received user input data and performs real-time emotion analysis using the emotion engine. The emotion engine analyzes the following elements:

[1198] Text analysis: Analyze the content of a user's messages to identify their emotional state.

[1199] Voice analysis: Analyzes the tone and pitch of a user's voice to identify their emotional state.

[1200] Facial Expression Analysis: Analyzes facial expressions from the user's facial recognition data to identify their emotional state.

[1201] For example, if a user enters the text "I'm feeling overwhelmed with work," and has a stressed facial expression, the emotion "anxious" is identified.

[1202] Intervention message selection

[1203] The server selects an appropriate intervention message based on the result of the sentiment analysis, which is chosen based on a predefined emotion-message mapping.

[1204] For example, the message selected to correspond to "anxious" is "Try some relaxation exercises."

[1205] Chat history recording

[1206] The server records the user's interactions as a chat history. The recorded data includes the user's input, the results of sentiment analysis, and selected intervention messages. This data is stored in a database.

[1207] Sending a response message

[1208] The server then sends the selected intervention message back to the terminal via the network, again in JSON format.

[1209] Displaying a response message

[1210] The terminal displays the intervention message received from the server to the user, for example, a message saying "Try some relaxation exercises" is displayed on the user interface.

[1211] User perception and behavior

[1212] The user checks the intervention message displayed on the terminal and performs the suggested action (e.g., relaxation exercises).

[1213] Specific examples

[1214] For example, consider a case where a user types the message "I'm feeling overwhelmed with work," speaking in a high-pitched voice and displaying facial expressions that indicate stress. This message and additional data are sent to the server via the device. The server performs emotion analysis using text, voice, and facial expression analysis modules, and identifies the emotion "anxious." The server then selects "Try some relaxation exercises" as an intervention message corresponding to "anxious" and records it in the chat history. The server then sends this intervention message to the device, which then displays it to the user. Finally, the user can confirm the message and actually try some relaxation exercises.

[1215] This system can respond to the user's diverse emotional states in real time and provide more personalized mental health care.

[1216] The processing flow will be explained below.

[1217] Step 1:

[1218] A user uses a terminal to enter a message into the system. For example, the user enters, "I'm feeling overwhelmed with work."

[1219] Step 2:

[1220] The device sends the input message along with voice and facial expression data to the server in JSON format or another suitable format.

[1221] Step 3:

[1222] The server takes the received user input data for analysis and stores it in an internal data structure.

[1223] Step 4:

[1224] The server loads the emotion engine and performs real-time emotion analysis using text, voice, and facial expression data. The emotion analysis is performed as follows:

[1225] The server performs text analysis to identify the emotion from the content of the input message.

[1226] The server performs voice analysis and identifies emotions from the tone and pitch of the user's voice.

[1227] The server performs facial expression analysis and identifies emotions from the user's facial recognition data.

[1228] Step 5:

[1229] The server aggregates the results of the emotion analysis and identifies an overall emotional state. For example, if the results of text, voice, and facial expression analysis all indicate "anxious," the server identifies the overall emotion as "anxious."

[1230] Step 6:

[1231] The server selects an appropriate intervention message based on the results of emotion analysis. For example, it selects "Try some relaxation exercises" as the message for "anxious" based on predefined emotion-message mapping.

[1232] Step 7:

[1233] The server records the current conversation as a chat history, including user input, sentiment analysis results, and selected intervening messages, and stores the recorded data in a database.

[1234] Step 8:

[1235] The server sends the selected intervention message back to the terminal over the network, again in JSON format or another suitable format.

[1236] Step 9:

[1237] The terminal displays the intervention message received from the server to the user, for example, a message saying "Try some relaxation exercises" is displayed on the user interface.

[1238] Step 10:

[1239] The user acknowledges the displayed intervention message and performs the suggested action (e.g., relaxation exercises).

[1240] This series of steps enables the system to more accurately grasp the user's mental health status through multifaceted emotion analysis and provide appropriate real-time mental health care.

[1241] Example 2

[1242] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1243] Mental health issues are serious in modern society, and many users are experiencing increasing stress and anxiety. However, systems that provide affordable and private mental health care are not yet widespread. Another challenge is the lack of real-time emotional analysis of users and appropriate interventions based on that analysis.

[1244] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1245] In this invention, the server includes means for receiving user input, means for transmitting the received user input to the server via a network, means for performing real-time emotion analysis based on the received user input, means for selecting an intervention message based on the emotion analysis result, means for providing the selected intervention message to the user, and means for recording the chat history, thereby enabling the user to understand their own emotional state in real time and receive appropriate mental health care.

[1246] A "means for receiving user input" is a device or software that provides an interface for a user to enter messages or data into the system.

[1247] The "means for transmitting received user input to a server over a network" is a device or software that communicates user input to a server over the Internet or other communications network.

[1248] "Means for performing real-time sentiment analysis based on received user input" refers to analytical equipment or software for analyzing received data and determining the emotional state of the user in real time.

[1249] The "means for selecting an intervention message based on the result of sentiment analysis" is a device or software that selects the most appropriate advice or message based on the result of sentiment analysis.

[1250] The "means for providing a selected intervention message to a user" refers to a device or software that displays or notifies a user of a selected intervention message.

[1251] The "means for recording chat history" is a database or storage device for storing interactions with users and analysis results.

[1252] A "means for providing access to a professional" is a device or software that provides contact and access for a user to obtain professional advice or counseling as needed.

[1253] The "means for tracking emotional changes and providing early intervention" is a device or software that monitors changes in the user's emotional state and takes prompt action if necessary.

[1254] MODE FOR CARRYING OUT THE INVENTION

[1255] The system of the present invention aims to provide preventative, affordable, and private mental health care for users. In particular, by adding an emotion engine that recognizes the user's emotions, more accurate emotion analysis and intervention can be achieved. The overall configuration and specific processing of this system are described below.

[1256] Receiving User Input

[1257] Users input messages into the system using devices such as smartphones or PCs. This input can be text input, voice input, or input using facial recognition. For example, a user might input "I'm feeling overwhelmed with work." This input is done through the interface on the device.

[1258] Sending input data

[1259] The device sends input data acquired from the user to the server in JSON format. Voice data and image data are also sent in the same way. Specifically, the API on the device generates a JSON-formatted data packet and sends it through the network interface.

[1260] Performing sentiment analysis

[1261] The server first stores the received user input data in a database, and then performs real-time emotion analysis using the emotion engine. The emotion engine consists of the following analysis modules:

[1262] Text Analysis Module: Uses a natural language processing (NLP) engine to analyze text data and identify emotional states.

[1263] Audio analysis module: Analyzes the tone and pitch of audio data.

[1264] Facial Expression Analysis Module: Analyzes facial expressions from image data by using machine learning algorithms to extract visual features and infer emotions.

[1265] For example, if a user enters the text "I'm feeling overwhelmed with work," and there is a high-pitched voice tone indicative of stress and facial recognition data, the emotion engine will identify the emotion "anxious."

[1266] Selection of intervention messages

[1267] The server selects an appropriate intervention message based on the emotion analysis results. This message is selected from a predefined emotion-message mapping table. Specifically, the message "Try some relaxation exercises" is selected as the message corresponding to the emotion "anxious."

[1268] Chat history recording

[1269] The server records user interactions as chat history in a database. The recorded data includes user input, sentiment analysis results, and selected intervention messages. Specifically, it performs appropriate escaping and executes INSERT queries in the database using techniques to prevent SQL injection.

[1270] Sending a response message

[1271] The server encodes the selected intervention message in JSON format and sends it to the terminal. Specifically, it sends the packet again using the network interface.

[1272] Displaying a response message

[1273] The terminal decodes the intervention message received from the server and displays it to the user. The user interface (UI) displays the message content. For example, the UI displays a message such as "Try some relaxation exercises."

[1274] User perception and behavior

[1275] The user checks the intervention message displayed on the device and actually performs the suggested action (e.g., relaxation exercises), thereby improving their mental health.

[1276] Specific examples

[1277] As a specific example, if a user inputs the message "I'm feeling overwhelmed with work," with a high-pitched voice tone and facial expression indicating stress, this message and additional data are sent via the device to the server. The server receives this message and sequentially invokes the text analysis module, voice analysis module, and facial expression analysis module to identify the emotion "anxious." The server then selects an intervention message, "Try some relaxation exercises," from the emotion-message mapping table and sends it to the device. The device receives this and displays it to the user. The user confirms this message, tries to do some relaxation exercises, and enters the results as feedback into the system, starting the next cycle.

[1278] Prompt Sentence Examples

[1279] Please explain the operation of your system to support users' mental health care, breaking it down into specific steps. Please explain the operation of each step in detail, paying particular attention to the specific processing involved in performing analysis and sending and receiving data.

[1280] In this way, the present invention makes it possible to provide personalized mental health care according to the user's emotional state.

[1281] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1282] Step 1:

[1283] A user inputs a message into the system using a device (e.g., a smartphone or a PC). This input can include not only text input, but also voice input and input using facial recognition. For example, a user inputs the text "I'm feeling overwhelmed with work." Voice input and facial recognition data are also processed simultaneously.

[1284] Input: User text, voice, and facial recognition data

[1285] Output: Getting input data via the terminal

[1286] Step 2:

[1287] The device converts the input data received from the user into JSON format. Specifically, it encodes text data, compresses audio data, and encodes image data. This operation is performed through the API on the device.

[1288] Input: User input data (text, voice, facial recognition data)

[1289] Output: JSON formatted data packet

[1290] Step 3:

[1291] The device sends the encoded JSON-formatted data packets through the network interface. Specifically, the device sends the data to the server using a network protocol, using a secure communication protocol such as HTTPS.

[1292] Input: JSON formatted data packet

[1293] Output: Send data to the server

[1294] Step 4:

[1295] The server decodes the received JSON-formatted user data and stores it in a database. Specifically, it executes an INSERT query using a database management system (DBMS) on the server and stores the user data in the database.

[1296] Input: User data in JSON format

[1297] Output: Data storage in database

[1298] Step 5:

[1299] The server retrieves the stored user data and performs analysis using the sentiment analysis engine. Specifically, the following analysis modules are called in sequence:

[1300] Text Analysis Module: Uses a natural language processing engine to identify emotional states from text data.

[1301] Voice Analysis Module: Analyzes the tone and pitch of voice data to identify emotional states.

[1302] Facial Expression Analysis Module: Analyzes facial expressions from image data to identify emotional states.

[1303] Input: User data retrieved from the database

[1304] Output: Emotion analysis results (emotional state)

[1305] Step 6:

[1306] The server selects an appropriate intervention message based on the results of emotion analysis. Specifically, it references a mapping table of emotions and messages to select the most appropriate message. For example, if the emotion analysis result is "anxious," the server selects the message "Try some relaxation exercises."

[1307] Input: Emotion analysis results (emotional state)

[1308] Output: Intervention message

[1309] Step 7:

[1310] The server encodes the selected intervention message in JSON format and sends it to the terminal. Specifically, the server generates packets using a network protocol and transmits the data to the terminal via secure communication such as HTTPS.

[1311] Input: Intervention message

[1312] Output: Intervention message packet in JSON format

[1313] Step 8:

[1314] The device decodes the JSON-formatted intervention message received from the server and displays it on the user interface (UI). Specifically, the device's decoding process and UI rendering engine work together to visually display the message. For example, a message such as "Try some relaxation exercises" is displayed on the UI.

[1315] Input: JSON formatted intervention message packet

[1316] Output: Intervention message displayed to the user

[1317] Step 9:

[1318] The user confirms the displayed intervention message and performs the suggested action (e.g., relaxation exercises). The user performs the action and inputs the results as feedback to the system, providing new data for the next analysis cycle.

[1319] Input: Intervention message displayed to the user

[1320] Output: User actions and feedback

[1321] (Application example 2)

[1322] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1323] In modern society, the number of individuals experiencing mental stress and anxiety is increasing, creating a demand for prompt and effective mental health care. However, many mental health care systems are expensive and lack privacy. It is necessary to solve these problems and provide a preventive mental health care system that is easy for users to use.

[1324] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving user input, means for performing real-time emotion analysis based on the received user input, means for selecting an intervention message based on the emotion analysis result, means for providing the selected intervention message to the user, means for recording the chat history, and means for recommending mental health care content based on the emotion analysis result. This makes it possible to provide appropriate intervention and mental health care content according to the user's emotional state.

[1325] "User input" refers to information provided by a system user through a terminal, and may be in the form of text, audio, image, or the like.

[1326] "Real-time" refers to processing that occurs immediately without delay, and refers to data analysis and response in real time.

[1327] "Sentiment analysis" refers to the process of analyzing user-input data to identify a user's emotional state, and includes text analysis, speech analysis, and facial recognition analysis.

[1328] An "intervention message" refers to a message of recommended actions or advice provided to a user based on the results of sentiment analysis.

[1329] "Chat history" refers to data that records the content of conversations between users and the system, and is used for later reference and analysis.

[1330] "Mental health care content" refers to information materials such as video, audio, and text that are intended to support the mental health of users.

[1331] The system of the present invention provides preventative mental health care for users and is composed of a server, a terminal, and a user. In particular, by using an emotion analysis engine, the system can analyze the user's real-time emotional state and recommend appropriate intervention messages and mental health care content.

[1332] System configuration

[1333] The system consists of the following:

[1334] Server: Uses Flask (a Python web application framework) to receive, process, and control user input data. EmotionEngine (a hypothetical emotion analysis engine library) is used for emotion analysis, and ContentRecommender (a hypothetical content recommendation library) is used to recommend mental health care content.

[1335] Terminal: A device used by a user, such as a smartphone or head-mounted display (HMD). These devices are responsible for acquiring user input (text, voice, facial recognition data, etc.) and sending it to a server.

[1336] User: An individual who uses the system. They input their emotional state using a terminal.

[1337] System Operation

[1338] 1. Receiving user input

[1339] Users input their emotional state using a smartphone or HMD. This input can be in various forms, such as text, voice, or facial recognition data. For example, when a user types a message like "I'm overwhelmed by work," their tone of voice and facial expressions, which indicate stress, are also captured.

[1340] 2. Emotion analysis by the server

[1341] The server analyzes the user input data received from the device. During this analysis, the Emotion Engine is used to identify the user's emotional state through text analysis, voice analysis, and facial recognition analysis. For example, if the user's voice tone acquired along with the text message is high and the user's facial expression indicates stress, the emotional state of "anxiety" is identified.

[1342] 3. Recommending mental health care content

[1343] Based on the results of the emotion analysis, the server uses ContentRecommender to recommend appropriate mental health care content. For example, for the emotional state of "anxiety," content such as relaxing videos and calming music is recommended. This content is sent to the user's device and displayed.

[1344] 4. Chat history recording

[1345] All user interactions are recorded by the server as chat history, which is stored in a database for later analysis and intervention.

[1346] Specific examples

[1347] For example, if a user types "I'm overwhelmed by work," and their voice tone is high and their facial expression indicates stress, this data is sent from the device to the server. The server uses the Emotion Engine to analyze the text, voice, and facial expression data to identify the emotional state of "anxiety." It then uses the Content Recommender to recommend appropriate mental health content, such as "exercise videos to help you relax." This content is then sent to the user's device and displayed.

[1348] Example prompts to input to a generative AI model:

[1349] "If a user types in 'I'm overwhelmed at work,' and has a tone of voice and facial expression that indicates stress, what kind of mental health care content would be best?"

[1350] This system enables prompt and appropriate responses to the user's emotional state, providing personalized mental health care.

[1351] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1352] Step 1:

[1353] Users input their emotional state using a smartphone or head-mounted display (HMD). This input can include text, voice, or facial recognition data. For example, a message such as "I'm overwhelmed by work," a voice with a stressful tone, or image data of a stressful facial expression can be input. This input data is sent from the device to the server in JSON format.

[1354] Step 2:

[1355] The server receives the user's input data sent from the device. This input data includes text, voice, and facial recognition data. The received data is first temporarily stored in a database. This stored data is then extracted for analysis and provided to the emotion analysis engine (EmotionEngine).

[1356] Step 3:

[1357] The server uses the Emotion Engine to analyze the received user data. This involves text analysis, voice analysis, and facial recognition analysis to identify the user's emotional state. For example, the text message "I'm overwhelmed by work," high-pitched voice, and facial expression data indicating stress are combined to identify the emotional state of "anxiety."

[1358] Step 4:

[1359] The server selects an appropriate intervention message based on the emotion analysis results. This selection is performed using a content recommendation engine (ContentRecommender). Specifically, for the emotion "anxiety," it recommends mental health care content, including exercise videos and calming music for relaxation. This recommended intervention message is generated in JSON format.

[1360] Step 5:

[1361] The server sends the selected intervention message and the emotion analysis results to the user's device in JSON format as mental health care content. This sent data includes the emotion analysis results (e.g., "anxiety") and the recommended intervention message (e.g., "exercise video for relaxation").

[1362] Step 6:

[1363] The device displays the intervention message received from the server to the user. Specifically, a message such as "Please watch an exercise video to help you relax" is displayed on the user interface. The device also plays the recommended mental health care content (e.g., a video).

[1364] Step 7:

[1365] The server records the user's interactions and changes in emotional state and saves them as a chat history, including the user's input data, emotion analysis results, and recommended intervention messages. This history data is used for future analysis and further service improvement.

[1366] Through the specific actions of each step, users can enjoy personalized mental health care. The system supports users' mental health by quickly and accurately analyzing their emotional state and providing appropriate interventions.

[1367] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1368] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1369] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1370] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1371] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1372] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1373] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1374] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1375] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1376] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1377] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1378] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1379] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1380] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1381] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1382] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1383] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1384] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1385] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1386] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1387] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1388] The following is further disclosed regarding the above embodiment.

[1389] (Claim 1)

[1390] means for receiving user input;

[1391] means for performing real-time sentiment analysis based on received user input;

[1392] means for selecting an intervention message based on the sentiment analysis results;

[1393] means for providing a selected intervention message to a user;

[1394] a means for recording chat history;

[1395] A system including:

[1396] (Claim 2)

[1397] 10. The system of claim 1, further comprising means for providing access to an expert based on the sentiment analysis results.

[1398] (Claim 3)

[1399] 10. The system of claim 1, further comprising means for tracking changes in a user's emotions and providing early intervention.

[1400] "Example 1"

[1401] (Claim 1)

[1402] means for receiving user input;

[1403] means for transmitting the received user input to a server over an internet connection;

[1404] means for performing real-time sentiment analysis based on received user input;

[1405] means for selecting and recording in a database an intervention message based on the sentiment analysis results;

[1406] means for transmitting the selected intervention message to the user terminal;

[1407] a means for recording chat history;

[1408] A system including:

[1409] (Claim 2)

[1410] 10. The system of claim 1, further comprising means for using an artificial intelligence model for sentiment analysis to generate the selected intervention message.

[1411] (Claim 3)

[1412] 10. The system of claim 1, further comprising means for tracking changes in a user's emotions based on chat history and for providing new interventions at an early stage in response to the changes in emotions.

[1413] "Application Example 1"

[1414] (Claim 1)

[1415] means for receiving user input;

[1416] means for performing real-time sentiment analysis based on received user input;

[1417] means for selecting an intervention message based on the sentiment analysis results;

[1418] means for providing a selected intervention message to a user;

[1419] a means for recording chat history;

[1420] A means of conducting mental health checks on passengers of autonomous vehicles while they are in operation; and

[1421] A system including:

[1422] (Claim 2)

[1423] 10. The system of claim 1, further comprising means for providing access to an expert based on the sentiment analysis results.

[1424] (Claim 3)

[1425] 10. The system of claim 1, further comprising means for tracking changes in a user's emotions and providing early intervention.

[1426] "Example 2: Combining Emotion Engines"

[1427] (Claim 1)

[1428] means for receiving user input;

[1429] means for transmitting the received user input to a server over a network;

[1430] means for performing real-time sentiment analysis based on received user input;

[1431] means for selecting an intervention message based on the sentiment analysis result;

[1432] means for providing a selected intervention message to a user;

[1433] a means for recording chat history;

[1434] A system including:

[1435] (Claim 2)

[1436] 10. The system of claim 1, further comprising means for providing access to an expert based on the sentiment analysis results.

[1437] (Claim 3)

[1438] 10. The system of claim 1, further comprising means for tracking changes in a user's emotions and providing early intervention.

[1439] "Application example 2 when combining emotion engines"

[1440] (Claim 1)

[1441] means for receiving user input;

[1442] means for performing real-time sentiment analysis based on received user input;

[1443] means for selecting an intervention message based on the sentiment analysis results;

[1444] means for providing a selected intervention message to a user;

[1445] a means for recording chat history;

[1446] A means for recommending mental health care content based on the results of sentiment analysis;

[1447] A system including:

[1448] (Claim 2)

[1449] 10. The system of claim 1, further comprising means for providing access to an expert based on the sentiment analysis results.

[1450] (Claim 3)

[1451] 10. The system of claim 1, further comprising means for tracking changes in a user's emotions and providing early intervention. [Explanation of symbols]

[1452] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving user input; means for performing real-time sentiment analysis based on received user input; means for selecting an intervention message based on the sentiment analysis results; means for providing a selected intervention message to a user; a means for recording chat history; A system including:

2. The system of claim 1 , further comprising means for providing access to an expert based on the sentiment analysis results.

3. The system of claim 1 further comprising means for tracking changes in a user's emotions and providing early intervention.

Images