system

Abstract

We provide the system. [Solution] A means of receiving user input information and recording it as profile data, A means of interacting with users, generating text data, and performing sentiment analysis, A method for scoring the user's mental state based on the analysis results and visualizing it over time, A means for receiving biometric data from multiple external devices, integrating it, and correcting the mental score, A means of providing users with appropriate advice and action suggestions, A system that includes this.

Description

Technical Field

[0004] , , , ,

[0005] , , , , , ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Individuals with mental illnesses and their families struggle to manage symptom waves and recurrence risks. There is also a problem that it is difficult for family members living in distant locations to grasp the situation and provide appropriate support. Therefore, there is a need for a system that can grasp individual mental states in real time and provide appropriate advice and support.

Means for Solving the Problems

[0005] This invention provides a system that records user input information, performs emotional analysis based on data obtained through dialogue, and scores the user's mental state. Furthermore, it integrates biometric data acquired from multiple external devices to correct the score and provides appropriate advice and behavioral suggestions. It also includes a function to share the mental score with authorized stakeholders and automatically adjusts the support plan based on score changes, thereby enhancing mental healthcare.

[0006] A "user" refers to an individual who uses this system to receive mental health care.

[0007] "Input information" refers to data about the user's basic profile and status.

[0008] "Profile data" refers to personal data, including the user's basic information and past mental health data.

[0009] "Dialogue" refers to the process by which a user and a system communicate using language or text.

[0010] "Text data" refers to data that has been organized into a sentence structure obtained from interactions with users.

[0011] "Sentiment analysis" refers to a technique that analyzes text data to infer a user's emotional state.

[0012] "Mental state" refers to the psychological and emotional state that the user is experiencing.

[0013] "Scoring" refers to the process of quantifying and evaluating a user's mental state.

[0014] "External devices" refer to wearable devices and other devices used to measure the user's biometric information.

[0015] "Biometric data" refers to information representing the physical state of a user, and refers to data including heart rate and sleep patterns.

[0016] "Advice" refers to advice and suggestions given based on the mental state of the user.

[0017] "Behavior proposal" refers to specific behaviors recommended for the user to improve mental health.

[0018] "Related person" refers to family members or supporters whom the user has permitted to share the mental score.

[0019] "Support plan" refers to a continuous support plan formulated to support the mental health of the user.

Brief Explanation of Drawings

[0020] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9]Shows an emotion map where multiple emotions are mapped. [Figure 10] Shows an emotion map where multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Modes for Carrying Out the Invention

[0021] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a processor with a reference number (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0024] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0025] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0026] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0028] [First Embodiment]

[0029] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0030] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0033] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0036] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0040] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0041] This invention provides an integrated system to support users' mental health, and is primarily implemented using smartphones and wearable devices. Users first download the application to their smartphone and create a personal account by entering basic profile information. This prepares the system to provide mental healthcare tailored to the user.

[0042] Users can use the app on a daily basis to interact with AI using natural language. The app, running on the device, accepts voice or text input, which is processed as data to understand the user's emotions. The server analyzes the user's statements and applies emotion recognition algorithms to quantify the user's mental state. This scored data visualizes the user's mental health status, and the user is notified if there are any changes.

[0043] In addition, the terminal connects with a wearable device attached as an external device. This allows it to acquire biometric data such as heart rate, exercise level, and sleep patterns, and transmit it to a server. This biometric data is used to adjust the user's mental score, enabling highly accurate state analysis.

[0044] For example, when user B feels stressed, the app recognizes that emotion and reflects it in the score. If the heart rate data is higher than normal, the system suggests a specific action to the user, such as, "Try breathing exercises to relax." In this way, the system provides customized mental support based on the user's real-time state.

[0045] Furthermore, because users can share their scores with authorized family members or healthcare providers, their mental health status can be monitored remotely, allowing for the provision of necessary support. This feature enables more comprehensive mental healthcare.

[0046] The following describes the processing flow.

[0047] Step 1:

[0048] The user installs the app and enters basic profile information, including age, gender, and mental health history.

[0049] Step 2:

[0050] The terminal sends user input information to the server, which then registers it in a database. This creates an individual user profile.

[0051] Step 3:

[0052] The user launches the app and begins interacting with the AI. The device receives the user's responses via voice or text input.

[0053] Step 4:

[0054] The terminal converts the user's speech into text data and sends it to the server. The server then performs natural language processing on this text data and conducts sentiment analysis.

[0055] Step 5:

[0056] The server quantifies the user's emotional state and records it in a database as a mental score. This allows for the accumulation of time-series data on the user's mental state.

[0057] Step 6:

[0058] The device connects with a wearable device to acquire biometric data (heart rate, sleep patterns, etc.). This data is also sent to the server.

[0059] Step 7:

[0060] The server uses biometric data to correct the mental score and perform a highly accurate analysis of the user's state. The score is then updated based on these results.

[0061] Step 8:

[0062] The device displays the user's latest mental score and provides specific advice. For example, it might suggest breathing exercises or physical exercises for relaxation.

[0063] Step 9:

[0064] If the user consents, the server will share the updated mental score with authorized stakeholders. This allows third parties to understand the user's condition and provide support as needed.

[0065] Step 10:

[0066] The server compares the data with past data and automatically adjusts the support plan if necessary. The user is notified of these adjustments via their terminal.

[0067] (Example 1)

[0068] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0069] In modern society, properly managing an individual's mental health is extremely important. However, conventional methods make it difficult to grasp an individual's mental state in real time and objectively, which hinders the implementation of appropriate measures. Therefore, more accurate assessment of mental state and appropriate support are needed.

[0070] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0071] In this invention, the server includes means for receiving user input information and recording it as profile information, means for interacting with the user, generating text information, performing sentiment analysis using language processing technology, and means for quantifying the user's mental state based on the analysis results and displaying it over time. This enables real-time, personalized mental health support for each individual user.

[0072] A "user" is someone who participates in this system, inputs their personal profile information, and has their mental state evaluated through dialogue.

[0073] "Profile information" refers to basic personal information provided by the user and is used as foundational data for mental health support.

[0074] "Text information" refers to linguistic data generated from user interactions and is used for sentiment analysis.

[0075] "Language processing technology" refers to methods for analyzing natural language and is used to evaluate a user's emotional state.

[0076] "Mental state" refers to the state of a user's psychological health and is quantified through emotion analysis.

[0077] "Biometric information" refers to data about the user's physical activity and physiological indicators, obtained from measuring devices, and used to correct mental scores.

[0078] The "mental score" is a numerical representation of the user's mental state, and it has the characteristic of changing over time.

[0079] "Stakeholders" are individuals or organizations that users have given permission to share their mental health scores with, and who play a role in supporting the users' mental well-being.

[0080] The system for implementing this invention consists of a user, a terminal, and a server. First, the user downloads a dedicated application to a terminal such as a smartphone and creates a personal account by entering basic profile information. This application has the function of receiving the user's input information and recording it as profile information. The terminal interacts with the user via voice or text and generates text information. The generated text information is sent to the server, where sentiment analysis is performed using language processing technology. In this process, specific natural language processing libraries and generative AI models can be used.

[0081] Furthermore, the device works in conjunction with a wearable device to acquire biometric information such as the user's heart rate, activity level, and sleep patterns. This biometric information is sent to a server and used to correct the mental score calculated based on the results of the emotion analysis. The server quantifies the user's mental state through comprehensive data analysis and visualizes this over time.

[0082] Based on the analysis results, the server can provide users with personalized guidance and behavioral suggestions via their devices. For example, if a user shows a high stress level, it can offer specific prompts such as, "Try breathing exercises to relax." Users can also share their generated mental health score with family members, healthcare professionals, and other relevant parties, thereby facilitating mutual support.

[0083] A concrete example of a prompt message might be, "Analyze your recent stress levels and suggest relaxation methods considering your heart rate data." In this way, the system can provide real-time, personalized support to improve the user's psychological well-being.

[0084] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0085] Step 1:

[0086] Users download the application to their smartphones and enter basic profile information. This information includes name, date of birth, average heart rate, etc., and is stored on the device. The device then transfers this information to a server to create the user's personal profile. This profile serves as the basis for personalized mental health care data.

[0087] Step 2:

[0088] Users use the app daily, inputting their emotions and thoughts via voice or text. For example, they might say, "I'm feeling stressed at work." The device converts the input information into text data and sends it to the server. The server receives text data from the user and uses natural language processing techniques to perform sentiment analysis based on this data. The output is analyzed data indicating the user's emotional state.

[0089] Step 3:

[0090] The server analyzes text data and performs sentiment analysis using a generative AI model. It applies an algorithm that classifies the data into sentiment categories such as positive, negative, and neutral. The server's output is a mental score, quantified based on the sentiment analysis. This score provides information that visualizes the user's mental state over time.

[0091] Step 4:

[0092] The device works in conjunction with wearable devices to periodically acquire biometric information such as heart rate, activity level, and sleep patterns. For example, if a user is wearing a fitness tracker, this data is sent to the device in real time. The device then sends this biometric information to a server. Based on the biometric information as input, the server corrects the mental score as output, enabling a more accurate state analysis.

[0093] Step 5:

[0094] The server analyzes the user's mental health status with high accuracy based on integrated data. If there is a significant change in the quantified mental score, the server automatically sends a notification to the user. For example, it generates a specific prompt and makes a suggestion to the user, such as, "Your stress level is high, try some relaxation techniques." This allows for immediate action.

[0095] Step 6:

[0096] Users can, if they wish, share their generated mental health score with family members or healthcare providers. The device provides the functionality to send the score to designated recipients, allowing them to remotely support the user's mental health. In this way, the system as a whole can provide highly personalized mental health support to the user.

[0097] (Application Example 1)

[0098] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0099] In modern society, managing individual mental health is a crucial issue, but it is not always easy for users to accurately understand their own mental state and improve it in an appropriate way. Furthermore, when remote support is needed, the means of sharing the situation with relevant parties in real time and proposing effective support plans are limited. In addition, mental healthcare support in virtual environments is still under development, and there are not enough mechanisms to provide customized experiences.

[0100] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0101] In this invention, the server includes means for receiving user input information and recording it as profile data; means for interacting with the user, generating text data, and performing sentiment analysis; means for scoring the user's mental state based on the analysis results and visualizing it in a time series; means for receiving biometric data from multiple external devices and integrating it to correct the mental score; means for providing the user with appropriate advice and action suggestions; and means for presenting a program based on the user's mental state in a virtual environment. This enables the user to understand their own mental health with high accuracy and in real time and receive support. In addition, they can effectively receive mental care through a customized experience in a virtual environment.

[0102] "User input information" refers to the individual data that users provide to the system, and is used for profile creation and individual mental health assessments.

[0103] "Profile data" is a dataset that compiles the individual attributes and characteristics of each user, and serves as the foundation for providing mental healthcare tailored to individual needs.

[0104] "Sentiment analysis" is the process of quantifying or visualizing emotional and mood states based on data collected from users.

[0105] "Scoring mental state" is a method of quantitatively measuring a user's mental health level based on the results of emotional analysis.

[0106] "Visualizing over time" is an approach that visually represents changes in mental scores over a certain period of time, making them easier for users to understand.

[0107] "Biometric data" refers to data related to a user's physical health status, including information such as heart rate and exercise level.

[0108] "External devices" refer to hardware used to acquire a user's biometric data, such as smartphones and wearable devices.

[0109] "Adjusting the mental score" is the process of using biometric data to adjust the score in order to more accurately reflect the user's mental state.

[0110] "Appropriate advice and action suggestions" refer to recommended actions and advice presented by the system based on the user's mental state, with the aim of improvement or maintenance.

[0111] A "virtual environment" is a digital space created using virtual reality or augmented reality technologies, allowing users to have an immersive experience.

[0112] "Presenting a program" means providing users with customized experiences or sessions that are tailored to their current mental state.

[0113] To implement this invention, the user must first install a dedicated application on their smartphone or virtual reality device. This application accepts user profile data input and has the function of setting up a personal account. The user's smartphone or VR device connects with a wearable device using Bluetooth or Wi-Fi to acquire biometric data (heart rate, activity level, etc.) in real time.

[0114] The device acquires voice and text input to analyze the user's emotional state and sends this data to a server. The server inputs the collected data into a generative AI model to perform emotion recognition. The software used here includes TENSORFLOW®, which is suitable for data analysis. This analysis generates a user's mental score, and its changes over time are visualized.

[0115] The server generates appropriate advice and action suggestions based on the user's mental score and provides feedback to the user. This feedback is presented as a program experience within an immersive virtual environment. The virtual environment is built using the Unity engine and promotes mental care by providing a customized experience based on the user's mental state.

[0116] Furthermore, users can share this mental score with multiple authorized stakeholders. This feature allows, for example, medical professionals to remotely monitor a user's health and provide necessary support. Specifically, if a user is detected as being in a high-stress state, the system can suggest actions such as "Try a meditation session in a virtual environment to relax."

[0117] An example of a prompt might be: "Suggest a virtual reality experience to improve the user's mental health. Based on the user's stress level, provide an appropriate relaxation program."

[0118] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0119] Step 1:

[0120] The user installs and launches a dedicated application on their smartphone or VR device. First, the user enters their profile information and sets up their account. This information is sent to the server and recorded as the user's personal profile data. This establishes the foundational data necessary for subsequent analysis and advice.

[0121] Step 2:

[0122] The device connects with a wearable device to acquire biometric data such as the user's heart rate and activity level in real time. This data is transferred from the device to a server using Bluetooth or Wi-Fi. The received biometric data is used to improve the accuracy of scoring the user's mental state. Specifically, the biometric data is smoothed and denoised using statistical models.

[0123] Step 3:

[0124] The user records their emotional state by inputting information via voice or text. The device sends this input to a server for emotion recognition. The server uses a generative AI model to recognize emotions from this input data and generate a mental score. The input data is processed using natural language processing and converted into a numerical emotion score.

[0125] Step 4:

[0126] The server saves the mental score generated in the previous step to a time-series database and visualizes the time-series changes in the mental state. This visualized result is sent to the user's terminal and displayed in a viewer. At this stage, the display is in a visually easy-to-understand graph format.

[0127] Step 5:

[0128] The server generates appropriate advice and action suggestions based on the generated mental score and biometric data. These suggestions are sent to the user's terminal and presented as a mental care program executable in a virtual environment. Here, based on the analysis results, the server suggests VR sessions for relaxation and stress reduction.

[0129] Step 6:

[0130] The user accepts the proposed program and experiences it in a virtual environment. The device uses software such as the Unity engine to run the virtual reality program. During this execution, the user's reactions and new biometric data are collected again and used for further mental health assessments and future recommendations.

[0131] Step 7:

[0132] The server shares mental health scores with authorized contacts. This shared information allows healthcare professionals and family members to remotely access it, enabling additional support. This sharing process is conducted using secure communication to protect privacy.

[0133] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0134] This invention is a system for accurately understanding a user's emotional state and providing appropriate mental healthcare based on that understanding. This system uses the user's smartphone as the primary operating terminal and functions in conjunction with wearable devices and an emotion engine.

[0135] The user first downloads the application to their smartphone and enters the required profile information. This allows the server to obtain basic information about the user and build a profile database. After that, the user can begin interacting with the app's AI. The interaction takes place in either voice or text format, and the user is encouraged to talk about everyday events and emotions.

[0136] The device transmits user input to an emotion engine in real time. This emotion engine uses natural language processing technology to recognize and identify the user's emotions. The recognized emotions are analyzed by a server and reflected in the user's mental score. The score is stored as time-series data, providing a function to visualize the progress of the user's mental state.

[0137] Furthermore, the device continuously acquires biometric data via the wearable device. This data is sent to a server and integrated with the results of the emotion engine to correct the mental score. This process improves the accuracy of the underlying data, resulting in better feedback and advice for the user.

[0138] For example, if user C tells the app, "I feel very tired from work today," the emotion engine analyzes this input and recognizes "fatigue" and "stress." As a result, the server adjusts the user's mental score, and the device suggests ways to relax to the user. For example, it might display advice such as, "We recommend you go to bed early tonight and get enough sleep."

[0139] This system enables more personalized mental healthcare based on each user's emotions and biological state. The system also allows for sharing emotional changes with relevant parties, enabling those supporting the user to provide more accurate support.

[0140] The following describes the processing flow.

[0141] Step 1:

[0142] Users download the app to their smartphones and complete the initial setup. By entering their profile information, they prepare to receive personalized mental health care.

[0143] Step 2:

[0144] The device sends the user's entered profile information to the server. The server registers this information in a profile database and builds a user-specific analytics platform.

[0145] Step 3:

[0146] The user launches the app and begins interacting with the AI. The device accepts voice or text input and prepares it to be passed to the emotion engine.

[0147] Step 4:

[0148] The device sends user input data to the emotion engine for sentiment analysis. The emotion engine uses natural language processing to identify the user's emotions from the input information.

[0149] Step 5:

[0150] The server receives the analysis results from the emotion engine and calculates the user's mental score based on them. This score is stored in a database to record the user's emotional state over time.

[0151] Step 6:

[0152] The device connects with wearable devices to acquire biometric information (heart rate, activity level, sleep patterns, etc.). This data reflects the user's current physical condition.

[0153] Step 7:

[0154] The device transmits acquired biometric information to the server. The server integrates this data with previously obtained emotional data and adjusts the mental score. The adjusted score becomes a detailed analysis of the user's state.

[0155] Step 8:

[0156] The device displays the user's latest mental score and provides personalized advice and action suggestions based on their emotional state. For example, it might suggest exercises to relieve stress.

[0157] Step 9:

[0158] The server sends the updated mental score to relevant parties that the user has authorized to share the score with. Based on the information provided, these parties can appropriately adjust the effectiveness and timing of the user's support.

[0159] Step 10:

[0160] The server references past records and automatically adjusts the support plan based on changes in the mental score. This information is then communicated to the user via their device, prompting them to take necessary action.

[0161] (Example 2)

[0162] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0163] In modern society, there is a need to quickly and accurately grasp fluctuations in the mental health of individual users and provide appropriate support. However, conventional systems have difficulty integrating and analyzing users' emotions and biometric data, limiting the accuracy of providing appropriate advice. Furthermore, there is a lack of mechanisms for sharing changes in a user's mental state with relevant parties, resulting in missed opportunities for support from stakeholders.

[0164] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0165] In this invention, the server includes means for acquiring user information and building a personal database, means for interacting with the user and identifying their emotional state using natural language processing technology, and means for evaluating the mental state based on the identified emotions and storing it as time-series data. This makes it possible to analyze the user's emotional state with high accuracy and provide appropriate mental healthcare.

[0166] A "personal database" is a collection of data that stores information obtained from users and is organized by user.

[0167] "Natural language processing technology" is a technology that enables computers to understand and analyze natural human language.

[0168] "Emotional state" refers to data that represents the user's mental and emotional condition.

[0169] "Mental state" is an index that indicates a user's psychological health, and is an evaluation metric calculated based on emotions and biometric data.

[0170] "Time-series data" is a collection of information arranged in order of time progression, and is used to track changes.

[0171] The system according to the present invention is designed to accurately grasp the user's emotional state and provide personalized mental healthcare based on that understanding. This system uses a smartphone as the main operating terminal and analyzes the user's psychological state in detail by linking it with wearable devices and an emotion analysis engine.

[0172] Specifically, users first download a dedicated application to their smartphones and enter the necessary profile information. Based on the user's input, the server builds a personal database. This organizes user-specific data, which is then used for subsequent analysis.

[0173] Next, the user interacts with the app in voice or text format, providing information about everyday events and emotions. This information is sent in real time via the device to an emotion analysis engine, where natural language processing technology is used to identify the emotional state. For example, in response to the input "I felt very tired from work today," the emotions "fatigue" and "stress" are analyzed.

[0174] The server evaluates the user's mental state based on these analysis results and stores it as time-series data. This generates a user mental score, enabling long-term tracking of the user's mental health. Furthermore, the terminal collects biometric data such as heart rate and activity level from wearable devices, and the server integrates this with emotional data to correct the evaluation and improve accuracy.

[0175] Ultimately, the device provides users with personalized healthcare advice. For example, it might offer specific action suggestions such as, "We recommend you go to bed early tonight and get plenty of rest."

[0176] Furthermore, the system includes a feature that allows users to share their generated mental score with relevant parties if they permit it, enabling family members and healthcare professionals to provide appropriate support.

[0177] An example of a prompt message might be: "Please explain how the server adjusts the mental score based on the sentiment data entered by the user."

[0178] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0179] Step 1:

[0180] The user downloads and launches the application on their smartphone. Next, they enter the required profile information. This information (e.g., name, age, gender, etc.) is sent to the server and recorded as an entry in the personal database. This creates a user-specific data environment.

[0181] Step 2:

[0182] Users provide information about daily events and emotions in voice or text format via the app. The input data is sent via the device to an emotion analysis engine. The emotion analysis engine uses natural language processing technology to identify emotional states from the input data. This data processing results in the output of specific emotions such as "fatigue" or "stress."

[0183] Step 3:

[0184] The server receives output from the emotion analysis engine and evaluates the user's mental state. Specifically, it calculates a specific mental score based on the emotional data. This mental score is recorded as time-series data and serves as an indicator of the user's mental health. This makes it possible to visualize changes in the user's mental state over the long term.

[0185] Step 4:

[0186] The terminal collects biometric data such as heart rate and body temperature from the wearable device worn by the user. This data is sent to a server and integrated with and corrected for the previously calculated mental score. By considering biometric data, the accuracy of the mental score is improved, enabling a more accurate assessment.

[0187] Step 5:

[0188] Based on the mental score adjusted by the server, the device provides the user with specific advice and action suggestions. This information, based on the user's emotions and biometric data, is offered as relaxation techniques and lifestyle improvements. For example, it might display advice such as, "We recommend going to bed early tonight and getting plenty of rest."

[0189] (Application Example 2)

[0190] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0191] In traditional commercial facilities, it has been difficult to grasp the individual emotional states of visitors in real time and provide personalized product recommendations based on that information. This has resulted in a limited quality of service for visitors. Furthermore, there is a need for improvement in the accuracy of systems that provide appropriate advice and product recommendations based on the user's emotional state.

[0192] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0193] In this invention, the server includes means for receiving user input information and recording it as profile data, means for interacting with the user, generating text data, and performing sentiment analysis, and means for scoring the user's mental state based on the analysis results and visualizing it in a time series. This makes it possible to provide personalized product suggestions based on the emotional state of visitors in commercial facilities.

[0194] A "user" is an individual who utilizes the system and provides input information.

[0195] "Profile data" refers to data used to record a user's basic information and individual characteristics.

[0196] "Sentiment analysis" is a technology that identifies a user's emotional state based on their dialogue and input information.

[0197] "Mental score" is an index that quantifies a user's emotional state based on certain criteria.

[0198] An "external device" is a device used to acquire biometric data from a user.

[0199] "Real-time understanding" refers to the process of immediately recognizing the current situation.

[0200] A "commercial facility" is a building or area where businesses operate as a place to offer products and provide customer services.

[0201] "Staff" refers to individuals responsible for providing services to visitors within a commercial facility.

[0202] "Product recommendations" refer to the presentation of products or services that are recommended based on the visitor's emotional state.

[0203] This invention realizes a system in which a user inputs their emotional state through an application installed on their smartphone, and based on this input, personalized product suggestions are made within a commercial facility.

[0204] The server records profile data from user input. This data is collected through voice and text input from the user and utilizes a sentiment analysis engine. This engine uses natural language processing technologies such as Google® Cloud Natural Language API to convert the input text data into emotional states.

[0205] The device uses smart glasses to detect visitors' facial expressions and voices, and performs real-time emotion analysis. This calculates the visitor's current mental score. This score is displayed on a tablet used by facility staff and is integrated with biometric data and input data.

[0206] As a concrete example, when a visitor responds with "I've been feeling stressed lately," the server uses an emotion engine to identify the "stress" and provides staff with information suggesting "aromatherapy products with relaxing effects." Based on this information, it becomes possible to make appropriate product recommendations within the commercial facility.

[0207] The prompts for the generating AI model include phrases like, "Determine the customer's emotional state and update their profile," and "Generate product suggestions based on their current emotional state." This allows the program to provide immediate responses to visitors in actual store operations.

[0208] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0209] Step 1:

[0210] Users input their emotional state and brief notes via text or voice through a smartphone app. This input information is transferred to a server as an initial processing step. Here, the server prepares to formalize the input data using natural language processing techniques.

[0211] Step 2:

[0212] The server uses the transmitted input information to perform sentiment analysis using its sentiment engine. Specifically, it extracts key phrases from the input text data and identifies sentiment labels (e.g., stress, fatigue, joy). This process quantifies the user's emotional state and records it as new profile data.

[0213] Step 3:

[0214] User profile data is stored chronologically in a server database for use within commercial facilities. Here, the user's mental score is continuously updated by a generating AI model, accumulating data.

[0215] Step 4:

[0216] Real-time emotional data of visitors is detected through external devices such as smart glasses and transmitted to a server. The terminal uses this data to perform immediate emotional analysis based on the visitor's facial expressions and voice. The obtained data is transmitted to a tablet terminal as a visualized emotional state.

[0217] Step 5:

[0218] The server provides product recommendations to staff within the commercial facility based on the analyzed emotional state. These recommendations select products and services that match the emotional state and deliver them as push notifications to tablets. This allows staff to quickly provide visitors with appropriate product recommendations and related services.

[0219] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0220] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0221] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0222] [Second Embodiment]

[0223] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0224] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0225] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0226] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0227] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0228] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0229] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0230] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0231] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0232] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0233] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0234] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0235] This invention provides an integrated system to support users' mental health, and is primarily implemented using smartphones and wearable devices. Users first download the application to their smartphone and create a personal account by entering basic profile information. This prepares the system to provide mental healthcare tailored to the user.

[0236] Users can use the app on a daily basis to interact with AI using natural language. The app, running on the device, accepts voice or text input, which is processed as data to understand the user's emotions. The server analyzes the user's statements and applies emotion recognition algorithms to quantify the user's mental state. This scored data visualizes the user's mental health status, and the user is notified if there are any changes.

[0237] In addition, the terminal interacts with a wearable device attached as an external device. This allows it to acquire biometric data such as heart rate, exercise level, and sleep patterns, and transmit it to a server. This biometric data is used to adjust the user's mental score, enabling highly accurate state analysis.

[0238] For example, when user B feels stressed, the app recognizes that emotion and reflects it in the score. If the heart rate data is higher than normal, the system suggests a specific action to the user, such as, "Try breathing exercises to relax." In this way, the system provides customized mental support based on the user's real-time state.

[0239] Furthermore, because users can share their scores with authorized family members or healthcare providers, their mental health status can be monitored remotely, allowing for the provision of necessary support. This feature enables more comprehensive mental healthcare.

[0240] The following describes the processing flow.

[0241] Step 1:

[0242] The user installs the app and enters basic profile information, including age, gender, and mental health history.

[0243] Step 2:

[0244] The terminal sends user input information to the server, which then registers it in a database. This creates an individual user profile.

[0245] Step 3:

[0246] The user launches the app and begins interacting with the AI. The device receives the user's responses via voice or text input.

[0247] Step 4:

[0248] The terminal converts the user's speech into text data and sends it to the server. The server then performs natural language processing on this text data and conducts sentiment analysis.

[0249] Step 5:

[0250] The server quantifies the user's emotional state and records it in a database as a mental score. This allows for the accumulation of time-series data on the user's mental state.

[0251] Step 6:

[0252] The device connects with a wearable device to acquire biometric data (heart rate, sleep patterns, etc.). This data is also sent to the server.

[0253] Step 7:

[0254] The server uses biometric data to correct the mental score and perform a highly accurate analysis of the user's state. The score is then updated based on these results.

[0255] Step 8:

[0256] The device displays the user's latest mental score and provides specific advice. For example, it might suggest breathing exercises or physical exercises for relaxation.

[0257] Step 9:

[0258] If the user consents, the server will share the updated mental score with authorized stakeholders. This allows third parties to understand the user's condition and provide support as needed.

[0259] Step 10:

[0260] The server compares the data with past data and automatically adjusts the support plan if necessary. The user is notified of these adjustments via their terminal.

[0261] (Example 1)

[0262] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0263] In modern society, properly managing an individual's mental health is extremely important. However, conventional methods make it difficult to grasp an individual's mental state in real time and objectively, which hinders the implementation of appropriate measures. Therefore, more accurate assessment of mental state and appropriate support are needed.

[0264] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0265] In this invention, the server includes means for receiving user input information and recording it as profile information, means for interacting with the user, generating text information, performing sentiment analysis using language processing technology, and means for quantifying the user's mental state based on the analysis results and displaying it over time. This enables real-time, personalized mental health support for each individual user.

[0266] A "user" is someone who participates in this system, inputs their personal profile information, and has their mental state evaluated through dialogue.

[0267] "Profile information" refers to basic personal information provided by the user and is used as foundational data for mental health support.

[0268] "Text information" refers to linguistic data generated from user interactions and is used for sentiment analysis.

[0269] "Language processing technology" refers to methods for analyzing natural language and is used to evaluate a user's emotional state.

[0270] "Mental state" refers to the state of a user's psychological health and is quantified through emotion analysis.

[0271] "Biometric information" refers to data about the user's physical activity and physiological indicators, obtained from measuring devices, and used to correct mental scores.

[0272] The "mental score" is a numerical representation of the user's mental state, and it has the characteristic of changing over time.

[0273] "Stakeholders" are individuals or organizations that users have given permission to share their mental health scores with, and who play a role in supporting the users' mental well-being.

[0274] The system for implementing this invention consists of a user, a terminal, and a server. First, the user downloads a dedicated application to a terminal such as a smartphone and creates a personal account by entering basic profile information. This application has the function of receiving the user's input information and recording it as profile information. The terminal interacts with the user via voice or text and generates text information. The generated text information is sent to the server, where sentiment analysis is performed using language processing technology. In this process, specific natural language processing libraries and generative AI models can be used.

[0275] Furthermore, the device works in conjunction with a wearable device to acquire biometric information such as the user's heart rate, activity level, and sleep patterns. This biometric information is sent to a server and used to correct the mental score calculated based on the results of the emotion analysis. The server quantifies the user's mental state through comprehensive data analysis and visualizes this over time.

[0276] Based on the analysis results, the server can provide users with personalized guidance and behavioral suggestions via their devices. For example, if a user shows a high stress level, it can offer specific prompts such as, "Try breathing exercises to relax." Users can also share their generated mental health score with family members, healthcare professionals, and other relevant parties, thereby facilitating mutual support.

[0277] A concrete example of a prompt message might be, "Analyze your recent stress levels and suggest relaxation methods considering your heart rate data." In this way, the system can provide real-time, personalized support to improve the user's psychological well-being.

[0278] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0279] Step 1:

[0280] Users download the application to their smartphones and enter basic profile information. This information includes name, date of birth, average heart rate, etc., and is stored on the device. The device then transfers this information to a server to create the user's personal profile. This profile serves as the basis for personalized mental health care data.

[0281] Step 2:

[0282] Users use the app daily and input their emotions and thoughts either by voice or text. For example, they make a statement like "I'm feeling stressed at work." The terminal converts the input information into text data and sends it to the server. The server's input is the text data from the user, and based on this, it performs sentiment analysis using language processing technology. The output is the analysis data indicating the user's emotional state.

[0283] Step 3:

[0284] The server analyzes the text data and performs sentiment analysis using a generated AI model. At this time, an algorithm for classifying into emotion categories such as positive, negative, and neutral is applied. The server's output is a numerical mental score based on the sentiment analysis. This score becomes information for visualizing the user's mental state over time.

[0285] Step 4:

[0286] The terminal cooperates with wearable devices and periodically acquires biological information such as heart rate, amount of exercise, and sleep pattern. For example, when the user is wearing a fitness tracker, this data is sent to the terminal in real time. The terminal sends this biological information to the server. Based on the biological information as input, the server corrects the mental score as output, enabling a more accurate state analysis.

[0287] Step 5:

[0288] The server analyzes the high-precision mental health state based on the integrated data. If there are significant fluctuations in the numerical mental score, the server automatically sends a notification to the user. For example, it generates a specific prompt like "Since your stress level is high, please try some ways to relax" and makes a proposal to the user. This enables immediate response.

[0289] Step 6:

[0290] Users can, if they wish, share their generated mental health score with family members or healthcare providers. The device provides the functionality to send the score to designated recipients, allowing them to remotely support the user's mental health. In this way, the system as a whole can provide highly personalized mental health support to the user.

[0291] (Application Example 1)

[0292] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0293] In modern society, managing individual mental health is a crucial issue, but it is not always easy for users to accurately understand their own mental state and improve it in an appropriate way. Furthermore, when remote support is needed, the means of sharing the situation with relevant parties in real time and proposing effective support plans are limited. In addition, mental healthcare support in virtual environments is still under development, and there are not enough mechanisms to provide customized experiences.

[0294] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0295] In this invention, the server includes means for receiving user input information and recording it as profile data; means for interacting with the user, generating text data, and performing sentiment analysis; means for scoring the user's mental state based on the analysis results and visualizing it in a time series; means for receiving biometric data from multiple external devices and integrating it to correct the mental score; means for providing the user with appropriate advice and action suggestions; and means for presenting a program based on the user's mental state in a virtual environment. This enables the user to understand their own mental health with high accuracy and in real time and receive support. In addition, they can effectively receive mental care through a customized experience in a virtual environment.

[0296] "User input information" refers to the individual data that users provide to the system, and is used for profile creation and individual mental health assessments.

[0297] "Profile data" is a dataset that compiles the individual attributes and characteristics of each user, and serves as the foundation for providing mental healthcare tailored to individual needs.

[0298] "Sentiment analysis" is the process of quantifying or visualizing emotional and mood states based on data collected from users.

[0299] "Scoring mental state" is a method of quantitatively measuring a user's mental health level based on the results of emotional analysis.

[0300] "Visualizing over time" is an approach that visually represents changes in mental scores over a certain period of time, making them easier for users to understand.

[0301] "Biometric data" refers to data related to a user's physical health status, including information such as heart rate and exercise level.

[0302] "External device" refers to hardware used to acquire a user's biological data, such as a smartphone or a wearable device.

[0303] "Correcting the mental score" is the process of adjusting the score to more accurately reflect the user's mental state by utilizing information from biological data.

[0304] "Appropriate advice and action proposals" are recommended actions and advice presented by the system for the purpose of improvement or maintenance based on the user's mental state.

[0305] "Virtual environment" is a digital space constructed using virtual reality or augmented reality technology, allowing users to have an immersive experience.

[0306] "Presenting a program" means providing the user with a customized experience or session corresponding to the user's current mental state.

[0307] To implement this invention, first, the user needs to install a dedicated application on a smartphone or a virtual reality device. This application has the function of accepting the input of the user's profile data and setting up an individual account. The user's smartphone or VR device uses Bluetooth or Wi-Fi to cooperate with a wearable device and acquire biological data (such as heart rate, amount of exercise, etc.) in real time.

[0308] The terminal acquires voice input or text input for analyzing the user's emotional state and transmits this data to the server. At the server, the collected data is input into a generated AI model for emotion recognition. The software used here includes TensorFlow, which is suitable for data analysis. Through this analysis, the user's mental score is generated, and the fluctuations over time are visualized.

[0309] The server generates appropriate advice and action suggestions based on the user's mental score and provides feedback to the user. This feedback is presented as a program experience within an immersive virtual environment. The virtual environment is built using the Unity engine and promotes mental care by providing a customized experience based on the user's mental state.

[0310] Furthermore, users can share this mental score with multiple authorized stakeholders. This feature allows, for example, medical professionals to remotely monitor a user's health and provide necessary support. Specifically, if a user is detected as being in a high-stress state, the system can suggest actions such as "Try a meditation session in a virtual environment to relax."

[0311] An example of a prompt might be: "Suggest a virtual reality experience to improve the user's mental health. Based on the user's stress level, provide an appropriate relaxation program."

[0312] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0313] Step 1:

[0314] The user installs and launches a dedicated application on their smartphone or VR device. First, the user enters their profile information and sets up their account. This information is sent to the server and recorded as the user's personal profile data. This establishes the foundational data necessary for subsequent analysis and advice.

[0315] Step 2:

[0316] The device connects with a wearable device to acquire biometric data such as the user's heart rate and activity level in real time. This data is transferred from the device to a server using Bluetooth or Wi-Fi. The received biometric data is used to improve the accuracy of scoring the user's mental state. Specifically, the biometric data is smoothed and denoised using statistical models.

[0317] Step 3:

[0318] The user records their emotional state by inputting information via voice or text. The device sends this input to a server for emotion recognition. The server uses a generative AI model to recognize emotions from this input data and generate a mental score. The input data is processed using natural language processing and converted into a numerical emotion score.

[0319] Step 4:

[0320] The server saves the mental score generated in the previous step to a time-series database and visualizes the time-series changes in the mental state. This visualized result is sent to the user's terminal and displayed in a viewer. At this stage, the display is in a visually easy-to-understand graph format.

[0321] Step 5:

[0322] The server generates appropriate advice and action suggestions based on the generated mental score and biometric data. These suggestions are sent to the user's terminal and presented as a mental care program executable in a virtual environment. Here, based on the analysis results, the server suggests VR sessions for relaxation and stress reduction.

[0323] Step 6:

[0324] The user accepts the proposed program and experiences it in a virtual environment. The device uses software such as the Unity engine to run the virtual reality program. During this execution, the user's reactions and new biometric data are collected again and used for further mental health assessments and future recommendations.

[0325] Step 7:

[0326] The server shares mental health scores with authorized contacts. This shared information allows healthcare professionals and family members to remotely access it, enabling additional support. This sharing process is conducted using secure communication to protect privacy.

[0327] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0328] This invention is a system for accurately understanding a user's emotional state and providing appropriate mental healthcare based on that understanding. This system uses the user's smartphone as the primary operating terminal and functions in conjunction with wearable devices and an emotion engine.

[0329] The user first downloads the application to their smartphone and enters the required profile information. This allows the server to obtain basic information about the user and build a profile database. After that, the user can begin interacting with the app's AI. The interaction takes place in either voice or text format, and the user is encouraged to talk about everyday events and emotions.

[0330] The device transmits user input to an emotion engine in real time. This emotion engine uses natural language processing technology to recognize and identify the user's emotions. The recognized emotions are analyzed by a server and reflected in the user's mental score. The score is stored as time-series data, providing a function to visualize the progress of the user's mental state.

[0331] Furthermore, the device continuously acquires biometric data via the wearable device. This data is sent to a server and integrated with the results of the emotion engine to correct the mental score. This process improves the accuracy of the underlying data, resulting in better feedback and advice for the user.

[0332] For example, if user C tells the app, "I feel very tired from work today," the emotion engine analyzes this input and recognizes "fatigue" and "stress." As a result, the server adjusts the user's mental score, and the device suggests ways to relax to the user. For example, it might display advice such as, "We recommend you go to bed early tonight and get enough sleep."

[0333] This system enables more personalized mental healthcare based on each user's emotions and biological state. The system also allows for sharing emotional changes with relevant parties, enabling those supporting the user to provide more accurate support.

[0334] The following describes the processing flow.

[0335] Step 1:

[0336] Users download the app to their smartphones and complete the initial setup. By entering their profile information, they prepare to receive personalized mental health care.

[0337] Step 2:

[0338] The device sends the user's entered profile information to the server. The server registers this information in a profile database and builds a user-specific analytics platform.

[0339] Step 3:

[0340] The user launches the app and begins interacting with the AI. The device accepts voice or text input and prepares it to be passed to the emotion engine.

[0341] Step 4:

[0342] The device sends user input data to the emotion engine for sentiment analysis. The emotion engine uses natural language processing to identify the user's emotions from the input information.

[0343] Step 5:

[0344] The server receives the analysis results from the emotion engine and calculates the user's mental score based on them. This score is stored in a database to record the user's emotional state over time.

[0345] Step 6:

[0346] The device connects with wearable devices to acquire biometric information (heart rate, activity level, sleep patterns, etc.). This data reflects the user's current physical condition.

[0347] Step 7:

[0348] The device transmits acquired biometric information to the server. The server integrates this data with previously obtained emotional data and adjusts the mental score. The adjusted score becomes a detailed analysis of the user's state.

[0349] Step 8:

[0350] The device displays the user's latest mental score and provides personalized advice and action suggestions based on their emotional state. For example, it might suggest exercises to relieve stress.

[0351] Step 9:

[0352] The server sends the updated mental score to relevant parties that the user has authorized to share the score with. Based on the information provided, these parties can appropriately adjust the effectiveness and timing of the user's support.

[0353] Step 10:

[0354] The server references past records and automatically adjusts the support plan based on changes in the mental score. This information is then communicated to the user via their device, prompting them to take necessary action.

[0355] (Example 2)

[0356] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0357] In modern society, there is a need to quickly and accurately grasp fluctuations in the mental health of individual users and provide appropriate support. However, conventional systems have difficulty integrating and analyzing users' emotions and biometric data, limiting the accuracy of providing appropriate advice. Furthermore, there is a lack of mechanisms for sharing changes in a user's mental state with relevant parties, resulting in missed opportunities for support from stakeholders.

[0358] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0359] In this invention, the server includes means for acquiring user information and building a personal database, means for interacting with the user and identifying their emotional state using natural language processing technology, and means for evaluating the mental state based on the identified emotions and storing it as time-series data. This makes it possible to analyze the user's emotional state with high accuracy and provide appropriate mental healthcare.

[0360] A "personal database" is a collection of data that stores information obtained from users and is organized by user.

[0361] "Natural language processing technology" is a technology that enables computers to understand and analyze natural human language.

[0362] "Emotional state" refers to data that represents the user's mental and emotional condition.

[0363] "Mental state" is an index that indicates a user's psychological health, and is an evaluation metric calculated based on emotions and biometric data.

[0364] "Time-series data" is a collection of information arranged in order of time progression, and is used to track changes.

[0365] The system according to the present invention is designed to accurately grasp the user's emotional state and provide personalized mental healthcare based on that understanding. This system uses a smartphone as the main operating terminal and analyzes the user's psychological state in detail by linking it with wearable devices and an emotion analysis engine.

[0366] Specifically, users first download a dedicated application to their smartphones and enter the necessary profile information. Based on the user's input, the server builds a personal database. This organizes user-specific data, which is then used for subsequent analysis.

[0367] Next, the user interacts with the app in voice or text format, providing information about everyday events and emotions. This information is sent in real time via the device to an emotion analysis engine, where natural language processing technology is used to identify the emotional state. For example, in response to the input "I felt very tired from work today," the emotions "fatigue" and "stress" are analyzed.

[0368] The server evaluates the user's mental state based on these analysis results and stores it as time-series data. This generates a user mental score, enabling long-term tracking of the user's mental health. Furthermore, the terminal collects biometric data such as heart rate and activity level from wearable devices, and the server integrates this with emotional data to correct the evaluation and improve accuracy.

[0369] Ultimately, the device provides users with personalized healthcare advice. For example, it might offer specific action suggestions such as, "We recommend you go to bed early tonight and get plenty of rest."

[0370] Furthermore, the system includes a feature that allows users to share their generated mental score with relevant parties if they permit it, enabling family members and healthcare professionals to provide appropriate support.

[0371] An example of a prompt message might be: "Please explain how the server adjusts the mental score based on the sentiment data entered by the user."

[0372] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0373] Step 1:

[0374] The user downloads and launches the application on their smartphone. Next, they enter the required profile information. This information (e.g., name, age, gender, etc.) is sent to the server and recorded as an entry in the personal database. This creates a user-specific data environment.

[0375] Step 2:

[0376] Users provide information about daily events and emotions in voice or text format via the app. The input data is sent via the device to an emotion analysis engine. The emotion analysis engine uses natural language processing technology to identify emotional states from the input data. This data processing results in the output of specific emotions such as "fatigue" or "stress."

[0377] Step 3:

[0378] The server receives output from the emotion analysis engine and evaluates the user's mental state. Specifically, it calculates a specific mental score based on the emotional data. This mental score is recorded as time-series data and serves as an indicator of the user's mental health. This makes it possible to visualize changes in the user's mental state over the long term.

[0379] Step 4:

[0380] The terminal collects biometric data such as heart rate and body temperature from the wearable device worn by the user. This data is sent to a server and integrated with and corrected for the previously calculated mental score. By considering biometric data, the accuracy of the mental score is improved, enabling a more accurate assessment.

[0381] Step 5:

[0382] Based on the mental score adjusted by the server, the device provides the user with specific advice and action suggestions. This information, based on the user's emotions and biometric data, is offered as relaxation techniques and lifestyle improvements. For example, it might display advice such as, "We recommend going to bed early tonight and getting plenty of rest."

[0383] (Application Example 2)

[0384] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0385] In traditional commercial facilities, it has been difficult to grasp the individual emotional states of visitors in real time and provide personalized product recommendations based on that information. This has resulted in a limited quality of service for visitors. Furthermore, there is a need for improvement in the accuracy of systems that provide appropriate advice and product recommendations based on the user's emotional state.

[0386] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0387] In this invention, the server includes means for receiving user input information and recording it as profile data, means for interacting with the user, generating text data, and performing sentiment analysis, and means for scoring the user's mental state based on the analysis results and visualizing it in a time series. This makes it possible to provide personalized product suggestions based on the emotional state of visitors in commercial facilities.

[0388] A "user" is an individual who utilizes the system and provides input information.

[0389] "Profile data" refers to data used to record a user's basic information and individual characteristics.

[0390] "Sentiment analysis" is a technology that identifies a user's emotional state based on their dialogue and input information.

[0391] "Mental score" is an index that quantifies a user's emotional state based on certain criteria.

[0392] An "external device" is a device used to acquire biometric data from a user.

[0393] "Real-time understanding" refers to the process of immediately recognizing the current situation.

[0394] A "commercial facility" is a building or area where businesses operate as a place to offer products and provide customer services.

[0395] "Staff" refers to individuals responsible for providing services to visitors within a commercial facility.

[0396] "Product recommendations" refer to the presentation of products or services that are recommended based on the visitor's emotional state.

[0397] This invention realizes a system in which a user inputs their emotional state through an application installed on their smartphone, and based on this input, personalized product suggestions are made within a commercial facility.

[0398] The server records profile data from user input. This data is collected through voice and text input from the user and utilizes a sentiment analysis engine. This engine uses natural language processing technologies such as the Google Cloud Natural Language API to convert the input text data into emotional states.

[0399] The device uses smart glasses to detect visitors' facial expressions and voices, and performs real-time emotion analysis. This calculates the visitor's current mental score. This score is displayed on a tablet used by facility staff and is integrated with biometric data and input data.

[0400] As a concrete example, when a visitor responds with "I've been feeling stressed lately," the server uses an emotion engine to identify the "stress" and provides staff with information suggesting "aromatherapy products with relaxing effects." Based on this information, it becomes possible to make appropriate product recommendations within the commercial facility.

[0401] The prompts for the generating AI model include phrases like, "Determine the customer's emotional state and update their profile," and "Generate product suggestions based on their current emotional state." This allows the program to provide immediate responses to visitors in actual store operations.

[0402] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0403] Step 1:

[0404] Users input their emotional state and brief notes via text or voice through a smartphone app. This input information is transferred to a server as an initial processing step. Here, the server prepares to formalize the input data using natural language processing techniques.

[0405] Step 2:

[0406] The server uses the transmitted input information to perform sentiment analysis using its sentiment engine. Specifically, it extracts key phrases from the input text data and identifies sentiment labels (e.g., stress, fatigue, joy). This process quantifies the user's emotional state and records it as new profile data.

[0407] Step 3:

[0408] User profile data is stored chronologically in a server database for use within commercial facilities. Here, the user's mental score is continuously updated by a generating AI model, accumulating data.

[0409] Step 4:

[0410] Real-time emotional data of visitors is detected through external devices such as smart glasses and transmitted to a server. The terminal uses this data to perform immediate emotional analysis based on the visitor's facial expressions and voice. The obtained data is then transmitted to a tablet terminal as a visualized emotional state.

[0411] Step 5:

[0412] The server provides product recommendations to staff within the commercial facility based on the analyzed emotional state. These recommendations select products and services that match the emotional state and deliver them as push notifications to tablets. This allows staff to quickly provide visitors with appropriate product recommendations and related services.

[0413] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0414] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0415] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0416] [Third Embodiment]

[0417] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0418] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0419] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0420] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0421] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0422] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0423] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0424] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0425] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0426] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0427] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0428] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0429] This invention provides an integrated system to support users' mental health, and is primarily implemented using smartphones and wearable devices. Users first download the application to their smartphone and create a personal account by entering basic profile information. This prepares the system to provide mental healthcare tailored to the user.

[0430] Users can use the app on a daily basis to interact with AI using natural language. The app, running on the device, accepts voice or text input, which is processed as data to understand the user's emotions. The server analyzes the user's statements and applies emotion recognition algorithms to quantify the user's mental state. This scored data visualizes the user's mental health status, and the user is notified if there are any changes.

[0431] In addition, the terminal interacts with a wearable device attached as an external device. This allows it to acquire biometric data such as heart rate, exercise level, and sleep patterns, and transmit it to a server. This biometric data is used to adjust the user's mental score, enabling highly accurate state analysis.

[0432] For example, when user B feels stressed, the app recognizes that emotion and reflects it in the score. If the heart rate data is higher than normal, the system suggests a specific action to the user, such as, "Try breathing exercises to relax." In this way, the system provides customized mental support based on the user's real-time state.

[0433] Furthermore, because users can share their scores with authorized family members or healthcare providers, their mental health status can be monitored remotely, allowing for the provision of necessary support. This feature enables more comprehensive mental healthcare.

[0434] The following describes the processing flow.

[0435] Step 1:

[0436] The user installs the app and enters basic profile information, including age, gender, and mental health history.

[0437] Step 2:

[0438] The terminal sends user input information to the server, which then registers it in a database. This creates an individual user profile.

[0439] Step 3:

[0440] The user launches the app and begins interacting with the AI. The device receives the user's responses via voice or text input.

[0441] Step 4:

[0442] The terminal converts the user's speech into text data and sends it to the server. The server then performs natural language processing on this text data and conducts sentiment analysis.

[0443] Step 5:

[0444] The server quantifies the user's emotional state and records it in a database as a mental score. This allows for the accumulation of time-series data on the user's mental state.

[0445] Step 6:

[0446] The device connects with a wearable device to acquire biometric data (heart rate, sleep patterns, etc.). This data is also sent to the server.

[0447] Step 7:

[0448] The server uses biometric data to correct the mental score and perform a highly accurate analysis of the user's state. The score is then updated based on these results.

[0449] Step 8:

[0450] The device displays the user's latest mental score and provides specific advice. For example, it might suggest breathing exercises or physical exercises for relaxation.

[0451] Step 9:

[0452] If the user consents, the server will share the updated mental score with authorized stakeholders. This allows third parties to understand the user's condition and provide support as needed.

[0453] Step 10:

[0454] The server compares the data with past data and automatically adjusts the support plan if necessary. The user is notified of these adjustments via their terminal.

[0455] (Example 1)

[0456] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0457] In modern society, properly managing an individual's mental health is extremely important. However, conventional methods make it difficult to grasp an individual's mental state in real time and objectively, which hinders the implementation of appropriate measures. Therefore, more accurate assessment of mental state and appropriate support are needed.

[0458] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0459] In this invention, the server includes means for receiving user input information and recording it as profile information, means for interacting with the user, generating text information, performing sentiment analysis using language processing technology, and means for quantifying the user's mental state based on the analysis results and displaying it over time. This enables real-time, personalized mental health support for each individual user.

[0460] A "user" is someone who participates in this system, inputs their personal profile information, and has their mental state evaluated through dialogue.

[0461] "Profile information" refers to basic personal information provided by the user and is used as foundational data for mental health support.

[0462] "Text information" refers to linguistic data generated from user interactions and is used for sentiment analysis.

[0463] "Language processing technology" refers to methods for analyzing natural language and is used to evaluate a user's emotional state.

[0464] "Mental state" refers to the state of a user's psychological health and is quantified through emotion analysis.

[0465] "Biometric information" refers to data about the user's physical activity and physiological indicators, obtained from measuring devices, and used to correct mental scores.

[0466] The "mental score" is a numerical representation of the user's mental state, and it has the characteristic of changing over time.

[0467] "Stakeholders" are individuals or organizations that users have given permission to share their mental health scores with, and who play a role in supporting the users' mental well-being.

[0468] The system for implementing this invention consists of a user, a terminal, and a server. First, the user downloads a dedicated application to a terminal such as a smartphone and creates a personal account by entering basic profile information. This application has the function of receiving the user's input information and recording it as profile information. The terminal interacts with the user via voice or text and generates text information. The generated text information is sent to the server, where sentiment analysis is performed using language processing technology. In this process, specific natural language processing libraries and generative AI models can be used.

[0469] Furthermore, the device works in conjunction with a wearable device to acquire biometric information such as the user's heart rate, activity level, and sleep patterns. This biometric information is sent to a server and used to correct the mental score calculated based on the results of the emotion analysis. The server quantifies the user's mental state through comprehensive data analysis and visualizes this over time.

[0470] Based on the analysis results, the server can provide users with personalized guidance and behavioral suggestions via their devices. For example, if a user shows a high stress level, it can offer specific prompts such as, "Try breathing exercises to relax." Users can also share their generated mental health score with family members, healthcare professionals, and other relevant parties, thereby facilitating mutual support.

[0471] A concrete example of a prompt message might be, "Analyze your recent stress levels and suggest relaxation methods considering your heart rate data." In this way, the system can provide real-time, personalized support to improve the user's psychological well-being.

[0472] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0473] Step 1:

[0474] Users download the application to their smartphones and enter basic profile information. This information includes name, date of birth, average heart rate, etc., and is stored on the device. The device then transfers this information to a server to create the user's personal profile. This profile serves as the basis for personalized mental health care data.

[0475] Step 2:

[0476] Users use the app daily, inputting their emotions and thoughts via voice or text. For example, they might say, "I'm feeling stressed at work." The device converts the input information into text data and sends it to the server. The server receives text data from the user and uses natural language processing techniques to perform sentiment analysis based on this data. The output is analyzed data indicating the user's emotional state.

[0477] Step 3:

[0478] The server analyzes text data and performs sentiment analysis using a generative AI model. It applies an algorithm that classifies the data into sentiment categories such as positive, negative, and neutral. The server's output is a mental score, quantified based on the sentiment analysis. This score provides information that visualizes the user's mental state over time.

[0479] Step 4:

[0480] The device works in conjunction with wearable devices to periodically acquire biometric information such as heart rate, activity level, and sleep patterns. For example, if a user is wearing a fitness tracker, this data is sent to the device in real time. The device then sends this biometric information to a server. Based on the biometric information as input, the server corrects the mental score as output, enabling a more accurate state analysis.

[0481] Step 5:

[0482] The server analyzes the user's mental health status with high accuracy based on integrated data. If there is a significant change in the quantified mental score, the server automatically sends a notification to the user. For example, it generates a specific prompt and makes a suggestion to the user, such as, "Your stress level is high, try some relaxation techniques." This allows for immediate action.

[0483] Step 6:

[0484] Users can, if they wish, share their generated mental health score with family members or healthcare providers. The device provides the functionality to send the score to designated recipients, allowing them to remotely support the user's mental health. In this way, the system as a whole can provide highly personalized mental health support to the user.

[0485] (Application Example 1)

[0486] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0487] In modern society, managing individual mental health is a crucial issue, but it is not always easy for users to accurately understand their own mental state and improve it in an appropriate way. Furthermore, when remote support is needed, the means of sharing the situation with relevant parties in real time and proposing effective support plans are limited. In addition, mental healthcare support in virtual environments is still under development, and there are not enough mechanisms to provide customized experiences.

[0488] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0489] In this invention, the server includes means for receiving user input information and recording it as profile data; means for interacting with the user, generating text data, and performing sentiment analysis; means for scoring the user's mental state based on the analysis results and visualizing it in a time series; means for receiving biometric data from multiple external devices and integrating it to correct the mental score; means for providing the user with appropriate advice and action suggestions; and means for presenting a program based on the user's mental state in a virtual environment. This enables the user to understand their own mental health with high accuracy and in real time and receive support. In addition, they can effectively receive mental care through a customized experience in a virtual environment.

[0490] "User input information" refers to the individual data that users provide to the system, and is used for profile creation and individual mental health assessments.

[0491] "Profile data" is a dataset that compiles the individual attributes and characteristics of each user, and serves as the foundation for providing mental healthcare tailored to individual needs.

[0492] "Sentiment analysis" is the process of quantifying or visualizing emotional and mood states based on data collected from users.

[0493] "Scoring mental state" is a method of quantitatively measuring a user's mental health level based on the results of emotional analysis.

[0494] "Visualizing over time" is an approach that visually represents changes in mental scores over a certain period of time, making them easier for users to understand.

[0495] "Biometric data" refers to data related to a user's physical health status, including information such as heart rate and exercise level.

[0496] "External devices" refer to hardware used to acquire a user's biometric data, such as smartphones and wearable devices.

[0497] "Adjusting the mental score" is the process of using biometric data to adjust the score in order to more accurately reflect the user's mental state.

[0498] "Appropriate advice and action suggestions" refer to recommended actions and advice presented by the system based on the user's mental state, with the aim of improvement or maintenance.

[0499] A "virtual environment" is a digital space created using virtual reality or augmented reality technologies, allowing users to have an immersive experience.

[0500] "Presenting a program" means providing users with customized experiences or sessions that are tailored to their current mental state.

[0501] To implement this invention, the user must first install a dedicated application on their smartphone or virtual reality device. This application accepts user profile data input and has the function of setting up a personal account. The user's smartphone or VR device connects with a wearable device using Bluetooth or Wi-Fi to acquire biometric data (heart rate, activity level, etc.) in real time.

[0502] The device acquires voice and text input to analyze the user's emotional state and sends this data to a server. The server inputs the collected data into a generative AI model to perform emotion recognition. The software used here includes TensorFlow, which is suitable for data analysis. This analysis generates a user's mental score, and its changes over time are visualized.

[0503] The server generates appropriate advice and action suggestions based on the user's mental score and provides feedback to the user. This feedback is presented as a program experience within an immersive virtual environment. The virtual environment is built using the Unity engine and promotes mental care by providing a customized experience based on the user's mental state.

[0504] Furthermore, users can share this mental score with multiple authorized stakeholders. This feature allows, for example, medical professionals to remotely monitor a user's health and provide necessary support. Specifically, if a user is detected as being in a high-stress state, the system can suggest actions such as "Try a meditation session in a virtual environment to relax."

[0505] An example of a prompt might be: "Suggest a virtual reality experience to improve the user's mental health. Based on the user's stress level, provide an appropriate relaxation program."

[0506] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0507] Step 1:

[0508] The user installs and launches a dedicated application on their smartphone or VR device. First, the user enters their profile information and sets up their account. This information is sent to the server and recorded as the user's personal profile data. This establishes the foundational data necessary for subsequent analysis and advice.

[0509] Step 2:

[0510] The device connects with a wearable device to acquire biometric data such as the user's heart rate and activity level in real time. This data is transferred from the device to a server using Bluetooth or Wi-Fi. The received biometric data is used to improve the accuracy of scoring the user's mental state. Specifically, the biometric data is smoothed and denoised using statistical models.

[0511] Step 3:

[0512] The user records their emotional state by inputting information via voice or text. The device sends this input to a server for emotion recognition. The server uses a generative AI model to recognize emotions from this input data and generate a mental score. The input data is processed using natural language processing and converted into a numerical emotion score.

[0513] Step 4:

[0514] The server saves the mental score generated in the previous step to a time-series database and visualizes the time-series changes in the mental state. This visualized result is sent to the user's terminal and displayed in a viewer. At this stage, the display is in a visually easy-to-understand graph format.

[0515] Step 5:

[0516] The server generates appropriate advice and action suggestions based on the generated mental score and biometric data. These suggestions are sent to the user's terminal and presented as a mental care program executable in a virtual environment. Here, based on the analysis results, the server suggests VR sessions for relaxation and stress reduction.

[0517] Step 6:

[0518] The user accepts the proposed program and experiences it in a virtual environment. The device uses software such as the Unity engine to run the virtual reality program. During this execution, the user's reactions and new biometric data are collected again and used for further mental health assessments and future recommendations.

[0519] Step 7:

[0520] The server shares mental health scores with authorized contacts. This shared information allows healthcare professionals and family members to remotely access it, enabling additional support. This sharing process is conducted using secure communication to protect privacy.

[0521] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0522] This invention is a system for accurately understanding a user's emotional state and providing appropriate mental healthcare based on that understanding. This system uses the user's smartphone as the primary operating terminal and functions in conjunction with wearable devices and an emotion engine.

[0523] The user first downloads the application to their smartphone and enters the required profile information. This allows the server to obtain basic information about the user and build a profile database. After that, the user can begin interacting with the app's AI. The interaction takes place in either voice or text format, and the user is encouraged to talk about everyday events and emotions.

[0524] The device transmits user input to an emotion engine in real time. This emotion engine uses natural language processing technology to recognize and identify the user's emotions. The recognized emotions are analyzed by a server and reflected in the user's mental score. The score is stored as time-series data, providing a function to visualize the progress of the user's mental state.

[0525] Furthermore, the device continuously acquires biometric data via the wearable device. This data is sent to a server and integrated with the results of the emotion engine to correct the mental score. This process improves the accuracy of the underlying data, resulting in better feedback and advice for the user.

[0526] For example, if user C tells the app, "I feel very tired from work today," the emotion engine analyzes this input and recognizes "fatigue" and "stress." As a result, the server adjusts the user's mental score, and the device suggests ways to relax to the user. For example, it might display advice such as, "We recommend you go to bed early tonight and get enough sleep."

[0527] This system enables more personalized mental healthcare based on each user's emotions and biological state. The system also allows for sharing emotional changes with relevant parties, enabling those supporting the user to provide more accurate support.

[0528] The following describes the processing flow.

[0529] Step 1:

[0530] Users download the app to their smartphones and complete the initial setup. By entering their profile information, they prepare to receive personalized mental health care.

[0531] Step 2:

[0532] The device sends the user's entered profile information to the server. The server registers this information in a profile database and builds a user-specific analytics platform.

[0533] Step 3:

[0534] The user launches the app and begins interacting with the AI. The device accepts voice or text input and prepares it to be passed to the emotion engine.

[0535] Step 4:

[0536] The device sends user input data to the emotion engine for sentiment analysis. The emotion engine uses natural language processing to identify the user's emotions from the input information.

[0537] Step 5:

[0538] The server receives the analysis results from the emotion engine and calculates the user's mental score based on them. This score is stored in a database to record the user's emotional state over time.

[0539] Step 6:

[0540] The device connects with wearable devices to acquire biometric information (heart rate, activity level, sleep patterns, etc.). This data reflects the user's current physical condition.

[0541] Step 7:

[0542] The device transmits acquired biometric information to the server. The server integrates this data with previously obtained emotional data and adjusts the mental score. The adjusted score becomes a detailed analysis of the user's state.

[0543] Step 8:

[0544] The device displays the user's latest mental score and provides personalized advice and action suggestions based on their emotional state. For example, it might suggest exercises to relieve stress.

[0545] Step 9:

[0546] The server sends the updated mental score to relevant parties that the user has authorized to share the score with. Based on the information provided, these parties can appropriately adjust the effectiveness and timing of the user's support.

[0547] Step 10:

[0548] The server references past records and automatically adjusts the support plan based on changes in the mental score. This information is then communicated to the user via their device, prompting them to take necessary action.

[0549] (Example 2)

[0550] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0551] In modern society, there is a need to quickly and accurately grasp fluctuations in the mental health of individual users and provide appropriate support. However, conventional systems have difficulty integrating and analyzing users' emotions and biometric data, limiting the accuracy of providing appropriate advice. Furthermore, there is a lack of mechanisms for sharing changes in a user's mental state with relevant parties, resulting in missed opportunities for support from stakeholders.

[0552] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0553] In this invention, the server includes means for acquiring user information and building a personal database, means for interacting with the user and identifying their emotional state using natural language processing technology, and means for evaluating the mental state based on the identified emotions and storing it as time-series data. This makes it possible to analyze the user's emotional state with high accuracy and provide appropriate mental healthcare.

[0554] A "personal database" is a collection of data that stores information obtained from users and is organized by user.

[0555] "Natural language processing technology" is a technology that enables computers to understand and analyze natural human language.

[0556] "Emotional state" refers to data that represents the user's mental and emotional condition.

[0557] "Mental state" is an index that indicates a user's psychological health, and is an evaluation metric calculated based on emotions and biometric data.

[0558] "Time-series data" is a collection of information arranged in order of time progression, and is used to track changes.

[0559] The system according to the present invention is designed to accurately grasp the user's emotional state and provide personalized mental healthcare based on that understanding. This system uses a smartphone as the main operating terminal and analyzes the user's psychological state in detail by linking it with wearable devices and an emotion analysis engine.

[0560] Specifically, users first download a dedicated application to their smartphones and enter the necessary profile information. Based on the user's input, the server builds a personal database. This organizes user-specific data, which is then used for subsequent analysis.

[0561] Next, the user interacts with the app in voice or text format, providing information about everyday events and emotions. This information is sent in real time via the device to an emotion analysis engine, where natural language processing technology is used to identify the emotional state. For example, in response to the input "I felt very tired from work today," the emotions "fatigue" and "stress" are analyzed.

[0562] The server evaluates the user's mental state based on these analysis results and stores it as time-series data. This generates a user mental score, enabling long-term tracking of the user's mental health. Furthermore, the terminal collects biometric data such as heart rate and activity level from wearable devices, and the server integrates this with emotional data to correct the evaluation and improve accuracy.

[0563] Ultimately, the device provides users with personalized healthcare advice. For example, it might offer specific action suggestions such as, "We recommend you go to bed early tonight and get plenty of rest."

[0564] Furthermore, the system includes a feature that allows users to share their generated mental score with relevant parties if they permit it, enabling family members and healthcare professionals to provide appropriate support.

[0565] An example of a prompt message might be: "Please explain how the server adjusts the mental score based on the sentiment data entered by the user."

[0566] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0567] Step 1:

[0568] The user downloads and launches the application on their smartphone. Next, they enter the required profile information. This information (e.g., name, age, gender, etc.) is sent to the server and recorded as an entry in the personal database. This creates a user-specific data environment.

[0569] Step 2:

[0570] Users provide information about daily events and emotions in voice or text format via the app. The input data is sent via the device to an emotion analysis engine. The emotion analysis engine uses natural language processing technology to identify emotional states from the input data. This data processing results in the output of specific emotions such as "fatigue" or "stress."

[0571] Step 3:

[0572] The server receives output from the emotion analysis engine and evaluates the user's mental state. Specifically, it calculates a specific mental score based on the emotional data. This mental score is recorded as time-series data and serves as an indicator of the user's mental health. This makes it possible to visualize changes in the user's mental state over the long term.

[0573] Step 4:

[0574] The terminal collects biometric data such as heart rate and body temperature from the wearable device worn by the user. This data is sent to a server and integrated with and corrected for the previously calculated mental score. By considering biometric data, the accuracy of the mental score is improved, enabling a more accurate assessment.

[0575] Step 5:

[0576] Based on the mental score adjusted by the server, the device provides the user with specific advice and action suggestions. This information, based on the user's emotions and biometric data, is offered as relaxation techniques and lifestyle improvements. For example, it might display advice such as, "We recommend going to bed early tonight and getting plenty of rest."

[0577] (Application Example 2)

[0578] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0579] In traditional commercial facilities, it has been difficult to grasp the individual emotional states of visitors in real time and provide personalized product recommendations based on that information. This has resulted in a limited quality of service for visitors. Furthermore, there is a need for improvement in the accuracy of systems that provide appropriate advice and product recommendations based on the user's emotional state.

[0580] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0581] In this invention, the server includes means for receiving user input information and recording it as profile data, means for interacting with the user, generating text data, and performing sentiment analysis, and means for scoring the user's mental state based on the analysis results and visualizing it in a time series. This makes it possible to provide personalized product suggestions based on the emotional state of visitors in commercial facilities.

[0582] A "user" is an individual who utilizes the system and provides input information.

[0583] "Profile data" refers to data used to record a user's basic information and individual characteristics.

[0584] "Sentiment analysis" is a technology that identifies a user's emotional state based on their dialogue and input information.

[0585] "Mental score" is an index that quantifies a user's emotional state based on certain criteria.

[0586] An "external device" is a device used to acquire biometric data from a user.

[0587] "Real-time understanding" refers to the process of immediately recognizing the current situation.

[0588] A "commercial facility" is a building or area where businesses operate as a place to offer products and provide customer services.

[0589] "Staff" refers to individuals responsible for providing services to visitors within a commercial facility.

[0590] "Product recommendations" refer to the presentation of products or services that are recommended based on the visitor's emotional state.

[0591] This invention realizes a system in which a user inputs their emotional state through an application installed on their smartphone, and based on this input, personalized product suggestions are made within a commercial facility.

[0592] The server records profile data from user input. This data is collected through voice and text input from the user and utilizes a sentiment analysis engine. This engine uses natural language processing technologies such as the Google Cloud Natural Language API to convert the input text data into emotional states.

[0593] The device uses smart glasses to detect visitors' facial expressions and voices, and performs real-time emotion analysis. This calculates the visitor's current mental score. This score is displayed on a tablet used by facility staff and is integrated with biometric data and input data.

[0594] As a concrete example, when a visitor responds with "I've been feeling stressed lately," the server uses an emotion engine to identify the "stress" and provides staff with information suggesting "aromatherapy products with relaxing effects." Based on this information, it becomes possible to make appropriate product recommendations within the commercial facility.

[0595] The prompts for the generating AI model include phrases like, "Determine the customer's emotional state and update their profile," and "Generate product suggestions based on their current emotional state." This allows the program to provide immediate responses to visitors in actual store operations.

[0596] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0597] Step 1:

[0598] Users input their emotional state and brief notes via text or voice through a smartphone app. This input information is transferred to a server as an initial processing step. Here, the server prepares to formalize the input data using natural language processing techniques.

[0599] Step 2:

[0600] The server uses the transmitted input information to perform sentiment analysis using its sentiment engine. Specifically, it extracts key phrases from the input text data and identifies sentiment labels (e.g., stress, fatigue, joy). This process quantifies the user's emotional state and records it as new profile data.

[0601] Step 3:

[0602] User profile data is stored chronologically in a server database for use within commercial facilities. Here, the user's mental score is continuously updated by a generating AI model, accumulating data.

[0603] Step 4:

[0604] Real-time emotional data of visitors is detected through external devices such as smart glasses and transmitted to a server. The terminal uses this data to perform immediate emotional analysis based on the visitor's facial expressions and voice. The obtained data is then transmitted to a tablet terminal as a visualized emotional state.

[0605] Step 5:

[0606] The server provides product recommendations to staff within the commercial facility based on the analyzed emotional state. These recommendations select products and services that match the emotional state and deliver them as push notifications to tablets. This allows staff to quickly provide visitors with appropriate product recommendations and related services.

[0607] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0608] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0609] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0610] [Fourth Embodiment]

[0611] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0612] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0613] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0614] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0615] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0616] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0617] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0618] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0619] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0620] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0621] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0622] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0623] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0624] This invention provides an integrated system to support users' mental health, and is primarily implemented using smartphones and wearable devices. Users first download the application to their smartphone and create a personal account by entering basic profile information. This prepares the system to provide mental healthcare tailored to the user.

[0625] Users can use the app on a daily basis to interact with AI using natural language. The app, running on the device, accepts voice or text input, which is processed as data to understand the user's emotions. The server analyzes the user's statements and applies emotion recognition algorithms to quantify the user's mental state. This scored data visualizes the user's mental health status, and the user is notified if there are any changes.

[0626] In addition, the terminal interacts with a wearable device attached as an external device. This allows it to acquire biometric data such as heart rate, exercise level, and sleep patterns, and transmit it to a server. This biometric data is used to adjust the user's mental score, enabling highly accurate state analysis.

[0627] For example, when user B feels stressed, the app recognizes that emotion and reflects it in the score. If the heart rate data is higher than normal, the system suggests a specific action to the user, such as, "Try breathing exercises to relax." In this way, the system provides customized mental support based on the user's real-time state.

[0628] Furthermore, because users can share their scores with authorized family members or healthcare providers, their mental health status can be monitored remotely, allowing for the provision of necessary support. This feature enables more comprehensive mental healthcare.

[0629] The following describes the processing flow.

[0630] Step 1:

[0631] The user installs the app and enters basic profile information, including age, gender, and mental health history.

[0632] Step 2:

[0633] The terminal sends user input information to the server, which then registers it in a database. This creates an individual user profile.

[0634] Step 3:

[0635] The user launches the app and begins interacting with the AI. The device receives the user's responses via voice or text input.

[0636] Step 4:

[0637] The terminal converts the user's speech into text data and sends it to the server. The server then performs natural language processing on this text data and conducts sentiment analysis.

[0638] Step 5:

[0639] The server quantifies the user's emotional state and records it in a database as a mental score. This allows for the accumulation of time-series data on the user's mental state.

[0640] Step 6:

[0641] The device connects with a wearable device to acquire biometric data (heart rate, sleep patterns, etc.). This data is also sent to the server.

[0642] Step 7:

[0643] The server uses biometric data to correct the mental score and perform a highly accurate analysis of the user's state. The score is then updated based on these results.

[0644] Step 8:

[0645] The device displays the user's latest mental score and provides specific advice. For example, it might suggest breathing exercises or physical exercises for relaxation.

[0646] Step 9:

[0647] If the user consents, the server will share the updated mental score with authorized stakeholders. This allows third parties to understand the user's condition and provide support as needed.

[0648] Step 10:

[0649] The server compares the data with past data and automatically adjusts the support plan if necessary. The user is notified of these adjustments via their terminal.

[0650] (Example 1)

[0651] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0652] In modern society, properly managing an individual's mental health is extremely important. However, conventional methods make it difficult to grasp an individual's mental state in real time and objectively, which hinders the implementation of appropriate measures. Therefore, more accurate assessment of mental state and appropriate support are needed.

[0653] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0654] In this invention, the server includes means for receiving user input information and recording it as profile information, means for interacting with the user, generating text information, performing sentiment analysis using language processing technology, and means for quantifying the user's mental state based on the analysis results and displaying it over time. This enables real-time, personalized mental health support for each individual user.

[0655] A "user" is someone who participates in this system, inputs their personal profile information, and has their mental state evaluated through dialogue.

[0656] "Profile information" refers to basic personal information provided by the user and is used as foundational data for mental health support.

[0657] "Text information" refers to linguistic data generated from user interactions and is used for sentiment analysis.

[0658] "Language processing technology" refers to methods for analyzing natural language and is used to evaluate a user's emotional state.

[0659] "Mental state" refers to the state of a user's psychological health and is quantified through emotion analysis.

[0660] "Biometric information" refers to data about the user's physical activity and physiological indicators, obtained from measuring devices, and used to correct mental scores.

[0661] The "mental score" is a numerical representation of the user's mental state, and it has the characteristic of changing over time.

[0662] "Stakeholders" are individuals or organizations that users have given permission to share their mental health scores with, and who play a role in supporting the users' mental well-being.

[0663] The system for implementing this invention consists of a user, a terminal, and a server. First, the user downloads a dedicated application to a terminal such as a smartphone and creates a personal account by entering basic profile information. This application has the function of receiving the user's input information and recording it as profile information. The terminal interacts with the user via voice or text and generates text information. The generated text information is sent to the server, where sentiment analysis is performed using language processing technology. In this process, specific natural language processing libraries and generative AI models can be used.

[0664] Furthermore, the device works in conjunction with a wearable device to acquire biometric information such as the user's heart rate, activity level, and sleep patterns. This biometric information is sent to a server and used to correct the mental score calculated based on the results of the emotion analysis. The server quantifies the user's mental state through comprehensive data analysis and visualizes this over time.

[0665] Based on the analysis results, the server can provide users with personalized guidance and behavioral suggestions via their devices. For example, if a user shows a high stress level, it can offer specific prompts such as, "Try breathing exercises to relax." Users can also share their generated mental health score with family members, healthcare professionals, and other relevant parties, thereby facilitating mutual support.

[0666] A concrete example of a prompt message might be, "Analyze your recent stress levels and suggest relaxation methods considering your heart rate data." In this way, the system can provide real-time, personalized support to improve the user's psychological well-being.

[0667] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0668] Step 1:

[0669] Users download the application to their smartphones and enter basic profile information. This information includes name, date of birth, average heart rate, etc., and is stored on the device. The device then transfers this information to a server to create the user's personal profile. This profile serves as the basis for personalized mental health care data.

[0670] Step 2:

[0671] Users use the app daily, inputting their emotions and thoughts via voice or text. For example, they might say, "I'm feeling stressed at work." The device converts the input information into text data and sends it to the server. The server receives text data from the user and uses natural language processing techniques to perform sentiment analysis based on this data. The output is analyzed data indicating the user's emotional state.

[0672] Step 3:

[0673] The server analyzes text data and performs sentiment analysis using a generative AI model. It applies an algorithm that classifies the data into sentiment categories such as positive, negative, and neutral. The server's output is a mental score, quantified based on the sentiment analysis. This score provides information that visualizes the user's mental state over time.

[0674] Step 4:

[0675] The device works in conjunction with wearable devices to periodically acquire biometric information such as heart rate, activity level, and sleep patterns. For example, if a user is wearing a fitness tracker, this data is sent to the device in real time. The device then sends this biometric information to a server. Based on the biometric information as input, the server corrects the mental score as output, enabling a more accurate state analysis.

[0676] Step 5:

[0677] The server analyzes the user's mental health status with high accuracy based on integrated data. If there is a significant change in the quantified mental score, the server automatically sends a notification to the user. For example, it generates a specific prompt and makes a suggestion to the user, such as, "Your stress level is high, try some relaxation techniques." This allows for immediate action.

[0678] Step 6:

[0679] Users can, if they wish, share their generated mental health score with family members or healthcare providers. The device provides the functionality to send the score to designated recipients, allowing them to remotely support the user's mental health. In this way, the system as a whole can provide highly personalized mental health support to the user.

[0680] (Application Example 1)

[0681] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0682] In modern society, managing individual mental health is a crucial issue, but it is not always easy for users to accurately understand their own mental state and improve it in an appropriate way. Furthermore, when remote support is needed, the means of sharing the situation with relevant parties in real time and proposing effective support plans are limited. In addition, mental healthcare support in virtual environments is still under development, and there are not enough mechanisms to provide customized experiences.

[0683] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0684] In this invention, the server includes means for receiving user input information and recording it as profile data; means for interacting with the user, generating text data, and performing sentiment analysis; means for scoring the user's mental state based on the analysis results and visualizing it in a time series; means for receiving biometric data from multiple external devices and integrating it to correct the mental score; means for providing the user with appropriate advice and action suggestions; and means for presenting a program based on the user's mental state in a virtual environment. This enables the user to understand their own mental health with high accuracy and in real time and receive support. In addition, they can effectively receive mental care through a customized experience in a virtual environment.

[0685] "User input information" refers to the individual data that users provide to the system, and is used for profile creation and individual mental health assessments.

[0686] "Profile data" is a dataset that compiles the individual attributes and characteristics of each user, and serves as the foundation for providing mental healthcare tailored to individual needs.

[0687] "Sentiment analysis" is the process of quantifying or visualizing emotional and mood states based on data collected from users.

[0688] "Scoring mental state" is a method of quantitatively measuring a user's mental health level based on the results of emotional analysis.

[0689] "Visualizing over time" is an approach that visually represents changes in mental scores over a certain period of time, making them easier for users to understand.

[0690] "Biometric data" refers to data related to a user's physical health status, including information such as heart rate and exercise level.

[0691] "External devices" refer to hardware used to acquire a user's biometric data, such as smartphones and wearable devices.

[0692] "Adjusting the mental score" is the process of using biometric data to adjust the score in order to more accurately reflect the user's mental state.

[0693] "Appropriate advice and action suggestions" refer to recommended actions and advice presented by the system based on the user's mental state, with the aim of improvement or maintenance.

[0694] A "virtual environment" is a digital space created using virtual reality or augmented reality technologies, allowing users to have an immersive experience.

[0695] "Presenting a program" means providing users with customized experiences or sessions that are tailored to their current mental state.

[0696] To implement this invention, the user must first install a dedicated application on their smartphone or virtual reality device. This application accepts user profile data input and has the function of setting up a personal account. The user's smartphone or VR device connects with a wearable device using Bluetooth or Wi-Fi to acquire biometric data (heart rate, activity level, etc.) in real time.

[0697] The device acquires voice and text input to analyze the user's emotional state and sends this data to a server. The server inputs the collected data into a generative AI model to perform emotion recognition. The software used here includes TensorFlow, which is suitable for data analysis. This analysis generates a user's mental score, and its changes over time are visualized.

[0698] The server generates appropriate advice and action suggestions based on the user's mental score and provides feedback to the user. This feedback is presented as a program experience within an immersive virtual environment. The virtual environment is built using the Unity engine and promotes mental care by providing a customized experience based on the user's mental state.

[0699] Furthermore, users can share this mental score with multiple authorized stakeholders. This feature allows, for example, medical professionals to remotely monitor a user's health and provide necessary support. Specifically, if a user is detected as being in a high-stress state, the system can suggest actions such as "Try a meditation session in a virtual environment to relax."

[0700] An example of a prompt might be: "Suggest a virtual reality experience to improve the user's mental health. Based on the user's stress level, provide an appropriate relaxation program."

[0701] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0702] Step 1:

[0703] The user installs and launches a dedicated application on their smartphone or VR device. First, the user enters their profile information and sets up their account. This information is sent to the server and recorded as the user's personal profile data. This establishes the foundational data necessary for subsequent analysis and advice.

[0704] Step 2:

[0705] The device connects with a wearable device to acquire biometric data such as the user's heart rate and activity level in real time. This data is transferred from the device to a server using Bluetooth or Wi-Fi. The received biometric data is used to improve the accuracy of scoring the user's mental state. Specifically, the biometric data is smoothed and denoised using statistical models.

[0706] Step 3:

[0707] The user records their emotional state by inputting information via voice or text. The device sends this input to a server for emotion recognition. The server uses a generative AI model to recognize emotions from this input data and generate a mental score. The input data is processed using natural language processing and converted into a numerical emotion score.

[0708] Step 4:

[0709] The server saves the mental score generated in the previous step to a time-series database and visualizes the time-series changes in the mental state. This visualized result is sent to the user's terminal and displayed in a viewer. At this stage, the display is in a visually easy-to-understand graph format.

[0710] Step 5:

[0711] The server generates appropriate advice and action suggestions based on the generated mental score and biometric data. These suggestions are sent to the user's terminal and presented as a mental care program executable in a virtual environment. Here, based on the analysis results, the server suggests VR sessions for relaxation and stress reduction.

[0712] Step 6:

[0713] The user accepts the proposed program and experiences it in a virtual environment. The device uses software such as the Unity engine to run the virtual reality program. During this execution, the user's reactions and new biometric data are collected again and used for further mental health assessments and future recommendations.

[0714] Step 7:

[0715] The server shares mental health scores with authorized contacts. This shared information allows healthcare professionals and family members to remotely access it, enabling additional support. This sharing process is conducted using secure communication to protect privacy.

[0716] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0717] This invention is a system for accurately understanding a user's emotional state and providing appropriate mental healthcare based on that understanding. This system uses the user's smartphone as the primary operating terminal and functions in conjunction with wearable devices and an emotion engine.

[0718] The user first downloads the application to their smartphone and enters the required profile information. This allows the server to obtain basic information about the user and build a profile database. After that, the user can begin interacting with the app's AI. The interaction takes place in either voice or text format, and the user is encouraged to talk about everyday events and emotions.

[0719] The device transmits user input to an emotion engine in real time. This emotion engine uses natural language processing technology to recognize and identify the user's emotions. The recognized emotions are analyzed by a server and reflected in the user's mental score. The score is stored as time-series data, providing a function to visualize the progress of the user's mental state.

[0720] Furthermore, the device continuously acquires biometric data via the wearable device. This data is sent to a server and integrated with the results of the emotion engine to correct the mental score. This process improves the accuracy of the underlying data, resulting in better feedback and advice for the user.

[0721] For example, if user C tells the app, "I feel very tired from work today," the emotion engine analyzes this input and recognizes "fatigue" and "stress." As a result, the server adjusts the user's mental score, and the device suggests ways to relax to the user. For example, it might display advice such as, "We recommend you go to bed early tonight and get enough sleep."

[0722] This system enables more personalized mental healthcare based on each user's emotions and biological state. The system also allows for sharing emotional changes with relevant parties, enabling those supporting the user to provide more accurate support.

[0723] The following describes the processing flow.

[0724] Step 1:

[0725] Users download the app to their smartphones and complete the initial setup. By entering their profile information, they prepare to receive personalized mental health care.

[0726] Step 2:

[0727] The device sends the user's entered profile information to the server. The server registers this information in a profile database and builds a user-specific analytics platform.

[0728] Step 3:

[0729] The user launches the app and begins interacting with the AI. The device accepts voice or text input and prepares it to be passed to the emotion engine.

[0730] Step 4:

[0731] The device sends user input data to the emotion engine for sentiment analysis. The emotion engine uses natural language processing to identify the user's emotions from the input information.

[0732] Step 5:

[0733] The server receives the analysis results from the emotion engine and calculates the user's mental score based on them. This score is stored in a database to record the user's emotional state over time.

[0734] Step 6:

[0735] The device connects with wearable devices to acquire biometric information (heart rate, activity level, sleep patterns, etc.). This data reflects the user's current physical condition.

[0736] Step 7:

[0737] The device transmits acquired biometric information to the server. The server integrates this data with previously obtained emotional data and adjusts the mental score. The adjusted score becomes a detailed analysis of the user's state.

[0738] Step 8:

[0739] The device displays the user's latest mental score and provides personalized advice and action suggestions based on their emotional state. For example, it might suggest exercises to relieve stress.

[0740] Step 9:

[0741] The server sends the updated mental score to relevant parties that the user has authorized to share the score with. Based on the information provided, these parties can appropriately adjust the effectiveness and timing of the user's support.

[0742] Step 10:

[0743] The server references past records and automatically adjusts the support plan based on changes in the mental score. This information is then communicated to the user via their device, prompting them to take necessary action.

[0744] (Example 2)

[0745] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0746] In modern society, there is a need to quickly and accurately grasp fluctuations in the mental health of individual users and provide appropriate support. However, conventional systems have difficulty integrating and analyzing users' emotions and biometric data, limiting the accuracy of providing appropriate advice. Furthermore, there is a lack of mechanisms for sharing changes in a user's mental state with relevant parties, resulting in missed opportunities for support from stakeholders.

[0747] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0748] In this invention, the server includes means for acquiring user information and building a personal database, means for interacting with the user and identifying their emotional state using natural language processing technology, and means for evaluating the mental state based on the identified emotions and storing it as time-series data. This makes it possible to analyze the user's emotional state with high accuracy and provide appropriate mental healthcare.

[0749] A "personal database" is a collection of data that stores information obtained from users and is organized by user.

[0750] "Natural language processing technology" is a technology that enables computers to understand and analyze natural human language.

[0751] "Emotional state" refers to data that represents the user's mental and emotional condition.

[0752] "Mental state" is an index that indicates a user's psychological health, and is an evaluation metric calculated based on emotions and biometric data.

[0753] "Time-series data" is a collection of information arranged in order of time progression, and is used to track changes.

[0754] The system according to the present invention is designed to accurately grasp the user's emotional state and provide personalized mental healthcare based on that understanding. This system uses a smartphone as the main operating terminal and analyzes the user's psychological state in detail by linking it with wearable devices and an emotion analysis engine.

[0755] Specifically, users first download a dedicated application to their smartphones and enter the necessary profile information. Based on the user's input, the server builds a personal database. This organizes user-specific data, which is then used for subsequent analysis.

[0756] Next, the user interacts with the app in voice or text format, providing information about everyday events and emotions. This information is sent in real time via the device to an emotion analysis engine, where natural language processing technology is used to identify the emotional state. For example, in response to the input "I felt very tired from work today," the emotions "fatigue" and "stress" are analyzed.

[0757] The server evaluates the user's mental state based on these analysis results and stores it as time-series data. This generates a user mental score, enabling long-term tracking of the user's mental health. Furthermore, the terminal collects biometric data such as heart rate and activity level from wearable devices, and the server integrates this with emotional data to correct the evaluation and improve accuracy.

[0758] Ultimately, the device provides users with personalized healthcare advice. For example, it might offer specific action suggestions such as, "We recommend you go to bed early tonight and get plenty of rest."

[0759] Furthermore, the system includes a feature that allows users to share their generated mental score with relevant parties if they permit it, enabling family members and healthcare professionals to provide appropriate support.

[0760] An example of a prompt message might be: "Please explain how the server adjusts the mental score based on the sentiment data entered by the user."

[0761] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0762] Step 1:

[0763] The user downloads and launches the application on their smartphone. Next, they enter the required profile information. This information (e.g., name, age, gender, etc.) is sent to the server and recorded as an entry in the personal database. This creates a user-specific data environment.

[0764] Step 2:

[0765] Users provide information about daily events and emotions in voice or text format via the app. The input data is sent via the device to an emotion analysis engine. The emotion analysis engine uses natural language processing technology to identify emotional states from the input data. This data processing results in the output of specific emotions such as "fatigue" or "stress."

[0766] Step 3:

[0767] The server receives output from the emotion analysis engine and evaluates the user's mental state. Specifically, it calculates a specific mental score based on the emotional data. This mental score is recorded as time-series data and serves as an indicator of the user's mental health. This makes it possible to visualize changes in the user's mental state over the long term.

[0768] Step 4:

[0769] The terminal collects biometric data such as heart rate and body temperature from the wearable device worn by the user. This data is sent to a server and integrated with and corrected for the previously calculated mental score. By considering biometric data, the accuracy of the mental score is improved, enabling a more accurate assessment.

[0770] Step 5:

[0771] Based on the mental score adjusted by the server, the device provides the user with specific advice and action suggestions. This information, based on the user's emotions and biometric data, is offered as relaxation techniques and lifestyle improvements. For example, it might display advice such as, "We recommend going to bed early tonight and getting plenty of rest."

[0772] (Application Example 2)

[0773] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0774] In traditional commercial facilities, it has been difficult to grasp the individual emotional states of visitors in real time and provide personalized product recommendations based on that information. This has resulted in a limited quality of service for visitors. Furthermore, there is a need for improvement in the accuracy of systems that provide appropriate advice and product recommendations based on the user's emotional state.

[0775] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0776] In this invention, the server includes means for receiving user input information and recording it as profile data, means for interacting with the user, generating text data, and performing sentiment analysis, and means for scoring the user's mental state based on the analysis results and visualizing it in a time series. This makes it possible to provide personalized product suggestions based on the emotional state of visitors in commercial facilities.

[0777] A "user" is an individual who utilizes the system and provides input information.

[0778] "Profile data" refers to data used to record a user's basic information and individual characteristics.

[0779] "Sentiment analysis" is a technology that identifies a user's emotional state based on their dialogue and input information.

[0780] "Mental score" is an index that quantifies a user's emotional state based on certain criteria.

[0781] An "external device" is a device used to acquire biometric data from a user.

[0782] "Real-time understanding" refers to the process of immediately recognizing the current situation.

[0783] A "commercial facility" is a building or area where businesses operate as a place to offer products and provide customer services.

[0784] "Staff" refers to individuals responsible for providing services to visitors within a commercial facility.

[0785] "Product recommendations" refer to the presentation of products or services that are recommended based on the visitor's emotional state.

[0786] This invention realizes a system in which a user inputs their emotional state through an application installed on their smartphone, and based on this input, personalized product suggestions are made within a commercial facility.

[0787] The server records profile data from user input. This data is collected through voice and text input from the user and utilizes a sentiment analysis engine. This engine uses natural language processing technologies such as the Google Cloud Natural Language API to convert the input text data into emotional states.

[0788] The device uses smart glasses to detect visitors' facial expressions and voices, and performs real-time emotion analysis. This calculates the visitor's current mental score. This score is displayed on a tablet used by facility staff and is integrated with biometric data and input data.

[0789] As a concrete example, when a visitor responds with "I've been feeling stressed lately," the server uses an emotion engine to identify the "stress" and provides staff with information suggesting "aromatherapy products with relaxing effects." Based on this information, it becomes possible to make appropriate product recommendations within the commercial facility.

[0790] The prompts for the generating AI model include phrases like, "Determine the customer's emotional state and update their profile," and "Generate product suggestions based on their current emotional state." This allows the program to provide immediate responses to visitors in actual store operations.

[0791] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0792] Step 1:

[0793] Users input their emotional state and brief notes via text or voice through a smartphone app. This input information is transferred to a server as an initial processing step. Here, the server prepares to formalize the input data using natural language processing techniques.

[0794] Step 2:

[0795] The server uses the transmitted input information to perform sentiment analysis using its sentiment engine. Specifically, it extracts key phrases from the input text data and identifies sentiment labels (e.g., stress, fatigue, joy). This process quantifies the user's emotional state and records it as new profile data.

[0796] Step 3:

[0797] User profile data is stored chronologically in a server database for use within commercial facilities. Here, the user's mental score is continuously updated by a generating AI model, accumulating data.

[0798] Step 4:

[0799] Real-time emotional data of visitors is detected through external devices such as smart glasses and transmitted to a server. The terminal uses this data to perform immediate emotional analysis based on the visitor's facial expressions and voice. The obtained data is then transmitted to a tablet terminal as a visualized emotional state.

[0800] Step 5:

[0801] The server provides product recommendations to staff within the commercial facility based on the analyzed emotional state. These recommendations select products and services that match the emotional state and deliver them as push notifications to tablets. This allows staff to quickly provide visitors with appropriate product recommendations and related services.

[0802] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0803] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0804] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0805] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0806] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0807] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0808] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0809] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0810] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0811] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0812] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0813] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0814] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0815] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0816] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0817] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0818] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0819] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0820] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0821] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0822] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0823] The following is further disclosed regarding the embodiments described above.

[0824] (Claim 1)

[0825] A means of receiving user input information and recording it as profile data,

[0826] A means of interacting with users, generating text data, and performing sentiment analysis,

[0827] A method for scoring the user's mental state based on the analysis results and visualizing it over time,

[0828] A means for receiving biometric data from multiple external devices, integrating it, and correcting the mental score,

[0829] A means of providing users with appropriate advice and action suggestions,

[0830] A system that includes this.

[0831] (Claim 2)

[0832] The system according to claim 1, further comprising means for sharing the generated mental score with multiple authorized parties.

[0833] (Claim 3)

[0834] The system according to claim 1, comprising means for automatically adjusting a support plan to suit the user's mental state based on changes in the score.

[0835] "Example 1"

[0836] (Claim 1)

[0837] A means of receiving user input information and recording it as profile information,

[0838] A means of interacting with users, generating text information, and performing sentiment analysis using language processing technology,

[0839] A means of quantifying the user's mental state based on the analysis results and displaying it over time,

[0840] A means for receiving and integrating biometric information from multiple measuring devices to correct the mental score,

[0841] A means of providing personalized guidance and behavioral suggestions to users,

[0842] A system that includes this.

[0843] (Claim 2)

[0844] The system according to claim 1, further comprising means for sharing the generated mental score with multiple stakeholders approved by the user.

[0845] (Claim 3)

[0846] The system according to claim 1, comprising means for automatically adjusting a support plan to suit the user's mental state based on changes in the score.

[0847] "Application Example 1"

[0848] (Claim 1)

[0849] A means of receiving user input information and recording it as profile data,

[0850] A means of interacting with users, generating text data, and performing sentiment analysis,

[0851] A method for scoring the user's mental state based on the analysis results and visualizing it over time,

[0852] A means for receiving biometric data from multiple external devices, integrating it, and correcting the mental score,

[0853] A means of providing users with appropriate advice and action suggestions,

[0854] A means of presenting a program based on the user's mental state in a virtual environment,

[0855] A system that includes this.

[0856] (Claim 2)

[0857] The system according to claim 1, further comprising means for sharing the generated mental score with multiple authorized parties.

[0858] (Claim 3)

[0859] The system according to claim 1, comprising means for supporting the improvement of a user's mental score through a software experience in a virtual environment.

[0860] "Example 2 of combining an emotion engine"

[0861] (Claim 1)

[0862] A means of obtaining user information and building a personal database,

[0863] A means of interacting with the user and identifying their emotional state using natural language processing technology,

[0864] A means of evaluating mental state based on identified emotions and storing it as time-series data,

[0865] A means of correcting evaluations by continuously receiving data from external sensor devices and integrating it with emotion analysis results,

[0866] A means of providing users with personalized healthcare advice,

[0867] A system that includes this.

[0868] (Claim 2)

[0869] The system according to claim 1, comprising means for sharing mental assessments with relevant parties designated by the user.

[0870] (Claim 3)

[0871] The system according to claim 1, comprising means for automatically optimizing support measures appropriate to the user's mental state based on evaluation data.

[0872] "Application example 2 when combining with an emotional engine"

[0873] (Claim 1)

[0874] A means of receiving user input information and recording it as profile data,

[0875] A means of interacting with users, generating text data, and performing sentiment analysis,

[0876] A method for scoring the user's mental state based on the analysis results and visualizing it over time,

[0877] A means for receiving biometric data from multiple external devices, integrating it, and correcting the mental score,

[0878] A means of providing users with appropriate advice and action suggestions,

[0879] A means of understanding the emotional state of visitors in real time and notifying staff within the commercial facility,

[0880] A means of generating product suggestions for visitors and providing them to staff within the commercial facility,

[0881] A system that includes this.

[0882] (Claim 2)

[0883] The system according to claim 1, further comprising means for sharing the generated mental score with multiple authorized parties.

[0884] (Claim 3)

[0885] The system according to claim 1, comprising means for automatically adjusting a support plan to suit the user's mental state based on score fluctuations, and means for adaptively changing product suggestions within a commercial facility based on visitor emotional data. [Explanation of symbols]

[0886] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of receiving user input information and recording it as profile data, A means of interacting with users, generating text data, and performing sentiment analysis, A method for scoring the user's mental state based on the analysis results and visualizing it over time, A means for receiving biometric data from multiple external devices, integrating it, and correcting the mental score, A means of providing users with appropriate advice and action suggestions, A system that includes this.

2. The system according to claim 1, further comprising means for sharing the generated mental score with multiple authorized parties.

3. The system according to claim 1, comprising means for automatically adjusting a support plan to suit the user's mental state based on changes in the score.

Images