system

Abstract

We provide the system. [Solution] A means comprising a device for detecting the user's biometric information, A means comprising a program that analyzes the user's emotional state based on the aforementioned biometric information, A means comprising an information processing system that generates a relaxation scenario suitable for the user based on the aforementioned emotional state, A means comprising a display device using virtual reality or augmented reality technology for presenting the relaxation scenario to the user, A means for providing a device that obtains feedback through interaction with the user, Information processing device including

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] With the increase in long - term digital device use and telework in modern society, many people are suffering from stress and loneliness. Under such circumstances, there is a need for a system that can effectively provide relaxation based on the individual physiological and emotional states of users. However, conventional methods have a problem that it is difficult to effectively analyze the specific emotional state and biometric information of users and provide an appropriate relaxation method.

Means for Solving the Problems

[0005] This invention accurately evaluates a user's emotional state by utilizing a group of sensors that detect the user's biometric information in real time and artificial intelligence that analyzes this information. Based on the evaluation results, a generative model generates individually optimized relaxation scenarios. These scenarios are provided to the user using virtual reality or augmented reality devices, allowing the user's mind and body to be healed through activities specifically designed for stress reduction. Furthermore, based on feedback received from the user, the system sequentially adjusts the relaxation scenario to provide a more effective individual experience.

[0006] "User biometric information" refers to data that indicates the user's physical condition, such as heart rate, body temperature, and stress level.

[0007] A "sensor" is a device that detects a user's biometric information in real time.

[0008] "Emotional state" refers to the user's current mental and emotional state, including elements such as stress and relaxation.

[0009] "Artificial intelligence" is a computer model used to analyze a user's biometric information and evaluate their emotional state.

[0010] A "generative model" is an algorithm that creates relaxation scenarios suitable for a user based on an assessment of their emotional state.

[0011] A "virtual reality or augmented reality device" is a device that provides users with visual and auditory experiences and enables interaction in a virtual space.

[0012] A "relaxation scenario" is a combination of activities and experiences designed to alleviate user stress and promote relaxation.

[0013] "Interaction" refers to two-way communication and actions between a user and a system, and specifically to interactions with virtual pets.

[0014] "Feedback" refers to information about the effects and satisfaction levels that users experienced after going through a scenario.

[0015] "Individualized optimization" is the process of customizing relaxation scenarios to suit each user's individual needs based on user feedback and experience data. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the language used in the following description will be explained.

[0019] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] The system of the present invention aims to evaluate the user's biometric information and provide an individually optimized relaxation environment. The embodiments thereof are described in detail below.

[0038] 1. Sensor installation and data collection

[0039] The user first wears a biofeedback sensor, which measures biometric data such as heart rate and body temperature in real time. The device periodically collects this biometric data and sends it to a server using a secure protocol.

[0040] 2. Data Analysis

[0041] The server stores the received biometric data in a database, and an AI agent analyzes this data. The artificial intelligence uses an emotion analysis algorithm to assess the user's current emotional state and determine how stressed the user is.

[0042] 3. Generating Relaxation Scenarios

[0043] The generative model on the server generates an optimal relaxation scenario based on an emotional state assessment. This scenario includes activities and experiences important for reducing user stress and is presented via a VR / AR device.

[0044] 4. Interaction and obtaining feedback

[0045] The device presents the generated scenario to the user. The user then begins interacting with the virtual pet using a VR / AR device. For example, the user can take a walk with the pet in a virtual space or enjoy a relaxing environment together. Throughout the interaction, the device continues to monitor the user's biometric data.

[0046] 5. Evaluation of results and improvement

[0047] The server re-analyzes biometric data after the interaction to evaluate the relaxation effect. Based on the feedback, the system sequentially adjusts the scenario for the next scenario generation, providing the user with a better, more personalized experience.

[0048] Specific example

[0049] For example, consider a user who experiences stress in their daily work using this system. A sensor attached to the user detects that their heart rate is higher than normal, and an AI agent determines that they are experiencing high stress levels. Based on this information, a generative model creates a scenario in which the user can relax with a virtual pet in nature. Through a VR device, the user experiences relaxation while exploring a virtual forest with their pet. This process stabilizes the user's heart rate and leads to overall relaxation.

[0050] In this way, the system can provide users with individually optimized relaxation experiences, promoting their physical and mental well-being.

[0051] The following describes the processing flow.

[0052] Step 1:

[0053] The device collects biometric information from biofeedback sensors attached to the user. The collected data includes heart rate, body temperature, and stress level. This data reflects the user's physiological state.

[0054] Step 2:

[0055] The device transmits collected biometric information to the server using a secure communication protocol. This transmission occurs in real time, enabling rapid monitoring of changes in the user's state.

[0056] Step 3:

[0057] The server records the biometric information transmitted from the terminal into a database and prepares it for analysis. In this step, the integrity of the data is verified to ensure that subsequent analysis is performed correctly.

[0058] Step 4:

[0059] An AI agent on the server analyzes the collected biometric information. In particular, it uses an emotion analysis algorithm to evaluate the user's stress level and emotional state. Based on this evaluation, it determines a relaxation scenario that meets the user's needs.

[0060] Step 5:

[0061] The generation model on the server generates an optimal relaxation scenario based on the evaluation results of the AI ​​agent. This scenario includes activities designed to reduce user stress and promote relaxation, and is presented to the user via a VR / AR device.

[0062] Step 6:

[0063] The device receives the generated relaxation scenario and presents it to the user. The user then uses a VR / AR device to experience interaction with a virtual pet in a virtual space. This interaction is primarily designed for the user's relaxation.

[0064] Step 7:

[0065] Users interact with virtual pets through virtual space interactions. The pets react to the user's actions, providing real-time feedback.

[0066] Step 8:

[0067] The device continuously monitors the user's biometric information during interaction and records changes in responses. This data is important for evaluating the user's relaxation effect.

[0068] Step 9:

[0069] The server re-analyzes the data after the interaction to evaluate how effective the relaxation was. Based on this, the server makes adjustments for generating future scenarios.

[0070] Step 10:

[0071] After a session ends, users receive feedback from the system regarding their performance and stress levels. This allows them to objectively understand their own state. Furthermore, this feedback serves as important data for customizing future experiences.

[0072] (Example 1)

[0073] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0074] In modern society, many individuals suffer from stress and emotional imbalances, which can negatively impact their health. In this context, there is a need to provide relaxation methods optimized for each individual user's condition. However, existing methods struggle to accurately assess individual emotional states and provide flexible and effective relaxation experiences based on those assessments. A solution to this challenge is urgently needed.

[0075] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0076] In this invention, the server includes means for measuring biometric data, means for analyzing individual emotional states, and means for designing a suitable relaxation scenario. This makes it possible to evaluate the user's emotional state in real time and provide a customized relaxation experience accordingly.

[0077] "Biometric data" includes information about the user's physical condition, such as heart rate and body temperature.

[0078] A "measuring device" refers to equipment used to collect a user's biometric data.

[0079] An "information processing device" refers to a computer system used to analyze a user's emotional state based on biometric data.

[0080] A "computational model" refers to a mechanism that includes algorithms for designing relaxation scenarios tailored to the user's emotional state.

[0081] A "visual device" refers to a device that presents a generated relaxation scenario to the user and embodies a virtual reality environment.

[0082] "Dialogue" refers to the process by which a user interacts with a system, and includes actions that enable the acquisition of feedback.

[0083] "Evaluation information" refers to information obtained through interaction with users to assess the quality of their experience.

[0084] An "adaptive control device" refers to a mechanism that dynamically adjusts relaxation scenarios based on evaluation information to individually optimize the user experience.

[0085] This invention is a system that utilizes a user's biometric information to provide an individually optimized relaxation experience. The system consists of a biofeedback sensor, a data processing device, a generative AI model, and a VR / AR device as its main components.

[0086] First, the device attaches a biofeedback sensor to the user, measuring biometric data such as heart rate and body temperature in real time. Next, the device collects this data at regular intervals and transmits it to a server using a secure protocol. This makes the biometric data available for use.

[0087] The server stores the received biometric data in a database. Furthermore, an AI agent installed on the server uses an emotion analysis algorithm to analyze the data and evaluate the user's emotional state. Based on this analysis, a generative AI model designs the optimal relaxation scenario. An example of a prompt would be, "Create a scenario for the user to relax with a pet in a natural environment."

[0088] The resulting relaxation scenarios are presented to the user via a device. The user wears a VR device and experiences the scenarios in virtual reality, enjoying activities such as exploring a virtual forest with a pet. Throughout this process, the device continuously monitors the user's biometric data and evaluates the relaxation effect in real time.

[0089] Finally, the server uses the feedback information to improve the next relaxation scenario, further enhancing the user's next experience. This system allows users to receive a personalized and enriching relaxation experience that leads to stress reduction.

[0090] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0091] Step 1:

[0092] The user wears a biofeedback sensor on their body. The sensor measures biometric data such as heart rate and body temperature in real time. Specifically, the sensor makes contact with the skin and acquires biometric information as an electrical signal. The input is the user's physical state, and the output is the measured biometric data.

[0093] Step 2:

[0094] The terminal acquires biometric data collected from sensors at regular time intervals and transmits it to the server using a secure communication protocol. Specifically, the terminal relays the data via communication methods such as Bluetooth or Wi-Fi. The input is biometric data from the sensors, and the output is data packets sent to the server.

[0095] Step 3:

[0096] The server records the received biometric data in a database. Based on this data, the server's AI agent performs emotion analysis. Specifically, the AI ​​analyzes heart rate fluctuations and changes in body temperature to estimate the user's emotional state. The input is data packets sent from the terminal, and the output is an evaluation result regarding the user's emotional state.

[0097] Step 4:

[0098] The server's generation AI model generates relaxation scenarios based on the results of emotional state evaluations. Specifically, the generation model uses prompts to design an environment and activities optimized for the user. The input is the emotional evaluation result, and the output is the relaxation scenario proposed to the user.

[0099] Step 5:

[0100] The device presents the generated relaxation scenario to the user through a VR or AR device. The user then begins interacting with the pet in the virtual environment. Specifically, the user puts on a VR device and takes a walk with the pet in a virtual forest. The input is the relaxation scenario, and the output is the virtual reality environment experienced by the user.

[0101] Step 6:

[0102] The server re-analyzes the user's biometric data after the interaction to evaluate the relaxation effect. Specifically, the server analyzes the newly acquired biometric data to determine whether stress has been reduced. The input is the biometric data after the interaction, and the output is the evaluation of the relaxation effect.

[0103] Step 7:

[0104] The server dynamically adjusts the scenario based on the evaluation results obtained, individually optimizing the next experience. Specifically, the server uses AI to analyze the feedback and redesign a scenario that is more suitable for the user. The input is the evaluation of the relaxation effect, and the output is the improved relaxation scenario.

[0105] (Application Example 1)

[0106] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0107] In modern society, many individuals are exposed to mental burden and stress, but there are limited methods to effectively alleviate this. Existing relaxation methods lack optimization based on individual emotional states and biometric information, making it difficult to provide a personalized experience. Therefore, there is a need for a system that can provide appropriate and sustained relaxation effects to individual users.

[0108] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0109] In this invention, the server includes means for a device that detects the user's biometric information, means for a program that analyzes the user's emotional state based on the biometric information, and means for an information processing system that generates a relaxation scenario suitable for the user based on the emotional state. This makes it possible to generate a scenario that enhances the relaxation effect according to the individual user's state.

[0110] A "device that detects a user's biometric information" is a measuring device that collects data on a user's physical condition, such as heart rate and body temperature.

[0111] A "program for analyzing emotional states" is software that uses a user's biometric information to estimate their emotions and psychological state at that time.

[0112] A "relaxation scenario generation information processing system" is a computer system that performs processing to design the optimal relaxation experience for the user based on the analyzed emotional state.

[0113] A "display device using virtual reality or augmented reality technology" is a visual display device used to present a generated relaxation scenario to a user, and is a device that can overlay virtual information onto a real-world environment.

[0114] A "feedback device" is a device or system for recording the reactions and results obtained from user interaction.

[0115] "Activities that reduce mental burden" refer to actions and experiences that users engage in to alleviate stress and tension and to calm their mind and body.

[0116] The "individualized optimization function" is a function that adjusts the relaxation experience provided based on the characteristics and needs of each individual user, in order to maximize its effectiveness.

[0117] The system implementing this invention consists of a user, a terminal, and a server. The user first wears a biofeedback sensor and accesses the system using smart glasses or a head-mounted display. The terminal collects biometric information (heart rate and body temperature) from the sensor in real time and transmits the data to the server using a secure communication protocol. Encryption technologies such as TLS are used for this communication.

[0118] On the server, an AI agent analyzes the received biometric information to estimate the user's emotional state. The Python scikit-learn library is expected to be used for emotion analysis. The analysis results are input into a generative AI model, which generates an optimal relaxation scenario based on the user's emotional state. Frameworks such as TENSORFLOW® and PyTorch are used for this generative AI model. The generated scenario is sent to the device as highly personalized content.

[0119] The device presents this scenario to the user using virtual reality or augmented reality technology. This allows the user to interact with their pet in a natural setting and experience relaxation. For example, a scene is provided in which the user takes a walk with their pet in a virtual forest. The user's interaction is continuously monitored by the device, and biometric data is sent back to the server. The server uses this data to evaluate the effectiveness of the user's experience and uses the feedback to adjust the next scenario.

[0120] As a concrete example, consider the case where an elderly person living alone uses this system. If the system detects that the user's heart rate is higher than normal, a relaxing scenario of taking a walk in the forest is generated and presented. As a result, the user's heart rate stabilizes, alleviating feelings of loneliness and stress.

[0121] An example of a prompt message would be, "The user's heart rate is high. Please generate a scenario where they spend time on a calm beach." This instruction would be input to the generation AI model.

[0122] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0123] Step 1:

[0124] The user wears a biofeedback sensor. This allows the device to begin measuring biometric information such as heart rate and body temperature in real time. The measured data is transmitted to the server in its original format using a secure protocol. The input is biometric information, and the output is data transfer to the server.

[0125] Step 2:

[0126] The server analyzes the received biometric information. The AI ​​agent takes in the data and performs emotion analysis using the Python scikit-learn library. In this analysis, the emotional state is quantified and evaluated. The input is biometric information, and the output is quantified emotional state data.

[0127] Step 3:

[0128] A server-based AI model generates the optimal relaxation scenario for the user based on emotional state data. This process utilizes TensorFlow or PyTorch, constructing the scenario based on prompt statements. Inputs are quantified emotional states and prompt statements, while output is the relaxation scenario.

[0129] Step 4:

[0130] The terminal presents the user with a relaxation scenario received from the server using a device that utilizes virtual reality or augmented reality technology. The user then begins the relaxation experience through this scenario. The input is the relaxation scenario, and the output is the virtual experience through sight and sound.

[0131] Step 5:

[0132] The user interacts with the presented scenario. The device continuously monitors the user's biometric information and collects new biometric data. This data is also securely transmitted to the server. The input is the biometric information during the interaction, and the output is the data transmitted to the server.

[0133] Step 6:

[0134] The server re-analyzes biometric data after the interaction. It uses feedback to evaluate the relaxation effect and utilize this information to generate the next scenario. This process continuously optimizes the user experience. The input is biometric data after the interaction, and the output is the feedback result.

[0135] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0136] This invention provides a model for a system that accurately analyzes a user's emotional state and provides relaxation scenarios based on that analysis. This system encompasses a series of functions, from collecting the user's biometric information and recognizing their emotional state to providing a relaxation experience tailored to their individual needs.

[0137] 1. Collection of biological and non-biological information

[0138] The user provides biometric information such as heart rate and body temperature through a biofeedback sensor worn on the device. Simultaneously, the device collects non-biometric information such as the user's voice, gaze, and facial expressions. This information is sent to the emotion engine.

[0139] 2. Recognition of emotional states by the emotion engine

[0140] The emotion engine installed on the server integrates and analyzes collected biometric and non-biometric information. This process utilizes machine learning algorithms and leverages the user's past data. The emotion engine determines the user's emotional state and passes the result to a relaxation scenario generation model.

[0141] 3. Generating Relaxation Scenarios

[0142] Based on the results of the emotion engine, a generative model on the server creates a relaxation scenario suitable for the user. This scenario includes activities that reduce the user's stress and promote mental stability.

[0143] 4. Scenario presentation using virtual devices

[0144] The selected relaxation scenario is presented to the user by the device via a VR / AR device. The user can then experience interaction with a virtual pet in this environment and relax.

[0145] 5. Monitoring and feedback on interactions

[0146] During user interaction, the device continuously monitors biometric information. This data is then sent back to the server for analysis and evaluation. The evaluation results provide information to individually optimize the user experience and are used to adjust scenarios for future interactions.

[0147] Specific example

[0148] For example, consider a scenario where a user is busy at work and the system is being used. Sensors detect an increase in the user's heart rate, and a camera observes frown lines indicating the user's tension. Based on this information, the emotion engine determines that the user is stressed, and the server's generative model creates a scenario in which the user relaxes in a virtual forest and plays with a virtual pet. The user then wears a VR headset and experiences relaxation through the created scenario.

[0149] Thus, the present invention enables an optimal relaxation experience tailored to the user's emotional state through advanced analysis combining biological and non-biological information.

[0150] The following describes the processing flow.

[0151] Step 1:

[0152] The user wears a device equipped with biofeedback sensors to measure heart rate and body temperature, as well as a camera and microphone to recognize facial expressions and voice. This initiates the collection of biometric and non-biometric information.

[0153] Step 2:

[0154] The device transmits collected biometric information (heart rate, body temperature, etc.) and non-biometric information (facial expressions, voice, gaze, etc.) to the server in a single batch. Real-time data transmission enables rapid analysis.

[0155] Step 3:

[0156] The server inputs the received data into the emotion engine. The emotion engine uses machine learning algorithms to analyze this data and accurately evaluate the user's emotional state. Past data is also referenced to perform more accurate emotion recognition.

[0157] Step 4:

[0158] The server's generation model uses emotional state information provided by the emotion engine to generate relaxation scenarios. These scenarios are designed to include activities that reduce user stress and promote psychological stability.

[0159] Step 5:

[0160] The terminal presents relaxation scenarios provided by the server to the user via a VR / AR device. The user experiences relaxation while interacting with a virtual pet in this virtual space.

[0161] Step 6:

[0162] During the interaction, the device continuously monitors the user's biometric information and records the user's responses. This information is then sent back to the server for analysis of emotional changes and stress reduction effects during the experience.

[0163] Step 7:

[0164] Based on feedback after the interaction, the server individually optimizes the next relaxation scenario. This adjustment allows users to experience more effective relaxation in subsequent sessions.

[0165] (Example 2)

[0166] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0167] Traditional relaxation systems have struggled to accurately grasp a user's emotional state and provide an optimal relaxation scenario accordingly. As a result, user experiences tend to be uniform, making it difficult to meet individual needs. Furthermore, feedback cannot be fully utilized, and scenarios are not incorporated into future sessions.

[0168] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0169] In this invention, the server includes means for a device that detects biometric and non-biometric information, means for artificial intelligence that analyzes the user's emotional state, and means for a generative AI model that generates a suitable relaxation scenario. This makes it possible to provide a personalized relaxation experience tailored to the user's emotional state.

[0170] "Biometric information" refers to data about the user's body, such as heart rate and body temperature.

[0171] "Non-biometric information" refers to data related to a user's behavior and state other than their physical body, such as their voice, gaze, and facial expressions.

[0172] "Artificial intelligence" refers to technology that analyzes a user's biometric and non-biometric information to recognize their emotional state.

[0173] A "generative AI model" refers to a technology that generates relaxation scenarios suitable for the user based on their analyzed emotional state.

[0174] A "relaxation scenario" refers to a set of activities and environments designed to reduce user stress and promote mental well-being.

[0175] A "virtual reality display device" refers to a device that provides users with a virtual experience.

[0176] An "augmented reality display device" refers to a device that combines and displays information from the real world with digital information.

[0177] "Feedback" refers to information obtained during a user's relaxation scenario experience, which is used to optimize the scenario for future sessions.

[0178] This invention provides a specific embodiment of a system that analyzes a user's emotional state with high accuracy and provides relaxation scenarios based on that analysis. This system collects the user's biometric and non-biometric information and uses it to provide an individually optimized relaxation experience.

[0179] The system configuration involves the user wearing a biofeedback sensor to detect biometric information such as heart rate and body temperature. Furthermore, the terminal simultaneously collects non-biometric information such as the user's voice, gaze, and facial expressions using a camera and microphone. This collected information is analyzed by artificial intelligence (AI) installed on a server. The AI ​​utilizes machine learning algorithms to analyze past user data and recognize the user's emotional state.

[0180] Subsequently, the server uses a generative AI model to generate relaxation scenarios based on the analyzed emotional state. This generative AI model references various data to create scenarios designed to reduce user stress and promote mental stability. The generated scenarios are then visually presented to the user through a virtual reality (VR) or augmented reality (AR) display device.

[0181] For example, if a user is experiencing excessive stress at work, sensors might detect an increased heart rate and a tense facial expression. Based on this information, the server's artificial intelligence determines that the user is anxious and generates a virtual beach relaxation scenario through a generative AI model. This scenario includes the sound of ocean waves and a tranquil landscape, which the user experiences through a virtual reality device.

[0182] A concrete example of a prompt message might be the instruction, "Generate a relaxation scenario to provide the optimal calming environment based on the user's current emotional state."

[0183] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0184] Step 1:

[0185] The user wears biofeedback sensors that detect heart rate and body temperature, while the device uses a camera and microphone to collect non-biometric information such as the user's voice, gaze, and facial expressions. Both biometric and non-biometric information are used as input, and this data is necessary for analyzing the user's emotional state. The device then transmits this input data to a server.

[0186] Step 2:

[0187] Based on biometric and non-biometric data sent to the server, artificial intelligence analyzes the user's emotional state. This process utilizes machine learning algorithms and also references the user's past data. Integrated biometric and non-biometric data is used as input, analyzed to determine the user's emotional state, and the analysis results are passed to a generating AI model as output.

[0188] Step 3:

[0189] The server's AI model generates relaxation scenarios tailored to the user based on emotion analysis results. This model considers past data and existing relaxation techniques to output the optimal scenario for reducing user stress. The input is the result of emotional state analysis, and the output is the generated relaxation scenario.

[0190] Step 4:

[0191] The device presents relaxation scenarios generated by a generated AI model to the user via a VR / AR device. This allows the user to experience relaxation in a virtual environment. The input is the generated relaxation scenario, and the output is presented as the user's experience.

[0192] Step 5:

[0193] While the user experiences the scenario, the device continuously monitors biometric information and sends this data to the server. The server analyzes the received data and evaluates the effectiveness of the relaxation scenario as feedback. The input is biometric information during the experience, and the output is feedback data that is used to optimize the next scenario.

[0194] (Application Example 2)

[0195] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0196] Conventional relaxation systems have struggled to accurately analyze a user's emotional state and provide an individually optimized relaxation experience. Furthermore, they lacked the means to create relaxation scenarios that adapt to the user in real time and to effectively reduce user stress by controlling acoustic stimuli and aromas.

[0197] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0198] In this invention, the server includes means for analyzing the user's emotional state based on biometric information, means for generating a relaxation scenario based on the emotional state, and means for controlling acoustic stimuli and aromas based on interaction data. This makes it possible to provide an optimal relaxation experience that is adapted to the user's emotional state in real time.

[0199] "Biometric information" refers to biological data such as the user's heart rate and body temperature, which indicates the user's current physical condition.

[0200] "Emotional state" refers to the user's mood and psychological state, and is indicated by the results of the analysis of biometric and non-biometric information.

[0201] A "relaxation scenario" is a plan of activities and environments designed to reduce user stress and promote mental well-being.

[0202] A "visual device" is a device used to present virtual images or information to a user, and typically includes virtual reality and augmented reality devices.

[0203] "Feedback" refers to information collected from user interactions, including their reactions and results, which is used to adjust the system.

[0204] "Acoustic stimuli" refer to sounds or music used to influence a user's emotional state.

[0205] "Fragrance" refers to scents released through diffusers or other means to improve the user's emotional state.

[0206] The system for implementing this invention is designed to analyze the user's emotional state and provide an appropriate relaxation scenario.

[0207] The server works in conjunction with sensor devices to detect the user's biometric information, such as heart rate and body temperature, and collects this data. The collected data is processed by an artificial intelligence system that analyzes emotions, and the user's current emotional state is evaluated. In this process, the server uses machine learning algorithms to determine the emotional state, utilizing past data as well.

[0208] Based on these results, a generative model on the server creates an optimal relaxation scenario for the user. The generated scenario is presented to the user through visual devices and devices that enhance the sense of presence, and includes control over acoustic stimuli and aromas. Feedback obtained through user interaction is collected as sensor data during the interaction and sent back to the server. This feedback is used to sequentially adjust the relaxation scenario and optimize the user experience.

[0209] As a concrete example, if the server detects an increase in the user's heart rate and facial tension, it generates a virtual environment, including a "relaxing experience in a forest," based on this information and presents it through a VR headset. During this time, healing music plays, and a relaxing scent is diffused.

[0210] The prompt given to the generating AI model is in the form of, "Based on the user's heart rate data and past stress levels, what combination of music and scent is optimal?"

[0211] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0212] Step 1:

[0213] The device collects biometric information such as heart rate and body temperature in real time from sensor devices worn by the user. In addition, it uses a camera and microphone to capture the user's facial expressions and voice. This information is compiled as biometric and non-biometric information and transmitted to a server.

[0214] Step 2:

[0215] The server uses received biometric and non-biometric information as input to analyze the user's emotional state based on machine learning algorithms. Referring to previously collected data, it accurately determines the user's stress level and psychological state. As a result, the user's current emotional state is output.

[0216] Step 3:

[0217] The server's generation model automatically generates relaxation scenarios based on analysis results. Specifically, it creates scenarios incorporating combinations of music, video, and scent, ensuring that these provide the most effective relaxation experience for the user.

[0218] Step 4:

[0219] The user's device presents the generated relaxation scenario to the user through visual devices. Specifically, it uses VR headsets or AR devices to provide the user with a virtual environment. Simultaneously, it plays appropriate music using sound equipment and activates an aroma diffuser to release fragrance.

[0220] Step 5:

[0221] During interaction, the device continuously monitors the user's biometric information and acquires new data. This data is recorded as feedback and resent to the server. This feedback serves as foundational data for optimizing the relaxation scenario for future sessions.

[0222] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0223] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0224] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0225] [Second Embodiment]

[0226] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0227] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0228] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0229] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0230] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0231] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0232] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0233] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0234] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0235] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0236] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0237] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0238] The system of the present invention aims to evaluate the user's biometric information and provide an individually optimized relaxation environment. The embodiments thereof are described in detail below.

[0239] 1. Sensor installation and data collection

[0240] The user first wears a biofeedback sensor, which measures biometric data such as heart rate and body temperature in real time. The device periodically collects this biometric data and sends it to a server using a secure protocol.

[0241] 2. Data Analysis

[0242] The server stores the received biometric data in a database, and an AI agent analyzes this data. The artificial intelligence uses an emotion analysis algorithm to assess the user's current emotional state and determine how stressed the user is.

[0243] 3. Generating Relaxation Scenarios

[0244] The generative model on the server generates an optimal relaxation scenario based on an emotional state assessment. This scenario includes activities and experiences important for reducing user stress and is presented via a VR / AR device.

[0245] 4. Interaction and obtaining feedback

[0246] The device presents the generated scenario to the user. The user then begins interacting with the virtual pet using a VR / AR device. For example, the user can take a walk with the pet in a virtual space or enjoy a relaxing environment together. Throughout the interaction, the device continues to monitor the user's biometric data.

[0247] 5. Evaluation of results and improvement

[0248] The server re-analyzes biometric data after the interaction to evaluate the relaxation effect. Based on the feedback, the system sequentially adjusts the scenario for the next scenario generation, providing the user with a better, more personalized experience.

[0249] Specific example

[0250] For example, consider a user who experiences stress in their daily work using this system. A sensor attached to the user detects that their heart rate is higher than normal, and an AI agent determines that they are experiencing high stress levels. Based on this information, a generative model creates a scenario in which the user can relax with a virtual pet in nature. Through a VR device, the user experiences relaxation while exploring a virtual forest with their pet. This process stabilizes the user's heart rate and leads to overall relaxation.

[0251] In this way, the system can provide users with individually optimized relaxation experiences, promoting their physical and mental well-being.

[0252] The following describes the processing flow.

[0253] Step 1:

[0254] The device collects biometric information from biofeedback sensors attached to the user. The collected data includes heart rate, body temperature, and stress level. This data reflects the user's physiological state.

[0255] Step 2:

[0256] The device transmits collected biometric information to the server using a secure communication protocol. This transmission occurs in real time, enabling rapid monitoring of changes in the user's state.

[0257] Step 3:

[0258] The server records the biometric information transmitted from the terminal into a database and prepares it for analysis. In this step, the integrity of the data is verified to ensure that subsequent analysis is performed correctly.

[0259] Step 4:

[0260] An AI agent on the server analyzes the collected biometric information. In particular, it uses an emotion analysis algorithm to evaluate the user's stress level and emotional state. Based on this evaluation, it determines a relaxation scenario that meets the user's needs.

[0261] Step 5:

[0262] The server-side generative model generates an optimal relaxation scenario based on the evaluation results of the AI ​​agent. This scenario includes activities designed to reduce user stress and promote relaxation, and is presented to the user via a VR / AR device.

[0263] Step 6:

[0264] The device receives the generated relaxation scenario and presents it to the user. The user then uses a VR / AR device to experience interaction with a virtual pet in a virtual space. This interaction is primarily designed for the user's relaxation.

[0265] Step 7:

[0266] Users interact with virtual pets through virtual space interactions. The pets react to the user's actions, providing real-time feedback.

[0267] Step 8:

[0268] The device continuously monitors the user's biometric information during interaction and records changes in responses. This data is important for evaluating the user's relaxation effect.

[0269] Step 9:

[0270] The server re-analyzes the data after the interaction to evaluate how effective the relaxation was. Based on this, the server makes adjustments for generating future scenarios.

[0271] Step 10:

[0272] After a session ends, users receive feedback from the system regarding their performance and stress levels. This allows them to objectively understand their own state. Furthermore, this feedback serves as important data for customizing future experiences.

[0273] (Example 1)

[0274] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0275] In modern society, many individuals suffer from stress and emotional imbalances, which can negatively impact their health. In this context, there is a need to provide relaxation methods optimized for each individual user's condition. However, existing methods struggle to accurately assess individual emotional states and provide flexible and effective relaxation experiences based on those assessments. A solution to this challenge is urgently needed.

[0276] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0277] In this invention, the server includes means for measuring biometric data, means for analyzing individual emotional states, and means for designing a suitable relaxation scenario. This makes it possible to evaluate the user's emotional state in real time and provide a customized relaxation experience accordingly.

[0278] "Biometric data" includes information about the user's physical condition, such as heart rate and body temperature.

[0279] The "measurement device" refers to a device for collecting the user's biological data.

[0280] The "information processing device" refers to a computer system used to analyze the user's emotional state based on biological data.

[0281] The "computing model" refers to a mechanism including an algorithm for designing a relaxation scenario adapted based on the user's emotional state.

[0282] The "visual device" refers to a device for presenting the generated relaxation scenario to the user and realizing a virtual reality environment.

[0283] "Interaction" refers to the process in which the user interacts with the system and includes actions that enable the acquisition of feedback.

[0284] The "evaluation information" refers to information obtained through interaction with the user for evaluating the quality of the experience.

[0285] The "adaptive control device" refers to a mechanism for dynamically adjusting the relaxation scenario based on the evaluation information and individually optimizing the user experience.

[0286] The present invention is a system that utilizes the user's biological information to provide an individually optimized relaxation experience. This system is mainly composed of a biofeedback sensor, a data processing device, a generation AI model, and a VR / AR device.

[0287] First, the terminal makes the user wear a biofeedback sensor and measures biological data such as heart rate and body temperature in real time. Next, the terminal collects this data at regular intervals and transmits it to the server using a secure protocol. Thereby, the biological data becomes available.

[0288] The server stores the received biometric data in a database. Furthermore, an AI agent installed on the server uses an emotion analysis algorithm to analyze the data and evaluate the user's emotional state. Based on this analysis, a generative AI model designs the optimal relaxation scenario. An example of a prompt would be, "Create a scenario for the user to relax with a pet in a natural environment."

[0289] The resulting relaxation scenarios are presented to the user via a device. The user wears a VR device and experiences the scenarios in virtual reality, enjoying activities such as exploring a virtual forest with a pet. Throughout this process, the device continuously monitors the user's biometric data and evaluates the relaxation effect in real time.

[0290] Finally, the server uses the feedback information to improve the next relaxation scenario, further enhancing the user's next experience. This system allows users to receive a personalized and enriching relaxation experience that leads to stress reduction.

[0291] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0292] Step 1:

[0293] The user wears a biofeedback sensor on their body. The sensor measures biometric data such as heart rate and body temperature in real time. Specifically, the sensor makes contact with the skin and acquires biometric information as an electrical signal. The input is the user's physical state, and the output is the measured biometric data.

[0294] Step 2:

[0295] The terminal acquires biometric data collected from sensors at regular time intervals and transmits it to the server using a secure communication protocol. Specifically, the terminal relays the data via communication methods such as Bluetooth or Wi-Fi. The input is biometric data from the sensors, and the output is data packets sent to the server.

[0296] Step 3:

[0297] The server records the received biometric data in a database. Based on this data, the server's AI agent performs emotion analysis. Specifically, the AI ​​analyzes heart rate fluctuations and changes in body temperature to estimate the user's emotional state. The input is data packets sent from the terminal, and the output is an evaluation result regarding the user's emotional state.

[0298] Step 4:

[0299] The server's generation AI model generates relaxation scenarios based on the results of emotional state evaluations. Specifically, the generation model uses prompts to design an environment and activities optimized for the user. The input is the emotional evaluation result, and the output is the relaxation scenario proposed to the user.

[0300] Step 5:

[0301] The device presents the generated relaxation scenario to the user through a VR or AR device. The user then begins interacting with the pet in the virtual environment. Specifically, the user puts on a VR device and takes a walk with the pet in a virtual forest. The input is the relaxation scenario, and the output is the virtual reality environment experienced by the user.

[0302] Step 6:

[0303] The server re-analyzes the biometric data after the user's interaction and evaluates the relaxation effect. Specifically, the server analyzes the new biometric data it has obtained to determine whether the stress has been reduced. The input is the biometric data after the interaction, and the output is the evaluation of the relaxation effect.

[0304] Step 7:

[0305] Based on the obtained evaluation results, the server dynamically adjusts the scenario to individually optimize the next experience. As a specific operation, the server uses AI to analyze the feedback and redesign a scenario more suitable for the user. The input is the evaluation of the relaxation effect, and the output is the improved relaxation scenario.

[0306] (Application Example 1)

[0307] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0308] In modern society, many individuals are exposed to mental stress, but the methods for effectively alleviating this are limited. Existing relaxation methods lack optimization based on individual emotional states and biometric information, making it difficult to provide a personalized experience. Therefore, a system that can provide an appropriate and sustainable relaxation effect for each user is required.

[0309] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0310] In this invention, the server includes means for a device that detects the user's biometric information, means for a program that analyzes the user's emotional state based on the biometric information, and means for an information processing system that generates a relaxation scenario suitable for the user based on the emotional state. This makes it possible to generate a scenario that enhances the relaxation effect according to the individual user's state.

[0311] A "device that detects a user's biometric information" is a measuring device that collects data on a user's physical condition, such as heart rate and body temperature.

[0312] A "program for analyzing emotional states" is software that uses a user's biometric information to estimate their emotions and psychological state at that time.

[0313] A "relaxation scenario generation information processing system" is a computer system that performs processing to design the optimal relaxation experience for the user based on the analyzed emotional state.

[0314] A "display device using virtual reality or augmented reality technology" is a visual display device used to present a generated relaxation scenario to a user, and is a device that can overlay virtual information onto a real-world environment.

[0315] A "feedback device" is a device or system for recording the reactions and results obtained from user interaction.

[0316] "Activities that reduce mental burden" refer to actions and experiences that users engage in to alleviate stress and tension and to calm their mind and body.

[0317] The "individualized optimization function" is a function that adjusts the relaxation experience provided based on the characteristics and needs of each individual user, in order to maximize its effectiveness.

[0318] The system implementing this invention consists of a user, a terminal, and a server. The user first wears a biofeedback sensor and accesses the system using smart glasses or a head-mounted display. The terminal collects biometric information (heart rate and body temperature) from the sensor in real time and transmits the data to the server using a secure communication protocol. Encryption technologies such as TLS are used for this communication.

[0319] On the server, an AI agent analyzes the received biometric information to estimate the user's emotional state. The Python scikit-learn library is expected to be used for emotion analysis. The analysis results are input into a generative AI model, which generates an optimal relaxation scenario based on the user's emotional state. Frameworks such as TensorFlow and PyTorch are used for this generative AI model. The generated scenario is sent to the device as highly personalized content.

[0320] The device presents this scenario to the user using virtual reality or augmented reality technology. This allows the user to interact with their pet in a natural setting and experience relaxation. For example, a scene is provided in which the user takes a walk with their pet in a virtual forest. The user's interaction is continuously monitored by the device, and biometric data is sent back to the server. The server uses this data to evaluate the effectiveness of the user's experience and uses the feedback to adjust the next scenario.

[0321] As a concrete example, consider the case where an elderly person living alone uses this system. If the system detects that the user's heart rate is higher than normal, a relaxing scenario of taking a walk in the forest is generated and presented. As a result, the user's heart rate stabilizes, alleviating feelings of loneliness and stress.

[0322] An example of a prompt message would be, "The user's heart rate is high. Please generate a scenario where they spend time on a calm beach." This instruction would be input to the generation AI model.

[0323] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0324] Step 1:

[0325] The user wears a biofeedback sensor. This allows the device to begin measuring biometric information such as heart rate and body temperature in real time. The measured data is transmitted to the server in its original format using a secure protocol. The input is biometric information, and the output is data transfer to the server.

[0326] Step 2:

[0327] The server analyzes the received biometric information. The AI ​​agent takes in the data and performs emotion analysis using the Python scikit-learn library. In this analysis, the emotional state is quantified and evaluated. The input is biometric information, and the output is quantified emotional state data.

[0328] Step 3:

[0329] A server-based AI model generates the optimal relaxation scenario for the user based on emotional state data. This process utilizes TensorFlow or PyTorch, constructing the scenario based on prompt statements. Inputs are quantified emotional states and prompt statements, while output is the relaxation scenario.

[0330] Step 4:

[0331] The terminal presents the user with a relaxation scenario received from the server using a device that utilizes virtual reality or augmented reality technology. The user then begins the relaxation experience through this scenario. The input is the relaxation scenario, and the output is the virtual experience through sight and sound.

[0332] Step 5:

[0333] The user interacts with the presented scenario. The device continuously monitors the user's biometric information and collects new biometric data. This data is also securely transmitted to the server. The input is the biometric information during the interaction, and the output is the data transmitted to the server.

[0334] Step 6:

[0335] The server re-analyzes biometric data after the interaction. It uses feedback to evaluate the relaxation effect and utilize this information to generate the next scenario. This process continuously optimizes the user experience. The input is biometric data after the interaction, and the output is the feedback result.

[0336] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0337] This invention provides a model for a system that accurately analyzes a user's emotional state and provides relaxation scenarios based on that analysis. This system encompasses a series of functions, from collecting the user's biometric information and recognizing their emotional state to providing a relaxation experience tailored to their individual needs.

[0338] 1. Collection of biological and non-biological information

[0339] The user provides biometric information such as heart rate and body temperature through a biofeedback sensor worn on the device. Simultaneously, the device collects non-biometric information such as the user's voice, gaze, and facial expressions. This information is sent to the emotion engine.

[0340] 2. Recognition of emotional states by the emotion engine

[0341] The emotion engine installed on the server integrates and analyzes collected biometric and non-biometric information. This process utilizes machine learning algorithms and leverages the user's past data. The emotion engine determines the user's emotional state and passes the result to a relaxation scenario generation model.

[0342] 3. Generating Relaxation Scenarios

[0343] Based on the results of the emotion engine, a generative model on the server creates a relaxation scenario suitable for the user. This scenario includes activities that reduce the user's stress and promote mental stability.

[0344] 4. Scenario presentation using virtual devices

[0345] The selected relaxation scenario is presented to the user by the device via a VR / AR device. The user can then experience interaction with a virtual pet in this environment and relax.

[0346] 5. Monitoring and feedback on interactions

[0347] During user interaction, the device continuously monitors biometric information. This data is then sent back to the server for analysis and evaluation. The evaluation results provide information to individually optimize the user experience and are used to adjust scenarios for future interactions.

[0348] Specific example

[0349] For example, consider a scenario where a user is busy at work and the system is being used. Sensors detect an increase in the user's heart rate, and a camera observes frown lines indicating the user's tension. Based on this information, the emotion engine determines that the user is stressed, and the server's generative model creates a scenario in which the user relaxes in a virtual forest and plays with a virtual pet. The user then wears a VR headset and experiences relaxation through the created scenario.

[0350] Thus, the present invention enables an optimal relaxation experience tailored to the user's emotional state through advanced analysis combining biological and non-biological information.

[0351] The following describes the processing flow.

[0352] Step 1:

[0353] The user wears a device equipped with biofeedback sensors to measure heart rate and body temperature, as well as a camera and microphone to recognize facial expressions and voice. This initiates the collection of biometric and non-biometric information.

[0354] Step 2:

[0355] The device transmits collected biometric information (heart rate, body temperature, etc.) and non-biometric information (facial expressions, voice, gaze, etc.) to the server in a single batch. Real-time data transmission enables rapid analysis.

[0356] Step 3:

[0357] The server inputs the received data into the emotion engine. The emotion engine uses machine learning algorithms to analyze this data and accurately evaluate the user's emotional state. Past data is also referenced to perform more accurate emotion recognition.

[0358] Step 4:

[0359] The server's generation model uses emotional state information provided by the emotion engine to generate relaxation scenarios. These scenarios are designed to include activities that reduce user stress and promote psychological stability.

[0360] Step 5:

[0361] The terminal presents relaxation scenarios provided by the server to the user via a VR / AR device. The user experiences relaxation while interacting with a virtual pet in this virtual space.

[0362] Step 6:

[0363] During the interaction, the device continuously monitors the user's biometric information and records the user's responses. This information is then sent back to the server for analysis of emotional changes and stress reduction effects during the experience.

[0364] Step 7:

[0365] Based on feedback after the interaction, the server individually optimizes the next relaxation scenario. This adjustment allows users to experience more effective relaxation in subsequent sessions.

[0366] (Example 2)

[0367] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0368] Traditional relaxation systems have struggled to accurately grasp a user's emotional state and provide an optimal relaxation scenario accordingly. As a result, user experiences tend to be uniform, making it difficult to meet individual needs. Furthermore, feedback cannot be fully utilized, and scenarios are not incorporated into future sessions.

[0369] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0370] In this invention, the server includes means for a device that detects biometric and non-biometric information, means for artificial intelligence that analyzes the user's emotional state, and means for a generative AI model that generates a suitable relaxation scenario. This makes it possible to provide a personalized relaxation experience tailored to the user's emotional state.

[0371] "Biometric information" refers to data about the user's body, such as heart rate and body temperature.

[0372] "Non-biometric information" refers to data related to a user's behavior and state other than their physical body, such as their voice, gaze, and facial expressions.

[0373] "Artificial intelligence" refers to technology that analyzes a user's biometric and non-biometric information to recognize their emotional state.

[0374] A "generative AI model" refers to a technology that generates relaxation scenarios suitable for the user based on their analyzed emotional state.

[0375] A "relaxation scenario" refers to a set of activities and environments designed to reduce user stress and promote mental well-being.

[0376] A "virtual reality display device" refers to a device that provides users with a virtual experience.

[0377] An "augmented reality display device" refers to a device that combines and displays information from the real world with digital information.

[0378] "Feedback" refers to information obtained during a user's relaxation scenario experience, which is used to optimize the scenario for future sessions.

[0379] This invention provides a specific embodiment of a system that analyzes a user's emotional state with high accuracy and provides relaxation scenarios based on that analysis. This system collects the user's biometric and non-biometric information and uses it to provide an individually optimized relaxation experience.

[0380] The system configuration involves the user wearing a biofeedback sensor to detect biometric information such as heart rate and body temperature. Furthermore, the terminal simultaneously collects non-biometric information such as the user's voice, gaze, and facial expressions using a camera and microphone. This collected information is analyzed by artificial intelligence (AI) installed on a server. The AI ​​utilizes machine learning algorithms to analyze past user data and recognize the user's emotional state.

[0381] Subsequently, the server uses a generative AI model to generate relaxation scenarios based on the analyzed emotional state. This generative AI model references various data to create scenarios designed to reduce user stress and promote mental stability. The generated scenarios are then visually presented to the user through a virtual reality (VR) or augmented reality (AR) display device.

[0382] For example, if a user is experiencing excessive stress at work, sensors might detect an increased heart rate and a tense facial expression. Based on this information, the server's artificial intelligence determines that the user is anxious and generates a virtual beach relaxation scenario through a generative AI model. This scenario includes the sound of ocean waves and a tranquil landscape, which the user experiences through a virtual reality device.

[0383] A concrete example of a prompt message might be the instruction, "Generate a relaxation scenario to provide the optimal calming environment based on the user's current emotional state."

[0384] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0385] Step 1:

[0386] The user wears biofeedback sensors that detect heart rate and body temperature, while the device uses a camera and microphone to collect non-biometric information such as the user's voice, gaze, and facial expressions. Both biometric and non-biometric information are used as input, and this data is necessary for analyzing the user's emotional state. The device then transmits this input data to a server.

[0387] Step 2:

[0388] Based on biometric and non-biometric data sent to the server, artificial intelligence analyzes the user's emotional state. This process utilizes machine learning algorithms and also references the user's past data. Integrated biometric and non-biometric data is used as input, analyzed to determine the user's emotional state, and the analysis results are passed to a generating AI model as output.

[0389] Step 3:

[0390] The server's AI model generates relaxation scenarios tailored to the user based on emotion analysis results. This model considers past data and existing relaxation techniques to output the optimal scenario for reducing user stress. The input is the result of emotional state analysis, and the output is the generated relaxation scenario.

[0391] Step 4:

[0392] The device presents relaxation scenarios generated by a generated AI model to the user via a VR / AR device. This allows the user to experience relaxation in a virtual environment. The input is the generated relaxation scenario, and the output is presented as the user's experience.

[0393] Step 5:

[0394] While the user experiences the scenario, the device continuously monitors biometric information and sends this data to the server. The server analyzes the received data and evaluates the effectiveness of the relaxation scenario as feedback. The input is biometric information during the experience, and the output is feedback data that is used to optimize the next scenario.

[0395] (Application Example 2)

[0396] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0397] Conventional relaxation systems have struggled to accurately analyze a user's emotional state and provide an individually optimized relaxation experience. Furthermore, they lacked the means to create relaxation scenarios that adapt to the user in real time and to effectively reduce user stress by controlling acoustic stimuli and aromas.

[0398] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0399] In this invention, the server includes means for analyzing the user's emotional state based on biometric information, means for generating a relaxation scenario based on the emotional state, and means for controlling acoustic stimuli and aromas based on interaction data. This makes it possible to provide an optimal relaxation experience that is adapted to the user's emotional state in real time.

[0400] "Biometric information" refers to biological data such as the user's heart rate and body temperature, which indicates the user's current physical condition.

[0401] "Emotional state" refers to the user's mood and psychological state, and is indicated by the results of the analysis of biometric and non-biometric information.

[0402] A "relaxation scenario" is a plan of activities and environments designed to reduce user stress and promote mental well-being.

[0403] A "visual device" is a device used to present virtual images or information to a user, and typically includes virtual reality and augmented reality devices.

[0404] "Feedback" refers to information collected from user interactions, including their reactions and results, which is used to adjust the system.

[0405] "Acoustic stimuli" refer to sounds or music used to influence a user's emotional state.

[0406] "Fragrance" refers to scents released through diffusers or other means to improve the user's emotional state.

[0407] The system for implementing this invention is designed to analyze the user's emotional state and provide an appropriate relaxation scenario.

[0408] The server works in conjunction with sensor devices to detect the user's biometric information, such as heart rate and body temperature, and collects this data. The collected data is processed by an artificial intelligence system that analyzes emotions, and the user's current emotional state is evaluated. In this process, the server uses machine learning algorithms to determine the emotional state, utilizing past data as well.

[0409] Based on these results, a generative model on the server creates an optimal relaxation scenario for the user. The generated scenario is presented to the user through visual devices and devices that enhance the sense of presence, and includes control over acoustic stimuli and aromas. Feedback obtained through user interaction is collected as sensor data during the interaction and sent back to the server. This feedback is used to sequentially adjust the relaxation scenario and optimize the user experience.

[0410] As a concrete example, if the server detects an increase in the user's heart rate and facial tension, it generates a virtual environment, including a "relaxing experience in a forest," based on this information and presents it through a VR headset. During this time, healing music plays, and a relaxing scent is diffused.

[0411] The prompt given to the generating AI model is in the form of, "Based on the user's heart rate data and past stress levels, what combination of music and scent is optimal?"

[0412] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0413] Step 1:

[0414] The device collects biometric information such as heart rate and body temperature in real time from sensor devices worn by the user. In addition, it uses a camera and microphone to capture the user's facial expressions and voice. This information is compiled as biometric and non-biometric information and transmitted to a server.

[0415] Step 2:

[0416] The server uses received biometric and non-biometric information as input to analyze the user's emotional state based on machine learning algorithms. Referring to previously collected data, it accurately determines the user's stress level and psychological state. As a result, the user's current emotional state is output.

[0417] Step 3:

[0418] The server's generation model automatically generates relaxation scenarios based on analysis results. Specifically, it creates scenarios incorporating combinations of music, video, and scent, ensuring that these provide the most effective relaxation experience for the user.

[0419] Step 4:

[0420] The user's device presents the generated relaxation scenario to the user through visual devices. Specifically, it uses VR headsets or AR devices to provide the user with a virtual environment. Simultaneously, it plays appropriate music using sound equipment and activates an aroma diffuser to release fragrance.

[0421] Step 5:

[0422] During interaction, the device continuously monitors the user's biometric information and acquires new data. This data is recorded as feedback and resent to the server. This feedback serves as foundational data for optimizing the relaxation scenario for future sessions.

[0423] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0424] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0425] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0426] [Third Embodiment]

[0427] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0428] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0429] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0430] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0431] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0432] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0433] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0434] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0435] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0436] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0437] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0438] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0439] The system of the present invention aims to evaluate the user's biometric information and provide an individually optimized relaxation environment. The embodiments thereof are described in detail below.

[0440] 1. Sensor installation and data collection

[0441] The user first wears a biofeedback sensor, which measures biometric data such as heart rate and body temperature in real time. The device periodically collects this biometric data and sends it to a server using a secure protocol.

[0442] 2. Data Analysis

[0443] The server stores the received biometric data in a database, and an AI agent analyzes this data. The artificial intelligence uses an emotion analysis algorithm to assess the user's current emotional state and determine how stressed the user is.

[0444] 3. Generating Relaxation Scenarios

[0445] The generative model on the server generates an optimal relaxation scenario based on an emotional state assessment. This scenario includes activities and experiences important for reducing user stress and is presented via a VR / AR device.

[0446] 4. Interaction and obtaining feedback

[0447] The device presents the generated scenario to the user. The user then begins interacting with the virtual pet using a VR / AR device. For example, the user can take a walk with the pet in a virtual space or enjoy a relaxing environment together. Throughout the interaction, the device continues to monitor the user's biometric data.

[0448] 5. Evaluation of results and improvement

[0449] The server re-analyzes biometric data after the interaction to evaluate the relaxation effect. Based on the feedback, the system sequentially adjusts the scenario for the next scenario generation, providing the user with a better, more personalized experience.

[0450] Specific example

[0451] For example, consider a user who experiences stress in their daily work using this system. A sensor attached to the user detects that their heart rate is higher than normal, and an AI agent determines that they are experiencing high stress levels. Based on this information, a generative model creates a scenario in which the user can relax with a virtual pet in nature. Through a VR device, the user experiences relaxation while exploring a virtual forest with their pet. This process stabilizes the user's heart rate and leads to overall relaxation.

[0452] In this way, the system can provide users with individually optimized relaxation experiences, promoting their physical and mental well-being.

[0453] The following describes the processing flow.

[0454] Step 1:

[0455] The device collects biometric information from biofeedback sensors attached to the user. The collected data includes heart rate, body temperature, and stress level. This data reflects the user's physiological state.

[0456] Step 2:

[0457] The device transmits collected biometric information to the server using a secure communication protocol. This transmission occurs in real time, enabling rapid monitoring of changes in the user's state.

[0458] Step 3:

[0459] The server records the biometric information transmitted from the terminal into a database and prepares it for analysis. In this step, the integrity of the data is verified to ensure that subsequent analysis is performed correctly.

[0460] Step 4:

[0461] An AI agent on the server analyzes the collected biometric information. In particular, it uses an emotion analysis algorithm to evaluate the user's stress level and emotional state. Based on this evaluation, it determines a relaxation scenario that meets the user's needs.

[0462] Step 5:

[0463] The server-side generative model generates an optimal relaxation scenario based on the evaluation results of the AI ​​agent. This scenario includes activities designed to reduce user stress and promote relaxation, and is presented to the user via a VR / AR device.

[0464] Step 6:

[0465] The device receives the generated relaxation scenario and presents it to the user. The user then uses a VR / AR device to experience interaction with a virtual pet in a virtual space. This interaction is primarily designed for the user's relaxation.

[0466] Step 7:

[0467] Users interact with virtual pets through virtual space interactions. The pets react to the user's actions, providing real-time feedback.

[0468] Step 8:

[0469] The device continuously monitors the user's biometric information during interaction and records changes in responses. This data is important for evaluating the user's relaxation effect.

[0470] Step 9:

[0471] The server re-analyzes the data after the interaction to evaluate how effective the relaxation was. Based on this, the server makes adjustments for generating future scenarios.

[0472] Step 10:

[0473] After a session ends, users receive feedback from the system regarding their performance and stress levels. This allows them to objectively understand their own state. Furthermore, this feedback serves as important data for customizing future experiences.

[0474] (Example 1)

[0475] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0476] In modern society, many individuals suffer from stress and emotional imbalances, which can negatively impact their health. In this context, there is a need to provide relaxation methods optimized for each individual user's condition. However, existing methods struggle to accurately assess individual emotional states and provide flexible and effective relaxation experiences based on those assessments. A solution to this challenge is urgently needed.

[0477] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0478] In this invention, the server includes means for measuring biometric data, means for analyzing individual emotional states, and means for designing a suitable relaxation scenario. This makes it possible to evaluate the user's emotional state in real time and provide a customized relaxation experience accordingly.

[0479] "Biometric data" includes information about the user's physical condition, such as heart rate and body temperature.

[0480] A "measuring device" refers to equipment used to collect a user's biometric data.

[0481] An "information processing device" refers to a computer system used to analyze a user's emotional state based on biometric data.

[0482] A "computational model" refers to a mechanism that includes algorithms for designing relaxation scenarios tailored to the user's emotional state.

[0483] A "visual device" refers to a device that presents a generated relaxation scenario to the user and embodies a virtual reality environment.

[0484] "Dialogue" refers to the process by which a user interacts with a system, and includes actions that enable the acquisition of feedback.

[0485] "Evaluation information" refers to information obtained through interaction with users to assess the quality of their experience.

[0486] An "adaptive control device" refers to a mechanism that dynamically adjusts relaxation scenarios based on evaluation information to individually optimize the user experience.

[0487] This invention is a system that utilizes a user's biometric information to provide an individually optimized relaxation experience. The system consists of a biofeedback sensor, a data processing device, a generative AI model, and a VR / AR device as its main components.

[0488] First, the device attaches a biofeedback sensor to the user, measuring biometric data such as heart rate and body temperature in real time. Next, the device collects this data at regular intervals and transmits it to a server using a secure protocol. This makes the biometric data available for use.

[0489] The server stores the received biometric data in a database. Furthermore, an AI agent installed on the server uses an emotion analysis algorithm to analyze the data and evaluate the user's emotional state. Based on this analysis, a generative AI model designs the optimal relaxation scenario. An example of a prompt would be, "Create a scenario for the user to relax with a pet in a natural environment."

[0490] The resulting relaxation scenarios are presented to the user via a device. The user wears a VR device and experiences the scenarios in virtual reality, enjoying activities such as exploring a virtual forest with a pet. Throughout this process, the device continuously monitors the user's biometric data and evaluates the relaxation effect in real time.

[0491] Finally, the server uses the feedback information to improve the next relaxation scenario, further enhancing the user's experience. This system allows users to receive a personalized and enriching relaxation experience that leads to stress reduction.

[0492] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0493] Step 1:

[0494] The user wears a biofeedback sensor on their body. The sensor measures biometric data such as heart rate and body temperature in real time. Specifically, the sensor makes contact with the skin and acquires biometric information as an electrical signal. The input is the user's physical state, and the output is the measured biometric data.

[0495] Step 2:

[0496] The terminal acquires biometric data collected from sensors at regular time intervals and transmits it to the server using a secure communication protocol. Specifically, the terminal relays the data via communication methods such as Bluetooth or Wi-Fi. The input is biometric data from the sensors, and the output is data packets sent to the server.

[0497] Step 3:

[0498] The server records the received biometric data in a database. Based on this data, the server's AI agent performs emotion analysis. Specifically, the AI ​​analyzes heart rate fluctuations and changes in body temperature to estimate the user's emotional state. The input is data packets sent from the terminal, and the output is an evaluation result regarding the user's emotional state.

[0499] Step 4:

[0500] The server's generation AI model generates relaxation scenarios based on the results of emotional state evaluations. Specifically, the generation model uses prompts to design an environment and activities optimized for the user. The input is the emotional evaluation result, and the output is the relaxation scenario proposed to the user.

[0501] Step 5:

[0502] The device presents the generated relaxation scenario to the user through a VR or AR device. The user then begins interacting with the pet in the virtual environment. Specifically, the user puts on a VR device and takes a walk with the pet in a virtual forest. The input is the relaxation scenario, and the output is the virtual reality environment experienced by the user.

[0503] Step 6:

[0504] The server re-analyzes the user's biometric data after the interaction to evaluate the relaxation effect. Specifically, the server analyzes the newly acquired biometric data to determine whether stress has been reduced. The input is the biometric data after the interaction, and the output is the evaluation of the relaxation effect.

[0505] Step 7:

[0506] The server dynamically adjusts the scenario based on the evaluation results obtained, individually optimizing the next experience. Specifically, the server uses AI to analyze the feedback and redesign a scenario that is more suitable for the user. The input is the evaluation of the relaxation effect, and the output is the improved relaxation scenario.

[0507] (Application Example 1)

[0508] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0509] In modern society, many individuals are exposed to mental burden and stress, but there are limited methods to effectively alleviate this. Existing relaxation methods lack optimization based on individual emotional states and biometric information, making it difficult to provide a personalized experience. Therefore, there is a need for a system that can provide appropriate and sustained relaxation effects to individual users.

[0510] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0511] In this invention, the server includes means for a device that detects the user's biometric information, means for a program that analyzes the user's emotional state based on the biometric information, and means for an information processing system that generates a relaxation scenario suitable for the user based on the emotional state. This makes it possible to generate a scenario that enhances the relaxation effect according to the individual user's state.

[0512] A "device that detects a user's biometric information" is a measuring device that collects data on a user's physical condition, such as heart rate and body temperature.

[0513] A "program for analyzing emotional states" is software that uses a user's biometric information to estimate their emotions and psychological state at that time.

[0514] A "relaxation scenario generation information processing system" is a computer system that performs processing to design the optimal relaxation experience for the user based on the analyzed emotional state.

[0515] A "display device using virtual reality or augmented reality technology" is a visual display device used to present a generated relaxation scenario to a user, and is a device that can overlay virtual information onto a real-world environment.

[0516] A "feedback device" is a device or system for recording the reactions and results obtained from user interaction.

[0517] "Activities that reduce mental burden" refer to actions and experiences that users engage in to alleviate stress and tension and to calm their mind and body.

[0518] The "individualized optimization function" is a function that adjusts the relaxation experience provided based on the characteristics and needs of each individual user, in order to maximize its effectiveness.

[0519] The system implementing this invention consists of a user, a terminal, and a server. The user first wears a biofeedback sensor and accesses the system using smart glasses or a head-mounted display. The terminal collects biometric information (heart rate and body temperature) from the sensor in real time and transmits the data to the server using a secure communication protocol. Encryption technologies such as TLS are used for this communication.

[0520] On the server, an AI agent analyzes the received biometric information to estimate the user's emotional state. The Python scikit-learn library is expected to be used for emotion analysis. The analysis results are input into a generative AI model, which generates an optimal relaxation scenario based on the user's emotional state. Frameworks such as TensorFlow and PyTorch are used for this generative AI model. The generated scenario is sent to the device as highly personalized content.

[0521] The device presents this scenario to the user using virtual reality or augmented reality technology. This allows the user to interact with their pet in a natural setting and experience relaxation. For example, a scene is provided in which the user takes a walk with their pet in a virtual forest. The user's interaction is continuously monitored by the device, and biometric data is sent back to the server. The server uses this data to evaluate the effectiveness of the user's experience and uses the feedback to adjust the next scenario.

[0522] As a concrete example, consider the case where an elderly person living alone uses this system. If the system detects that the user's heart rate is higher than normal, a relaxing scenario of taking a walk in the forest is generated and presented. As a result, the user's heart rate stabilizes, alleviating feelings of loneliness and stress.

[0523] An example of a prompt message would be, "The user's heart rate is high. Please generate a scenario where they spend time on a calm beach." This instruction would be input to the generation AI model.

[0524] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0525] Step 1:

[0526] The user wears a biofeedback sensor. This allows the device to begin measuring biometric information such as heart rate and body temperature in real time. The measured data is transmitted to the server in its original format using a secure protocol. The input is biometric information, and the output is data transfer to the server.

[0527] Step 2:

[0528] The server analyzes the received biometric information. The AI ​​agent takes in the data and performs sentiment analysis using the Python scikit-learn library. In this analysis, the emotional state is quantified and evaluated. The input is biometric information, and the output is quantified emotional state data.

[0529] Step 3:

[0530] A server-based AI model generates the optimal relaxation scenario for the user based on emotional state data. This process utilizes TensorFlow or PyTorch, constructing the scenario based on prompt statements. Inputs are quantified emotional states and prompt statements, while output is the relaxation scenario.

[0531] Step 4:

[0532] The terminal presents the user with a relaxation scenario received from the server using a device that utilizes virtual reality or augmented reality technology. The user then begins the relaxation experience through this scenario. The input is the relaxation scenario, and the output is the virtual experience through sight and sound.

[0533] Step 5:

[0534] The user interacts with the presented scenario. The device continuously monitors the user's biometric information and collects new biometric data. This data is also securely transmitted to the server. The input is the biometric information during the interaction, and the output is the data transmitted to the server.

[0535] Step 6:

[0536] The server re-analyzes biometric data after the interaction. It uses feedback to evaluate the relaxation effect and utilize this information to generate the next scenario. This process continuously optimizes the user experience. The input is biometric data after the interaction, and the output is the feedback result.

[0537] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0538] This invention provides a model for a system that accurately analyzes a user's emotional state and provides relaxation scenarios based on that analysis. This system encompasses a series of functions, from collecting the user's biometric information and recognizing their emotional state to providing a relaxation experience tailored to their individual needs.

[0539] 1. Collection of biological and non-biological information

[0540] The user provides biometric information such as heart rate and body temperature through a biofeedback sensor worn on the device. Simultaneously, the device collects non-biometric information such as the user's voice, gaze, and facial expressions. This information is sent to the emotion engine.

[0541] 2. Recognition of emotional states by the emotion engine

[0542] The emotion engine installed on the server integrates and analyzes collected biometric and non-biometric information. This process utilizes machine learning algorithms and leverages the user's past data. The emotion engine determines the user's emotional state and passes the result to a relaxation scenario generation model.

[0543] 3. Generating Relaxation Scenarios

[0544] Based on the results of the emotion engine, a generative model on the server creates a relaxation scenario suitable for the user. This scenario includes activities that reduce the user's stress and promote mental stability.

[0545] 4. Scenario presentation using virtual devices

[0546] The selected relaxation scenario is presented to the user by the device via a VR / AR device. The user can then experience interaction with a virtual pet in this environment and relax.

[0547] 5. Monitoring and feedback on interactions

[0548] During user interaction, the device continuously monitors biometric information. This data is then sent back to the server for analysis and evaluation. The evaluation results provide information to individually optimize the user experience and are used to adjust scenarios for future interactions.

[0549] Specific example

[0550] For example, consider a scenario where a user is busy at work and the system is being used. Sensors detect an increase in the user's heart rate, and a camera observes frown lines indicating the user's tension. Based on this information, the emotion engine determines that the user is stressed, and the server's generative model creates a scenario in which the user relaxes in a virtual forest and plays with a virtual pet. The user then wears a VR headset and experiences relaxation through the created scenario.

[0551] Thus, the present invention enables an optimal relaxation experience tailored to the user's emotional state through advanced analysis combining biological and non-biological information.

[0552] The following describes the processing flow.

[0553] Step 1:

[0554] The user wears a device equipped with biofeedback sensors to measure heart rate and body temperature, as well as a camera and microphone to recognize facial expressions and voice. This initiates the collection of both biological and non-biometric information.

[0555] Step 2:

[0556] The device transmits collected biometric information (heart rate, body temperature, etc.) and non-biometric information (facial expressions, voice, gaze, etc.) to the server in a single batch. Real-time data transmission enables rapid analysis.

[0557] Step 3:

[0558] The server inputs the received data into the emotion engine. The emotion engine uses machine learning algorithms to analyze this data and accurately evaluate the user's emotional state. Past data is also referenced to perform more accurate emotion recognition.

[0559] Step 4:

[0560] The server's generation model uses emotional state information provided by the emotion engine to generate relaxation scenarios. These scenarios are designed to include activities that reduce user stress and promote psychological stability.

[0561] Step 5:

[0562] The terminal presents relaxation scenarios provided by the server to the user via a VR / AR device. The user experiences relaxation while interacting with a virtual pet in this virtual space.

[0563] Step 6:

[0564] During the interaction, the device continuously monitors the user's biometric information and records the user's responses. This information is then sent back to the server for analysis of emotional changes and stress reduction effects during the experience.

[0565] Step 7:

[0566] Based on feedback after the interaction, the server individually optimizes the next relaxation scenario. This adjustment allows users to experience more effective relaxation in subsequent sessions.

[0567] (Example 2)

[0568] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0569] Traditional relaxation systems have struggled to accurately grasp a user's emotional state and provide an optimal relaxation scenario accordingly. As a result, user experiences tend to be uniform, making it difficult to meet individual needs. Furthermore, feedback cannot be fully utilized, and scenarios are not incorporated into future sessions.

[0570] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0571] In this invention, the server includes means for a device that detects biometric and non-biometric information, means for artificial intelligence that analyzes the user's emotional state, and means for a generative AI model that generates a suitable relaxation scenario. This makes it possible to provide a personalized relaxation experience tailored to the user's emotional state.

[0572] "Biometric information" refers to data about the user's body, such as heart rate and body temperature.

[0573] "Non-biometric information" refers to data related to a user's behavior and state other than their physical body, such as their voice, gaze, and facial expressions.

[0574] "Artificial intelligence" refers to technology that analyzes a user's biometric and non-biometric information to recognize their emotional state.

[0575] A "generative AI model" refers to a technology that generates relaxation scenarios suitable for the user based on their analyzed emotional state.

[0576] A "relaxation scenario" refers to a set of activities and environments designed to reduce user stress and promote mental well-being.

[0577] A "virtual reality display device" refers to a device that provides users with a virtual experience.

[0578] An "augmented reality display device" refers to a device that combines and displays information from the real world with digital information.

[0579] "Feedback" refers to information obtained during a user's relaxation scenario experience, which is used to optimize the scenario for the next time.

[0580] This invention provides a specific embodiment of a system that analyzes a user's emotional state with high accuracy and provides relaxation scenarios based on that analysis. This system collects the user's biometric and non-biometric information and uses it to provide an individually optimized relaxation experience.

[0581] The system configuration involves the user wearing a biofeedback sensor to detect biometric information such as heart rate and body temperature. Furthermore, the terminal simultaneously collects non-biometric information such as the user's voice, gaze, and facial expressions using a camera and microphone. This collected information is analyzed by artificial intelligence (AI) installed on a server. The AI ​​utilizes machine learning algorithms to analyze past user data and recognize the user's emotional state.

[0582] Subsequently, the server uses a generative AI model to generate relaxation scenarios based on the analyzed emotional state. This generative AI model references various data to create scenarios designed to reduce user stress and promote mental stability. The generated scenarios are then visually presented to the user through a virtual reality (VR) or augmented reality (AR) display device.

[0583] For example, if a user is experiencing excessive stress at work, sensors might detect an increased heart rate and a tense facial expression. Based on this information, the server's artificial intelligence determines that the user is anxious and generates a virtual beach relaxation scenario through a generative AI model. This scenario includes the sound of ocean waves and a tranquil landscape, which the user experiences through a virtual reality device.

[0584] A concrete example of a prompt message might be the instruction, "Generate a relaxation scenario to provide the optimal calming environment based on the user's current emotional state."

[0585] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0586] Step 1:

[0587] The user wears biofeedback sensors that detect heart rate and body temperature, while the device uses a camera and microphone to collect non-biometric information such as the user's voice, gaze, and facial expressions. Both biometric and non-biometric information are used as input, and this data is necessary for analyzing the user's emotional state. The device then transmits this input data to a server.

[0588] Step 2:

[0589] Based on biometric and non-biometric data sent to the server, artificial intelligence analyzes the user's emotional state. This process utilizes machine learning algorithms and also references the user's past data. Integrated biometric and non-biometric data is used as input, analyzed to determine the user's emotional state, and the analysis results are passed to a generating AI model as output.

[0590] Step 3:

[0591] The server's AI model generates relaxation scenarios tailored to the user based on emotion analysis results. This model considers past data and existing relaxation techniques to output the optimal scenario for reducing user stress. The input is the result of emotional state analysis, and the output is the generated relaxation scenario.

[0592] Step 4:

[0593] The device presents relaxation scenarios generated by a generated AI model to the user via a VR / AR device. This allows the user to experience relaxation in a virtual environment. The input is the generated relaxation scenario, and the output is presented as the user's experience.

[0594] Step 5:

[0595] While the user experiences the scenario, the device continuously monitors biometric information and sends this data to the server. The server analyzes the received data and evaluates the effectiveness of the relaxation scenario as feedback. The input is biometric information during the experience, and the output is feedback data that is used to optimize the next scenario.

[0596] (Application Example 2)

[0597] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0598] Conventional relaxation systems have struggled to accurately analyze a user's emotional state and provide an individually optimized relaxation experience. Furthermore, they lacked the means to create relaxation scenarios that adapt to the user in real time and to effectively reduce user stress by controlling acoustic stimuli and aromas.

[0599] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0600] In this invention, the server includes means for analyzing the user's emotional state based on biometric information, means for generating a relaxation scenario based on the emotional state, and means for controlling acoustic stimuli and aromas based on interaction data. This makes it possible to provide an optimal relaxation experience that is adapted to the user's emotional state in real time.

[0601] "Biometric information" refers to biological data such as the user's heart rate and body temperature, which indicates the user's current physical condition.

[0602] "Emotional state" refers to the user's mood and psychological state, and is indicated by the results of the analysis of biometric and non-biometric information.

[0603] A "relaxation scenario" is a plan of activities and environments designed to reduce user stress and promote mental well-being.

[0604] A "visual device" is a device used to present virtual images or information to a user, and typically includes virtual reality and augmented reality devices.

[0605] "Feedback" refers to information collected from user interactions, including their reactions and results, which is used to adjust the system.

[0606] "Acoustic stimuli" refer to sounds or music used to influence a user's emotional state.

[0607] "Fragrance" refers to scents released through diffusers or other means to improve the user's emotional state.

[0608] The system for implementing this invention is designed to analyze the user's emotional state and provide an appropriate relaxation scenario.

[0609] The server works in conjunction with sensor devices to detect the user's biometric information, such as heart rate and body temperature, and collects this data. The collected data is processed by an artificial intelligence system that analyzes emotions, and the user's current emotional state is evaluated. In this process, the server uses machine learning algorithms to determine the emotional state, utilizing past data as well.

[0610] Based on these results, a generative model on the server creates an optimal relaxation scenario for the user. The generated scenario is presented to the user through visual devices and devices that enhance the sense of presence, and includes control over acoustic stimuli and aromas. Feedback obtained through user interaction is collected as sensor data during the interaction and sent back to the server. This feedback is used to sequentially adjust the relaxation scenario and optimize the user experience.

[0611] As a concrete example, if the server detects an increase in the user's heart rate and facial tension, it generates a virtual environment, including a "relaxing experience in a forest," based on this information and presents it through a VR headset. During this time, healing music plays, and a relaxing scent is diffused.

[0612] The prompt given to the generating AI model is in the form of, "Based on the user's heart rate data and past stress levels, what combination of music and scent is optimal?"

[0613] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0614] Step 1:

[0615] The device collects biometric information such as heart rate and body temperature in real time from sensor devices worn by the user. In addition, it uses a camera and microphone to capture the user's facial expressions and voice. This information is compiled as biometric and non-biometric information and transmitted to a server.

[0616] Step 2:

[0617] The server uses received biometric and non-biometric information as input to analyze the user's emotional state based on machine learning algorithms. Referring to previously collected data, it accurately determines the user's stress level and psychological state. As a result, the user's current emotional state is output.

[0618] Step 3:

[0619] The server's generation model automatically generates relaxation scenarios based on analysis results. Specifically, it creates scenarios incorporating combinations of music, video, and scent, ensuring that these provide the most effective relaxation experience for the user.

[0620] Step 4:

[0621] The user's device presents the generated relaxation scenario to the user through visual devices. Specifically, it uses VR headsets or AR devices to provide the user with a virtual environment. Simultaneously, it plays appropriate music using sound equipment and activates an aroma diffuser to release fragrance.

[0622] Step 5:

[0623] During interaction, the device continuously monitors the user's biometric information and acquires new data. This data is recorded as feedback and resent to the server. This feedback serves as foundational data for optimizing the relaxation scenario for future sessions.

[0624] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0625] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0626] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0627] [Fourth Embodiment]

[0628] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0629] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0630] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0631] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0632] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0633] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0634] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0635] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0636] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0637] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0638] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0639] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0640] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0641] The system of the present invention aims to evaluate the user's biometric information and provide an individually optimized relaxation environment. The embodiments thereof are described in detail below.

[0642] 1. Sensor installation and data collection

[0643] The user first wears a biofeedback sensor, which measures biometric data such as heart rate and body temperature in real time. The device periodically collects this biometric data and sends it to a server using a secure protocol.

[0644] 2. Data Analysis

[0645] The server stores the received biometric data in a database, and an AI agent analyzes this data. The artificial intelligence uses an emotion analysis algorithm to assess the user's current emotional state and determine how stressed the user is.

[0646] 3. Generating Relaxation Scenarios

[0647] The generative model on the server generates an optimal relaxation scenario based on an emotional state assessment. This scenario includes activities and experiences important for reducing user stress and is presented via a VR / AR device.

[0648] 4. Interaction and obtaining feedback

[0649] The device presents the generated scenario to the user. The user then begins interacting with the virtual pet using a VR / AR device. For example, the user can take a walk with the pet in a virtual space or enjoy a relaxing environment together. Throughout the interaction, the device continues to monitor the user's biometric data.

[0650] 5. Evaluation of results and improvement

[0651] The server re-analyzes biometric data after the interaction to evaluate the relaxation effect. Based on the feedback, the system sequentially adjusts the scenario for the next scenario generation, providing the user with a better, more personalized experience.

[0652] Specific example

[0653] For example, consider a user who experiences stress in their daily work using this system. A sensor attached to the user detects that their heart rate is higher than normal, and an AI agent determines that they are experiencing high stress levels. Based on this information, a generative model creates a scenario in which the user can relax with a virtual pet in nature. Through a VR device, the user experiences relaxation while exploring a virtual forest with their pet. This process stabilizes the user's heart rate and leads to overall relaxation.

[0654] In this way, the system can provide users with individually optimized relaxation experiences, promoting their physical and mental well-being.

[0655] The following describes the processing flow.

[0656] Step 1:

[0657] The device collects biometric information from biofeedback sensors attached to the user. The collected data includes heart rate, body temperature, and stress level. This data reflects the user's physiological state.

[0658] Step 2:

[0659] The device transmits collected biometric information to the server using a secure communication protocol. This transmission occurs in real time, enabling rapid monitoring of changes in the user's state.

[0660] Step 3:

[0661] The server records the biometric information transmitted from the terminal into a database and prepares it for analysis. In this step, the integrity of the data is verified to ensure that subsequent analysis is performed correctly.

[0662] Step 4:

[0663] An AI agent on the server analyzes the collected biometric information. In particular, it uses an emotion analysis algorithm to evaluate the user's stress level and emotional state. Based on this evaluation, it determines a relaxation scenario that meets the user's needs.

[0664] Step 5:

[0665] The server-side generative model generates an optimal relaxation scenario based on the evaluation results of the AI ​​agent. This scenario includes activities designed to reduce user stress and promote relaxation, and is presented to the user via a VR / AR device.

[0666] Step 6:

[0667] The device receives the generated relaxation scenario and presents it to the user. The user then uses a VR / AR device to experience interaction with a virtual pet in a virtual space. This interaction is primarily designed for the user's relaxation.

[0668] Step 7:

[0669] Users interact with virtual pets through virtual space interactions. The pets react to the user's actions, providing real-time feedback.

[0670] Step 8:

[0671] The device continuously monitors the user's biometric information during interaction and records changes in responses. This data is important for evaluating the user's relaxation effect.

[0672] Step 9:

[0673] The server re-analyzes the data after the interaction to evaluate how effective the relaxation was. Based on this, the server makes adjustments for generating future scenarios.

[0674] Step 10:

[0675] After a session ends, users receive feedback from the system regarding their performance and stress levels. This allows them to objectively understand their own state. Furthermore, this feedback serves as important data for customizing future experiences.

[0676] (Example 1)

[0677] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0678] In modern society, many individuals suffer from stress and emotional imbalances, which can negatively impact their health. In this context, there is a need to provide relaxation methods optimized for each individual user's condition. However, existing methods struggle to accurately assess individual emotional states and provide flexible and effective relaxation experiences based on those assessments. A solution to this challenge is urgently needed.

[0679] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0680] In this invention, the server includes means for measuring biometric data, means for analyzing individual emotional states, and means for designing a suitable relaxation scenario. This makes it possible to evaluate the user's emotional state in real time and provide a customized relaxation experience accordingly.

[0681] "Biometric data" includes information about the user's physical condition, such as heart rate and body temperature.

[0682] A "measuring device" refers to equipment used to collect a user's biometric data.

[0683] An "information processing device" refers to a computer system used to analyze a user's emotional state based on biometric data.

[0684] A "computational model" refers to a mechanism that includes algorithms for designing relaxation scenarios tailored to the user's emotional state.

[0685] A "visual device" refers to a device that presents a generated relaxation scenario to the user and embodies a virtual reality environment.

[0686] "Dialogue" refers to the process by which a user interacts with a system, and includes actions that enable the acquisition of feedback.

[0687] "Evaluation information" refers to information obtained through interaction with users to assess the quality of their experience.

[0688] An "adaptive control device" refers to a mechanism that dynamically adjusts relaxation scenarios based on evaluation information to individually optimize the user experience.

[0689] This invention is a system that utilizes a user's biometric information to provide an individually optimized relaxation experience. The system consists of a biofeedback sensor, a data processing device, a generative AI model, and a VR / AR device as its main components.

[0690] First, the device attaches a biofeedback sensor to the user, measuring biometric data such as heart rate and body temperature in real time. Next, the device collects this data at regular intervals and transmits it to a server using a secure protocol. This makes the biometric data available for use.

[0691] The server stores the received biometric data in a database. Furthermore, an AI agent installed on the server uses an emotion analysis algorithm to analyze the data and evaluate the user's emotional state. Based on this analysis, a generative AI model designs the optimal relaxation scenario. An example of a prompt would be, "Create a scenario for the user to relax with a pet in a natural environment."

[0692] The resulting relaxation scenarios are presented to the user via a device. The user wears a VR device and experiences the scenarios in virtual reality, enjoying activities such as exploring a virtual forest with a pet. Throughout this process, the device continuously monitors the user's biometric data and evaluates the relaxation effect in real time.

[0693] Finally, the server uses the feedback information to improve the next relaxation scenario, further enhancing the user's experience. This system allows users to receive a personalized and enriching relaxation experience that leads to stress reduction.

[0694] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0695] Step 1:

[0696] The user wears a biofeedback sensor on their body. The sensor measures biometric data such as heart rate and body temperature in real time. Specifically, the sensor makes contact with the skin and acquires biometric information as an electrical signal. The input is the user's physical state, and the output is the measured biometric data.

[0697] Step 2:

[0698] The terminal acquires biometric data collected from sensors at regular time intervals and transmits it to the server using a secure communication protocol. Specifically, the terminal relays the data via communication methods such as Bluetooth or Wi-Fi. The input is biometric data from the sensors, and the output is data packets sent to the server.

[0699] Step 3:

[0700] The server records the received biometric data in a database. Based on this data, the server's AI agent performs emotion analysis. Specifically, the AI ​​analyzes heart rate fluctuations and changes in body temperature to estimate the user's emotional state. The input is data packets sent from the terminal, and the output is an evaluation result regarding the user's emotional state.

[0701] Step 4:

[0702] The server's generation AI model generates relaxation scenarios based on the results of emotional state evaluations. Specifically, the generation model uses prompts to design an environment and activities optimized for the user. The input is the emotional evaluation result, and the output is the relaxation scenario proposed to the user.

[0703] Step 5:

[0704] The device presents the generated relaxation scenario to the user through a VR or AR device. The user then begins interacting with the pet in the virtual environment. Specifically, the user puts on a VR device and takes a walk with the pet in a virtual forest. The input is the relaxation scenario, and the output is the virtual reality environment experienced by the user.

[0705] Step 6:

[0706] The server re-analyzes the user's biometric data after the interaction to evaluate the relaxation effect. Specifically, the server analyzes the newly acquired biometric data to determine whether stress has been reduced. The input is the biometric data after the interaction, and the output is the evaluation of the relaxation effect.

[0707] Step 7:

[0708] The server dynamically adjusts the scenario based on the evaluation results obtained, individually optimizing the next experience. Specifically, the server uses AI to analyze the feedback and redesign a scenario that is more suitable for the user. The input is the evaluation of the relaxation effect, and the output is the improved relaxation scenario.

[0709] (Application Example 1)

[0710] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0711] In modern society, many individuals are exposed to mental burden and stress, but there are limited methods to effectively alleviate this. Existing relaxation methods lack optimization based on individual emotional states and biometric information, making it difficult to provide a personalized experience. Therefore, there is a need for a system that can provide appropriate and sustained relaxation effects to individual users.

[0712] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0713] In this invention, the server includes means for a device that detects the user's biometric information, means for a program that analyzes the user's emotional state based on the biometric information, and means for an information processing system that generates a relaxation scenario suitable for the user based on the emotional state. This makes it possible to generate a scenario that enhances the relaxation effect according to the individual user's state.

[0714] A "device that detects a user's biometric information" is a measuring device that collects data on a user's physical condition, such as heart rate and body temperature.

[0715] A "program for analyzing emotional states" is software that uses a user's biometric information to estimate their emotions and psychological state at that time.

[0716] A "relaxation scenario generation information processing system" is a computer system that performs processing to design the optimal relaxation experience for the user based on the analyzed emotional state.

[0717] A "display device using virtual reality or augmented reality technology" is a visual display device used to present a generated relaxation scenario to a user, and is a device that can overlay virtual information onto a real-world environment.

[0718] A "feedback device" is a device or system for recording the reactions and results obtained from user interaction.

[0719] "Activities that reduce mental burden" refer to actions and experiences that users engage in to alleviate stress and tension and to calm their mind and body.

[0720] The "individualized optimization function" is a function that adjusts the relaxation experience provided based on the characteristics and needs of each individual user, in order to maximize its effectiveness.

[0721] The system implementing this invention consists of a user, a terminal, and a server. The user first wears a biofeedback sensor and accesses the system using smart glasses or a head-mounted display. The terminal collects biometric information (heart rate and body temperature) from the sensor in real time and transmits the data to the server using a secure communication protocol. Encryption technologies such as TLS are used for this communication.

[0722] On the server, an AI agent analyzes the received biometric information to estimate the user's emotional state. The Python scikit-learn library is expected to be used for emotion analysis. The analysis results are input into a generative AI model, which generates an optimal relaxation scenario based on the user's emotional state. Frameworks such as TensorFlow and PyTorch are used for this generative AI model. The generated scenario is sent to the device as highly personalized content.

[0723] The device presents this scenario to the user using virtual reality or augmented reality technology. This allows the user to interact with their pet in a natural setting and experience relaxation. For example, a scene is provided in which the user takes a walk with their pet in a virtual forest. The user's interaction is continuously monitored by the device, and biometric data is sent back to the server. The server uses this data to evaluate the effectiveness of the user's experience and uses the feedback to adjust the next scenario.

[0724] As a concrete example, consider the case where an elderly person living alone uses this system. If the system detects that the user's heart rate is higher than normal, a relaxing scenario of taking a walk in the forest is generated and presented. As a result, the user's heart rate stabilizes, alleviating feelings of loneliness and stress.

[0725] An example of a prompt message would be, "The user's heart rate is high. Please generate a scenario where they spend time on a calm beach." This instruction would be input to the generation AI model.

[0726] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0727] Step 1:

[0728] The user wears a biofeedback sensor. This allows the device to begin measuring biometric information such as heart rate and body temperature in real time. The measured data is transmitted to the server in its original format using a secure protocol. The input is biometric information, and the output is data transfer to the server.

[0729] Step 2:

[0730] The server analyzes the received biometric information. The AI ​​agent takes in the data and performs sentiment analysis using the Python scikit-learn library. In this analysis, the emotional state is quantified and evaluated. The input is biometric information, and the output is quantified emotional state data.

[0731] Step 3:

[0732] A server-based AI model generates the optimal relaxation scenario for the user based on emotional state data. This process utilizes TensorFlow or PyTorch, constructing the scenario based on prompt statements. Inputs are quantified emotional states and prompt statements, while output is the relaxation scenario.

[0733] Step 4:

[0734] The terminal presents the user with a relaxation scenario received from the server using a device that utilizes virtual reality or augmented reality technology. The user then begins the relaxation experience through this scenario. The input is the relaxation scenario, and the output is the virtual experience through sight and sound.

[0735] Step 5:

[0736] The user interacts with the presented scenario. The device continuously monitors the user's biometric information and collects new biometric data. This data is also securely transmitted to the server. The input is the biometric information during the interaction, and the output is the data transmitted to the server.

[0737] Step 6:

[0738] The server re-analyzes biometric data after the interaction. It uses feedback to evaluate the relaxation effect and utilize this information to generate the next scenario. This process continuously optimizes the user experience. The input is biometric data after the interaction, and the output is the feedback result.

[0739] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0740] This invention provides a model for a system that accurately analyzes a user's emotional state and provides relaxation scenarios based on that analysis. This system encompasses a series of functions, from collecting the user's biometric information and recognizing their emotional state to providing a relaxation experience tailored to their individual needs.

[0741] 1. Collection of biological and non-biological information

[0742] The user provides biometric information such as heart rate and body temperature through a biofeedback sensor worn on the device. Simultaneously, the device collects non-biometric information such as the user's voice, gaze, and facial expressions. This information is sent to the emotion engine.

[0743] 2. Recognition of emotional states by the emotion engine

[0744] The emotion engine installed on the server integrates and analyzes collected biometric and non-biometric information. This process utilizes machine learning algorithms and leverages the user's past data. The emotion engine determines the user's emotional state and passes the result to a relaxation scenario generation model.

[0745] 3. Generating Relaxation Scenarios

[0746] Based on the results of the emotion engine, a generative model on the server creates a relaxation scenario suitable for the user. This scenario includes activities that reduce the user's stress and promote mental stability.

[0747] 4. Scenario presentation using virtual devices

[0748] The selected relaxation scenario is presented to the user by the device via a VR / AR device. The user can then experience interaction with a virtual pet in this environment and relax.

[0749] 5. Monitoring and feedback on interactions

[0750] During user interaction, the device continuously monitors biometric information. This data is then sent back to the server for analysis and evaluation. The evaluation results provide information to individually optimize the user experience and are used to adjust scenarios for future interactions.

[0751] Specific example

[0752] For example, consider a scenario where a user is busy at work and the system is being used. Sensors detect an increase in the user's heart rate, and a camera observes frown lines indicating the user's tension. Based on this information, the emotion engine determines that the user is stressed, and the server's generative model creates a scenario in which the user relaxes in a virtual forest and plays with a virtual pet. The user then wears a VR headset and experiences relaxation through the created scenario.

[0753] Thus, the present invention enables an optimal relaxation experience tailored to the user's emotional state through advanced analysis combining biological and non-biological information.

[0754] The following describes the processing flow.

[0755] Step 1:

[0756] The user wears a device equipped with biofeedback sensors to measure heart rate and body temperature, as well as a camera and microphone to recognize facial expressions and voice. This initiates the collection of both biological and non-biometric information.

[0757] Step 2:

[0758] The device transmits collected biometric information (heart rate, body temperature, etc.) and non-biometric information (facial expressions, voice, gaze, etc.) to the server in a single batch. Real-time data transmission enables rapid analysis.

[0759] Step 3:

[0760] The server inputs the received data into the emotion engine. The emotion engine uses machine learning algorithms to analyze this data and accurately evaluate the user's emotional state. Past data is also referenced to perform more accurate emotion recognition.

[0761] Step 4:

[0762] The server's generation model uses emotional state information provided by the emotion engine to generate relaxation scenarios. These scenarios are designed to include activities that reduce user stress and promote psychological stability.

[0763] Step 5:

[0764] The terminal presents relaxation scenarios provided by the server to the user via a VR / AR device. The user experiences relaxation while interacting with a virtual pet in this virtual space.

[0765] Step 6:

[0766] During the interaction, the device continuously monitors the user's biometric information and records the user's responses. This information is then sent back to the server for analysis of emotional changes and stress reduction effects during the experience.

[0767] Step 7:

[0768] Based on feedback after the interaction, the server individually optimizes the next relaxation scenario. This adjustment allows users to experience more effective relaxation in subsequent sessions.

[0769] (Example 2)

[0770] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0771] Traditional relaxation systems have struggled to accurately grasp a user's emotional state and provide an optimal relaxation scenario accordingly. As a result, user experiences tend to be uniform, making it difficult to meet individual needs. Furthermore, feedback cannot be fully utilized, and scenarios are not incorporated into future sessions.

[0772] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0773] In this invention, the server includes means for a device that detects biometric and non-biometric information, means for artificial intelligence that analyzes the user's emotional state, and means for a generative AI model that generates a suitable relaxation scenario. This makes it possible to provide a personalized relaxation experience tailored to the user's emotional state.

[0774] "Biometric information" refers to data about the user's body, such as heart rate and body temperature.

[0775] "Non-biometric information" refers to data related to a user's behavior and state other than their physical body, such as their voice, gaze, and facial expressions.

[0776] "Artificial intelligence" refers to technology that analyzes a user's biometric and non-biometric information to recognize their emotional state.

[0777] A "generative AI model" refers to a technology that generates relaxation scenarios suitable for the user based on their analyzed emotional state.

[0778] A "relaxation scenario" refers to a set of activities and environments designed to reduce user stress and promote mental well-being.

[0779] A "virtual reality display device" refers to a device that provides users with a virtual experience.

[0780] An "augmented reality display device" refers to a device that combines and displays information from the real world with digital information.

[0781] "Feedback" refers to information obtained during a user's relaxation scenario experience, which is used to optimize the scenario for the next time.

[0782] This invention provides a specific embodiment of a system that analyzes a user's emotional state with high accuracy and provides relaxation scenarios based on that analysis. This system collects the user's biometric and non-biometric information and uses it to provide an individually optimized relaxation experience.

[0783] The system configuration involves the user wearing a biofeedback sensor to detect biometric information such as heart rate and body temperature. Furthermore, the terminal simultaneously collects non-biometric information such as the user's voice, gaze, and facial expressions using a camera and microphone. This collected information is analyzed by artificial intelligence (AI) installed on a server. The AI ​​utilizes machine learning algorithms to analyze past user data and recognize the user's emotional state.

[0784] Subsequently, the server uses a generative AI model to generate relaxation scenarios based on the analyzed emotional state. This generative AI model references various data to create scenarios designed to reduce user stress and promote mental stability. The generated scenarios are then visually presented to the user through a virtual reality (VR) or augmented reality (AR) display device.

[0785] For example, if a user is experiencing excessive stress at work, sensors might detect an increased heart rate and a tense facial expression. Based on this information, the server's artificial intelligence determines that the user is anxious and generates a virtual beach relaxation scenario through a generative AI model. This scenario includes the sound of ocean waves and a tranquil landscape, which the user experiences through a virtual reality device.

[0786] A concrete example of a prompt message might be the instruction, "Generate a relaxation scenario to provide the optimal calming environment based on the user's current emotional state."

[0787] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0788] Step 1:

[0789] The user wears biofeedback sensors that detect heart rate and body temperature, while the device uses a camera and microphone to collect non-biometric information such as the user's voice, gaze, and facial expressions. Both biometric and non-biometric information are used as input, and this data is necessary for analyzing the user's emotional state. The device then transmits this input data to a server.

[0790] Step 2:

[0791] Based on biometric and non-biometric data sent to the server, artificial intelligence analyzes the user's emotional state. This process utilizes machine learning algorithms and also references the user's past data. Integrated biometric and non-biometric data is used as input, analyzed to determine the user's emotional state, and the analysis results are passed to a generating AI model as output.

[0792] Step 3:

[0793] The server's AI model generates relaxation scenarios tailored to the user based on emotion analysis results. This model considers past data and existing relaxation techniques to output the optimal scenario for reducing user stress. The input is the result of emotional state analysis, and the output is the generated relaxation scenario.

[0794] Step 4:

[0795] The device presents relaxation scenarios generated by a generated AI model to the user via a VR / AR device. This allows the user to experience relaxation in a virtual environment. The input is the generated relaxation scenario, and the output is presented as the user's experience.

[0796] Step 5:

[0797] While the user experiences the scenario, the device continuously monitors biometric information and sends this data to the server. The server analyzes the received data and evaluates the effectiveness of the relaxation scenario as feedback. The input is biometric information during the experience, and the output is feedback data that is used to optimize the next scenario.

[0798] (Application Example 2)

[0799] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0800] Conventional relaxation systems have struggled to accurately analyze a user's emotional state and provide an individually optimized relaxation experience. Furthermore, they lacked the means to create relaxation scenarios that adapt to the user in real time and to effectively reduce user stress by controlling acoustic stimuli and aromas.

[0801] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0802] In this invention, the server includes means for analyzing the user's emotional state based on biometric information, means for generating a relaxation scenario based on the emotional state, and means for controlling acoustic stimuli and aromas based on interaction data. This makes it possible to provide an optimal relaxation experience that is adapted to the user's emotional state in real time.

[0803] "Biometric information" refers to biological data such as the user's heart rate and body temperature, which indicates the user's current physical condition.

[0804] "Emotional state" refers to the user's mood and psychological state, and is indicated by the results of the analysis of biometric and non-biometric information.

[0805] A "relaxation scenario" is a plan of activities and environments designed to reduce user stress and promote mental well-being.

[0806] A "visual device" is a device used to present virtual images or information to a user, and typically includes virtual reality and augmented reality devices.

[0807] "Feedback" refers to information collected from user interactions, including their reactions and results, which is used to adjust the system.

[0808] "Acoustic stimuli" refer to sounds or music used to influence a user's emotional state.

[0809] "Fragrance" refers to scents released through diffusers or other means to improve the user's emotional state.

[0810] The system for implementing this invention is designed to analyze the user's emotional state and provide an appropriate relaxation scenario.

[0811] The server works in conjunction with sensor devices to detect the user's biometric information, such as heart rate and body temperature, and collects this data. The collected data is processed by an artificial intelligence system that analyzes emotions, and the user's current emotional state is evaluated. In this process, the server uses machine learning algorithms to determine the emotional state, utilizing past data as well.

[0812] Based on these results, a generative model on the server creates an optimal relaxation scenario for the user. The generated scenario is presented to the user through visual devices and devices that enhance the sense of presence, and includes control over acoustic stimuli and aromas. Feedback obtained through user interaction is collected as sensor data during the interaction and sent back to the server. This feedback is used to sequentially adjust the relaxation scenario and optimize the user experience.

[0813] As a concrete example, if the server detects an increase in the user's heart rate and facial tension, it generates a virtual environment, including a "relaxing experience in a forest," based on this information and presents it through a VR headset. During this time, healing music plays, and a relaxing scent is diffused.

[0814] The prompt given to the generating AI model is in the form of, "Based on the user's heart rate data and past stress levels, what combination of music and scent is optimal?"

[0815] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0816] Step 1:

[0817] The device collects biometric information such as heart rate and body temperature in real time from sensor devices worn by the user. In addition, it uses a camera and microphone to capture the user's facial expressions and voice. This information is compiled as biometric and non-biometric information and transmitted to a server.

[0818] Step 2:

[0819] The server uses received biometric and non-biometric information as input to analyze the user's emotional state based on machine learning algorithms. Referring to previously collected data, it accurately determines the user's stress level and psychological state. As a result, the user's current emotional state is output.

[0820] Step 3:

[0821] The server's generation model automatically generates relaxation scenarios based on analysis results. Specifically, it creates scenarios incorporating combinations of music, video, and scent, ensuring that these provide the most effective relaxation experience for the user.

[0822] Step 4:

[0823] The user's device presents the generated relaxation scenario to the user through visual devices. Specifically, it uses VR headsets or AR devices to provide the user with a virtual environment. Simultaneously, it plays appropriate music using sound equipment and activates an aroma diffuser to release fragrance.

[0824] Step 5:

[0825] During interaction, the device continuously monitors the user's biometric information and acquires new data. This data is recorded as feedback and resent to the server. This feedback serves as foundational data for optimizing the relaxation scenario for future sessions.

[0826] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0827] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0828] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0829] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0830] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. In the upper and lower directions of the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. Also, the upper side of the concentric circles is where "pleasant" emotions are located, and the lower side is where "unpleasant" emotions are located. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0831] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0832] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0833] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0834] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0835] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0836] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0837] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0838] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0839] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0840] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0841] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0842] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0843] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0844] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0845] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0846] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0847] The following is further disclosed regarding the embodiments described above.

[0848] (Claim 1)

[0849] A means equipped with a sensor that detects the user's biometric information,

[0850] A means equipped with artificial intelligence that analyzes the user's emotional state based on the aforementioned biometric information,

[0851] Means comprising a generative model that generates a relaxation scenario suitable for the user based on the aforementioned emotional state,

[0852] Means comprising a virtual reality or augmented reality device for presenting the aforementioned relaxation scenario to the user,

[0853] A means of obtaining feedback through interaction with users,

[0854] A system that includes this.

[0855] (Claim 2)

[0856] The system according to claim 1, wherein the relaxation scenario includes activities specifically designed to reduce the user's stress level.

[0857] (Claim 3)

[0858] The system according to claim 1, further comprising means for sequentially adjusting relaxation scenarios based on the aforementioned feedback to individually optimize the user experience.

[0859] "Example 1"

[0860] (Claim 1)

[0861] A means comprising a measuring device for measuring the user's biometric data,

[0862] A means comprising an information processing device for analyzing the user's emotional state based on the aforementioned biometric data,

[0863] A means comprising a computational model for designing a relaxation scenario suited to the aforementioned emotional state,

[0864] Means comprising a visual device that embodies a virtual environment for presenting the aforementioned relaxation scenario,

[0865] A means of obtaining evaluation information through interaction with users,

[0866] A means for dynamically adjusting scenarios based on evaluation information and providing an adaptive control device for individually optimizing the experience,

[0867] A system that includes this.

[0868] (Claim 2)

[0869] The system according to claim 1, wherein the relaxation scenario is designed based on an individual stress assessment.

[0870] (Claim 3)

[0871] The system according to claim 1, further comprising a feedback function for improving the quality of the next experience based on the analysis results of the aforementioned evaluation information.

[0872] "Application Example 1"

[0873] (Claim 1)

[0874] A means comprising a device for detecting the user's biometric information,

[0875] A means comprising a program that analyzes the user's emotional state based on the aforementioned biometric information,

[0876] A means comprising an information processing system that generates a relaxation scenario suitable for the user based on the aforementioned emotional state,

[0877] A means comprising a display device using virtual reality or augmented reality technology for presenting the relaxation scenario to the user,

[0878] A means for providing a device that obtains feedback through interaction with the user,

[0879] Information processing device including

[0880] (Claim 2)

[0881] The information processing apparatus according to claim 1, wherein the relaxation scenario includes activities that reduce the user's mental burden.

[0882] (Claim 3)

[0883] The information processing apparatus according to claim 1, comprising a function to repeatedly adjust the relaxation scenario based on the aforementioned feedback information and individually optimize the user experience.

[0884] "Example 2 of combining an emotion engine"

[0885] (Claim 1)

[0886] A means comprising a device for detecting the user's biometric and non-biometric information,

[0887] A means comprising artificial intelligence that analyzes the user's emotional state based on the aforementioned biometric and non-biometric information,

[0888] Means comprising a generative AI model that generates a relaxation scenario suitable for the user based on the aforementioned emotional state,

[0889] A means comprising a virtual reality or augmented reality display device for presenting the aforementioned relaxation scenario to the user,

[0890] A means of obtaining feedback through user interaction and reflecting that feedback in subsequent scenarios,

[0891] A system that includes this.

[0892] (Claim 2)

[0893] The system according to claim 1, wherein the relaxation scenario includes activities that have the effect of reducing the user's psychological stress level.

[0894] (Claim 3)

[0895] The system according to claim 1, further comprising means for dynamically adjusting relaxation scenarios based on the aforementioned feedback and individually optimizing the user experience.

[0896] "Application example 2 when combining with an emotional engine"

[0897] (Claim 1)

[0898] A means equipped with a sensor that detects the user's biometric information,

[0899] A means equipped with artificial intelligence that analyzes the user's emotional state based on the aforementioned biometric information,

[0900] Means comprising a generative model that generates a relaxation scenario suitable for the user based on the aforementioned emotional state,

[0901] Means comprising a visual device or a device for enhancing the sense of presence for presenting the relaxation scenario to the user,

[0902] A means of obtaining feedback through interaction with users,

[0903] Based on the data relating to the aforementioned interaction, means for controlling acoustic stimuli and aromas,

[0904] A system that includes this.

[0905] (Claim 2)

[0906] The system according to claim 1, wherein the relaxation scenario includes activities specifically designed to reduce the user's stress level.

[0907] (Claim 3)

[0908] The system according to claim 1, further comprising means for sequentially adjusting relaxation scenarios based on the aforementioned feedback to individually optimize the user experience. [Explanation of Symbols]

[0909] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means comprising a device for detecting the user's biometric information, A means comprising a program that analyzes the user's emotional state based on the aforementioned biometric information, A means comprising an information processing system that generates a relaxation scenario suitable for the user based on the aforementioned emotional state, A means comprising a display device using virtual reality or augmented reality technology for presenting the relaxation scenario to the user, A means for providing a device that obtains feedback through interaction with the user, Information processing device including

2. The information processing apparatus according to claim 1, wherein the relaxation scenario includes activities that reduce the user's mental burden.

3. The information processing apparatus according to claim 1, comprising a function to repeatedly adjust the relaxation scenario based on the aforementioned feedback information and individually optimize the user experience.

Images