system

Abstract

We provide the system. [Solution] A data acquisition device for collecting operational data, A data analysis device that runs machine learning algorithms to analyze the collected motion data, A feedback device that provides real-time feedback to the user based on the analysis results, A system including an augmented reality device to provide a training experience in a virtual environment.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern society, many individuals have difficulty obtaining opportunities for continuous and effective health management due to busy lifestyles and high personal training costs. As a result, there are problems that correct form of exercise and maintenance of appropriate lifestyle habits are hindered, and the motivation for exercise cannot be sustained. Furthermore, there is a lack of personalized feedback according to individual physical conditions and goals.

Means for Solving the Problems

[0005] The present invention solves the above problems by providing a system that includes a data collection device for collecting motion data, a data analysis device for analyzing data using machine learning algorithms, a feedback device that provides real-time feedback to the user based on the analysis results, and an augmented reality device for providing a training experience in a virtual environment. Furthermore, by providing a progress management device that calculates reward points and manages the user's progress, and a health management device that generates health improvement advice based on motion data and lifestyle data, the invention supports personalized and continuous training and health management.

[0006] A "data acquisition device" is a device used to acquire user movement data and sensor information from the body.

[0007] A "data analysis device" is a device that uses machine learning algorithms to analyze collected motion data and evaluate the accuracy and effectiveness of the user's actions.

[0008] A "feedback device" is a device that provides real-time instructions and advice to the user based on the analysis results generated by a data analysis device.

[0009] An "augmented reality device" is a device that visually provides a virtual training environment, allowing users to experience training sessions in a virtual space.

[0010] A "progress management device" is a device that monitors the user's training progress and uses reward points to maintain motivation.

[0011] A "health management device" is a device that evaluates a user's health status and generates improvement advice based on their activity data and lifestyle information.

[0012] A "machine learning algorithm" is a learning model used in the data analysis process to evaluate actions and generate feedback based on motion data.

[0013] "Reward points" are points used to show users their training progress and are an indicator that contributes to maintaining motivation. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] The system of the present invention enables efficient health management and training for users and has a multi-layered configuration including a data acquisition device, a data analysis device, a feedback device, an augmented reality device, a progress management device, and a health management device. The operation of each device and the overall system will be described below.

[0036] The device functions as a data collection device to collect user movements and biometric data. Examples include smartphone cameras and wearable devices with heart rate sensors. Users record their body movements using these devices while performing their usual exercise.

[0037] The collected data is transmitted via the internet to a server in the cloud. On the server, a data analysis device uses machine learning algorithms to analyze this data and evaluate postural imbalances and the effects of movement. This analysis is performed using a pre-trained model and can take into account the movement characteristics of each user.

[0038] The analysis results are transmitted to the terminal via a feedback device and displayed in real time on the user's smartphone or tablet as advice for improving their form. For example, the terminal provides specific feedback such as "Bend your knees a little more" in voice or text. This allows the user to correct their form on the spot.

[0039] Augmented reality devices provide users with a virtual training environment. Through augmented reality-enabled devices, users can experience interaction with virtual trainers and receive visual and intuitive instruction. This makes it possible to receive instruction equivalent to professional training from the comfort of one's home.

[0040] Furthermore, the server uses a progress management device to record the user's training history and calculates reward points based on that data. Users can check their points on their devices, which helps to improve their exercise endurance.

[0041] Furthermore, the health management device evaluates the user's health status based on their activity data and registered lifestyle data, and provides personalized health improvement advice. For example, it considers the user's dietary history and makes specific suggestions such as, "You should increase your protein intake."

[0042] As a concrete example, when a user starts a new exercise, the device provides an initial setup guide and positions the camera appropriately. Once training begins, movements are recorded in real time, and the server immediately analyzes the data. Receiving immediate feedback, the user can immerse themselves in the virtual environment and perform focused training. In this way, the system of the present invention enables continuous and effective health management even when training at home.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] Users prepare to begin exercising by putting on a smartphone or wearable device and launching an application. The device collects the user's movement data in real time through cameras and sensors.

[0046] Step 2:

[0047] The device organizes the collected motion data according to a pre-configured format and transmits it to a server via the internet. This data includes video data and biometric data.

[0048] Step 3:

[0049] The server analyzes the received data using machine learning algorithms via a data analysis device. Here, joint angles and movement patterns are calculated to evaluate the accuracy and efficiency of the movements.

[0050] Step 4:

[0051] Based on the analysis results, the server identifies areas for improvement for the user and generates feedback data, which includes voice guidance and text messages.

[0052] Step 5:

[0053] The device displays feedback data sent from the server to the user in real time. The feedback is provided either visually or audibly as specific instructions, such as "Please lower your shoulders a little more."

[0054] Step 6:

[0055] Based on the feedback provided, users adjust their exercise form and continue training to achieve optimal movement.

[0056] Step 7:

[0057] The device utilizes augmented reality to provide users with videos of a virtual trainer guiding them through exercises. Users then use these videos to verify and correct their movements.

[0058] Step 8:

[0059] The server records the user's training progress and calculates reward points. This allows users to track their achievements and maintain motivation.

[0060] Step 9:

[0061] The server generates health improvement advice based on collected behavioral data and lifestyle information. This advice is then communicated to the user via the app.

[0062] (Example 1)

[0063] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0064] In modern society, individual health conditions and exercise habits are becoming increasingly diverse, creating a need for technologies that provide individually optimized health management and exercise guidance. However, current systems have problems in providing accurate real-time feedback and effective training experiences using virtual environments. Furthermore, mechanisms for providing specific health improvement advice based on movement and lifestyle information are insufficient.

[0065] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0066] In this invention, the server includes information gathering means for acquiring information about its operation, information analysis means for executing a machine learning algorithm to analyze the acquired information, and response providing means for providing an immediate response to the user based on the analysis results. This enables real-time feedback to the user tailored to their individual health condition.

[0067] "Information gathering means for acquiring information about movement" refers to devices used to collect data on a user's physical movements and physiological processes, and includes technologies that utilize sensors, cameras, etc., to acquire accurate data.

[0068] "Information analysis means that executes machine learning algorithms to analyze acquired information" refers to a device that utilizes machine learning technology to analyze collected data and perform pattern recognition or anomaly detection.

[0069] "A means of providing immediate responses to users based on analysis results" refers to a technology that generates and displays real-time feedback to users via voice or text based on the results of the analysis.

[0070] "Augmented reality means for providing exercise experiences in a virtual environment" refers to a technology that uses computer graphics to overlay digital information onto the real world, and is a method for providing users with virtual trainers and exercise guides.

[0071] "Presentation means for presenting generated guidance audibly and visually" refers to technology that conveys advice and guidance generated based on analysis results to users through displays or speakers.

[0072] A "progress management system that calculates reward points based on the degree of achievement and manages the progress of users" is a system that manages the activity history of users, converts their results into points and makes them visible, and is a method aimed at improving motivation.

[0073] "A health management tool for generating health improvement guidance based on user activity and lifestyle information" refers to a technology that evaluates a user's health status based on individual activity and lifestyle data and provides specific advice for improvement.

[0074] This invention utilizes a multi-layered IoT system to support users' daily health management and training. The main components and operation of the system are described below.

[0075] Information gathering

[0076] The device acquires information about the user's movements using smartphones and wearable devices. For example, it utilizes the camera built into the smartphone or a heart rate sensor in a wristwatch. These devices monitor the user's movements in real time and collect data including heart rate and posture.

[0077] Data Analysis

[0078] Once data is collected, the device sends this data to a server via the internet. On the server, a data analysis device receives this data and performs analysis using a pre-trained generative AI model. This model uses machine learning algorithms to evaluate the user's movements and posture. For example, it evaluates squat form and detects knee angles and postural irregularities.

[0079] Real-time feedback

[0080] The analysis results are sent from the server to the terminal, which then provides immediate feedback to the user via voice or text. This allows the user to receive specific instructions, such as "Bend your knees a little more," and correct their movements on the spot.

[0081] virtual environment

[0082] Furthermore, users can gain a virtual training experience using augmented reality devices. A virtual trainer is superimposed onto the real world, providing visual and intuitive guidance to the user. This makes it possible to receive high-quality instruction equivalent to professional training at home.

[0083] Progress management and health improvement

[0084] The server records the user's training progress and calculates reward points based on their achievement level. Users can check these points on their device, which motivates them to continue exercising. In addition, a health management device, based on the user's movement data and lifestyle data, provides specific advice for improving health. For example, it might generate suggestions such as, "You should increase your protein intake."

[0085] For example, when a user tries a new exercise, the device provides an initial setup guide and assists in properly positioning the device. Once data collection begins, the server analyzes the data in real time and immediately sends feedback to the device. This process is an effective means for users to engage in more focused training and manage their health.

[0086] An example of a prompt message would be, "How can I monitor new exercises and provide users with immediate feedback?"

[0087] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0088] Step 1:

[0089] The device uses the smartphone's camera and wearable devices to collect data about the user's movements in real time. Inputs include the user's physical movements and heart rate, and all of this data is continuously captured by sensors and built-in software. As output, the collected raw data is temporarily stored inside the device.

[0090] Step 2:

[0091] The device transmits the collected data to a cloud server via the internet. The input is the raw data collected in step 1. The data is transferred through a secure protocol, and the output is stored on the server. During this process, the data is encrypted to prevent data loss and unauthorized access.

[0092] Step 3:

[0093] The server processes the received data using a data analysis device. The input consists of user behavior and physiological data stored in the cloud. The server analyzes the data using machine learning algorithms and evaluates the user's behavior. Generative AI models are utilized to generate evaluation results and areas for improvement as output.

[0094] Step 4:

[0095] The server generates feedback for the user based on the analysis results. The input is the evaluation results generated in step 3. The server generates meaningful and specific advice for the user and prepares the feedback as output in text or audio format.

[0096] Step 5:

[0097] The terminal presents the user with feedback received from the server. The input is the specific advice generated in step 4. The feedback is displayed on the screen or provided audibly through the speaker. This allows the user to correct their actions in real time.

[0098] Step 6:

[0099] Users engage in a virtual training experience using augmented reality devices. Inputs consist of real-world environmental data and instructional information from a server. Outputs include visual and audio guidance in a training environment that integrates reality and virtuality. This interaction allows users to intuitively understand the instructions and perform exercises effectively.

[0100] Step 7:

[0101] The server manages the user's training progress using a progress tracking device and calculates reward points based on the goals achieved. The input is past training data. The output is the calculated points, which the user can view on their terminal.

[0102] Step 8:

[0103] The server uses a health management device to assess the user's overall health status. Inputs include the user's activity data and lifestyle data. The output is personalized health improvement advice, provided to the user, allowing them to gain concrete directions for improving their health.

[0104] (Application Example 1)

[0105] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0106] In today's fitness scene, the presence of a professional instructor is essential for receiving effective training guidance. However, environments where instructor support is always available are limited, and when training on one's own, there is a high risk of incorrect form and ineffective movements. Furthermore, there is often a lack of individualized feedback, and progress management and guidance for health improvement are insufficient. This invention aims to solve these problems and provide an environment in which users can continue training effectively and accurately at their own pace.

[0107] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0108] In this invention, the server includes information gathering means for collecting user behavior data, information analysis means for executing machine learning algorithms for analyzing the collected behavior data, response means for providing immediate feedback to the user based on the analysis results, and display means for enabling visual instruction with a virtual instructor through an extended entity device for providing a training experience in a virtual environment. This allows the user to correct their own behavior in real time and receive expert instruction.

[0109] "Information gathering means" refers to devices that collect user actions and biometric information, and are used to acquire data using sensors and cameras.

[0110] An "information analysis device" is a device that executes machine learning algorithms based on collected data to evaluate user behavior and identify areas for improvement.

[0111] A "response device" is a device that immediately informs the user of the analyzed results and provides guidance for correct operation or improvement in the form of audio or visual instructions.

[0112] An "extended physical device" is a technology that presents a virtual environment to the user, enabling a more effective training experience, and displays it overlaid on the real environment.

[0113] A "display means" is a device that presents information to enable interaction with a virtual instructor and to provide visual and intuitive instruction.

[0114] A system that implements an application example of this invention includes an information gathering means that collects motion data in real time through smart glasses worn by the user or a smartphone carried by the user. This means incorporates a camera and biosensors to measure posture, heart rate, and other parameters during exercise.

[0115] The collected data is transmitted to a cloud server via wireless communication. The server performs information analysis using machine learning algorithms such as TENSORFLOW® to analyze the user's behavior patterns. The analysis results identify errors in behavior and areas for improvement, and respond in real time based on this.

[0116] The feedback provided to the user is displayed in audio or text format on smart glasses or a smartphone, and visual guidance is also provided using augmented physical devices. A virtual instructor is overlaid on the user's field of view, demonstrating the correct actions. This allows the user to receive interactive instruction within the virtual environment.

[0117] As a concrete example, consider a user practicing the correct squat form at the gym. The smart glasses worn by the user analyze the user's posture in real time during the exercise and provide voice feedback such as, "Bring your knees a little further inward." A virtual instructor visually demonstrates the correct squat form, allowing the user to correct their movement accordingly. This feature provides the effect of personalized instruction without compromising privacy.

[0118] Furthermore, an example of a prompt message is: "Use this system to provide real-time advice to users aiming for ideal squat form. Specifically point out which movements need improvement and synchronize them with the movements of a virtual trainer." This prompt allows the system to generate precise feedback based on the movement data.

[0119] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0120] Step 1:

[0121] The device collects user motion data in real time using the camera and sensors of the smart glasses or smartphone worn by the user. The input is the user's physical movements and biometric data, and the output is a motion data file in which this data is recorded in digital format. The device temporarily stores the motion data and prepares it for transfer to the cloud server in the next step.

[0122] Step 2:

[0123] The terminal sends data to the cloud server via the internet. The input here is an operational data file, which is sent to the server as output via network communication. This communication uses a secure protocol to prevent data leakage.

[0124] Step 3:

[0125] The server executes a machine learning algorithm to analyze the received motion data. The model uses a generative AI model, trained to analyze user motion patterns. In this step, motion data is received as input, and the analysis results extract evaluations of the user's posture and coaching points. The output is feedback information based on the analysis results.

[0126] Step 4:

[0127] The server sends the analysis results to the terminal as feedback information via a response mechanism. The input is the feedback information of the analysis results, and the output is the feedback message converted for display on the terminal side. This feedback is provided in audio or text format.

[0128] Step 5:

[0129] The terminal provides feedback to the user using an augmented physical device. The input is feedback messages sent from the server, and the output is displayed in the user's field of view in real time. Specifically, the movements and instructions of a virtual instructor are visually overlaid, and the user uses this as a guide to correct their actions.

[0130] Step 6:

[0131] Users modify their actions according to the feedback provided. Here, they receive visual and auditory feedback as input and improve their actions as output. By following the system's instructions, users can learn the correct training methods.

[0132] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0133] This invention is a system incorporating emotion recognition capabilities to enable effective training and health management for users. The system includes a data acquisition device, a data analysis device, a feedback device, an augmented reality device, a progress management device, a health management device, and an emotion engine. The operation of each device and the overall system is described below.

[0134] The device is equipped with equipment to collect user motion and emotional data. Users begin exercising while wearing a smartphone or wearable device. These devices include cameras and biosensors that detect the user's emotional state in real time from their facial expressions and voice.

[0135] The collected behavioral and emotional data is transmitted to a cloud server via the internet. On the server, a data analysis device and an emotion engine analyze the respective datasets and evaluate the user's physical and emotional state. The emotion engine classifies the data based on multiple emotional categories, such as joy, anger, and concentration.

[0136] The analysis results are transmitted to the terminal via a feedback device, allowing the user to receive real-time advice based on their actions and emotions. For example, if the system detects that the user is experiencing fatigue or stress, the feedback will suggest slowing down the pace of their exercise. Conversely, if the system detects that the user is highly motivated, it can suggest more challenging training.

[0137] Augmented reality devices provide users with a virtual training environment that adapts to their emotional state. If the user is in a relaxed mental state, the training will be adjusted to take place within a calm scenario.

[0138] The progress tracking device records data, including the user's emotional responses to the exercise, and calculates reward points. These points are used as an incentive to help users train without stress.

[0139] Furthermore, the health management device generates advice aimed at improving lifestyle habits based on movement and emotional data. For example, if data analysis reveals that the user has a high stress level, suggestions for relaxation will be made.

[0140] As a concrete example, when a user starts a morning training session, the device instantly collects and transmits behavioral and emotional data. The server analyzes the data and, if the user is showing signs of tension, provides feedback such as "slow down." In augmented reality, an emotionally resonant environment is provided, such as a training scene on a beach. In this way, the system aims to enhance the sustainability and effectiveness of training.

[0141] The following describes the processing flow.

[0142] Step 1:

[0143] The user launches a smartphone application and prepares for a training session. The device captures the user's body movements through cameras and sensors to collect motion data. It also acquires emotional data from facial expressions and tone of voice using microphones and cameras.

[0144] Step 2:

[0145] The device organizes the collected behavioral and emotional data and sends data packets to the server in real time. The data includes real-time user feedback.

[0146] Step 3:

[0147] The server analyzes the received motion data using a data analysis device to evaluate the user's physical movements. Machine learning algorithms are used for the analysis to determine the accuracy and appropriateness of the movements.

[0148] Step 4:

[0149] The server uses an emotion engine to analyze emotional data and identify the user's emotional state. Here, it classifies the emotion as either joy, focus, or anxiety.

[0150] Step 5:

[0151] The server integrates the results of behavioral and emotional analysis to generate comprehensive feedback for the user. For example, if the user is showing signs of fatigue, it might include a suggestion such as, "Let's take a short break."

[0152] Step 6:

[0153] The device presents the user with feedback sent from the server. This includes voice guidance and text messages, which the user can use to adjust their exercise routine.

[0154] Step 7:

[0155] The device generates a virtual environment tailored to the user's emotional state via an augmented reality system. For example, during training in a relaxed state, a calm background scene is provided.

[0156] Step 8:

[0157] The server records the user's progress and calculates reward points based on their training performance. Users can check their points through their device to gauge their motivation.

[0158] Step 9:

[0159] The server generates health advice that leads to lifestyle improvements based on accumulated behavioral and emotional data. For example, it may include recommendations for improving sleep or reducing stress.

[0160] Step 10:

[0161] Users can review the health advice displayed on their devices and use it to improve their health by reviewing restrictive lifestyle habits.

[0162] (Example 2)

[0163] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0164] Providing effective, real-time feedback that takes into account individual physical and emotional states is challenging when users engage in fitness training. Furthermore, maintaining motivation during training is a challenge, and there is a need to utilize emotional data to provide advice that supports a healthier lifestyle.

[0165] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0166] In this invention, the server includes data collection means for collecting behavioral data and emotional data, data analysis means for analyzing the collected behavioral data, and emotion determination means for analyzing the collected emotional data and classifying the user's emotional state into multiple emotion categories. This makes it possible to provide appropriate training content and feedback to individual users in real time.

[0167] "Motion data" is a collection of information that shows the user's physical movements and physiological state, and is used to evaluate performance and health status during training.

[0168] "Emotional data" is a collection of information indicating a user's psychological state, obtained from their facial expressions, tone of voice, and other biosignals, and is used to evaluate the user's emotions and mental state.

[0169] "Data collection means" refers to devices or technologies used to acquire user behavioral data and emotional data, and includes smartphones and wearable devices.

[0170] "Data analysis means" refers to methods and devices for processing and analyzing collected motion data, and in particular, they evaluate movement patterns and physical conditions by using machine learning algorithms.

[0171] An "emotion determination tool" is a method or system for processing collected emotional data and classifying it into emotional categories such as joy, anger, and concentration.

[0172] A "feedback mechanism" is a device or technology used to provide users with advice and information in real time based on analysis results.

[0173] "Augmented reality means" are technologies that provide a virtual training environment tailored to the user's emotional state, overlaying digital information onto the real world visually or audibly.

[0174] "Reward points" are units of incentives awarded to users to evaluate their training progress and achievements, and to improve their motivation.

[0175] A "progress management method" refers to a method or system for recording a user's training history and emotional responses, and for calculating reward points.

[0176] "Health management tools" refer to methods and systems that utilize users' behavioral and emotional data to provide personalized health improvement advice and lifestyle suggestions.

[0177] In this invention, the user starts training using a smartphone and a wearable device, thereby collecting motion and emotional data in real time. The device is equipped with a camera and biosensors, which can acquire physiological indicators such as the user's facial expressions, heart rate, and body temperature. This data is transmitted to a server in the cloud via a secure protocol.

[0178] The server executes a program that includes machine learning algorithms as a data analysis tool to process behavioral data. Simultaneously, an emotion determination tool analyzes emotional data and classifies the user's psychological state into categories such as joy and concentration. This allows the user's fitness level and emotional condition to be measured.

[0179] The feedback system provides real-time, tailored advice to the user based on these analysis results. For example, if user fatigue is detected, the feedback can recommend "slow down." Furthermore, augmented reality is used to generate a virtual training environment tailored to the user's emotional state, such as presenting a calm beach scene if the user is relaxed.

[0180] Furthermore, the server uses progress management tools to record the user's exercise progress and emotional responses, and calculates reward points. This makes it possible to enhance the effectiveness of training while maintaining the user's motivation. The health management tools utilize the collected data to provide users with advice on improving their lifestyle. Specifically, when stress levels are high, relaxation techniques can be suggested.

[0181] As a concrete example, when a user starts a morning training session, the device instantly collects data and sends it to the server. The server analyzes the data and provides feedback to the user, such as "Let's slow down." In the virtual environment, a training scene on a beach is created, providing an environment that responds to the user's emotions. An example of a prompt using a generative AI model might be: "Based on user feedback, please provide training adjustment suggestions in real time. User's emotional state: motivated, physical state: slightly fatigued."

[0182] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0183] Step 1:

[0184] The user begins training by wearing a smartphone or wearable device. The device uses cameras and biosensors to collect the user's movement and emotional data. Specifically, information such as heart rate, body temperature, and facial expressions is acquired in real time. The input for this step is the user's physiological and behavioral information, and the output is transferred to a cloud server.

[0185] Step 2:

[0186] The device transmits collected behavioral and emotional data to a cloud server via the internet. This data transfer ensures that all physiological and emotional data is centrally managed on the server. The input consists of the data collected by the device, and the output consists of the data stored on the server after secure data transmission.

[0187] Step 3:

[0188] The server analyzes motion data using data analysis tools. Specifically, the server executes machine learning algorithms and evaluates the user's fitness status by analyzing motion patterns. Motion data stored in the cloud is used as input, and the analyzed fitness status is obtained as output.

[0189] Step 4:

[0190] The server analyzes the collected emotional data using an emotion determination tool. This classifies the user's emotions into categories such as joy, anger, and concentration, based on their voice and facial expressions. The input is emotional data, and the output is the identified emotional state.

[0191] Step 5:

[0192] The server generates feedback to the user using feedback mechanisms based on the analysis results. For example, if the user's data indicates fatigue, it will generate advice such as "Slow down." The results of behavioral and sentiment analysis are used as input, and the output is a feedback message.

[0193] Step 6:

[0194] The server sends the generated feedback to the terminal, and the terminal displays advice to the user in real time. Here, the input is the feedback data sent from the server, and the output is the feedback displayed on the user's terminal.

[0195] Step 7:

[0196] The device generates a virtual training environment using augmented reality. Depending on the user's emotional state, training scenes that match their mental state, such as a calm beach or mountain landscape, are presented. The input is analyzed emotional data, and the output is the customized virtual environment presented to the user.

[0197] Step 8:

[0198] The server uses progress management tools to record the user's training history and emotional responses, and calculates reward points. Cumulative data and responses are used as input, and visualized progress and incentive points are generated as output.

[0199] Step 9:

[0200] The server uses health management tools to generate lifestyle improvement advice based on the user's behavioral and emotional data. For example, users with high stress levels will be offered relaxation suggestions. The input is continuously collected data, and the output is customized lifestyle improvement advice.

[0201] (Application Example 2)

[0202] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0203] The challenge lies in how to alleviate the physical and mental burden faced by transportation workers in harsh working conditions, and how to improve work efficiency and safety. Furthermore, there is a need for a system that can monitor workers' health and emotional states in real time and respond appropriately.

[0204] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0205] In this invention, the server includes information gathering means for collecting operational information, information analysis means for executing machine learning algorithms to analyze the collected operational information, feedback means for providing real-time feedback to the user based on the execution results, emotion analysis means for monitoring emotional states in real time, and transportation work support means for providing information to improve the condition of workers in transportation operations. This makes it possible to maintain the health of transportation workers and improve work efficiency.

[0206] "Motion information" refers to information that records and acquires the user's physical movements as digital data.

[0207] "Information gathering means" refers to a device or system for detecting and collecting data on a user's actions and emotional state.

[0208] "Information analysis means" refers to a device or software that performs computational processing to evaluate the user's state and performance based on collected data.

[0209] A "machine learning algorithm" is a mathematical and statistical technique used in the data analysis process, a method that learns from past data to make predictions and classifications.

[0210] A "feedback device" is a device or system that provides users with advice and information regarding their behavior and emotions based on analysis results.

[0211] "Augmented reality means" refers to a device or system based on technology that overlays and displays virtual information onto real space.

[0212] "Emotional analysis methods" are technologies used to evaluate a user's emotions and stress levels based on data such as their voice and facial expressions.

[0213] A "transportation work support device" is a device or system that provides information in real time in order to improve the efficiency of transportation operations and reduce the burden on workers.

[0214] This invention provides a system for improving the physical and emotional state of workers in transportation operations. The following describes specific embodiments of this system.

[0215] The server includes information gathering means, information analysis means, feedback means, sentiment analysis means, and transportation work support means. The information gathering means acquires motion information such as heart rate and movement via smart devices worn by transportation workers. Smartphones and smartwatches are used as hardware for this purpose.

[0216] The information analysis method utilizes machine learning algorithms to analyze collected data on a cloud server. For the analysis, software such as "Google Cloud Platform" and emotion recognition APIs are used as information analysis platforms to evaluate the user's stress and fatigue in real time.

[0217] The emotion analysis system classifies the user's emotional state based on acquired voice data and heart rate. This allows the system to understand whether the user is in a state of excitement or tension, and prepares to provide optimal feedback through the transportation work support system.

[0218] The feedback mechanism provides users with real-time advice and break suggestions based on the analyzed results. For example, if a worker's stress level is high, a notification such as "You should take a short break" is sent to their smart device.

[0219] For example, if a delivery worker suddenly shows signs of stress, the feedback system will send the worker advice such as, "We recommend you take a short break immediately." In this way, the system can support the health of delivery workers while enhancing work safety.

[0220] As an example of a prompt, it can be used in the following format: "Design a prototype application that evaluates the stress level of delivery drivers in real time using emotion recognition data and suggests relaxation music as needed."

[0221] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0222] Step 1:

[0223] The device collects motion information, including heart rate, location information, and voice data, via the smart device (smartphone or smartwatch) worn by the user. This allows the device to obtain data as input to understand the user's current status in real time.

[0224] Step 2:

[0225] The device sends collected operational information to a cloud server via its internet connection. The server uses a data analysis platform (e.g., Google Cloud Platform) to cleanse and preprocess the data, transforming it into an analyzable format.

[0226] Step 3:

[0227] The server executes machine learning algorithms and analyzes the preprocessed data. Here, it uses an emotion recognition API to classify the user's emotional state. This process outputs data that specifically evaluates whether the user is currently feeling stressed or relaxed.

[0228] Step 4:

[0229] Based on the analysis results, the server generates feedback. Specifically, if it determines that the stress level is high, it will generate a notification such as, "Please consider taking a break." This feedback is a suggestion of action that the user can take.

[0230] Step 5:

[0231] The server sends the generated feedback to the device. The device displays this feedback on the user's smart device screen. The user can then view advice about their status in real time and take action based on it.

[0232] Step 6:

[0233] The device records the receipt of feedback and the user's response, and transmits this information to the progress management system. This information is used for the ongoing management of the user's health status and for awarding points.

[0234] These processes enable transport workers to efficiently manage stress and fatigue, allowing them to perform their duties in a healthy and safe manner.

[0235] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0236] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0237] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0238] [Second Embodiment]

[0239] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0240] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0241] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0242] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0243] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0244] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0245] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0246] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0247] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0248] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0249] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0250] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0251] The system of the present invention enables efficient health management and training for users and has a multi-layered configuration including a data acquisition device, a data analysis device, a feedback device, an augmented reality device, a progress management device, and a health management device. The operation of each device and the overall system will be described below.

[0252] The device functions as a data collection device to collect user movements and biometric data. Examples include smartphone cameras and wearable devices with heart rate sensors. Users record their body movements using these devices while performing their usual exercise.

[0253] The collected data is transmitted via the internet to a server in the cloud. On the server, a data analysis device uses machine learning algorithms to analyze this data and evaluate postural imbalances and the effects of movement. This analysis is performed using a pre-trained model and can take into account the movement characteristics of each user.

[0254] The analysis results are transmitted to the terminal via a feedback device and displayed in real time on the user's smartphone or tablet as advice for improving their form. For example, the terminal provides specific feedback such as "Bend your knees a little more" in voice or text. This allows the user to correct their form on the spot.

[0255] Augmented reality devices provide users with a virtual training environment. Through augmented reality-enabled devices, users can experience interaction with virtual trainers and receive visual and intuitive instruction. This makes it possible to receive instruction equivalent to professional training from the comfort of one's home.

[0256] Furthermore, the server uses a progress management device to record the user's training history and calculates reward points based on that data. Users can check their points on their devices, which helps to improve their exercise endurance.

[0257] Furthermore, the health management device evaluates the user's health status based on their activity data and registered lifestyle data, and provides personalized health improvement advice. For example, it considers the user's dietary history and makes specific suggestions such as, "You should increase your protein intake."

[0258] As a concrete example, when a user starts a new exercise, the device provides an initial setup guide and positions the camera appropriately. Once training begins, movements are recorded in real time, and the server immediately analyzes the data. Receiving immediate feedback, the user can immerse themselves in the virtual environment and perform focused training. In this way, the system of the present invention enables continuous and effective health management even when training at home.

[0259] The following describes the processing flow.

[0260] Step 1:

[0261] Users prepare to begin exercising by putting on their smartphone or wearable device and launching the application. The device collects the user's movement data in real time through cameras and sensors.

[0262] Step 2:

[0263] The device organizes the collected motion data according to a pre-configured format and transmits it to a server via the internet. This data includes video data and biometric data.

[0264] Step 3:

[0265] The server analyzes the received data using machine learning algorithms via a data analysis device. Here, joint angles and movement patterns are calculated to evaluate the accuracy and efficiency of the movements.

[0266] Step 4:

[0267] Based on the analysis results, the server identifies areas for improvement for the user and generates feedback data, which includes voice guidance and text messages.

[0268] Step 5:

[0269] The device displays feedback data sent from the server to the user in real time. The feedback is provided either visually or audibly as specific instructions, such as "Please lower your shoulders a little more."

[0270] Step 6:

[0271] Based on the feedback provided, users adjust their exercise form and continue training to achieve optimal movement.

[0272] Step 7:

[0273] The device utilizes augmented reality to provide users with videos of a virtual trainer guiding them through exercises. Users then use these videos to verify and correct their movements.

[0274] Step 8:

[0275] The server records the user's training progress and calculates reward points. This allows users to track their achievements and maintain motivation.

[0276] Step 9:

[0277] The server generates health improvement advice based on collected behavioral data and lifestyle information. This advice is then communicated to the user via the app.

[0278] (Example 1)

[0279] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0280] In modern society, due to the diversification of individual health conditions and exercise habits, there is a need for technologies to perform health management and exercise guidance optimized individually. However, in the current system, there are problems such as difficulty in providing accurate real-time feedback and providing users with an effective training experience utilizing a virtual environment. Also, the mechanism for providing specific health improvement advice based on motion information and lifestyle information is not sufficient.

[0281] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0282] In this invention, the server includes an information collection means for acquiring information regarding motion, an information analysis means for executing a machine learning algorithm to analyze the acquired information, and a response providing means for immediately providing a response to the user based on the analysis result. Thereby, it becomes possible to provide real-time feedback according to the individual health conditions of the user.

[0283] The "information collection means for acquiring information regarding motion" is a device used to collect the physical motions and physiological data of the user, and refers to a technology for acquiring accurate data by utilizing sensors, cameras, etc.

[0284] The "information analysis means for executing a machine learning algorithm to analyze the acquired information" is a device that utilizes machine learning technology to analyze the collected data and perform pattern recognition and anomaly detection.

[0285] The "response providing means for immediately providing a response to the user based on the analysis result" indicates a technology for generating and displaying real-time feedback to the user through voice or text based on the result of the analysis.

[0286] The "augmented reality means for providing a virtual sports experience" is a technology that overlays digital information on the real world using computer graphics, and is a method for providing users with virtual trainers and sports guides.

[0287] The "presentation means for presenting the generated guidance both audibly and visually" refers to a technology that conveys advice and guides generated based on the analysis results to the user through a display and a speaker.

[0288] The "progress management means for calculating reward points based on the degree of achievement and managing the progress of the user" is a mechanism that manages the user's activity history, points the results, and visualizes them, and is a method aimed at improving motivation.

[0289] The "health management means for generating health improvement guidance based on information on the user's actions and lifestyle habits" refers to a technology that evaluates the health status based on individual action data and lifestyle habit data and provides specific advice for improvement.

[0290] This invention uses a multi-layer IoT system to assist in the user's daily health management and training. The main components and operations of the system will be described below.

[0291] Information collection

[0292] The terminal acquires information on the user's actions using a smartphone or a wearable device. For example, a camera built into the smartphone or a wristwatch-type heart rate sensor is utilized. These devices monitor the user's actions in real time and collect data including the heart rate and posture.

[0293] Data analysis

[0294] Once data is collected, the device sends this data to a server via the internet. On the server, a data analysis device receives this data and performs analysis using a pre-trained generative AI model. This model uses machine learning algorithms to evaluate the user's movements and posture. For example, it evaluates squat form and detects knee angles and postural irregularities.

[0295] Real-time feedback

[0296] The analysis results are sent from the server to the terminal, which then provides immediate feedback to the user via voice or text. This allows the user to receive specific instructions, such as "Bend your knees a little more," and correct their movements on the spot.

[0297] virtual environment

[0298] Furthermore, users can gain a virtual training experience using augmented reality devices. A virtual trainer is superimposed onto the real world, providing visual and intuitive guidance to the user. This makes it possible to receive high-quality instruction equivalent to professional training at home.

[0299] Progress management and health improvement

[0300] The server records the user's training progress and calculates reward points based on their achievement level. Users can check these points on their device, which motivates them to continue exercising. In addition, a health management device, based on the user's movement data and lifestyle data, provides specific advice for improving health. For example, it might generate suggestions such as, "You should increase your protein intake."

[0301] As a specific example, when a user challenges a new exercise, the terminal provides an initial setting guide and assists in appropriately setting the device position. When data collection starts, the server analyzes the data in real time and immediately sends feedback to the terminal. This process is an effective means for the user to perform more concentrated training and achieve health management.

[0302] As an example of a prompt sentence, "Please teach me how to monitor a new exercise and provide immediate feedback to the user" can be cited.

[0303] The flow of the specific process in Example 1 will be described using FIG. 11.

[0304] Step 1:

[0305] The terminal uses the camera of the smartphone or a wearable device to collect data on the user's movements in real time. The input is the user's physical movements, heart rate, etc., and all these data are continuously captured by the software integrated with the sensor. As output, the collected raw data is temporarily stored inside the terminal.

[0306] Step 2:

[0307] The terminal sends the collected data to the cloud server via the Internet. The input is the raw data collected in Step 1. The data is transferred through a secure protocol, and as output, the data is stored on the server. In this process, the data is encrypted to prevent data loss and unauthorized access.

[0308] Step 3:

[0309] The server processes the received data using a data analysis device. The input consists of user behavior and physiological data stored in the cloud. The server analyzes the data using machine learning algorithms and evaluates the user's behavior. Generative AI models are utilized to generate evaluation results and areas for improvement as output.

[0310] Step 4:

[0311] The server generates feedback for the user based on the analysis results. The input is the evaluation results generated in step 3. The server generates meaningful and specific advice for the user and prepares the feedback as output in text or audio format.

[0312] Step 5:

[0313] The terminal presents the user with feedback received from the server. The input is the specific advice generated in step 4. The feedback is displayed on the screen or provided audibly through the speaker. This allows the user to correct their actions in real time.

[0314] Step 6:

[0315] Users engage in a virtual training experience using augmented reality devices. Inputs consist of real-world environmental data and instructional information from a server. Outputs include visual and audio guidance in a training environment that integrates reality and virtuality. This interaction allows users to intuitively understand the instructions and perform exercises effectively.

[0316] Step 7:

[0317] The server manages the user's training progress using a progress tracking device and calculates reward points based on the goals achieved. The input is past training data. The output is the calculated points, which the user can view on their terminal.

[0318] Step 8:

[0319] The server uses a health management device to assess the user's overall health status. Inputs include the user's activity data and lifestyle data. The output is personalized health improvement advice, provided to the user, allowing them to gain concrete directions for improving their health.

[0320] (Application Example 1)

[0321] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0322] In today's fitness scene, the presence of a professional instructor is essential for receiving effective training guidance. However, environments where instructor support is always available are limited, and when training on one's own, there is a high risk of incorrect form and ineffective movements. Furthermore, there is often a lack of individualized feedback, and progress management and guidance for health improvement are insufficient. This invention aims to solve these problems and provide an environment in which users can continue training effectively and accurately at their own pace.

[0323] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0324] In this invention, the server includes information gathering means for collecting user behavior data, information analysis means for executing machine learning algorithms for analyzing the collected behavior data, response means for providing immediate feedback to the user based on the analysis results, and display means for enabling visual instruction with a virtual instructor through an extended entity device for providing a training experience in a virtual environment. This allows the user to correct their own behavior in real time and receive expert instruction.

[0325] "Information gathering means" refers to devices that collect user actions and biometric information, and are used to acquire data using sensors and cameras.

[0326] An "information analysis device" is a device that executes machine learning algorithms based on collected data to evaluate user behavior and identify areas for improvement.

[0327] A "response device" is a device that immediately informs the user of the analyzed results and provides guidance for correct operation or improvement in the form of audio or visual instructions.

[0328] An "extended physical device" is a technology that presents a virtual environment to the user, enabling a more effective training experience, and displays it overlaid on the real environment.

[0329] A "display means" is a device that presents information to enable interaction with a virtual instructor and to provide visual and intuitive instruction.

[0330] A system that implements an application example of this invention includes an information gathering means that collects motion data in real time through smart glasses worn by the user or a smartphone carried by the user. This means incorporates a camera and biosensors to measure posture, heart rate, and other parameters during exercise.

[0331] The collected data is transmitted to a cloud server via wireless communication. The server performs information analysis using machine learning algorithms such as TensorFlow to analyze the user's behavior patterns. The analysis results identify errors in behavior and areas for improvement, and based on this, a real-time response is provided.

[0332] The feedback provided to the user is displayed in audio or text format on smart glasses or a smartphone, and visual guidance is also provided using augmented physical devices. A virtual instructor is overlaid on the user's field of view, demonstrating the correct actions. This allows the user to receive interactive instruction within the virtual environment.

[0333] As a concrete example, consider a user practicing the correct squat form at the gym. The smart glasses worn by the user analyze the user's posture in real time during the exercise and provide voice feedback such as, "Bring your knees a little further inward." A virtual instructor visually demonstrates the correct squat form, allowing the user to correct their movement accordingly. This feature provides the effect of personalized instruction without compromising privacy.

[0334] Furthermore, an example of a prompt message is: "Use this system to provide real-time advice to users aiming for ideal squat form. Specifically point out which movements need improvement and synchronize them with the movements of a virtual trainer." This prompt allows the system to generate precise feedback based on the movement data.

[0335] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0336] Step 1:

[0337] The device collects user motion data in real time using the camera and sensors of the smart glasses or smartphone worn by the user. The input is the user's physical movements and biometric data, and the output is a motion data file in which this data is recorded in digital format. The device temporarily stores the motion data and prepares it for transfer to the cloud server in the next step.

[0338] Step 2:

[0339] The terminal sends data to the cloud server via the internet. The input here is an operational data file, which is sent to the server as output via network communication. This communication uses a secure protocol to prevent data leakage.

[0340] Step 3:

[0341] The server executes a machine learning algorithm to analyze the received motion data. The model uses a generative AI model, trained to analyze user motion patterns. In this step, motion data is received as input, and the analysis results extract evaluations of the user's posture and coaching points. The output is feedback information based on the analysis results.

[0342] Step 4:

[0343] The server sends the analysis results to the terminal as feedback information via a response mechanism. The input is the feedback information of the analysis results, and the output is the feedback message converted for display on the terminal side. This feedback is provided in audio or text format.

[0344] Step 5:

[0345] The terminal provides feedback to the user using an augmented physical device. The input is feedback messages sent from the server, and the output is displayed in the user's field of view in real time. Specifically, the movements and instructions of a virtual instructor are visually overlaid, and the user uses this as a guide to correct their actions.

[0346] Step 6:

[0347] Users modify their actions according to the feedback provided. Here, they receive visual and auditory feedback as input and improve their actions as output. By following the system's instructions, users can learn the correct training methods.

[0348] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0349] This invention is a system incorporating emotion recognition capabilities to enable effective training and health management for users. The system includes a data acquisition device, a data analysis device, a feedback device, an augmented reality device, a progress management device, a health management device, and an emotion engine. The operation of each device and the overall system is described below.

[0350] The device is equipped with equipment to collect user motion and emotional data. Users begin exercising while wearing a smartphone or wearable device. These devices include cameras and biosensors that detect the user's emotional state in real time from their facial expressions and voice.

[0351] The collected behavioral and emotional data is transmitted to a cloud server via the internet. On the server, a data analysis device and an emotion engine analyze the respective datasets and evaluate the user's physical and emotional state. The emotion engine classifies the data based on multiple emotional categories, such as joy, anger, and concentration.

[0352] The analysis results are transmitted to the terminal via a feedback device, allowing the user to receive real-time advice based on their actions and emotions. For example, if the system detects that the user is experiencing fatigue or stress, the feedback will suggest slowing down the pace of their exercise. Conversely, if the system detects that the user is highly motivated, it can suggest more challenging training.

[0353] Augmented reality devices provide users with a virtual training environment that adapts to their emotional state. If the user is in a relaxed mental state, the training will be adjusted to take place within a calm scenario.

[0354] The progress tracking device records data, including the user's emotional responses to the exercise, and calculates reward points. These points are used as an incentive to help users train without stress.

[0355] Furthermore, the health management device generates advice aimed at improving lifestyle habits based on movement and emotional data. For example, if data analysis reveals that the user has a high stress level, suggestions for relaxation will be made.

[0356] As a concrete example, when a user starts a morning training session, the device instantly collects and transmits behavioral and emotional data. The server analyzes the data and, if the user is showing signs of tension, provides feedback such as "slow down." In augmented reality, an emotionally resonant environment is provided, such as a training scene on a beach. In this way, the system aims to enhance the sustainability and effectiveness of training.

[0357] The following describes the processing flow.

[0358] Step 1:

[0359] The user launches a smartphone application and prepares for a training session. The device captures the user's body movements through cameras and sensors to collect motion data. It also acquires emotional data from facial expressions and tone of voice using microphones and cameras.

[0360] Step 2:

[0361] The device organizes the collected behavioral and emotional data and sends data packets to the server in real time. The data includes real-time user feedback.

[0362] Step 3:

[0363] The server analyzes the received motion data using a data analysis device to evaluate the user's physical movements. Machine learning algorithms are used for the analysis to determine the accuracy and appropriateness of the movements.

[0364] Step 4:

[0365] The server uses an emotion engine to analyze emotional data and identify the user's emotional state. Here, it classifies the emotion as either joy, focus, or anxiety.

[0366] Step 5:

[0367] The server integrates the results of behavioral and emotional analysis to generate comprehensive feedback for the user. For example, if the user is showing signs of fatigue, it might include a suggestion such as, "Let's take a short break."

[0368] Step 6:

[0369] The device presents the user with feedback sent from the server. This includes voice guidance and text messages, which the user can use to adjust their exercise routine.

[0370] Step 7:

[0371] The device generates a virtual environment tailored to the user's emotional state via an augmented reality system. For example, during training in a relaxed state, a calm background scene is provided.

[0372] Step 8:

[0373] The server records the user's progress and calculates reward points based on their training performance. Users can check their points through their device to gauge their motivation.

[0374] Step 9:

[0375] The server generates health advice that leads to lifestyle improvements based on accumulated behavioral and emotional data. For example, it may include recommendations for improving sleep or reducing stress.

[0376] Step 10:

[0377] Users can review the health advice displayed on their devices and use it to improve their health by reviewing restrictive lifestyle habits.

[0378] (Example 2)

[0379] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0380] Providing effective, real-time feedback that takes into account individual physical and emotional states is challenging when users engage in fitness training. Furthermore, maintaining motivation during training is a challenge, and there is a need to utilize emotional data to provide advice that supports a healthier lifestyle.

[0381] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0382] In this invention, the server includes data collection means for collecting behavioral data and emotional data, data analysis means for analyzing the collected behavioral data, and emotion determination means for analyzing the collected emotional data and classifying the user's emotional state into multiple emotion categories. This makes it possible to provide appropriate training content and feedback to individual users in real time.

[0383] "Motion data" is a collection of information that shows the user's physical movements and physiological state, and is used to evaluate performance and health status during training.

[0384] "Emotional data" is a collection of information indicating a user's psychological state, obtained from their facial expressions, tone of voice, and other biosignals, and is used to evaluate the user's emotions and mental state.

[0385] "Data collection means" refers to devices or technologies used to acquire user behavioral data and emotional data, and includes smartphones and wearable devices.

[0386] "Data analysis means" refers to methods and devices for processing and analyzing collected motion data, and in particular, they evaluate movement patterns and physical conditions by using machine learning algorithms.

[0387] An "emotion determination tool" is a method or system for processing collected emotional data and classifying it into emotional categories such as joy, anger, and concentration.

[0388] A "feedback mechanism" is a device or technology used to provide users with advice and information in real time based on analysis results.

[0389] "Augmented reality means" are technologies that provide a virtual training environment tailored to the user's emotional state, overlaying digital information onto the real world visually or audibly.

[0390] "Reward points" are units of incentives awarded to users to evaluate their training progress and achievements, and to improve their motivation.

[0391] A "progress management method" refers to a method or system for recording a user's training history and emotional responses, and for calculating reward points.

[0392] "Health management tools" refer to methods and systems that utilize users' behavioral and emotional data to provide personalized health improvement advice and lifestyle suggestions.

[0393] In this invention, the user starts training using a smartphone and a wearable device, thereby collecting motion and emotional data in real time. The device is equipped with a camera and biosensors, which can acquire physiological indicators such as the user's facial expressions, heart rate, and body temperature. This data is transmitted to a server in the cloud via a secure protocol.

[0394] The server executes a program that includes machine learning algorithms as a data analysis tool to process behavioral data. Simultaneously, an emotion determination tool analyzes emotional data and classifies the user's psychological state into categories such as joy and concentration. This allows the user's fitness level and emotional condition to be measured.

[0395] The feedback system provides real-time, tailored advice to the user based on these analysis results. For example, if user fatigue is detected, the feedback can recommend "slow down." Furthermore, augmented reality is used to generate a virtual training environment tailored to the user's emotional state, such as presenting a calm beach scene if the user is relaxed.

[0396] Furthermore, the server uses progress management tools to record the user's exercise progress and emotional responses, and calculates reward points. This makes it possible to enhance the effectiveness of training while maintaining the user's motivation. The health management tools utilize the collected data to provide users with advice on improving their lifestyle. Specifically, when stress levels are high, relaxation techniques can be suggested.

[0397] As a concrete example, when a user starts a morning training session, the device instantly collects data and sends it to the server. The server analyzes the data and provides feedback to the user, such as "Let's slow down." In the virtual environment, a training scene on a beach is created, providing an environment that responds to the user's emotions. An example of a prompt using a generative AI model might be: "Based on user feedback, please provide training adjustment suggestions in real time. User's emotional state: motivated, physical state: slightly fatigued."

[0398] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0399] Step 1:

[0400] The user begins training by wearing a smartphone or wearable device. The device uses cameras and biosensors to collect the user's movement and emotional data. Specifically, information such as heart rate, body temperature, and facial expressions is acquired in real time. The input for this step is the user's physiological and behavioral information, and the output is transferred to a cloud server.

[0401] Step 2:

[0402] The device transmits collected behavioral and emotional data to a cloud server via the internet. This data transfer ensures that all physiological and emotional data is centrally managed on the server. The input consists of the data collected by the device, and the output consists of the data stored on the server after secure data transmission.

[0403] Step 3:

[0404] The server analyzes motion data using data analysis tools. Specifically, the server executes machine learning algorithms and evaluates the user's fitness status by analyzing motion patterns. Motion data stored in the cloud is used as input, and the analyzed fitness status is obtained as output.

[0405] Step 4:

[0406] The server analyzes the collected emotional data using an emotion determination tool. This classifies the user's emotions into categories such as joy, anger, and concentration, based on their voice and facial expressions. The input is emotional data, and the output is the identified emotional state.

[0407] Step 5:

[0408] The server generates feedback to the user using feedback mechanisms based on the analysis results. For example, if the user's data indicates fatigue, it will generate advice such as "Slow down." The results of behavioral and sentiment analysis are used as input, and the output is a feedback message.

[0409] Step 6:

[0410] The server sends the generated feedback to the terminal, and the terminal displays advice to the user in real time. Here, the input is the feedback data sent from the server, and the output is the feedback displayed on the user's terminal.

[0411] Step 7:

[0412] The device generates a virtual training environment using augmented reality. Depending on the user's emotional state, training scenes that match their mental state, such as a calm beach or mountain landscape, are presented. The input is analyzed emotional data, and the output is the customized virtual environment presented to the user.

[0413] Step 8:

[0414] The server uses progress management tools to record the user's training history and emotional responses, and calculates reward points. Cumulative data and responses are used as input, and visualized progress and incentive points are generated as output.

[0415] Step 9:

[0416] The server uses health management tools to generate lifestyle improvement advice based on the user's behavioral and emotional data. For example, users with high stress levels will be offered relaxation suggestions. The input is continuously collected data, and the output is customized lifestyle improvement advice.

[0417] (Application Example 2)

[0418] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0419] The challenge lies in how to alleviate the physical and mental burden faced by transportation workers in harsh working conditions, and how to improve work efficiency and safety. Furthermore, there is a need for a system that can monitor workers' health and emotional states in real time and respond appropriately.

[0420] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0421] In this invention, the server includes information gathering means for collecting operational information, information analysis means for executing machine learning algorithms to analyze the collected operational information, feedback means for providing real-time feedback to the user based on the execution results, emotion analysis means for monitoring emotional states in real time, and transportation work support means for providing information to improve the condition of workers in transportation operations. This makes it possible to maintain the health of transportation workers and improve work efficiency.

[0422] "Motion information" refers to information that records and acquires the user's physical movements as digital data.

[0423] "Information gathering means" refers to a device or system for detecting and collecting data on a user's actions and emotional state.

[0424] "Information analysis means" refers to a device or software that performs computational processing to evaluate the user's state and performance based on collected data.

[0425] A "machine learning algorithm" is a mathematical and statistical technique used in the data analysis process, a method that learns from past data to make predictions and classifications.

[0426] A "feedback device" is a device or system that provides users with advice and information regarding their behavior and emotions based on analysis results.

[0427] "Augmented reality means" refers to a device or system based on technology that overlays and displays virtual information onto real space.

[0428] "Emotional analysis methods" are technologies used to evaluate a user's emotions and stress levels based on data such as their voice and facial expressions.

[0429] A "transportation work support device" is a device or system that provides information in real time in order to improve the efficiency of transportation operations and reduce the burden on workers.

[0430] This invention provides a system for improving the physical and emotional state of workers in transportation operations. The following describes specific embodiments of this system.

[0431] The server includes information gathering means, information analysis means, feedback means, sentiment analysis means, and transportation work support means. The information gathering means acquires motion information such as heart rate and movement via smart devices worn by transportation workers. Smartphones and smartwatches are used as hardware for this purpose.

[0432] The information analysis method utilizes machine learning algorithms to analyze collected data on a cloud server. The analysis uses software such as "Google Cloud Platform" and emotion recognition APIs as information analysis platforms, evaluating the user's stress and fatigue in real time.

[0433] The emotion analysis system classifies the user's emotional state based on acquired voice data and heart rate. This allows the system to understand whether the user is in a state of excitement or tension, and prepares to provide optimal feedback through the transportation work support system.

[0434] The feedback mechanism provides users with real-time advice and break suggestions based on the analyzed results. For example, if a worker's stress level is high, a notification such as "You should take a short break" is sent to their smart device.

[0435] For example, if a delivery worker suddenly shows signs of stress, the feedback system will send the worker advice such as, "We recommend you take a short break immediately." In this way, the system can support the health of delivery workers while enhancing work safety.

[0436] As an example of a prompt, it can be used in the following format: "Design a prototype application that evaluates the stress level of delivery drivers in real time using emotion recognition data and suggests relaxation music as needed."

[0437] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0438] Step 1:

[0439] The device collects motion information, including heart rate, location information, and voice data, via the smart device (smartphone or smartwatch) worn by the user. This allows the device to obtain data as input to understand the user's current status in real time.

[0440] Step 2:

[0441] The device sends collected operational information to a cloud server via its internet connection. The server uses a data analysis platform (e.g., Google Cloud Platform) to cleanse and preprocess the data, transforming it into an analyzable format.

[0442] Step 3:

[0443] The server executes machine learning algorithms and analyzes the preprocessed data. Here, it uses an emotion recognition API to classify the user's emotional state. This process outputs data that specifically evaluates whether the user is currently feeling stressed or relaxed.

[0444] Step 4:

[0445] Based on the analysis results, the server generates feedback. Specifically, if it determines that the stress level is high, it will generate a notification such as, "Please consider taking a break." This feedback is a suggestion of action that the user can take.

[0446] Step 5:

[0447] The server sends the generated feedback to the device. The device displays this feedback on the user's smart device screen. The user can then view advice about their status in real time and take action based on it.

[0448] Step 6:

[0449] The device records the receipt of feedback and the user's response, and transmits this information to the progress management system. This information is used for the ongoing management of the user's health status and for awarding points.

[0450] These processes enable transport workers to efficiently manage stress and fatigue, allowing them to perform their duties in a healthy and safe manner.

[0451] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0452] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0453] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0454] [Third Embodiment]

[0455] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0456] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0457] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0458] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0459] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0460] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0461] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0462] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0463] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0464] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0465] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0466] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0467] The system of the present invention enables efficient health management and training for users and has a multi-layered configuration including a data acquisition device, a data analysis device, a feedback device, an augmented reality device, a progress management device, and a health management device. The operation of each device and the overall system will be described below.

[0468] The device functions as a data collection device to collect user movements and biometric data. Examples include smartphone cameras and wearable devices with heart rate sensors. Users record their body movements using these devices while performing their usual exercise.

[0469] The collected data is transmitted via the internet to a server in the cloud. On the server, a data analysis device uses machine learning algorithms to analyze this data and evaluate postural imbalances and the effects of movement. This analysis is performed using a pre-trained model and can take into account the movement characteristics of each user.

[0470] The analysis results are transmitted to the terminal via a feedback device and displayed in real time on the user's smartphone or tablet as advice for improving their form. For example, the terminal provides specific feedback such as "Bend your knees a little more" in voice or text. This allows the user to correct their form on the spot.

[0471] Augmented reality devices provide users with a virtual training environment. Through augmented reality-enabled devices, users can experience interaction with virtual trainers and receive visual and intuitive instruction. This makes it possible to receive instruction equivalent to professional training from the comfort of one's home.

[0472] Furthermore, the server uses a progress management device to record the user's training history and calculates reward points based on that data. Users can check their points on their devices, which helps to improve their exercise endurance.

[0473] Furthermore, the health management device evaluates the user's health status based on their activity data and registered lifestyle data, and provides personalized health improvement advice. For example, it considers the user's dietary history and makes specific suggestions such as, "You should increase your protein intake."

[0474] As a concrete example, when a user starts a new exercise, the device provides an initial setup guide and positions the camera appropriately. Once training begins, movements are recorded in real time, and the server immediately analyzes the data. Receiving immediate feedback, the user can immerse themselves in the virtual environment and perform focused training. In this way, the system of the present invention enables continuous and effective health management even when training at home.

[0475] The following describes the processing flow.

[0476] Step 1:

[0477] Users prepare to begin exercising by putting on their smartphone or wearable device and launching the application. The device collects the user's movement data in real time through cameras and sensors.

[0478] Step 2:

[0479] The device organizes the collected motion data according to a pre-configured format and transmits it to a server via the internet. This data includes video data and biometric data.

[0480] Step 3:

[0481] The server analyzes the received data using machine learning algorithms via a data analysis device. Here, joint angles and movement patterns are calculated to evaluate the accuracy and efficiency of the movements.

[0482] Step 4:

[0483] Based on the analysis results, the server identifies areas for improvement for the user and generates feedback data, which includes voice guidance and text messages.

[0484] Step 5:

[0485] The device displays feedback data sent from the server to the user in real time. The feedback is provided either visually or audibly as specific instructions, such as "Please lower your shoulders a little more."

[0486] Step 6:

[0487] Based on the feedback provided, users adjust their exercise form and continue training to achieve optimal movement.

[0488] Step 7:

[0489] The device utilizes augmented reality to provide users with videos of a virtual trainer guiding them through exercises. Users then use these videos to verify and correct their movements.

[0490] Step 8:

[0491] The server records the user's training progress and calculates reward points. This allows users to track their achievements and maintain motivation.

[0492] Step 9:

[0493] The server generates health improvement advice based on collected behavioral data and lifestyle information. This advice is then communicated to the user via the app.

[0494] (Example 1)

[0495] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0496] In modern society, individual health conditions and exercise habits are becoming increasingly diverse, creating a need for technologies that provide individually optimized health management and exercise guidance. However, current systems have problems in providing accurate real-time feedback and effective training experiences using virtual environments. Furthermore, mechanisms for providing specific health improvement advice based on movement and lifestyle information are insufficient.

[0497] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0498] In this invention, the server includes information gathering means for acquiring information about its operation, information analysis means for executing a machine learning algorithm to analyze the acquired information, and response providing means for providing an immediate response to the user based on the analysis results. This enables real-time feedback to the user tailored to their individual health condition.

[0499] "Information gathering means for acquiring information about movement" refers to devices used to collect data on a user's physical movements and physiological processes, and includes technologies that utilize sensors, cameras, etc., to acquire accurate data.

[0500] "Information analysis means that executes machine learning algorithms to analyze acquired information" refers to a device that utilizes machine learning technology to analyze collected data and perform pattern recognition or anomaly detection.

[0501] "A means of providing immediate responses to users based on analysis results" refers to a technology that generates and displays real-time feedback to users via voice or text based on the results of the analysis.

[0502] "Augmented reality means for providing exercise experiences in a virtual environment" refers to a technology that uses computer graphics to overlay digital information onto the real world, and is a method for providing users with virtual trainers and exercise guides.

[0503] "Presentation means for presenting generated guidance audibly and visually" refers to technology that conveys advice and guidance generated based on analysis results to users through displays or speakers.

[0504] "A progress management system that calculates reward points based on the degree of achievement and manages the progress of users" is a system that manages the activity history of users, converts their results into points and makes them visible, and is a method aimed at improving motivation.

[0505] "A health management tool for generating health improvement guidance based on user activity and lifestyle information" refers to a technology that evaluates a user's health status based on individual activity and lifestyle data and provides specific advice for improvement.

[0506] This invention utilizes a multi-layered IoT system to support users' daily health management and training. The main components and operation of the system are described below.

[0507] Information gathering

[0508] The device acquires information about the user's movements using smartphones and wearable devices. For example, it utilizes the camera built into the smartphone or a heart rate sensor in a wristwatch. These devices monitor the user's movements in real time and collect data including heart rate and posture.

[0509] Data Analysis

[0510] Once data is collected, the device sends this data to a server via the internet. On the server, a data analysis device receives this data and performs analysis using a pre-trained generative AI model. This model uses machine learning algorithms to evaluate the user's movements and posture. For example, it evaluates squat form and detects knee angles and postural irregularities.

[0511] Real-time feedback

[0512] The analysis results are sent from the server to the terminal, which then provides immediate feedback to the user via voice or text. This allows the user to receive specific instructions, such as "Bend your knees a little more," and correct their movements on the spot.

[0513] virtual environment

[0514] Furthermore, users can gain a virtual training experience using augmented reality devices. A virtual trainer is superimposed onto the real world, providing visual and intuitive guidance to the user. This makes it possible to receive high-quality instruction equivalent to professional training at home.

[0515] Progress management and health improvement

[0516] The server records the user's training progress and calculates reward points based on their achievement level. Users can check these points on their device, which motivates them to continue exercising. In addition, a health management device, based on the user's movement data and lifestyle data, provides specific advice for improving health. For example, it might generate suggestions such as, "You should increase your protein intake."

[0517] For example, when a user tries a new exercise, the device provides an initial setup guide and assists in properly positioning the device. Once data collection begins, the server analyzes the data in real time and immediately sends feedback to the device. This process is an effective means for users to engage in more focused training and manage their health.

[0518] An example of a prompt message would be, "How can I monitor new exercises and provide users with immediate feedback?"

[0519] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0520] Step 1:

[0521] The device uses the smartphone's camera and wearable devices to collect data about the user's movements in real time. Inputs include the user's physical movements and heart rate, and all of this data is continuously captured by sensors and built-in software. As output, the collected raw data is temporarily stored inside the device.

[0522] Step 2:

[0523] The device transmits the collected data to a cloud server via the internet. The input is the raw data collected in step 1. The data is transferred through a secure protocol, and the output is stored on the server. During this process, the data is encrypted to prevent data loss and unauthorized access.

[0524] Step 3:

[0525] The server processes the received data using a data analysis device. The input consists of user behavior and physiological data stored in the cloud. The server analyzes the data using machine learning algorithms and evaluates the user's behavior. Generative AI models are utilized to generate evaluation results and areas for improvement as output.

[0526] Step 4:

[0527] The server generates feedback for the user based on the analysis results. The input is the evaluation results generated in step 3. The server generates meaningful and specific advice for the user and prepares the feedback as output in text or audio format.

[0528] Step 5:

[0529] The terminal presents the user with feedback received from the server. The input is the specific advice generated in step 4. The feedback is displayed on the screen or provided audibly through the speaker. This allows the user to correct their actions in real time.

[0530] Step 6:

[0531] Users engage in a virtual training experience using augmented reality devices. Inputs consist of real-world environmental data and instructional information from a server. Outputs include visual and audio guidance in a training environment that integrates reality and virtuality. This interaction allows users to intuitively understand the instructions and perform exercises effectively.

[0532] Step 7:

[0533] The server manages the user's training progress using a progress tracking device and calculates reward points based on the goals achieved. The input is past training data. The output is the calculated points, which the user can view on their terminal.

[0534] Step 8:

[0535] The server uses a health management device to assess the user's overall health status. Inputs include the user's activity data and lifestyle data. The output is personalized health improvement advice, provided to the user, allowing them to gain concrete directions for improving their health.

[0536] (Application Example 1)

[0537] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0538] In today's fitness scene, the presence of a professional instructor is essential for receiving effective training guidance. However, environments where instructor support is always available are limited, and when training on one's own, there is a high risk of incorrect form and ineffective movements. Furthermore, there is often a lack of individualized feedback, and progress management and guidance for health improvement are insufficient. This invention aims to solve these problems and provide an environment in which users can continue training effectively and accurately at their own pace.

[0539] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0540] In this invention, the server includes information gathering means for collecting user behavior data, information analysis means for executing machine learning algorithms for analyzing the collected behavior data, response means for providing immediate feedback to the user based on the analysis results, and display means for enabling visual instruction with a virtual instructor through an extended entity device for providing a training experience in a virtual environment. This allows the user to correct their own behavior in real time and receive expert instruction.

[0541] "Information gathering means" refers to devices that collect user actions and biometric information, and are used to acquire data using sensors and cameras.

[0542] An "information analysis tool" is a device that executes machine learning algorithms based on collected data to evaluate user behavior and identify areas for improvement.

[0543] A "response device" is a device that immediately informs the user of the analyzed results and provides guidance for correct operation or improvement in the form of audio or visual instructions.

[0544] An "extended physical device" is a technology that presents a virtual environment to the user, enabling a more effective training experience, and displays it overlaid on the real environment.

[0545] A "display means" is a device that presents information to enable interaction with a virtual instructor and to provide visual and intuitive instruction.

[0546] A system that implements an application example of this invention includes an information gathering means that collects motion data in real time through smart glasses worn by the user or a smartphone carried by the user. This means incorporates a camera and biosensors to measure posture, heart rate, and other parameters during exercise.

[0547] The collected data is transmitted to a cloud server via wireless communication. The server performs information analysis using machine learning algorithms such as TensorFlow to analyze the user's behavior patterns. The analysis results identify errors in behavior and areas for improvement, and based on this, a real-time response is provided.

[0548] The feedback provided to the user is displayed in audio or text format on smart glasses or a smartphone, and visual guidance is also provided using augmented physical devices. A virtual instructor is overlaid on the user's field of view, demonstrating the correct actions. This allows the user to receive interactive instruction within the virtual environment.

[0549] As a concrete example, consider a user practicing the correct squat form at the gym. The smart glasses worn by the user analyze the user's posture in real time during the exercise and provide voice feedback such as, "Bring your knees a little further inward." A virtual instructor visually demonstrates the correct squat form, allowing the user to correct their movement accordingly. This feature provides the effect of personalized instruction without compromising privacy.

[0550] Furthermore, an example of a prompt message is: "Use this system to provide real-time advice to users aiming for ideal squat form. Specifically point out which movements need improvement and synchronize them with the movements of a virtual trainer." This prompt allows the system to generate precise feedback based on the movement data.

[0551] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0552] Step 1:

[0553] The device collects user motion data in real time using the camera and sensors of the smart glasses or smartphone worn by the user. The input is the user's physical movements and biometric data, and the output is a motion data file in which this data is recorded in digital format. The device temporarily stores the motion data and prepares it for transfer to the cloud server in the next step.

[0554] Step 2:

[0555] The terminal sends data to the cloud server via the internet. The input here is an operational data file, which is sent to the server as output via network communication. This communication uses a secure protocol to prevent data leakage.

[0556] Step 3:

[0557] The server executes a machine learning algorithm to analyze the received motion data. The model uses a generative AI model, trained to analyze user motion patterns. In this step, motion data is received as input, and the analysis results extract evaluations of the user's posture and coaching points. The output is feedback information based on the analysis results.

[0558] Step 4:

[0559] The server sends the analysis results to the terminal as feedback information via a response mechanism. The input is the feedback information of the analysis results, and the output is the feedback message converted for display on the terminal side. This feedback is provided in audio or text format.

[0560] Step 5:

[0561] The terminal provides feedback to the user using an augmented physical device. The input is feedback messages sent from the server, and the output is displayed in the user's field of view in real time. Specifically, the movements and instructions of a virtual instructor are visually overlaid, and the user uses this as a guide to correct their actions.

[0562] Step 6:

[0563] Users modify their actions according to the feedback provided. Here, they receive visual and auditory feedback as input and improve their actions as output. By following the system's instructions, users can learn the correct training methods.

[0564] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0565] This invention is a system incorporating emotion recognition capabilities to enable effective training and health management for users. The system includes a data acquisition device, a data analysis device, a feedback device, an augmented reality device, a progress management device, a health management device, and an emotion engine. The operation of each device and the overall system is described below.

[0566] The device is equipped with equipment to collect user motion and emotional data. Users begin exercising while wearing a smartphone or wearable device. These devices include cameras and biosensors that detect the user's emotional state in real time from their facial expressions and voice.

[0567] The collected behavioral and emotional data is transmitted to a cloud server via the internet. On the server, a data analysis device and an emotion engine analyze the respective datasets and evaluate the user's physical and emotional state. The emotion engine classifies the data based on multiple emotional categories, such as joy, anger, and concentration.

[0568] The analysis results are transmitted to the terminal via a feedback device, allowing the user to receive real-time advice based on their actions and emotions. For example, if the system detects that the user is experiencing fatigue or stress, the feedback will suggest slowing down the pace of their exercise. Conversely, if the system detects that the user is highly motivated, it can suggest more challenging training.

[0569] Augmented reality devices provide users with a virtual training environment that adapts to their emotional state. If the user is in a relaxed mental state, the training will be adjusted to take place within a calm scenario.

[0570] The progress tracking device records data, including the user's emotional responses to the exercise, and calculates reward points. These points are used as an incentive to help users train without stress.

[0571] Furthermore, the health management device generates advice aimed at improving lifestyle habits based on movement and emotional data. For example, if data analysis reveals that the user has a high stress level, suggestions for relaxation will be made.

[0572] As a concrete example, when a user starts a morning training session, the device instantly collects and transmits behavioral and emotional data. The server analyzes the data and, if the user is showing signs of tension, provides feedback such as "slow down." In augmented reality, an emotionally resonant environment is provided, such as a training scene on a beach. In this way, the system aims to enhance the sustainability and effectiveness of training.

[0573] The following describes the processing flow.

[0574] Step 1:

[0575] The user launches a smartphone application and prepares for a training session. The device captures the user's body movements through cameras and sensors to collect motion data. It also acquires emotional data from facial expressions and tone of voice using microphones and cameras.

[0576] Step 2:

[0577] The device organizes the collected behavioral and emotional data and sends data packets to the server in real time. The data includes real-time user feedback.

[0578] Step 3:

[0579] The server analyzes the received motion data using a data analysis device to evaluate the user's physical movements. Machine learning algorithms are used for the analysis to determine the accuracy and appropriateness of the movements.

[0580] Step 4:

[0581] The server uses an emotion engine to analyze emotional data and identify the user's emotional state. Here, it classifies the emotion as either joy, focus, or anxiety.

[0582] Step 5:

[0583] The server integrates the results of behavioral and emotional analysis to generate comprehensive feedback for the user. For example, if the user is showing signs of fatigue, it might include a suggestion such as, "Let's take a short break."

[0584] Step 6:

[0585] The device presents the user with feedback sent from the server. This includes voice guidance and text messages, which the user can use to adjust their exercise routine.

[0586] Step 7:

[0587] The device generates a virtual environment tailored to the user's emotional state via an augmented reality device. For example, during training in a relaxed state, a calm background scene is provided.

[0588] Step 8:

[0589] The server records the user's progress and calculates reward points based on their training performance. Users can check their points through their device to gauge their motivation.

[0590] Step 9:

[0591] The server generates health advice that leads to lifestyle improvements based on accumulated behavioral and emotional data. For example, it may include recommendations for improving sleep or reducing stress.

[0592] Step 10:

[0593] Users can review the health advice displayed on their devices and use it to improve their health by reviewing restrictive lifestyle habits.

[0594] (Example 2)

[0595] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0596] Providing effective, real-time feedback that takes into account individual physical and emotional states is challenging when users engage in fitness training. Furthermore, maintaining motivation during training is a challenge, and there is a need to utilize emotional data to provide advice that supports a healthier lifestyle.

[0597] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0598] In this invention, the server includes data collection means for collecting behavioral data and emotional data, data analysis means for analyzing the collected behavioral data, and emotion determination means for analyzing the collected emotional data and classifying the user's emotional state into multiple emotion categories. This makes it possible to provide appropriate training content and feedback to individual users in real time.

[0599] "Motion data" is a collection of information that shows the user's physical movements and physiological state, and is used to evaluate performance and health status during training.

[0600] "Emotional data" is a collection of information indicating a user's psychological state, obtained from their facial expressions, tone of voice, and other biosignals, and is used to evaluate the user's emotions and mental state.

[0601] "Data collection means" refers to devices or technologies used to acquire user behavioral data and emotional data, and includes smartphones and wearable devices.

[0602] "Data analysis means" refers to methods and devices for processing and analyzing collected motion data, and in particular, they evaluate movement patterns and physical conditions by using machine learning algorithms.

[0603] An "emotion determination tool" is a method or system for processing collected emotional data and classifying it into emotional categories such as joy, anger, and concentration.

[0604] A "feedback mechanism" is a device or technology used to provide users with advice and information in real time based on analysis results.

[0605] "Augmented reality means" are technologies that provide a virtual training environment tailored to the user's emotional state, overlaying digital information onto the real world visually or audibly.

[0606] "Reward points" are units of incentives awarded to users to evaluate their training progress and achievements, and to improve their motivation.

[0607] A "progress management method" refers to a method or system for recording a user's training history and emotional responses, and for calculating reward points.

[0608] "Health management tools" refer to methods and systems that utilize users' behavioral and emotional data to provide personalized health improvement advice and lifestyle suggestions.

[0609] In this invention, the user starts training using a smartphone and a wearable device, thereby collecting motion and emotional data in real time. The device is equipped with a camera and biosensors, which can acquire physiological indicators such as the user's facial expressions, heart rate, and body temperature. This data is transmitted to a server in the cloud via a secure protocol.

[0610] The server executes a program that includes machine learning algorithms as a data analysis tool to process behavioral data. Simultaneously, an emotion determination tool analyzes emotional data and classifies the user's psychological state into categories such as joy and concentration. This allows the user's fitness level and emotional condition to be measured.

[0611] The feedback system provides real-time, tailored advice to the user based on these analysis results. For example, if user fatigue is detected, the feedback can recommend "slow down." Furthermore, augmented reality is used to generate a virtual training environment tailored to the user's emotional state, such as presenting a calm beach scene if the user is relaxed.

[0612] Furthermore, the server uses progress management tools to record the user's exercise progress and emotional responses, and calculates reward points. This makes it possible to enhance the effectiveness of training while maintaining the user's motivation. The health management tools utilize the collected data to provide users with advice on improving their lifestyle. Specifically, when stress levels are high, relaxation techniques can be suggested.

[0613] As a concrete example, when a user starts a morning training session, the device instantly collects data and sends it to the server. The server analyzes the data and provides feedback to the user, such as "Let's slow down." In the virtual environment, a training scene on a beach is created, providing an environment that responds to the user's emotions. An example of a prompt using a generative AI model might be: "Based on user feedback, please provide training adjustment suggestions in real time. User's emotional state: motivated, physical state: slightly fatigued."

[0614] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0615] Step 1:

[0616] The user begins training by wearing a smartphone or wearable device. The device uses cameras and biosensors to collect the user's movement and emotional data. Specifically, information such as heart rate, body temperature, and facial expressions is acquired in real time. The input for this step is the user's physiological and behavioral information, and the output is transferred to a cloud server.

[0617] Step 2:

[0618] The device transmits collected behavioral and emotional data to a cloud server via the internet. This data transfer ensures that all physiological and emotional data is centrally managed on the server. The input consists of the data collected by the device, and the output consists of the data stored on the server after secure data transmission.

[0619] Step 3:

[0620] The server analyzes motion data using data analysis tools. Specifically, the server executes machine learning algorithms and evaluates the user's fitness status by analyzing motion patterns. Motion data stored in the cloud is used as input, and the analyzed fitness status is obtained as output.

[0621] Step 4:

[0622] The server analyzes the collected emotional data using an emotion determination tool. This classifies the user's emotions into categories such as joy, anger, and concentration, based on their voice and facial expressions. The input is emotional data, and the output is the identified emotional state.

[0623] Step 5:

[0624] The server generates feedback to the user using feedback mechanisms based on the analysis results. For example, if the user's data indicates fatigue, it will generate advice such as "Slow down." The results of behavioral and sentiment analysis are used as input, and the output is a feedback message.

[0625] Step 6:

[0626] The server sends the generated feedback to the terminal, and the terminal displays advice to the user in real time. Here, the input is the feedback data sent from the server, and the output is the feedback displayed on the user's terminal.

[0627] Step 7:

[0628] The device generates a virtual training environment using augmented reality. Depending on the user's emotional state, training scenes that match their mental state, such as a calm beach or mountain landscape, are presented. The input is analyzed emotional data, and the output is the customized virtual environment presented to the user.

[0629] Step 8:

[0630] The server uses progress management tools to record the user's training history and emotional responses, and calculates reward points. Cumulative data and responses are used as input, and visualized progress and incentive points are generated as output.

[0631] Step 9:

[0632] The server uses health management tools to generate lifestyle improvement advice based on the user's behavioral and emotional data. For example, users with high stress levels will be offered relaxation suggestions. The input is continuously collected data, and the output is customized lifestyle improvement advice.

[0633] (Application Example 2)

[0634] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0635] The challenge lies in how to alleviate the physical and mental burden faced by transportation workers in harsh working conditions, and how to improve work efficiency and safety. Furthermore, there is a need for a system that can monitor workers' health and emotional states in real time and respond appropriately.

[0636] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0637] In this invention, the server includes information gathering means for collecting operational information, information analysis means for executing machine learning algorithms to analyze the collected operational information, feedback means for providing real-time feedback to the user based on the execution results, emotion analysis means for monitoring emotional states in real time, and transportation work support means for providing information to improve the condition of workers in transportation operations. This makes it possible to maintain the health of transportation workers and improve work efficiency.

[0638] "Motion information" refers to information that records and acquires the user's physical movements as digital data.

[0639] "Information gathering means" refers to a device or system for detecting and collecting data on a user's actions and emotional state.

[0640] "Information analysis means" refers to a device or software that performs computational processing to evaluate the user's state and performance based on collected data.

[0641] A "machine learning algorithm" is a mathematical and statistical technique used in the data analysis process, which learns from past data to make predictions and classifications.

[0642] A "feedback device" is a device or system that provides users with advice and information regarding their behavior and emotions based on analysis results.

[0643] "Augmented reality means" refers to a device or system based on technology that overlays and displays virtual information onto real space.

[0644] "Emotional analysis methods" are technologies used to evaluate a user's emotions and stress levels based on data such as their voice and facial expressions.

[0645] A "transportation work support device" is a device or system that provides information in real time in order to improve the efficiency of transportation operations and reduce the burden on workers.

[0646] This invention provides a system for improving the physical and emotional state of workers in transportation operations. The following describes specific embodiments of this system.

[0647] The server includes information gathering means, information analysis means, feedback means, sentiment analysis means, and transportation work support means. The information gathering means acquires motion information such as heart rate and movement via smart devices worn by transportation workers. Smartphones and smartwatches are used as hardware for this purpose.

[0648] The information analysis method utilizes machine learning algorithms to analyze collected data on a cloud server. The analysis uses software such as "Google Cloud Platform" and emotion recognition APIs as information analysis platforms, evaluating the user's stress and fatigue in real time.

[0649] The emotion analysis system classifies the user's emotional state based on acquired voice data and heart rate. This allows the system to understand whether the user is in a state of excitement or tension, and prepares to provide optimal feedback through the transportation work support system.

[0650] The feedback mechanism provides users with real-time advice and break suggestions based on the analyzed results. For example, if a worker's stress level is high, a notification such as "You should take a short break" is sent to their smart device.

[0651] For example, if a delivery worker suddenly shows signs of stress, the feedback system will send the worker advice such as, "We recommend you take a short break immediately." In this way, the system can support the health of delivery workers while enhancing work safety.

[0652] As an example of a prompt, it can be used in the following format: "Design a prototype application that evaluates the stress level of delivery drivers in real time using emotion recognition data and suggests relaxation music as needed."

[0653] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0654] Step 1:

[0655] The device collects motion information, including heart rate, location information, and voice data, via the smart device (smartphone or smartwatch) worn by the user. This allows the device to obtain data as input to understand the user's current status in real time.

[0656] Step 2:

[0657] The device sends collected operational information to a cloud server via its internet connection. The server uses a data analysis platform (e.g., Google Cloud Platform) to cleanse and preprocess the data, transforming it into an analyzable format.

[0658] Step 3:

[0659] The server executes machine learning algorithms and analyzes the preprocessed data. Here, it uses an emotion recognition API to classify the user's emotional state. This process outputs data that specifically evaluates whether the user is currently feeling stressed or relaxed.

[0660] Step 4:

[0661] Based on the analysis results, the server generates feedback. Specifically, if it determines that the stress level is high, it will generate a notification such as, "Please consider taking a break." This feedback is a suggestion of action that the user can take.

[0662] Step 5:

[0663] The server sends the generated feedback to the device. The device displays this feedback on the user's smart device screen. The user can then view advice about their status in real time and take action based on it.

[0664] Step 6:

[0665] The device records the receipt of feedback and the user's response, and transmits this information to the progress management system. This information is used for the ongoing management of the user's health status and for awarding points.

[0666] These processes enable transport workers to efficiently manage stress and fatigue, allowing them to perform their duties in a healthy and safe manner.

[0667] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0668] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0669] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0670] [Fourth Embodiment]

[0671] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0672] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0673] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0674] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0675] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0676] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0677] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0678] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0679] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0680] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0681] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0682] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0683] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0684] The system of the present invention enables efficient health management and training for users and has a multi-layered configuration including a data acquisition device, a data analysis device, a feedback device, an augmented reality device, a progress management device, and a health management device. The operation of each device and the overall system will be described below.

[0685] The device functions as a data collection device to collect user movements and biometric data. Examples include smartphone cameras and wearable devices with heart rate sensors. Users record their body movements using these devices while performing their usual exercise.

[0686] The collected data is transmitted via the internet to a server in the cloud. On the server, a data analysis device uses machine learning algorithms to analyze this data and evaluate postural imbalances and the effects of movement. This analysis is performed using a pre-trained model and can take into account the movement characteristics of each user.

[0687] The analysis results are transmitted to the terminal via a feedback device and displayed in real time on the user's smartphone or tablet as advice for improving their form. For example, the terminal provides specific feedback such as "Bend your knees a little more" in voice or text. This allows the user to correct their form on the spot.

[0688] Augmented reality devices provide users with a virtual training environment. Through augmented reality-enabled devices, users can experience interaction with virtual trainers and receive visual and intuitive instruction. This makes it possible to receive instruction equivalent to professional training from the comfort of one's home.

[0689] Furthermore, the server uses a progress management device to record the user's training history and calculates reward points based on that data. Users can check their points on their devices, which helps to improve their exercise endurance.

[0690] Furthermore, the health management device evaluates the user's health status based on their activity data and registered lifestyle data, and provides personalized health improvement advice. For example, it considers the user's dietary history and makes specific suggestions such as, "You should increase your protein intake."

[0691] As a concrete example, when a user starts a new exercise, the device provides an initial setup guide and positions the camera appropriately. Once training begins, movements are recorded in real time, and the server immediately analyzes the data. Receiving immediate feedback, the user can immerse themselves in the virtual environment and perform focused training. In this way, the system of the present invention enables continuous and effective health management even when training at home.

[0692] The following describes the processing flow.

[0693] Step 1:

[0694] Users prepare to begin exercising by putting on their smartphone or wearable device and launching the application. The device collects the user's movement data in real time through cameras and sensors.

[0695] Step 2:

[0696] The device organizes the collected motion data according to a pre-configured format and transmits it to a server via the internet. This data includes video data and biometric data.

[0697] Step 3:

[0698] The server analyzes the received data using machine learning algorithms via a data analysis device. Here, joint angles and movement patterns are calculated to evaluate the accuracy and efficiency of the movements.

[0699] Step 4:

[0700] Based on the analysis results, the server identifies areas for improvement for the user and generates feedback data, which includes voice guidance and text messages.

[0701] Step 5:

[0702] The device displays feedback data sent from the server to the user in real time. The feedback is provided either visually or audibly as specific instructions, such as "Please lower your shoulders a little more."

[0703] Step 6:

[0704] Based on the feedback provided, users adjust their exercise form and continue training to achieve optimal movement.

[0705] Step 7:

[0706] The device utilizes augmented reality to provide users with videos of a virtual trainer guiding them through exercises. Users then use these videos to verify and correct their movements.

[0707] Step 8:

[0708] The server records the user's training progress and calculates reward points. This allows users to track their achievements and maintain motivation.

[0709] Step 9:

[0710] The server generates health improvement advice based on collected behavioral data and lifestyle information. This advice is then communicated to the user via the app.

[0711] (Example 1)

[0712] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0713] In modern society, individual health conditions and exercise habits are becoming increasingly diverse, creating a need for technologies that provide individually optimized health management and exercise guidance. However, current systems have problems in providing accurate real-time feedback and effective training experiences using virtual environments. Furthermore, mechanisms for providing specific health improvement advice based on movement and lifestyle information are insufficient.

[0714] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0715] In this invention, the server includes information gathering means for acquiring information about its operation, information analysis means for executing a machine learning algorithm to analyze the acquired information, and response providing means for providing an immediate response to the user based on the analysis results. This enables real-time feedback to the user tailored to their individual health condition.

[0716] "Information gathering means for acquiring information about movement" refers to devices used to collect data on a user's physical movements and physiological processes, and includes technologies that utilize sensors, cameras, etc., to acquire accurate data.

[0717] "Information analysis means that executes machine learning algorithms to analyze acquired information" refers to a device that utilizes machine learning technology to analyze collected data and perform pattern recognition or anomaly detection.

[0718] "A means of providing immediate responses to users based on analysis results" refers to a technology that generates and displays real-time feedback to users via voice or text based on the results of the analysis.

[0719] "Augmented reality means for providing exercise experiences in a virtual environment" refers to a technology that uses computer graphics to overlay digital information onto the real world, and is a method for providing users with virtual trainers and exercise guides.

[0720] "Presentation means for presenting generated guidance audibly and visually" refers to technology that conveys advice and guidance generated based on analysis results to users through displays or speakers.

[0721] "A progress management system that calculates reward points based on the degree of achievement and manages the progress of users" is a system that manages the activity history of users, converts their results into points and makes them visible, and is a method aimed at improving motivation.

[0722] "A health management tool for generating health improvement guidance based on user activity and lifestyle information" refers to a technology that evaluates a user's health status based on individual activity and lifestyle data and provides specific advice for improvement.

[0723] This invention utilizes a multi-layered IoT system to support users' daily health management and training. The main components and operation of the system are described below.

[0724] Information gathering

[0725] The device acquires information about the user's movements using smartphones and wearable devices. For example, it utilizes the camera built into the smartphone or a heart rate sensor in a wristwatch. These devices monitor the user's movements in real time and collect data including heart rate and posture.

[0726] Data Analysis

[0727] Once data is collected, the device sends this data to a server via the internet. On the server, a data analysis device receives this data and performs analysis using a pre-trained generative AI model. This model uses machine learning algorithms to evaluate the user's movements and posture. For example, it evaluates squat form and detects knee angles and postural irregularities.

[0728] Real-time feedback

[0729] The analysis results are sent from the server to the terminal, which then provides immediate feedback to the user via voice or text. This allows the user to receive specific instructions, such as "Bend your knees a little more," and correct their movements on the spot.

[0730] virtual environment

[0731] Furthermore, users can gain a virtual training experience using augmented reality devices. A virtual trainer is superimposed onto the real world, providing visual and intuitive guidance to the user. This makes it possible to receive high-quality instruction equivalent to professional training at home.

[0732] Progress management and health improvement

[0733] The server records the user's training progress and calculates reward points based on their achievement level. Users can check these points on their device, which motivates them to continue exercising. In addition, a health management device, based on the user's movement data and lifestyle data, provides specific advice for improving health. For example, it might generate suggestions such as, "You should increase your protein intake."

[0734] For example, when a user tries a new exercise, the device provides an initial setup guide and assists in properly positioning the device. Once data collection begins, the server analyzes the data in real time and immediately sends feedback to the device. This process is an effective means for users to engage in more focused training and manage their health.

[0735] An example of a prompt message would be, "How can I monitor new exercises and provide users with immediate feedback?"

[0736] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0737] Step 1:

[0738] The device uses the smartphone's camera and wearable devices to collect data about the user's movements in real time. Inputs include the user's physical movements and heart rate, and all of this data is continuously captured by sensors and built-in software. As output, the collected raw data is temporarily stored inside the device.

[0739] Step 2:

[0740] The device transmits the collected data to a cloud server via the internet. The input is the raw data collected in step 1. The data is transferred through a secure protocol, and the output is stored on the server. During this process, the data is encrypted to prevent data loss and unauthorized access.

[0741] Step 3:

[0742] The server processes the received data using a data analysis device. The input consists of user behavior and physiological data stored in the cloud. The server analyzes the data using machine learning algorithms and evaluates the user's behavior. Generative AI models are utilized to generate evaluation results and areas for improvement as output.

[0743] Step 4:

[0744] The server generates feedback for the user based on the analysis results. The input is the evaluation results generated in step 3. The server generates meaningful and specific advice for the user and prepares the feedback as output in text or audio format.

[0745] Step 5:

[0746] The terminal presents the user with feedback received from the server. The input is the specific advice generated in step 4. The feedback is displayed on the screen or provided audibly through the speaker. This allows the user to correct their actions in real time.

[0747] Step 6:

[0748] Users engage in a virtual training experience using augmented reality devices. Inputs consist of real-world environmental data and instructional information from a server. Outputs include visual and audio guidance in a training environment that integrates reality and virtuality. This interaction allows users to intuitively understand the instructions and perform exercises effectively.

[0749] Step 7:

[0750] The server manages the user's training progress using a progress tracking device and calculates reward points based on the goals achieved. The input is past training data. The output is the calculated points, which the user can view on their terminal.

[0751] Step 8:

[0752] The server uses a health management device to assess the user's overall health status. Inputs include the user's activity data and lifestyle data. The output is personalized health improvement advice, provided to the user, allowing them to gain concrete directions for improving their health.

[0753] (Application Example 1)

[0754] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0755] In today's fitness scene, the presence of a professional instructor is essential for receiving effective training guidance. However, environments where instructor support is always available are limited, and when training on one's own, there is a high risk of incorrect form and ineffective movements. Furthermore, there is often a lack of individualized feedback, and progress management and guidance for health improvement are insufficient. This invention aims to solve these problems and provide an environment in which users can continue training effectively and accurately at their own pace.

[0756] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0757] In this invention, the server includes information gathering means for collecting user behavior data, information analysis means for executing machine learning algorithms for analyzing the collected behavior data, response means for providing immediate feedback to the user based on the analysis results, and display means for enabling visual instruction with a virtual instructor through an extended entity device for providing a training experience in a virtual environment. This allows the user to correct their own behavior in real time and receive expert instruction.

[0758] "Information gathering means" refers to devices that collect user actions and biometric information, and are used to acquire data using sensors and cameras.

[0759] An "information analysis tool" is a device that executes machine learning algorithms based on collected data to evaluate user behavior and identify areas for improvement.

[0760] A "response device" is a device that immediately informs the user of the analyzed results and provides guidance for correct operation or improvement in the form of audio or visual instructions.

[0761] An "extended physical device" is a technology that presents a virtual environment to the user, enabling a more effective training experience, and displays it overlaid on the real environment.

[0762] A "display means" is a device that presents information to enable interaction with a virtual instructor and to provide visual and intuitive instruction.

[0763] A system that implements an application example of this invention includes an information gathering means that collects motion data in real time through smart glasses worn by the user or a smartphone carried by the user. This means incorporates a camera and biosensors to measure posture, heart rate, and other parameters during exercise.

[0764] The collected data is transmitted to a cloud server via wireless communication. The server performs information analysis using machine learning algorithms such as TensorFlow to analyze the user's behavior patterns. The analysis results identify errors in behavior and areas for improvement, and based on this, a real-time response is provided.

[0765] The feedback provided to the user is displayed in audio or text format on smart glasses or a smartphone, and visual guidance is also provided using augmented physical devices. A virtual instructor is overlaid on the user's field of view, demonstrating the correct actions. This allows the user to receive interactive instruction within the virtual environment.

[0766] As a concrete example, consider a user practicing the correct squat form at the gym. The smart glasses worn by the user analyze the user's posture in real time during the exercise and provide voice feedback such as, "Bring your knees a little further inward." A virtual instructor visually demonstrates the correct squat form, allowing the user to correct their movement accordingly. This feature provides the effect of personalized instruction without compromising privacy.

[0767] Furthermore, an example of a prompt message is: "Use this system to provide real-time advice to users aiming for ideal squat form. Specifically point out which movements need improvement and synchronize them with the movements of a virtual trainer." This prompt allows the system to generate precise feedback based on the movement data.

[0768] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0769] Step 1:

[0770] The device collects user motion data in real time using the camera and sensors of the smart glasses or smartphone worn by the user. The input is the user's physical movements and biometric data, and the output is a motion data file in which this data is recorded in digital format. The device temporarily stores the motion data and prepares it for transfer to the cloud server in the next step.

[0771] Step 2:

[0772] The terminal sends data to the cloud server via the internet. The input here is an operational data file, which is sent to the server as output via network communication. This communication uses a secure protocol to prevent data leakage.

[0773] Step 3:

[0774] The server executes a machine learning algorithm to analyze the received motion data. The model uses a generative AI model, trained to analyze user motion patterns. In this step, motion data is received as input, and the analysis results extract evaluations of the user's posture and coaching points. The output is feedback information based on the analysis results.

[0775] Step 4:

[0776] The server sends the analysis results to the terminal as feedback information via a response mechanism. The input is the feedback information of the analysis results, and the output is the feedback message converted for display on the terminal side. This feedback is provided in audio or text format.

[0777] Step 5:

[0778] The terminal provides feedback to the user using an augmented physical device. The input is feedback messages sent from the server, and the output is displayed in the user's field of view in real time. Specifically, the movements and instructions of a virtual instructor are visually overlaid, and the user uses this as a guide to correct their actions.

[0779] Step 6:

[0780] Users modify their actions according to the feedback provided. Here, they receive visual and auditory feedback as input and improve their actions as output. By following the system's instructions, users can learn the correct training methods.

[0781] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0782] This invention is a system incorporating emotion recognition capabilities to enable effective training and health management for users. The system includes a data acquisition device, a data analysis device, a feedback device, an augmented reality device, a progress management device, a health management device, and an emotion engine. The operation of each device and the overall system is described below.

[0783] The device is equipped with equipment to collect user motion and emotional data. Users begin exercising while wearing a smartphone or wearable device. These devices include cameras and biosensors that detect the user's emotional state in real time from their facial expressions and voice.

[0784] The collected behavioral and emotional data is transmitted to a cloud server via the internet. On the server, a data analysis device and an emotion engine analyze the respective datasets and evaluate the user's physical and emotional state. The emotion engine classifies the data based on multiple emotional categories, such as joy, anger, and concentration.

[0785] The analysis results are transmitted to the terminal via a feedback device, allowing the user to receive real-time advice based on their actions and emotions. For example, if the system detects that the user is experiencing fatigue or stress, the feedback will suggest slowing down the pace of their exercise. Conversely, if the system detects that the user is highly motivated, it can suggest more challenging training.

[0786] Augmented reality devices provide users with a virtual training environment that adapts to their emotional state. If the user is in a relaxed mental state, the training will be adjusted to take place within a calm scenario.

[0787] The progress tracking device records data, including the user's emotional responses to the exercise, and calculates reward points. These points are used as an incentive to help users train without stress.

[0788] Furthermore, the health management device generates advice aimed at improving lifestyle habits based on movement and emotional data. For example, if data analysis reveals that the user has a high stress level, suggestions for relaxation will be made.

[0789] As a concrete example, when a user starts a morning training session, the device instantly collects and transmits behavioral and emotional data. The server analyzes the data and, if the user is showing signs of tension, provides feedback such as "slow down." In augmented reality, an emotionally resonant environment is provided, such as a training scene on a beach. In this way, the system aims to enhance the sustainability and effectiveness of training.

[0790] The following describes the processing flow.

[0791] Step 1:

[0792] The user launches a smartphone application and prepares for a training session. The device captures the user's body movements through cameras and sensors to collect motion data. It also acquires emotional data from facial expressions and tone of voice using microphones and cameras.

[0793] Step 2:

[0794] The device organizes the collected behavioral and emotional data and sends data packets to the server in real time. The data includes real-time user feedback.

[0795] Step 3:

[0796] The server analyzes the received motion data using a data analysis device to evaluate the user's physical movements. Machine learning algorithms are used for the analysis to determine the accuracy and appropriateness of the movements.

[0797] Step 4:

[0798] The server uses an emotion engine to analyze emotional data and identify the user's emotional state. Here, it classifies the emotion as either joy, focus, or anxiety.

[0799] Step 5:

[0800] The server integrates the results of behavioral and emotional analysis to generate comprehensive feedback for the user. For example, if the user is showing signs of fatigue, it might include a suggestion such as, "Let's take a short break."

[0801] Step 6:

[0802] The device presents the user with feedback sent from the server. This includes voice guidance and text messages, which the user can use to adjust their exercise routine.

[0803] Step 7:

[0804] The device generates a virtual environment tailored to the user's emotional state via an augmented reality device. For example, during training in a relaxed state, a calm background scene is provided.

[0805] Step 8:

[0806] The server records the user's progress and calculates reward points based on their training performance. Users can check their points through their device to gauge their motivation.

[0807] Step 9:

[0808] The server generates health advice that leads to lifestyle improvements based on accumulated behavioral and emotional data. For example, it may include recommendations for improving sleep or reducing stress.

[0809] Step 10:

[0810] Users can review the health advice displayed on their devices and use it to improve their health by reviewing restrictive lifestyle habits.

[0811] (Example 2)

[0812] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0813] Providing effective, real-time feedback that takes into account individual physical and emotional states is challenging when users engage in fitness training. Furthermore, maintaining motivation during training is a challenge, and there is a need to utilize emotional data to provide advice that supports a healthier lifestyle.

[0814] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0815] In this invention, the server includes data collection means for collecting behavioral data and emotional data, data analysis means for analyzing the collected behavioral data, and emotion determination means for analyzing the collected emotional data and classifying the user's emotional state into multiple emotion categories. This makes it possible to provide appropriate training content and feedback to individual users in real time.

[0816] "Motion data" is a collection of information that shows the user's physical movements and physiological state, and is used to evaluate performance and health status during training.

[0817] "Emotional data" is a collection of information indicating a user's psychological state, obtained from their facial expressions, tone of voice, and other biosignals, and is used to evaluate the user's emotions and mental state.

[0818] "Data collection means" refers to devices or technologies used to acquire user behavioral data and emotional data, and includes smartphones and wearable devices.

[0819] "Data analysis means" refers to methods and devices for processing and analyzing collected motion data, and in particular, they evaluate movement patterns and physical conditions by using machine learning algorithms.

[0820] An "emotion determination tool" is a method or system for processing collected emotional data and classifying it into emotional categories such as joy, anger, and concentration.

[0821] A "feedback mechanism" is a device or technology used to provide users with advice and information in real time based on analysis results.

[0822] "Augmented reality means" are technologies that provide a virtual training environment tailored to the user's emotional state, overlaying digital information onto the real world visually or audibly.

[0823] "Reward points" are units of incentives awarded to users to evaluate their training progress and achievements, and to improve their motivation.

[0824] A "progress management method" refers to a method or system for recording a user's training history and emotional responses, and for calculating reward points.

[0825] "Health management tools" refer to methods and systems that utilize users' behavioral and emotional data to provide personalized health improvement advice and lifestyle suggestions.

[0826] In this invention, the user starts training using a smartphone and a wearable device, thereby collecting motion and emotional data in real time. The device is equipped with a camera and biosensors, which can acquire physiological indicators such as the user's facial expressions, heart rate, and body temperature. This data is transmitted to a server in the cloud via a secure protocol.

[0827] The server executes a program that includes machine learning algorithms as a data analysis tool to process behavioral data. Simultaneously, an emotion determination tool analyzes emotional data and classifies the user's psychological state into categories such as joy and concentration. This allows the user's fitness level and emotional condition to be measured.

[0828] The feedback system provides real-time, tailored advice to the user based on these analysis results. For example, if user fatigue is detected, the feedback can recommend "slow down." Furthermore, augmented reality is used to generate a virtual training environment tailored to the user's emotional state, such as presenting a calm beach scene if the user is relaxed.

[0829] Furthermore, the server uses progress management tools to record the user's exercise progress and emotional responses, and calculates reward points. This makes it possible to enhance the effectiveness of training while maintaining the user's motivation. The health management tools utilize the collected data to provide users with advice on improving their lifestyle. Specifically, when stress levels are high, relaxation techniques can be suggested.

[0830] As a concrete example, when a user starts a morning training session, the device instantly collects data and sends it to the server. The server analyzes the data and provides feedback to the user, such as "Let's slow down." In the virtual environment, a training scene on a beach is created, providing an environment that responds to the user's emotions. An example of a prompt using a generative AI model might be: "Based on user feedback, please provide training adjustment suggestions in real time. User's emotional state: motivated, physical state: slightly fatigued."

[0831] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0832] Step 1:

[0833] The user begins training by wearing a smartphone or wearable device. The device uses cameras and biosensors to collect the user's movement and emotional data. Specifically, information such as heart rate, body temperature, and facial expressions is acquired in real time. The input for this step is the user's physiological and behavioral information, and the output is transferred to a cloud server.

[0834] Step 2:

[0835] The device transmits collected behavioral and emotional data to a cloud server via the internet. This data transfer ensures that all physiological and emotional data is centrally managed on the server. The input consists of the data collected by the device, and the output consists of the data stored on the server after secure data transmission.

[0836] Step 3:

[0837] The server analyzes motion data using data analysis tools. Specifically, the server executes machine learning algorithms and evaluates the user's fitness status by analyzing motion patterns. Motion data stored in the cloud is used as input, and the analyzed fitness status is obtained as output.

[0838] Step 4:

[0839] The server analyzes the collected emotional data using an emotion determination tool. This classifies the user's emotions into categories such as joy, anger, and concentration, based on their voice and facial expressions. The input is emotional data, and the output is the identified emotional state.

[0840] Step 5:

[0841] The server generates feedback to the user using feedback mechanisms based on the analysis results. For example, if the user's data indicates fatigue, it will generate advice such as "Slow down." The results of behavioral and sentiment analysis are used as input, and the output is a feedback message.

[0842] Step 6:

[0843] The server sends the generated feedback to the terminal, and the terminal displays advice to the user in real time. Here, the input is the feedback data sent from the server, and the output is the feedback displayed on the user's terminal.

[0844] Step 7:

[0845] The device generates a virtual training environment using augmented reality. Depending on the user's emotional state, training scenes that match their mental state, such as a calm beach or mountain landscape, are presented. The input is analyzed emotional data, and the output is the customized virtual environment presented to the user.

[0846] Step 8:

[0847] The server uses progress management tools to record the user's training history and emotional responses, and calculates reward points. Cumulative data and responses are used as input, and visualized progress and incentive points are generated as output.

[0848] Step 9:

[0849] The server uses health management tools to generate lifestyle improvement advice based on the user's behavioral and emotional data. For example, users with high stress levels will be offered relaxation suggestions. The input is continuously collected data, and the output is customized lifestyle improvement advice.

[0850] (Application Example 2)

[0851] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0852] The challenge lies in how to alleviate the physical and mental burden faced by transportation workers in harsh working conditions, and how to improve work efficiency and safety. Furthermore, there is a need for a system that can monitor workers' health and emotional states in real time and respond appropriately.

[0853] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0854] In this invention, the server includes information gathering means for collecting operational information, information analysis means for executing machine learning algorithms to analyze the collected operational information, feedback means for providing real-time feedback to the user based on the execution results, emotion analysis means for monitoring emotional states in real time, and transportation work support means for providing information to improve the condition of workers in transportation operations. This makes it possible to maintain the health of transportation workers and improve work efficiency.

[0855] "Motion information" refers to information that records and acquires the user's physical movements as digital data.

[0856] "Information gathering means" refers to a device or system for detecting and collecting data on a user's actions and emotional state.

[0857] "Information analysis means" refers to a device or software that performs computational processing to evaluate the user's state and performance based on collected data.

[0858] A "machine learning algorithm" is a mathematical and statistical technique used in the data analysis process, which learns from past data to make predictions and classifications.

[0859] A "feedback device" is a device or system that provides users with advice and information regarding their behavior and emotions based on analysis results.

[0860] "Augmented reality means" refers to a device or system based on technology that overlays and displays virtual information onto real space.

[0861] "Emotional analysis methods" are technologies used to evaluate a user's emotions and stress levels based on data such as their voice and facial expressions.

[0862] A "transportation work support device" is a device or system that provides information in real time in order to improve the efficiency of transportation operations and reduce the burden on workers.

[0863] This invention provides a system for improving the physical and emotional state of workers in transportation operations. The following describes specific embodiments of this system.

[0864] The server includes information gathering means, information analysis means, feedback means, sentiment analysis means, and transportation work support means. The information gathering means acquires motion information such as heart rate and movement via smart devices worn by transportation workers. Smartphones and smartwatches are used as hardware for this purpose.

[0865] The information analysis method utilizes machine learning algorithms to analyze collected data on a cloud server. The analysis uses software such as "Google Cloud Platform" and emotion recognition APIs as information analysis platforms, evaluating the user's stress and fatigue in real time.

[0866] The emotion analysis system classifies the user's emotional state based on acquired voice data and heart rate. This allows the system to understand whether the user is in a state of excitement or tension, and prepares to provide optimal feedback through the transportation work support system.

[0867] The feedback mechanism provides users with real-time advice and break suggestions based on the analyzed results. For example, if a worker's stress level is high, a notification such as "You should take a short break" is sent to their smart device.

[0868] For example, if a delivery worker suddenly shows signs of stress, the feedback system will send the worker advice such as, "We recommend you take a short break immediately." In this way, the system can support the health of delivery workers while enhancing work safety.

[0869] As an example of a prompt, it can be used in the following format: "Design a prototype application that evaluates the stress level of delivery drivers in real time using emotion recognition data and suggests relaxation music as needed."

[0870] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0871] Step 1:

[0872] The device collects motion information, including heart rate, location information, and voice data, via the smart device (smartphone or smartwatch) worn by the user. This allows the device to obtain data as input to understand the user's current status in real time.

[0873] Step 2:

[0874] The device sends collected operational information to a cloud server via its internet connection. The server uses a data analysis platform (e.g., Google Cloud Platform) to cleanse and preprocess the data, transforming it into an analyzable format.

[0875] Step 3:

[0876] The server executes machine learning algorithms and analyzes the preprocessed data. Here, it uses an emotion recognition API to classify the user's emotional state. This process outputs data that specifically evaluates whether the user is currently feeling stressed or relaxed.

[0877] Step 4:

[0878] Based on the analysis results, the server generates feedback. Specifically, if it determines that the stress level is high, it will generate a notification such as, "Please consider taking a break." This feedback is a suggestion of action that the user can take.

[0879] Step 5:

[0880] The server sends the generated feedback to the device. The device displays this feedback on the user's smart device screen. The user can then view advice about their status in real time and take action based on it.

[0881] Step 6:

[0882] The device records the receipt of feedback and the user's response, and transmits this information to the progress management system. This information is used for the ongoing management of the user's health status and for awarding points.

[0883] These processes enable transport workers to efficiently manage stress and fatigue, allowing them to perform their duties in a healthy and safe manner.

[0884] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0885] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0886] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0887] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0888] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0889] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0890] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0891] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0892] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0893] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0894] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0895] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0896] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0897] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0898] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0899] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0900] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0901] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0902] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0903] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0904] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0905] The following is further disclosed regarding the embodiments described above.

[0906] (Claim 1)

[0907] A data acquisition device for collecting operational data,

[0908] A data analysis device that runs machine learning algorithms to analyze the collected motion data,

[0909] A feedback device that provides real-time feedback to the user based on the analysis results,

[0910] A system including an augmented reality device to provide a training experience in a virtual environment.

[0911] (Claim 2)

[0912] The system according to claim 1, further comprising a progress management device for calculating reward points and managing user progress.

[0913] (Claim 3)

[0914] The system according to claim 1, further comprising a health management device for generating health improvement advice based on user behavior data and lifestyle data.

[0915] "Example 1"

[0916] (Claim 1)

[0917] Information gathering means for obtaining information about operation,

[0918] An information analysis means that executes a machine learning algorithm to analyze the acquired information,

[0919] A response provision means that provides an immediate response to the user based on the analysis results,

[0920] Augmented reality means for providing an exercise experience in a virtual environment,

[0921] A presentation means for presenting the generated instruction audibly and visually,

[0922] A system that includes this.

[0923] (Claim 2)

[0924] The system according to claim 1, further comprising progress management means for calculating reward points based on the degree of achievement and for managing the progress of users.

[0925] (Claim 3)

[0926] The system according to claim 1, further comprising a health management means for generating health improvement guidance based on information about the user's actions and lifestyle habits.

[0927] "Application Example 1"

[0928] (Claim 1)

[0929] Information collection means for collecting user behavior data,

[0930] Information analysis means for executing machine learning algorithms to analyze collected motion data,

[0931] A response mechanism that provides immediate feedback to the user based on the analysis results,

[0932] An augmented physical device for providing a training experience in a virtual environment,

[0933] A display means that enables visual instruction with a virtual instructor through an extended physical device,

[0934] A system that includes this.

[0935] (Claim 2)

[0936] The system according to claim 1, further comprising progress management means for calculating reward evaluations and managing the progress of users.

[0937] (Claim 3)

[0938] The system according to claim 1, further comprising a health management means for generating health improvement suggestions based on the user's movement data and lifestyle data.

[0939] "Example 2 of combining an emotion engine"

[0940] (Claim 1)

[0941] A data collection means for collecting behavioral data and emotional data,

[0942] A data analysis means that executes a machine learning algorithm to analyze the collected motion data,

[0943] A means for determining emotions that analyzes collected emotional data and classifies the user's emotional state into multiple emotional categories,

[0944] A feedback mechanism that provides real-time feedback to the user based on the analysis results,

[0945] It provides a training experience in a virtual environment and augmented reality means that adapt to the user's emotional state,

[0946] A system that includes this.

[0947] (Claim 2)

[0948] The system according to claim 1, further comprising means for calculating reward points, managing user progress, and recording behavioral data and emotional responses.

[0949] (Claim 3)

[0950] The system according to claim 1, further comprising a health management means for generating health improvement advice based on user behavior data and emotional data.

[0951] "Application example 2 when combining with an emotional engine"

[0952] (Claim 1)

[0953] Information gathering means for collecting operational information,

[0954] Information analysis means for executing machine learning algorithms to analyze collected motion information,

[0955] A feedback mechanism that provides real-time feedback to the user based on the analysis results,

[0956] Augmented reality means to provide a training experience in a virtual environment,

[0957] A means of sentiment analysis for monitoring emotional states in real time,

[0958] Transportation work support means for providing information to improve the working conditions of workers in transportation operations.

[0959] A system that includes this.

[0960] (Claim 2)

[0961] The system according to claim 1, further comprising progress management means for calculating reward points and managing the user's progress.

[0962] (Claim 3)

[0963] The system according to claim 1, further comprising a health management means for generating health improvement advice based on the user's activity information and lifestyle information. [Explanation of Symbols]

[0964] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A data acquisition device for collecting operational data, A data analysis device that runs machine learning algorithms to analyze the collected motion data, A feedback device that provides real-time feedback to the user based on the analysis results, A system including an augmented reality device to provide a training experience in a virtual environment.

2. The system according to claim 1, further comprising a progress management device for calculating reward points and managing user progress.

3. The system according to claim 1, further comprising a health management device for generating health improvement advice based on user activity data and lifestyle data.

Images