system

Abstract

Provide a system. A method for collecting audio and video data includes: A means for analyzing the collected voice data to convert it into text and perform semantic analysis; A means for analyzing user motion information from the collected video data; A means for integrating the analyzed audio data and video data to grasp the user's health status; A means for generating advice and reminders based on the perceived health status; a means for notifying the user of the generated advice or reminder; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem that this invention aims to solve is to provide a system that supports patients with lifestyle-related diseases in efficiently managing their daily diet, exercise, mental and physical condition, and medication. In particular, there are challenges such as the difficulty of self-management, the effort required to record information, preventing forgetting, the complexity of dietary management, and maintaining motivation. The purpose of this is to contribute to improving the quality of life (QOL) of patients with lifestyle-related diseases and reducing medical costs. [Means for solving the problem]

[0005] The present invention provides a system including means for collecting audio data and video data, means for analyzing the collected audio data and performing text conversion and semantic analysis, means for analyzing user behavior information from the collected video data, means for integrating the analyzed audio data and video data to grasp the user's health status, means for generating advice and reminders based on the grasped health status, and means for notifying the user of the generated advice and reminders. Furthermore, by including means for storing the user's health status in a database from the audio data and video data to enable long-term analysis, and means for custom-generating advice and reminders based on the user's past health data, management suited to individual users can be performed.

[0006] "Voice data" refers to information or reports provided orally by a user that are recorded as digital signals.

[0007] "Video data" refers to video signal data recorded using a camera to record a user's movements and activities.

[0008] A "collection instrument" is a device or mechanism for recording and storing audio and video data.

[0009] "Analysis means" refers to the techniques and algorithms used to process collected data to extract and understand useful information.

[0010] "Text conversion" is the process of converting audio data into written information.

[0011] "Semantic analysis" is a natural language processing technique for understanding user intent and meaning from text data.

[0012] "Motion information" is data that indicates the user's physical activity and behavior.

[0013] "Integration" is the process of combining different types of data into one comprehensive piece of information.

[0014] "Health status" is information that indicates the user's current health condition and physical and psychological state.

[0015] "Generative means" is the process of creating new information and advice based on collected and analyzed data.

[0016] A "reminder" is a notification or alert that prompts a user to take a specific action or behavior.

[0017] "Notification means" refers to a method or device for communicating generated advice or reminders to the user.

[0018] A "database" is a system that efficiently stores and manages large amounts of data and allows quick access when needed.

[0019] "Custom-made" means a method or means that is specifically designed to meet the individual needs and requirements of a user. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The present invention is a system for supporting patients with lifestyle-related diseases to efficiently manage their daily diet, exercise, mental and physical condition, and medication, and specific embodiments thereof will be described below.

[0042] Data processing by the server

[0043] The server receives audio and video data sent from the user's device. The received audio data is converted into text using speech recognition technology. This text is then analyzed using natural language processing technology to extract meanings such as the user's health status and dietary details. Meanwhile, the video data is analyzed using video recognition technology to extract information about the user's movements.

[0044] The analyzed audio and video data are integrated to understand the user's health status. For example, if a user reports "chicken salad" as their diet and sends a video of themselves walking, the server analyzes this information and evaluates whether the user ate an appropriate diet or how much exercise they performed. Based on this evaluation, the generative AI generates customized advice and reminders.

[0045] The generated advice and reminders are stored in a database and managed as a long-term health record for the user, enabling more accurate personalized support based on the user's past data.

[0046] Device-based data collection and notification

[0047] The device has a means for collecting the user's voice and video. When the user reports verbally, "I feel fine this morning. I had chicken salad for dinner," the device records the voice via a microphone and sends it to the server. Similarly, when the user takes a video of themselves walking, the device records the video via a camera and sends it to the server.

[0048] When the device receives advice or reminders generated by the server, it notifies the user. Notifications are made using screen displays or voice synthesis technology, and specific instructions or reminders are presented to the user. For example, advice such as "We recommend you walk 30 minutes today" or a reminder such as "It's time to take your medication" is provided to the user.

[0049] User Interaction

[0050] Users can interact with the system simply by verbally reporting their health status. They can also record their daily health information easily by using the camera function on their smartphone or tablet to record their meals and exercise. This eliminates the need to record information and helps encourage continuous self-management.

[0051] As a concrete example, a user may report by voice, "I had oatmeal for breakfast today. I feel a little tired," and then film and send a video of their walking for the day. The server analyzes this information and generates advice based on their health condition and diet, such as, "Oatmeal is a good food choice. I recommend some light exercise today." The device notifies the user of this advice, and the user adjusts their lifestyle accordingly.

[0052] In this way, the present invention provides a system for supporting the user's daily lifestyle habits and for managing health efficiently and effectively.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] The user reports their health condition and dietary habits via voice and records their exercise using a smartphone or tablet camera. For example, they can report, "I had oatmeal and fruit for breakfast today. I feel great," and record themselves walking.

[0056] Step 2:

[0057] The device records the user's voice through a microphone and records video through a camera, thereby collecting audio and video data.

[0058] Step 3:

[0059] The terminal transmits the collected audio and video data to the server, and the transmitted data arrives at the server via a communication line.

[0060] Step 4:

[0061] The server analyzes the received voice data. First, it converts the voice data into text using voice recognition technology. Next, it uses natural language processing technology to analyze the meaning of the text and extract the user's health condition and dietary details.

[0062] Step 5:

[0063] The server analyzes the received video data and uses video recognition technology to extract information about the user's movements and analyze the amount and type of exercise.

[0064] Step 6:

[0065] The server combines the analyzed audio and video data to understand the user's health status, and provides a comprehensive assessment of the user's reported diet, physical condition, and exercise volume.

[0066] Step 7:

[0067] The AI ​​generates custom advice and reminders based on the user's health status, such as "Oatmeal is nutritious. Continue walking for 30 minutes today."

[0068] Step 8:

[0069] The server stores the generated advice and reminders in a database and manages them as long-term health data for the user, allowing for future data analysis and trend analysis.

[0070] Step 9:

[0071] The device receives advice and reminders from the server, and this information is obtained using the device's communication functions.

[0072] Step 10:

[0073] The device will notify the user of advice and reminders. Notifications are made using screen displays and voice synthesis technology, and provide specific instructions and reminders to the user. For example, a notification such as "We recommend walking for 30 minutes today" will be displayed.

[0074] Step 11:

[0075] Users receive advice and reminders to adjust their daily habits, for example by making time for walking and eating healthier.

[0076] Through the above steps, the system of the present invention supports patients with lifestyle-related diseases in efficiently managing their health.

[0077] Example 1

[0078] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0079] When patients with lifestyle-related diseases manage their daily health, they need a system that efficiently and effectively supports the management of diet, exercise, physical and mental condition, and medication. However, conventional systems have difficulty efficiently linking and using audio and video data, and lack the functionality to provide personalized advice and reminders. There is a need to resolve these issues and provide an environment in which users can sustainably manage their health.

[0080] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0081] In this invention, the server includes means for collecting audio data and video data, means for analyzing the collected audio data and performing text conversion and semantic analysis, means for analyzing user motion information from the collected video data, means for integrating the analyzed audio data and video data to grasp the user's health status, means for generating customized advice and reminders using a generative artificial intelligence model based on the grasped health status, and means for notifying the user of the generated advice and reminders, thereby enabling the user to receive efficient and personalized support based on their own health data.

[0082] "Audio data" refers to audio signals recorded and stored as digital data, and is generally collected using an input device such as a microphone.

[0083] "Video data" refers to visual information recorded using an input device such as a video camera or a smartphone camera and stored in digital format.

[0084] "Text conversion" refers to the process of analyzing audio data and converting its content into corresponding written information.

[0085] "Semantic analysis" refers to analysis performed using natural language processing techniques to extract important information and intent from text data.

[0086] "Movement information" is data relating to the physical movements of the user extracted from the video data, and includes walking, eating, exercise, and the like.

[0087] "Health status" refers to a set of data and information about a user's health condition, including diet, exercise, physical condition, etc.

[0088] A "generative artificial intelligence model" is an algorithm or system that generates responses in natural language based on given prompts, primarily using machine learning and deep learning techniques.

[0089] "Customized advice" means instructions or suggestions that are specially created based on a user's individual health data and patterns.

[0090] A "reminder" is a notification or message that prompts a user to take a specific action, such as taking medication or recommended exercise times.

[0091] MODE FOR CARRYING OUT THE INVENTION

[0092] The present invention is a system for supporting patients with lifestyle-related diseases to efficiently manage their daily health, and specific embodiments thereof will be described below.

[0093] Data processing by the server

[0094] The server receives audio and video data sent from the user's device. First, the received audio data is converted into text using speech recognition technology. This process uses technologies such as Google® Cloud Speech-to-Text and IBM Watson® Speech to Text. Next, this text is analyzed using natural language processing technology. Specifically, natural language processing technology using the Hugging Face Transformers library is used to extract meanings such as the user's health status and dietary details.

[0095] Meanwhile, video data is analyzed using video recognition technology, such as OpenCV and TENSORFLOW (registered trademark). User movement information is extracted from the video data, and the amount and pattern of movement are identified from a video of, for example, walking.

[0096] The analyzed audio and video data are integrated to understand the user's health status. For example, if a user reports "chicken salad" as their diet and sends a video of themselves walking, the server analyzes this information to evaluate whether the user ate a proper diet or how much exercise they performed. Based on this evaluation, a generative AI model (e.g., OpenAI® GPT-3®) generates customized advice and reminders.

[0097] The generated advice and reminders are stored in a database, typically using MongoDB or PostgreSQL. This data is managed as the user's long-term health data, enabling more accurate personalized support based on the user's past data.

[0098] Device-based data collection and notification

[0099] A user uses a device to collect daily health information. The device is equipped with a microphone and a camera, and records audio and video data. For example, if a user reports in voice, "I feel good this morning. I had chicken salad for dinner," and records a video of themselves walking, the device sends this data to a server.

[0100] Furthermore, the device receives advice and reminders generated by the server and notifies the user. Notifications are displayed on the device screen or using voice synthesis technology such as Google Text-to-Speech. For example, advice such as "We recommend walking 30 minutes today" or a reminder such as "It's time to take your medication" are provided.

[0101] User Interaction

[0102] Users can interact with the system simply by reporting their health status through voice input. They can also use the camera function on their smartphone or tablet to record their meals and exercise. This eliminates the need to record information and the risk of forgetting, enabling continuous self-management.

[0103] As a concrete example, a user may report by voice, "I had oatmeal for breakfast today. I feel a little tired," and then film and send a video of their daily walking. The server analyzes this information and generates advice based on their health condition and dietary details, such as, "Oatmeal is a good food choice. I recommend some light exercise today." The device then notifies the user of this advice, and the user adjusts their lifestyle accordingly.

[0104] Example prompts for generative AI models

[0105] Below are some example prompts to input to a generative AI model:

[0106] "The user's diet is oatmeal. Their current health condition is a little tired. They have sent a video of their exercise today. Please provide appropriate advice based on this."

[0107] In this way, the present invention provides a system for supporting the user's daily lifestyle habits and for managing health efficiently and effectively.

[0108] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0109] Step 1:

[0110] The user reports in voice, "I feel fine this morning. I had chicken salad for dinner," and films the walking video.

[0111] Specific operation: The user opens the voice recorder app on their smartphone and speaks into the microphone about what they ate and how they feel, then opens the camera app and records a video of their walking.

[0112] Input: User voice input, video recording

[0113] Output: Audio data, video data

[0114] Step 2:

[0115] The terminal records the user's voice via a microphone and records video data with a camera.

[0116] Specific operation: The device's recording function is activated and the recorded audio data is temporarily saved. At the same time, video data taken with the camera app is also saved.

[0117] Input: User's audio and video data

[0118] Output: Audio and video files temporarily saved on the device

[0119] Step 3:

[0120] The terminal transmits the collected audio and video data to the server.

[0121] Specific operation: The device uploads audio and video files to the specified endpoint on the server via Wi-Fi or mobile data communication using an HTTP / HTTPS request.

[0122] Input: Audio files, video files

[0123] Output: Audio and video files transferred to the server

[0124] Step 4:

[0125] The server converts the received voice data into text using the Google Cloud Speech-to-Text API.

[0126] Specific operation: The processing program on the server sends audio data to the API endpoint and receives the returned text data.

[0127] Input: Audio file

[0128] Output: Text data

[0129] Step 5:

[0130] The server uses Hugging Face's Transformers library to perform natural language processing on the text data, extracting meaning such as the user's health status and dietary details.

[0131] Specific operation: The server inputs the text data into a natural language processing model and extracts health status labels and dietary keywords as analysis results.

[0132] Input: Text data

[0133] Output: Health status label, dietary keywords

[0134] Step 6:

[0135] The server analyzes the video data using OpenCV and TensorFlow and extracts information about the user's movements.

[0136] Specific operation: Video data stored on the server is processed frame by frame, and walking movements are detected using a motion recognition algorithm.

[0137] Input: Video data

[0138] Output: Operation information

[0139] Step 7:

[0140] The server combines the analyzed audio and video data to determine the user's overall health status.

[0141] Specific operation: Combine the food information from the audio data and the exercise information from the video data into one dataset.

[0142] Input: Health status label, dietary keywords, behavior information

[0143] Output: Health status data

[0144] Step 8:

[0145] Based on the analysis results, the server inputs prompts into a generative AI model to generate custom advice and reminders.

[0146] Specific operation: Send a prompt to the generative AI model: "The user's meal is chicken salad. We have analyzed the walking video. Please provide appropriate advice." The generated result is returned and obtained as advice.

[0147] Input: Health status data, prompt text

[0148] Output: Custom advice and reminders

[0149] Step 9:

[0150] The generated advice and reminders are stored in a database.

[0151] Specific operation: The database connection program on the server inserts the advice data into the database.

[0152] Input: Custom advice, reminders

[0153] Output: Advice data stored in database, reminders

[0154] Step 10:

[0155] The server sends the generated advice and reminders to the device.

[0156] Specific operation: The server makes an HTTP / HTTPS request to send advice data to the device.

[0157] Input: Advice data stored in the database, reminders

[0158] Output: Advice data and reminders transferred to the device

[0159] Step 11:

[0160] The device notifies the user of advice and reminders received from the server.

[0161] Specific operation: Using the device's notification function, a message such as "We recommend walking for 30 minutes today" is displayed on the screen, and a voice notification is also given using voice synthesis technology such as Google Text-to-Speech.

[0162] Input: Advice data and reminders transferred to the device

[0163] Output: Advice and reminders sent to the user

[0164] The above are the specific processing steps of this system.

[0165] (Application example 1)

[0166] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0167] Currently, there are support systems that help patients with lifestyle-related diseases efficiently manage their daily diet, exercise, mental and physical condition, and medication. However, systems that collect and analyze users' health information in real time and provide appropriate advice and reminders while in a vehicle are not yet widespread. Furthermore, these systems are generally intended for use in fixed locations, making it difficult to receive the necessary support while traveling. Therefore, there is a need to provide a new system that provides appropriate advice and reminders by continuously managing health while in a vehicle and understanding the user's health condition in real time.

[0168] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0169] In this invention, the server includes means for collecting audio data and video data, means for analyzing the collected audio data and performing text conversion and semantic analysis, means for analyzing user motion information from the collected video data, means for ascertaining the user's health status by integrating the analyzed audio data and video data, means for generating advice and reminders based on the ascertained health status, means for notifying the user of the generated advice and reminders, and means for monitoring the user's health status using sensors and displays in the vehicle and providing feedback in real time, thereby enabling the user to effectively monitor their own health status even while in the vehicle and receive appropriate instructions and reminders in real time.

[0170] "Audio data" refers to data that is a digital recording of a user's voice or other acoustic information.

[0171] "Video data" refers to moving image information captured using a camera or other imaging device and recorded in digital format.

[0172] The term "collection means" refers to a device or method used to acquire data, and in the present invention, particularly refers to a device or method for acquiring audio data and video data.

[0173] "Means for analysis" refers to the algorithms or devices used to analyze the collected data, such as converting audio data into text or conducting semantic analysis, or performing motion analysis on video data.

[0174] "Text conversion" is the process of converting audio data into written data, and is accomplished using speech recognition technology.

[0175] "Semantic analysis" is the process of extracting meaning from data converted into text and understanding its content. It is done using natural language processing techniques.

[0176] "Motion information" refers to information about movements and actions extracted from video data.

[0177] "Health status" refers to information about the user's health condition and physical condition, and is understood from analyzed audio and video data.

[0178] "Advice and reminders" are instructions and warnings provided based on the user's health condition.

[0179] "In-vehicle sensors" refers to various types of sensors installed in the vehicle, including cameras and microphones.

[0180] A "display" is a device for visually displaying digital information, and is installed in a vehicle to provide information to a user.

[0181] "Means for providing feedback in real time" refers to technologies and devices that instantly analyze collected data and instantly provide appropriate information and instructions to users.

[0182] The present invention relates to a system for supporting a user's health management by utilizing audio data and video data. Specific embodiments of the system will be described below.

[0183] System Configuration

[0184] The system of the present invention includes means for collecting audio and video data, means for analyzing, means for generating advice and reminders, means for notifying, and means for providing feedback using sensors and displays within the vehicle.

[0185] Hardware and software used

[0186] Hardware: microphones, cameras, displays, speakers, and sensors in the car

[0187] Software: Speech recognition technology (Google Speech-to-Text), image recognition technology (OpenCV), natural language processing technology (GPT-3), generative AI (OpenAI)

[0188] Data Collection and Transmission

[0189] A microphone and a camera installed in the car collect the user's voice and video data. For example, if a user says, "I had yogurt this morning. I'm a little tired," the camera records the user's speech.

[0190] Data analysis and health status generation

[0191] The collected voice data is converted into text using voice recognition technology. The meaning of the text data is then analyzed using natural language processing technology. In parallel, the video data is analyzed using video recognition technology to extract information about the user's movements. These data are then integrated to generate the user's health status.

[0192] Generate advice and reminders

[0193] The server uses generative AI to generate custom advice and reminders based on the analyzed health status, for example, "Yogurt is a good choice. I recommend relaxed driving today."

[0194] Advice and reminder notifications

[0195] The generated advice and reminders are then sent to the user via the in-car display and speaker. For example, the display might say, "Yogurt is a good choice. We recommend a relaxed driving experience today."

[0196] Examples of concrete examples and prompts

[0197] Specific examples

[0198] The user reports in the car, "I had yogurt this morning. I'm a little tired," and the car's camera captures this information. The server analyzes this information and generates advice such as, "Yogurt is a good choice. I recommend you drive relaxed today," which is displayed on the in-car display.

[0199] Prompt Sentence Examples

[0200] User:

[0201] I had yogurt this morning. I'm a little tired.

[0202] Video data:

[0203] [Video of user relaxing in a lounge chair]

[0204] Analysis results:

[0205] The user's breakfast is healthy, but the user is feeling tired, so they should be encouraged to drive in a relaxed manner.

[0206] Generative AI advice:

[0207] Yogurt is a good choice. I recommend a relaxed drive today.

[0208] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0209] Step 1:

[0210] Audio and video data collection

[0211] The device uses the in-car microphone and camera to collect the user's voice and video data. When a user says, "I had yogurt this morning. I'm a little tired," the microphone records the voice, and the camera records the user's face and movements as video data. This data is sent to the server in real time.

[0212] Input: Audio data, video data

[0213] Output: Raw data transferred to the server

[0214] Step 2:

[0215] Speech data text conversion and semantic analysis

[0216] The server converts the received voice data into text using speech recognition technology (Google Speech-to-Text), then uses natural language processing technology (GPT-3) to analyze the meaning of the text data and extract the user's health condition and dietary details.

[0217] Specifically, the voice data is converted into text data such as "I had yogurt this morning. I feel a little tired," and then semantic analysis is used to identify "I had yogurt for breakfast" and "a complaint of fatigue" from the text.

[0218] Input: Audio data

[0219] Output: Text data and analysis results

[0220] Step 3:

[0221] Motion information analysis of video data

[0222] The server analyzes the received video data using video recognition technology (OpenCV) to extract information about the user's movements. For example, it analyzes the user's posture and facial expressions from the video data to determine whether the user is relaxed.

[0223] As for specific actions, the system analyzes the user's movements in the video and extracts motion information such as the user sitting in a relaxing chair.

[0224] Input: Video data

[0225] Output: Operation information

[0226] Step 4:

[0227] Integrated generation of health status

[0228] The server combines the analyzed text data and motion information to generate a health status for the user. For example, by combining the information "yogurt for breakfast," "complaint of fatigue," and "sitting in a lounge chair," the server determines that the user had a healthy breakfast but feels fatigued.

[0229] Input: Text data, operation information

[0230] Output: Health status

[0231] Step 5:

[0232] Generate advice and reminders

[0233] The server uses a generative AI model (OpenAI) to generate custom advice and reminders based on the generated health status, such as "Yogurt is a good choice. I recommend relaxed driving today."

[0234] Input: Health Status

[0235] Output: Advice, reminder

[0236] Step 6:

[0237] Advice and reminder notifications

[0238] The device then notifies the user of the generated advice and reminders through the display and speaker in the car. For example, the display might say, "Yogurt is a good choice. We recommend a relaxed driving experience today."

[0239] Input: Advice, Reminder

[0240] Output: Information notified to the user

[0241] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0242] The present invention is a system that supports patients with lifestyle-related diseases in efficiently managing their daily diet, exercise, mental and physical condition, and medication, and provides customized support by combining it with an emotion engine that recognizes the user's emotions. Specific embodiments of the system are described below.

[0243] Data processing by the server

[0244] The server receives audio and video data sent from the user's device. The received audio data is converted into text using speech recognition technology. This text is then analyzed using natural language processing technology to extract meanings such as the user's health condition and dietary details. Meanwhile, the video data is analyzed using video recognition technology to extract information about the user's movements. An emotion engine then recognizes the user's emotions from the audio and video data.

[0245] The analyzed voice, video, and emotional data are integrated to understand the user's health status. For example, if a user reports "chicken salad" as their diet, sends a video of themselves walking, and displays a "happy" expression, the server analyzes this information and evaluates that the user is eating a healthy diet, exercising, and feeling positive. Based on this evaluation, the generative AI generates customized advice and reminders.

[0246] The generated advice and reminders are stored in a database and managed as a long-term health record for the user, enabling more accurate personalized support based on past data, including the user's emotional state.

[0247] Device-based data collection and notification

[0248] The device has a means for collecting the user's voice and video. When the user reports in a voice message, "I feel fine this morning. I had chicken salad for dinner," and films himself walking and smiles, the device records the voice via a microphone and the video via a camera, and transmits them to a server.

[0249] When the device receives advice or reminders generated by the server, it notifies the user. Notifications are made using screen displays or voice synthesis technology, and specific instructions or reminders are presented to the user. For example, advice such as "I see you're enjoying your exercise. Keep up the same exercise," or a reminder such as "It's time to take your medication" is provided to the user.

[0250] User Interaction

[0251] Users can interact with the system simply by verbally reporting their health status. To easily record daily health information, users can also use the camera function on their smartphone or tablet to record videos of their meals and exercise. This eliminates the need to record information and the risk of forgetting, enabling continuous self-management. Additionally, the system takes the user's emotions into consideration, providing motivation and psychological support.

[0252] As a concrete example, a user may report by voice, "I had oatmeal for breakfast today. I feel a little tired," and then take and send a video of their daily walking. The server analyzes this information and generates advice based on their health condition, dietary content, and emotional data, such as, "Oatmeal is a good choice. I recommend some light exercise today. Considering your fatigue, you should do some stretching." The device then notifies the user of this advice, and the user adjusts their lifestyle accordingly.

[0253] In this way, the system of the present invention, combined with the emotion engine, supports the user's daily lifestyle habits and provides comprehensive and personalized health management.

[0254] The processing flow will be explained below.

[0255] Step 1:

[0256] The user reports their health condition and dietary habits via voice and records their exercise using a smartphone or tablet camera. For example, they can report, "I had oatmeal and fruit for breakfast today. I feel great," and record themselves walking.

[0257] Step 2:

[0258] The device records the user's voice through a microphone and records video through a camera, thereby collecting audio and video data.

[0259] Step 3:

[0260] The terminal transmits the collected audio and video data to the server, and the transmitted data arrives at the server via a communication line.

[0261] Step 4:

[0262] The server analyzes the received voice data. First, it converts the voice data into text using voice recognition technology. Next, it uses natural language processing technology to analyze the meaning of the text and extract the user's health condition and dietary details.

[0263] Step 5:

[0264] The server analyzes the received video data and uses video recognition technology to extract information about the user's movements and analyze the amount and type of exercise.

[0265] Step 6:

[0266] The emotion engine recognizes the user's emotions from the voice data, for example, determining emotions such as "joy" or "fatigue" from the tone of voice and speaking style.

[0267] Step 7:

[0268] The emotion engine recognizes the user's emotions from video data. For example, it uses facial expression recognition technology to analyze smiles and the presence or absence of wrinkles to determine emotions.

[0269] Step 8:

[0270] The server integrates the analyzed audio, video, and emotional data to understand the user's health status, and comprehensively evaluates the user's reported diet, physical condition, exercise, and emotions.

[0271] Step 9:

[0272] The AI ​​generates custom advice and reminders based on the user's health status, such as "Oatmeal is nutritious. Continue walking for 30 minutes today. Also, you seem tired, so take some time to relax."

[0273] Step 10:

[0274] The server stores the generated advice and reminders in a database and manages them as long-term health data for the user, allowing for future data analysis and trend analysis.

[0275] Step 11:

[0276] The device receives advice and reminders from the server, and this information is obtained using the device's communication functions.

[0277] Step 12:

[0278] The device will notify the user with advice and reminders. Notifications are made using screen displays and voice synthesis technology, and provide the user with specific instructions and reminders. For example, a notification might say, "I noticed you were enjoying your exercise. Please continue exercising. Also, you seem tired, so please get plenty of rest."

[0279] Step 13:

[0280] Users receive advice and reminders to adjust their daily habits, for example, by making time for walking, relaxing, or eating healthier.

[0281] Through these steps, the system of the present invention provides comprehensive and personalized health management that also takes into account the user's emotions.

[0282] Example 2

[0283] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0284] Patients with lifestyle-related diseases face the challenge of efficiently managing their daily diet, exercise, physical and mental health, and medication. Furthermore, there is a need to provide support tailored to each individual patient, taking into account their emotions and psychological state. However, conventional systems have not yet fully realized the integration and analysis of this complex data to provide highly accurate, personalized support.

[0285] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting audio data and video data, means for analyzing the collected audio data to perform text conversion and semantic analysis, and means for analyzing user movement information from the collected video data. This makes it possible to analyze the user's emotions from the collected audio data and video data. Furthermore, means for integrating the analyzed audio data, video data, and emotion data to grasp the user's health status is provided, and advice or notifications can be generated based on the grasped health status and notified to the user. This enables more accurate personalized support based on complex data including the user's emotions.

[0286] 1. "Voice data" means data that is a digital recording of voice information uttered by a user.

[0287] 2. "Video data" refers to data that captures a user's movements and facial expressions using a camera or other device and records them in digital format.

[0288] 3. "User movement information" refers to information about a user's movements and actions obtained by analyzing video data.

[0289] 4. "Emotional Data" means information about a user's emotional state obtained by analyzing audio and video data.

[0290] 5. "Health Status" refers to information about the user's physical and mental health status obtained by integrating audio data, video data, and emotional data.

[0291] 6. "Advice" means a health care suggestion or recommendation generated based on the analyzed health status.

[0292] 7. "Notification" means a server-generated display or audio output that conveys advice or reminders to the user.

[0293] 8. "Collection means" means the devices or methods used to capture audio and video data.

[0294] 9. "Analytical means" are techniques and methods used to process collected data and extract meaning from it.

[0295] 10. "Integration means" refers to techniques or methods for combining individually analyzed data to understand a single health status.

[0296] 11. "Means of understanding" refers to the technologies and methods used to understand a user's current health status from the integrated data.

[0297] 12. "Generating means" refers to techniques or methods for generating appropriate advice or reminders based on the analysis results.

[0298] The present invention is a system that supports patients with lifestyle-related diseases in efficiently managing their daily diet, exercise, mental and physical condition, and medication, and provides customized support by combining it with an emotion engine that recognizes the user's emotions. Specific embodiments of the system are described below.

[0299] Data processing by the server

[0300] The server receives audio and video data sent from the user's device. The received audio data is converted into text using Google Cloud Speech-to-Text, a speech recognition technology. This text is then analyzed using natural language processing technologies such as spaCy and Google Cloud Natural Language to extract meanings such as the user's health status and dietary details. Meanwhile, the video data is analyzed using OpenCV and Google Cloud Vision to extract information about the user's movements. Furthermore, the server uses emotion engines such as Affectiva and Microsoft® Azure® Emotion API to recognize the user's emotions from the audio and video data.

[0301] The analyzed audio, video, and emotional data are integrated and the server understands the user's health status. For example, if a user reports "chicken salad" as their diet, sends a video of themselves walking, and displays a "happy" expression, the server analyzes this information and evaluates that the user is eating a healthy diet, exercising, and feeling positive. Based on this evaluation, a generative AI (e.g., OpenAI's GPT-3) is used to generate customized advice and reminders.

[0302] The generated advice and reminders are stored in a database using MySQL (registered trademark) or PostgreSQL and managed as the user's long-term health data, allowing the server to provide more accurate personalized support based on past data, including the user's emotional data.

[0303] Device-based data collection and notification

[0304] The device has a means for collecting the user's voice and video. When the user uses the device to report, "I feel fine this morning. I had chicken salad for dinner," and takes a video of their day's walking and smiles, the device records the user's voice via a microphone and the video via a camera, and sends them to the server.

[0305] When the device receives advice or reminders generated by the server, it notifies the user. Notifications are displayed on the screen or using Google Text-to-Speech as speech synthesis technology, and specific instructions or reminders are presented to the user. For example, advice such as "I see you're enjoying your exercise. Keep doing it," or a reminder such as "It's time to take your medication" is provided to the user.

[0306] User Interaction

[0307] Users can interact with the system simply by verbally reporting their health status. To easily record daily health information, users can also use the camera function on their smartphone or tablet to record videos of their meals and exercise. This eliminates the need to record information and the risk of forgetting, enabling continuous self-management. Additionally, the system takes the user's emotions into consideration, providing motivation and psychological support.

[0308] Examples of concrete examples and prompts

[0309] As a concrete example, a user may report by voice, "I had oatmeal for breakfast today. I feel a little tired," and then take and send a video of their daily walking. The server analyzes this information and generates advice based on their health condition, dietary content, and emotional data, such as, "Oatmeal is a good choice. I recommend some light exercise today. Considering your fatigue, you should do some stretching." The device then notifies the user of this advice, and the user adjusts their lifestyle accordingly.

[0310] An example of a prompt for a generative AI model is:

[0311] "A user sends a video of themselves walking, saying, 'I had oatmeal for breakfast today. I'm feeling a little tired.' Please provide appropriate advice to the user based on their health condition, dietary habits, and emotional data."

[0312] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0313] Step 1: Collect data

[0314] The device collects audio and video data. The user uses a smartphone to report in audio, "I had oatmeal for breakfast today. I'm feeling a little tired," and records their walking.

[0315] Input: Voice spoken by the user into the device and captured video

[0316] Output: Collected audio and video data

[0317] Specifically, the device's built-in microphone records audio and the camera records video.

[0318] Step 2: Sending data

[0319] The device sends the collected audio and video data to a server over an internet connection using the HTTPS protocol.

[0320] Input: Audio and video data stored on the device

[0321] Output: Audio and video data sent to the server

[0322] Specifically, the device encrypts the collected data using HTTPS and sends it to the server.

[0323] Step 3: Convert audio data to text

[0324] The server receives the audio data and converts it into text using the Google Cloud Speech-to-Text API.

[0325] Input: Audio data sent to the server

[0326] Output: Text data (e.g., "I had oatmeal for breakfast today. I feel a little tired.")

[0327] Specifically, the audio file is sent to the API and the returned text data is retrieved.

[0328] Step 4: Semantic analysis of text data

[0329] The server analyzes the text data using natural language processing technology (e.g., spaCy or Google Cloud Natural Language API) to extract meanings such as the user's health status and dietary details.

[0330] Input: Converted text data

[0331] Output: Extracted information (e.g., "Breakfast: Oatmeal", "Health condition: Tired")

[0332] Specifically, the text data is passed to an analysis engine, and the target keywords and phrases are extracted.

[0333] Step 5: Analyze the video data

[0334] The server receives the video data, analyzes it using OpenCV and Google Cloud Vision, and extracts user behavior information.

[0335] Input: Video data sent to the server

[0336] Output: Extracted motion information (e.g., "Exercise: Walking")

[0337] To determine specific movements, the video data is analyzed frame by frame to detect movement characteristics.

[0338] Step 6: Sentiment Analysis

[0339] The server uses an emotion engine (such as Affectiva or Microsoft Azure's Emotion API) to analyze the user's emotions from the received audio and video data.

[0340] Input: Audio and video data

[0341] Output: Extracted emotion data (e.g., "Emotion: Tired")

[0342] Specifically, voice tone and facial expression data are passed to an analysis engine to classify emotions.

[0343] Step 7: Integrate the data

[0344] The server integrates the analyzed audio, video, and emotional data to understand the user's health status.

[0345] Input: Analyzed audio data, video data, emotion data

[0346] Output: Integrated health status (e.g., "Breakfast: Oatmeal", "Exercise: Walking", "Emotion: Tired")

[0347] Specifically, it integrates multiple data sources and compiles them into a single health status.

[0348] Step 8: Generate advice and reminders

[0349] The server uses generative AI (e.g., OpenAI's GPT-3) to generate custom advice and reminders based on the integrated data.

[0350] Input: Integrated Health Status

[0351] Output: Generated advice or reminders (e.g., "Oatmeal is a good choice. I recommend some light exercise today. Consider fatigue and do some stretching.")

[0352] Specifically, the prompt sentence is passed to the generation AI, which then generates advice.

[0353] Step 9: Advice and reminder notifications

[0354] The device receives advice and reminders generated by the server and notifies the user using a screen display or voice synthesis technology (e.g., Google Text-to-Speech).

[0355] Input: Advice and reminders sent from the server

[0356] Output: Advice and reminders sent to the user

[0357] Specifically, the system displays a message to the user via the smartphone's notification function.

[0358] (Application example 2)

[0359] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0360] In conventional factories, it was difficult to grasp the health status and emotions of workers in real time and provide appropriate advice. As a result, there were issues such as reduced work efficiency and safety and increased health risks for workers. In particular, the inability to properly manage workers' fatigue and stress levels led to problems such as long working hours and an increased risk of workplace accidents.

[0361] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0362] In this invention, the server includes means for collecting audio data and video data, means for analyzing the collected audio data and performing text conversion and semantic analysis, means for analyzing user motion information from the collected video data, means for ascertaining the user's health status by integrating the analyzed audio data and video data, means for generating advice and reminders based on the ascertained health status and emotional data, means for notifying the user of the generated advice and reminders, means for storing past data in a database and providing customized support based on the stored data, and means for generating advice for the user using a generative AI model. This makes it possible to ascertain the health status and emotions of workers in real time and provide appropriate advice and reminders.

[0363] "Audio data" refers to acoustic information collected by a sound recording device such as a microphone.

[0364] "Video data" refers to visual information collected by a video recording device such as a camera.

[0365] "Text conversion" is the process of converting voice data into text using voice recognition technology.

[0366] "Semantic analysis" is a natural language processing technique for extracting meaning from text data and understanding its content.

[0367] "Movement information" is information about the movements and postures of people detected from video data.

[0368] "Health status" refers to an individual's health condition as determined based on analyzed audio and video data.

[0369] "Emotion data" is information about an individual's emotional state that is analyzed from audio data and video data.

[0370] "Advice" refers to advice that supports a user's actions or choices based on analyzed data.

[0371] A "reminder" is a notification that reminds the user of a particular action or event.

[0372] A "database" is an information system for systematically storing and managing various data.

[0373] "Custom-made support" refers to personalized assistance based on the individual user's characteristics and past data.

[0374] A "generative AI model" is an artificial intelligence algorithm that is trained on large amounts of data and generates documents and data.

[0375] A "prompt" is an instruction given to a generative AI model to obtain a specific output.

[0376] This invention is a system in which a factory robot monitors the health and emotional state of workers in real time and provides appropriate advice and reminders, thereby improving work efficiency and safety and reducing health risks to workers.

[0377] System Configuration

[0378] This system is mainly composed of a server and terminals. Details of each element are described below.

[0379] Data processing by the server

[0380] The server processes data using the following software and hardware:

[0381] Speech Recognition Technology: We use technologies such as the Google Speech-to-Text API to convert collected voice data into text.

[0382] Video analysis technology: Using technologies such as OpenCV, motion information is extracted from collected video data.

[0383] Emotion recognition technology: Using technologies such as the Microsoft Azure Emotion API, emotional information is analyzed from audio and video data.

[0384] Natural language processing technology: Using technologies such as spaCy, text data can be analyzed to understand the health and emotional state of workers.

[0385] Database management system: Uses MySQL, PostgreSQL, etc. to store and manage collected data for the long term.

[0386] Generative AI models: Use pre-trained generative AI models (e.g., OpenAI's GPT-3) to generate advice and reminders for users.

[0387] Device-based data collection and notification

[0388] The device uses the following hardware and software to collect data and send notifications.

[0389] Microphone: Collects the voice of the worker.

[0390] Camera: Collects footage of workers.

[0391] Display and speaker: Notifies workers of advice and reminders generated by the server.

[0392] Operating procedure

[0393] 1. Data collection: The device collects the worker's voice and video and sends the data to the server.

[0394] 2. Data analysis: The server converts the voice data into text and analyzes the meaning using natural language processing technology. It also extracts motion information from the video data and analyzes the emotional data using emotion recognition technology.

[0395] 3. Integration and understanding: The server integrates the analyzed audio data, video data, and emotion data to understand the worker's health status and emotional state.

[0396] 4. Advice generation: Use generative AI models to generate custom advice and reminders.

[0397] 5. Notification: The device notifies the worker of generated advice and reminders via a display or voice announcement.

[0398] Specific examples

[0399] The worker reports, "I'm a little tired, but my morning work is going well," and the camera captures this. The device collects this information and sends it to the server. The server analyzes the audio data using speech recognition and natural language processing technology, and then checks the level of fatigue from the video data. Based on this information, the generative AI model generates advice such as "Take a 10-minute break to relieve fatigue," and notifies the worker via the device's display and speaker.

[0400] Prompt Sentence Examples

[0401] Today's worker reports, "I'm a little tired, but the morning's work went well." Video analysis shows that the worker appears fatigued. Please take this situation into consideration and provide appropriate rest advice.

[0402] The above is an embodiment of the present invention, which makes it possible to grasp the health condition and emotions of workers in real time and provide appropriate support.

[0403] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0404] Step 1:

[0405] The terminal collects audio and video of the workers.

[0406] Input: Audio and video data of the worker.

[0407] How it works: The microphone collects audio data and the camera collects video data. These data are collected in real time and sent to a server.

[0408] Output: The collected audio and video data is sent to a server.

[0409] Step 2:

[0410] The server converts the audio data into text.

[0411] Input: Collected audio data.

[0412] What it does: Uses speech recognition technology (e.g., Google Speech-to-Text API) to convert voice data into text.

[0413] Output: The converted text data.

[0414] Step 3:

[0415] The server analyzes the text data and extracts meaning.

[0416] Input: The converted text data.

[0417] How it works: Natural language processing techniques (e.g., spaCy) are used to analyze text data and extract information about the worker's health and emotional state.

[0418] Output: Parsed health and emotional state data.

[0419] Step 4:

[0420] The server analyzes the video data and extracts motion information and emotion data.

[0421] Input: Collected video data.

[0422] Movement: Video analysis technology (e.g., OpenCV) is used to extract movement information from video data, and emotion recognition technology (e.g., Microsoft Azure Emotion API) is used to analyze emotional states.

[0423] Output: Extracted motion information and emotion data.

[0424] Step 5:

[0425] The server integrates health status data, movement information, and emotional data to understand the overall health status.

[0426] Input: Analyzed health status data, motion information, and emotion data.

[0427] How it works: Integrates various data to assess the overall health status of workers.

[0428] Output: Consolidated health status data.

[0429] Step 6:

[0430] The server uses generative AI models to generate custom advice and reminders.

[0431] Input: Consolidated health status data.

[0432] How it works: It uses a generative AI model (e.g., OpenAI's GPT-3) to generate specific advice and reminders for workers based on a prompt.

[0433] Output: The generated advice or reminder.

[0434] Step 7:

[0435] The server sends advice and reminders to the device.

[0436] Input: Generated advice or reminder.

[0437] Operation: The server sends generated advice and reminders to the device.

[0438] Output: Advice or reminder sent.

[0439] Step 8:

[0440] The device will provide workers with advice and reminders.

[0441] Input: Advice or reminder sent.

[0442] How it works: Provides advice and reminders to workers through display and audio announcements.

[0443] Output: Workers receive advice and reminders.

[0444] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0445] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0446] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0447] [Second embodiment]

[0448] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0449] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0450] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0451] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0452] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0453] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0454] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0455] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0456] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0457] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0458] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0459] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0460] The present invention is a system for supporting patients with lifestyle-related diseases to efficiently manage their daily diet, exercise, mental and physical condition, and medication, and specific embodiments thereof will be described below.

[0461] Data processing by the server

[0462] The server receives audio and video data sent from the user's device. The received audio data is converted into text using speech recognition technology. This text is then analyzed using natural language processing technology to extract meanings such as the user's health status and dietary details. Meanwhile, the video data is analyzed using video recognition technology to extract information about the user's movements.

[0463] The analyzed audio and video data are integrated to understand the user's health status. For example, if a user reports "chicken salad" as their diet and sends a video of themselves walking, the server analyzes this information and evaluates whether the user ate an appropriate diet or how much exercise they performed. Based on this evaluation, the generative AI generates customized advice and reminders.

[0464] The generated advice and reminders are stored in a database and managed as a long-term health record for the user, enabling more accurate personalized support based on the user's past data.

[0465] Device-based data collection and notification

[0466] The device has a means for collecting the user's voice and video. When the user reports verbally, "I feel fine this morning. I had chicken salad for dinner," the device records the voice via a microphone and sends it to the server. Similarly, when the user takes a video of themselves walking, the device records the video via a camera and sends it to the server.

[0467] When the device receives advice or reminders generated by the server, it notifies the user. Notifications are made using screen displays or voice synthesis technology, and specific instructions or reminders are presented to the user. For example, advice such as "We recommend you walk 30 minutes today" or a reminder such as "It's time to take your medication" is provided to the user.

[0468] User Interaction

[0469] Users can interact with the system simply by verbally reporting their health status. They can also record their daily health information easily by using the camera function on their smartphone or tablet to record their meals and exercise. This eliminates the need to record information and helps encourage continuous self-management.

[0470] As a concrete example, a user may report by voice, "I had oatmeal for breakfast today. I feel a little tired," and then film and send a video of their walking for the day. The server analyzes this information and generates advice based on their health condition and diet, such as, "Oatmeal is a good food choice. I recommend some light exercise today." The device notifies the user of this advice, and the user adjusts their lifestyle accordingly.

[0471] In this way, the present invention provides a system for supporting the user's daily lifestyle habits and for managing health efficiently and effectively.

[0472] The processing flow will be explained below.

[0473] Step 1:

[0474] The user reports their health condition and dietary habits via voice and records their exercise using a smartphone or tablet camera. For example, they can report, "I had oatmeal and fruit for breakfast today. I feel great," and record themselves walking.

[0475] Step 2:

[0476] The device records the user's voice through a microphone and records video through a camera, thereby collecting audio and video data.

[0477] Step 3:

[0478] The terminal transmits the collected audio and video data to the server, and the transmitted data arrives at the server via a communication line.

[0479] Step 4:

[0480] The server analyzes the received voice data. First, it converts the voice data into text using voice recognition technology. Next, it uses natural language processing technology to analyze the meaning of the text and extract the user's health condition and dietary details.

[0481] Step 5:

[0482] The server analyzes the received video data and uses video recognition technology to extract information about the user's movements and analyze the amount and type of exercise.

[0483] Step 6:

[0484] The server combines the analyzed audio and video data to understand the user's health status, and provides a comprehensive assessment of the user's reported diet, physical condition, and exercise volume.

[0485] Step 7:

[0486] The AI ​​generates custom advice and reminders based on the user's health status, such as "Oatmeal is nutritious. Continue walking for 30 minutes today."

[0487] Step 8:

[0488] The server stores the generated advice and reminders in a database and manages them as long-term health data for the user, allowing for future data analysis and trend analysis.

[0489] Step 9:

[0490] The device receives advice and reminders from the server, and this information is obtained using the device's communication functions.

[0491] Step 10:

[0492] The device will notify the user of advice and reminders. Notifications are made using screen displays and voice synthesis technology, and provide specific instructions and reminders to the user. For example, a notification such as "We recommend walking for 30 minutes today" will be displayed.

[0493] Step 11:

[0494] Users receive advice and reminders to adjust their daily habits, for example by making time for walking and eating healthier.

[0495] Through the above steps, the system of the present invention supports patients with lifestyle-related diseases in efficiently managing their health.

[0496] Example 1

[0497] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0498] When patients with lifestyle-related diseases manage their daily health, they need a system that efficiently and effectively supports the management of diet, exercise, physical and mental condition, and medication. However, conventional systems have difficulty efficiently linking and using audio and video data, and lack the functionality to provide personalized advice and reminders. There is a need to resolve these issues and provide an environment in which users can sustainably manage their health.

[0499] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0500] In this invention, the server includes means for collecting audio data and video data, means for analyzing the collected audio data and performing text conversion and semantic analysis, means for analyzing user motion information from the collected video data, means for integrating the analyzed audio data and video data to grasp the user's health status, means for generating customized advice and reminders using a generative artificial intelligence model based on the grasped health status, and means for notifying the user of the generated advice and reminders, thereby enabling the user to receive efficient and personalized support based on their own health data.

[0501] "Audio data" refers to audio signals recorded and stored as digital data, and is generally collected using an input device such as a microphone.

[0502] "Video data" refers to visual information recorded using an input device such as a video camera or a smartphone camera and stored in digital format.

[0503] "Text conversion" refers to the process of analyzing audio data and converting its content into corresponding written information.

[0504] "Semantic analysis" refers to analysis performed using natural language processing techniques to extract important information and intent from text data.

[0505] "Movement information" is data relating to the physical movements of the user extracted from the video data, and includes walking, eating, exercise, and the like.

[0506] "Health status" refers to a set of data and information about a user's health condition, including diet, exercise, physical condition, etc.

[0507] A "generative artificial intelligence model" is an algorithm or system that generates responses in natural language based on given prompts, primarily using machine learning and deep learning techniques.

[0508] "Customized advice" means instructions or suggestions that are specially created based on a user's individual health data and patterns.

[0509] A "reminder" is a notification or message that prompts a user to take a specific action, such as taking medication or recommended exercise times.

[0510] MODE FOR CARRYING OUT THE INVENTION

[0511] The present invention is a system for supporting patients with lifestyle-related diseases to efficiently manage their daily health, and specific embodiments thereof will be described below.

[0512] Data processing by the server

[0513] The server receives audio and video data sent from the user's device. First, the received audio data is converted into text using speech recognition technology. This process uses technologies such as Google Cloud Speech-to-Text and IBM Watson Speech to Text. Next, this text is analyzed using natural language processing technology. Specifically, natural language processing technology using Hugging Face's Transformers library is used to extract meanings such as the user's health status and dietary details.

[0514] Meanwhile, video data is analyzed using video recognition technology, such as OpenCV and TensorFlow. User movement information is extracted from the video data, and the amount and pattern of movement can be identified from a video of someone walking, for example.

[0515] The analyzed audio and video data are integrated to understand the user's health status. For example, if a user reports "chicken salad" as their diet and sends a video of themselves walking, the server analyzes this information to evaluate whether the user ate a proper diet or how much exercise they performed. Based on this evaluation, a generative AI model (e.g., OpenAI GPT-3) generates customized advice and reminders.

[0516] The generated advice and reminders are stored in a database, typically using MongoDB or PostgreSQL. This data is managed as the user's long-term health data, enabling more accurate personalized support based on the user's past data.

[0517] Device-based data collection and notification

[0518] A user uses a device to collect daily health information. The device is equipped with a microphone and a camera, and records audio and video data. For example, if a user reports in voice, "I feel good this morning. I had chicken salad for dinner," and records a video of themselves walking, the device sends this data to a server.

[0519] Furthermore, the device receives advice and reminders generated by the server and notifies the user. Notifications are displayed on the device screen or using voice synthesis technology such as Google Text-to-Speech. For example, advice such as "We recommend walking 30 minutes today" or a reminder such as "It's time to take your medication" are provided.

[0520] User Interaction

[0521] Users can interact with the system simply by reporting their health status through voice input. They can also use the camera function on their smartphone or tablet to record their meals and exercise. This eliminates the need to record information and the risk of forgetting, enabling continuous self-management.

[0522] As a concrete example, a user may report by voice, "I had oatmeal for breakfast today. I feel a little tired," and then film and send a video of their daily walking. The server analyzes this information and generates advice based on their health condition and dietary details, such as, "Oatmeal is a good food choice. I recommend some light exercise today." The device then notifies the user of this advice, and the user adjusts their lifestyle accordingly.

[0523] Example prompts for generative AI models

[0524] Below are some example prompts to input to a generative AI model:

[0525] "The user's diet is oatmeal. Their current health condition is a little tired. They have sent a video of their exercise today. Please provide appropriate advice based on this."

[0526] In this way, the present invention provides a system for supporting the user's daily lifestyle habits and for managing health efficiently and effectively.

[0527] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0528] Step 1:

[0529] The user reports in voice, "I feel fine this morning. I had chicken salad for dinner," and films the walking video.

[0530] Specific operation: The user opens the voice recorder app on their smartphone and speaks into the microphone about what they ate and how they feel, then opens the camera app and records a video of their walking.

[0531] Input: User voice input, video recording

[0532] Output: Audio data, video data

[0533] Step 2:

[0534] The terminal records the user's voice via a microphone and records video data with a camera.

[0535] Specific operation: The device's recording function is activated and the recorded audio data is temporarily saved. At the same time, video data taken with the camera app is also saved.

[0536] Input: User's audio and video data

[0537] Output: Audio and video files temporarily saved on the device

[0538] Step 3:

[0539] The terminal transmits the collected audio and video data to the server.

[0540] Specific operation: The device uploads audio and video files to the specified endpoint on the server via Wi-Fi or mobile data communication using an HTTP / HTTPS request.

[0541] Input: Audio files, video files

[0542] Output: Audio and video files transferred to the server

[0543] Step 4:

[0544] The server converts the received voice data into text using the Google Cloud Speech-to-Text API.

[0545] Specific operation: The processing program on the server sends audio data to the API endpoint and receives the returned text data.

[0546] Input: Audio file

[0547] Output: Text data

[0548] Step 5:

[0549] The server uses Hugging Face's Transformers library to perform natural language processing on the text data, extracting meaning such as the user's health status and dietary details.

[0550] Specific operation: The server inputs the text data into a natural language processing model and extracts health status labels and dietary keywords as analysis results.

[0551] Input: Text data

[0552] Output: Health status label, dietary keywords

[0553] Step 6:

[0554] The server analyzes the video data using OpenCV and TensorFlow and extracts information about the user's movements.

[0555] Specific operation: Video data stored on the server is processed frame by frame, and walking movements are detected using a motion recognition algorithm.

[0556] Input: Video data

[0557] Output: Operation information

[0558] Step 7:

[0559] The server combines the analyzed audio and video data to determine the user's overall health status.

[0560] Specific operation: Combine the food information from the audio data and the exercise information from the video data into one dataset.

[0561] Input: Health status label, dietary keywords, behavior information

[0562] Output: Health status data

[0563] Step 8:

[0564] Based on the analysis results, the server inputs prompts into a generative AI model to generate custom advice and reminders.

[0565] Specific operation: Send a prompt to the generative AI model: "The user's meal is chicken salad. We have analyzed the walking video. Please provide appropriate advice." The generated result is returned and obtained as advice.

[0566] Input: Health status data, prompt text

[0567] Output: Custom advice and reminders

[0568] Step 9:

[0569] The generated advice and reminders are stored in a database.

[0570] Specific operation: The database connection program on the server inserts the advice data into the database.

[0571] Input: Custom advice, reminders

[0572] Output: Advice data stored in database, reminders

[0573] Step 10:

[0574] The server sends the generated advice and reminders to the device.

[0575] Specific operation: The server makes an HTTP / HTTPS request to send advice data to the device.

[0576] Input: Advice data stored in the database, reminders

[0577] Output: Advice data and reminders transferred to the device

[0578] Step 11:

[0579] The device notifies the user of advice and reminders received from the server.

[0580] Specific operation: Using the device's notification function, a message such as "We recommend walking for 30 minutes today" is displayed on the screen, and a voice notification is also given using voice synthesis technology such as Google Text-to-Speech.

[0581] Input: Advice data and reminders transferred to the device

[0582] Output: Advice and reminders sent to the user

[0583] The above are the specific processing steps of this system.

[0584] (Application example 1)

[0585] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0586] Currently, there are support systems that help patients with lifestyle-related diseases efficiently manage their daily diet, exercise, mental and physical condition, and medication. However, systems that collect and analyze users' health information in real time and provide appropriate advice and reminders while in a vehicle are not yet widespread. Furthermore, these systems are generally intended for use in fixed locations, making it difficult to receive the necessary support while traveling. Therefore, there is a need to provide a new system that provides appropriate advice and reminders by continuously managing health while in a vehicle and understanding the user's health condition in real time.

[0587] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0588] In this invention, the server includes means for collecting audio data and video data, means for analyzing the collected audio data and performing text conversion and semantic analysis, means for analyzing user motion information from the collected video data, means for ascertaining the user's health status by integrating the analyzed audio data and video data, means for generating advice and reminders based on the ascertained health status, means for notifying the user of the generated advice and reminders, and means for monitoring the user's health status using sensors and displays in the vehicle and providing feedback in real time, thereby enabling the user to effectively monitor their own health status even while in the vehicle and receive appropriate instructions and reminders in real time.

[0589] "Audio data" refers to data that is a digital recording of a user's voice or other acoustic information.

[0590] "Video data" refers to moving image information captured using a camera or other imaging device and recorded in digital format.

[0591] The term "collection means" refers to a device or method used to acquire data, and in the present invention, particularly refers to a device or method for acquiring audio data and video data.

[0592] "Means for analysis" refers to the algorithms or devices used to analyze the collected data, such as converting audio data into text or conducting semantic analysis, or performing motion analysis on video data.

[0593] "Text conversion" is the process of converting audio data into written data, and is accomplished using speech recognition technology.

[0594] "Semantic analysis" is the process of extracting meaning from data converted into text and understanding its content. It is done using natural language processing techniques.

[0595] "Motion information" refers to information about movements and actions extracted from video data.

[0596] "Health status" refers to information about the user's health condition and physical condition, and is understood from analyzed audio and video data.

[0597] "Advice and reminders" are instructions and warnings provided based on the user's health condition.

[0598] "In-vehicle sensors" refers to various types of sensors installed in the vehicle, including cameras and microphones.

[0599] A "display" is a device for visually displaying digital information, and is installed in a vehicle to provide information to a user.

[0600] "Means for providing feedback in real time" refers to technologies and devices that instantly analyze collected data and instantly provide appropriate information and instructions to users.

[0601] The present invention relates to a system for supporting a user's health management by utilizing audio data and video data. Specific embodiments of the system will be described below.

[0602] System Configuration

[0603] The system of the present invention includes means for collecting audio and video data, means for analyzing, means for generating advice and reminders, means for notifying, and means for providing feedback using sensors and displays within the vehicle.

[0604] Hardware and software used

[0605] Hardware: microphones, cameras, displays, speakers, and sensors in the car

[0606] Software: Speech recognition technology (Google Speech-to-Text), image recognition technology (OpenCV), natural language processing technology (GPT-3), generative AI (OpenAI)

[0607] Data Collection and Transmission

[0608] A microphone and a camera installed in the car collect the user's voice and video data. For example, if a user says, "I had yogurt this morning. I'm a little tired," the camera records the user's speech.

[0609] Data analysis and health status generation

[0610] The collected voice data is converted into text using voice recognition technology. The meaning of the text data is then analyzed using natural language processing technology. In parallel, the video data is analyzed using video recognition technology to extract information about the user's movements. These data are then integrated to generate the user's health status.

[0611] Generate advice and reminders

[0612] The server uses generative AI to generate custom advice and reminders based on the analyzed health status, for example, "Yogurt is a good choice. I recommend relaxed driving today."

[0613] Advice and reminder notifications

[0614] The generated advice and reminders are then sent to the user via the in-car display and speaker. For example, the display might say, "Yogurt is a good choice. We recommend a relaxed driving experience today."

[0615] Examples of concrete examples and prompts

[0616] Specific examples

[0617] The user reports in the car, "I had yogurt this morning. I'm a little tired," and the car's camera captures this information. The server analyzes this information and generates advice such as, "Yogurt is a good choice. I recommend you drive relaxed today," which is displayed on the in-car display.

[0618] Prompt Sentence Examples

[0619] User:

[0620] I had yogurt this morning. I'm a little tired.

[0621] Video data:

[0622] [Video of user relaxing in a lounge chair]

[0623] Analysis results:

[0624] The user's breakfast is healthy, but the user is feeling tired, so they should be encouraged to drive in a relaxed manner.

[0625] Generative AI advice:

[0626] Yogurt is a good choice. I recommend a relaxed drive today.

[0627] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0628] Step 1:

[0629] Audio and video data collection

[0630] The device uses the in-car microphone and camera to collect the user's voice and video data. When a user says, "I had yogurt this morning. I'm a little tired," the microphone records the voice, and the camera records the user's face and movements as video data. This data is sent to the server in real time.

[0631] Input: Audio data, video data

[0632] Output: Raw data transferred to the server

[0633] Step 2:

[0634] Speech data text conversion and semantic analysis

[0635] The server converts the received voice data into text using speech recognition technology (Google Speech-to-Text), then uses natural language processing technology (GPT-3) to analyze the meaning of the text data and extract the user's health condition and dietary details.

[0636] Specifically, the voice data is converted into text data such as "I had yogurt this morning. I feel a little tired," and then semantic analysis is used to identify "I had yogurt for breakfast" and "a complaint of fatigue" from the text.

[0637] Input: Audio data

[0638] Output: Text data and analysis results

[0639] Step 3:

[0640] Motion information analysis of video data

[0641] The server analyzes the received video data using video recognition technology (OpenCV) to extract information about the user's movements. For example, it analyzes the user's posture and facial expressions from the video data to determine whether the user is relaxed.

[0642] As for specific actions, the system analyzes the user's movements in the video and extracts motion information such as the user sitting in a relaxing chair.

[0643] Input: Video data

[0644] Output: Operation information

[0645] Step 4:

[0646] Integrated generation of health status

[0647] The server combines the analyzed text data and motion information to generate a health status for the user. For example, by combining the information "yogurt for breakfast," "complaint of fatigue," and "sitting in a lounge chair," the server determines that the user had a healthy breakfast but feels fatigued.

[0648] Input: Text data, operation information

[0649] Output: Health status

[0650] Step 5:

[0651] Generate advice and reminders

[0652] The server uses a generative AI model (OpenAI) to generate custom advice and reminders based on the generated health status, such as "Yogurt is a good choice. I recommend relaxed driving today."

[0653] Input: Health Status

[0654] Output: Advice, reminder

[0655] Step 6:

[0656] Advice and reminder notifications

[0657] The device then notifies the user of the generated advice and reminders through the display and speaker in the car. For example, the display might say, "Yogurt is a good choice. We recommend a relaxed driving experience today."

[0658] Input: Advice, Reminder

[0659] Output: Information notified to the user

[0660] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0661] The present invention is a system that supports patients with lifestyle-related diseases in efficiently managing their daily diet, exercise, mental and physical condition, and medication, and provides customized support by combining it with an emotion engine that recognizes the user's emotions. Specific embodiments of the system are described below.

[0662] Data processing by the server

[0663] The server receives audio and video data sent from the user's device. The received audio data is converted into text using speech recognition technology. This text is then analyzed using natural language processing technology to extract meanings such as the user's health condition and dietary details. Meanwhile, the video data is analyzed using video recognition technology to extract information about the user's movements. An emotion engine then recognizes the user's emotions from the audio and video data.

[0664] The analyzed voice, video, and emotional data are integrated to understand the user's health status. For example, if a user reports "chicken salad" as their diet, sends a video of themselves walking, and displays a "happy" expression, the server analyzes this information and evaluates that the user is eating a healthy diet, exercising, and feeling positive. Based on this evaluation, the generative AI generates customized advice and reminders.

[0665] The generated advice and reminders are stored in a database and managed as a long-term health record for the user, enabling more accurate personalized support based on past data, including the user's emotional state.

[0666] Device-based data collection and notification

[0667] The device has a means for collecting the user's voice and video. When the user reports in a voice message, "I feel fine this morning. I had chicken salad for dinner," and films himself walking and smiles, the device records the voice via a microphone and the video via a camera, and transmits them to a server.

[0668] When the device receives advice or reminders generated by the server, it notifies the user. Notifications are made using screen displays or voice synthesis technology, and specific instructions or reminders are presented to the user. For example, advice such as "I see you're enjoying your exercise. Keep up the same exercise," or a reminder such as "It's time to take your medication" is provided to the user.

[0669] User Interaction

[0670] Users can interact with the system simply by verbally reporting their health status. To easily record daily health information, users can also use the camera function on their smartphone or tablet to record videos of their meals and exercise. This eliminates the need to record information and the risk of forgetting, enabling continuous self-management. Additionally, the system takes the user's emotions into consideration, providing motivation and psychological support.

[0671] As a concrete example, a user may report by voice, "I had oatmeal for breakfast today. I feel a little tired," and then take and send a video of their daily walking. The server analyzes this information and generates advice based on their health condition, dietary content, and emotional data, such as, "Oatmeal is a good choice. I recommend some light exercise today. Considering your fatigue, you should do some stretching." The device then notifies the user of this advice, and the user adjusts their lifestyle accordingly.

[0672] In this way, the system of the present invention, combined with the emotion engine, supports the user's daily lifestyle habits and provides comprehensive and personalized health management.

[0673] The processing flow will be explained below.

[0674] Step 1:

[0675] The user reports their health condition and dietary habits via voice and records their exercise using a smartphone or tablet camera. For example, they can report, "I had oatmeal and fruit for breakfast today. I feel great," and record themselves walking.

[0676] Step 2:

[0677] The device records the user's voice through a microphone and records video through a camera, thereby collecting audio and video data.

[0678] Step 3:

[0679] The terminal transmits the collected audio and video data to the server, and the transmitted data arrives at the server via a communication line.

[0680] Step 4:

[0681] The server analyzes the received voice data. First, it converts the voice data into text using voice recognition technology. Next, it uses natural language processing technology to analyze the meaning of the text and extract the user's health condition and dietary details.

[0682] Step 5:

[0683] The server analyzes the received video data and uses video recognition technology to extract information about the user's movements and analyze the amount and type of exercise.

[0684] Step 6:

[0685] The emotion engine recognizes the user's emotions from the voice data, for example, determining emotions such as "joy" or "fatigue" from the tone of voice and speaking style.

[0686] Step 7:

[0687] The emotion engine recognizes the user's emotions from video data. For example, it uses facial expression recognition technology to analyze smiles and the presence or absence of wrinkles to determine emotions.

[0688] Step 8:

[0689] The server integrates the analyzed audio, video, and emotional data to understand the user's health status, and comprehensively evaluates the user's reported diet, physical condition, exercise, and emotions.

[0690] Step 9:

[0691] The AI ​​generates custom advice and reminders based on the user's health status, such as "Oatmeal is nutritious. Continue walking for 30 minutes today. Also, you seem tired, so take some time to relax."

[0692] Step 10:

[0693] The server stores the generated advice and reminders in a database and manages them as long-term health data for the user, allowing for future data analysis and trend analysis.

[0694] Step 11:

[0695] The device receives advice and reminders from the server, and this information is obtained using the device's communication functions.

[0696] Step 12:

[0697] The device will notify the user with advice and reminders. Notifications are made using screen displays and voice synthesis technology, and provide the user with specific instructions and reminders. For example, a notification might say, "I noticed you were enjoying your exercise. Please continue exercising. Also, you seem tired, so please get plenty of rest."

[0698] Step 13:

[0699] Users receive advice and reminders to adjust their daily habits, for example, by making time for walking, relaxing, or eating healthier.

[0700] Through these steps, the system of the present invention provides comprehensive and personalized health management that also takes into account the user's emotions.

[0701] Example 2

[0702] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0703] Patients with lifestyle-related diseases face the challenge of efficiently managing their daily diet, exercise, physical and mental health, and medication. Furthermore, there is a need to provide support tailored to each individual patient, taking into account their emotions and psychological state. However, conventional systems have not yet fully realized the integration and analysis of this complex data to provide highly accurate, personalized support.

[0704] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting audio data and video data, means for analyzing the collected audio data to perform text conversion and semantic analysis, and means for analyzing user movement information from the collected video data. This makes it possible to analyze the user's emotions from the collected audio data and video data. Furthermore, means for integrating the analyzed audio data, video data, and emotion data to grasp the user's health status is provided, and advice or notifications can be generated based on the grasped health status and notified to the user. This enables more accurate personalized support based on complex data including the user's emotions.

[0705] 1. "Voice data" means data that is a digital recording of voice information uttered by a user.

[0706] 2. "Video data" refers to data that captures a user's movements and facial expressions using a camera or other device and records them in digital format.

[0707] 3. "User movement information" refers to information about a user's movements and actions obtained by analyzing video data.

[0708] 4. "Emotional Data" means information about a user's emotional state obtained by analyzing audio and video data.

[0709] 5. "Health Status" refers to information about the user's physical and mental health status obtained by integrating audio data, video data, and emotional data.

[0710] 6. "Advice" means a health care suggestion or recommendation generated based on the analyzed health status.

[0711] 7. "Notification" means a server-generated display or audio output that conveys advice or reminders to the user.

[0712] 8. "Collection means" means the devices or methods used to capture audio and video data.

[0713] 9. "Analytical means" are techniques and methods used to process collected data and extract meaning from it.

[0714] 10. "Integration means" refers to techniques or methods for combining individually analyzed data to understand a single health status.

[0715] 11. "Means of understanding" refers to the technologies and methods used to understand a user's current health status from the integrated data.

[0716] 12. "Generating means" refers to techniques or methods for generating appropriate advice or reminders based on the analysis results.

[0717] The present invention is a system that supports patients with lifestyle-related diseases in efficiently managing their daily diet, exercise, mental and physical condition, and medication, and provides customized support by combining it with an emotion engine that recognizes the user's emotions. Specific embodiments of the system are described below.

[0718] Data processing by the server

[0719] The server receives audio and video data sent from the user's device. The received audio data is converted into text using Google Cloud Speech-to-Text, a speech recognition technology. This text is then analyzed using natural language processing technologies such as spaCy and Google Cloud Natural Language to extract meanings such as the user's health condition and dietary details. Meanwhile, the video data is analyzed using OpenCV and Google Cloud Vision to extract information about the user's movements. Furthermore, the server uses emotion engines such as Affectiva and Microsoft Azure's Emotion API to recognize the user's emotions from the audio and video data.

[0720] The analyzed audio, video, and emotional data are integrated and the server understands the user's health status. For example, if a user reports "chicken salad" as their diet, sends a video of themselves walking, and displays a "happy" expression, the server analyzes this information and evaluates that the user is eating a healthy diet, exercising, and feeling positive. Based on this evaluation, a generative AI (e.g., OpenAI's GPT-3) is used to generate customized advice and reminders.

[0721] The generated advice and reminders are stored in a database using MySQL or PostgreSQL and managed as a long-term health record for the user, allowing the server to provide more accurate personalized support based on past data, including the user's emotional state.

[0722] Device-based data collection and notification

[0723] The device has a means for collecting the user's voice and video. When the user uses the device to report, "I feel fine this morning. I had chicken salad for dinner," and takes a video of their day's walking and smiles, the device records the user's voice via a microphone and the video via a camera, and sends them to the server.

[0724] When the device receives advice or reminders generated by the server, it notifies the user. Notifications are displayed on the screen or using Google Text-to-Speech as speech synthesis technology, and specific instructions or reminders are presented to the user. For example, advice such as "I see you're enjoying your exercise. Keep doing it," or a reminder such as "It's time to take your medication" is provided to the user.

[0725] User Interaction

[0726] Users can interact with the system simply by verbally reporting their health status. To easily record daily health information, users can also use the camera function on their smartphone or tablet to record videos of their meals and exercise. This eliminates the need to record information and the risk of forgetting, enabling continuous self-management. Additionally, the system takes the user's emotions into consideration, providing motivation and psychological support.

[0727] Examples of concrete examples and prompts

[0728] As a concrete example, a user may report by voice, "I had oatmeal for breakfast today. I feel a little tired," and then take and send a video of their daily walking. The server analyzes this information and generates advice based on their health condition, dietary content, and emotional data, such as, "Oatmeal is a good choice. I recommend some light exercise today. Considering your fatigue, you should do some stretching." The device then notifies the user of this advice, and the user adjusts their lifestyle accordingly.

[0729] An example of a prompt for a generative AI model is:

[0730] "A user sends a video of themselves walking, saying, 'I had oatmeal for breakfast today. I'm feeling a little tired.' Please provide appropriate advice to the user based on their health condition, dietary habits, and emotional data."

[0731] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0732] Step 1: Collect data

[0733] The device collects audio and video data. The user uses a smartphone to report in audio, "I had oatmeal for breakfast today. I'm feeling a little tired," and records their walking.

[0734] Input: Voice spoken by the user into the device and captured video

[0735] Output: Collected audio and video data

[0736] Specifically, the device's built-in microphone records audio and the camera records video.

[0737] Step 2: Sending data

[0738] The device sends the collected audio and video data to a server over an internet connection using the HTTPS protocol.

[0739] Input: Audio and video data stored on the device

[0740] Output: Audio and video data sent to the server

[0741] Specifically, the device encrypts the collected data using HTTPS and sends it to the server.

[0742] Step 3: Convert audio data to text

[0743] The server receives the audio data and converts it into text using the Google Cloud Speech-to-Text API.

[0744] Input: Audio data sent to the server

[0745] Output: Text data (e.g., "I had oatmeal for breakfast today. I feel a little tired.")

[0746] Specifically, the audio file is sent to the API and the returned text data is retrieved.

[0747] Step 4: Semantic analysis of text data

[0748] The server analyzes the text data using natural language processing technology (e.g., spaCy or Google Cloud Natural Language API) to extract meanings such as the user's health status and dietary details.

[0749] Input: Converted text data

[0750] Output: Extracted information (e.g., "Breakfast: Oatmeal", "Health condition: Tired")

[0751] Specifically, the text data is passed to an analysis engine, and the target keywords and phrases are extracted.

[0752] Step 5: Analyze the video data

[0753] The server receives the video data, analyzes it using OpenCV and Google Cloud Vision, and extracts user behavior information.

[0754] Input: Video data sent to the server

[0755] Output: Extracted motion information (e.g., "Exercise: Walking")

[0756] To determine specific movements, the video data is analyzed frame by frame to detect movement characteristics.

[0757] Step 6: Sentiment Analysis

[0758] The server uses an emotion engine (such as Affectiva or Microsoft Azure's Emotion API) to analyze the user's emotions from the received audio and video data.

[0759] Input: Audio and video data

[0760] Output: Extracted emotion data (e.g., "Emotion: Tired")

[0761] Specifically, voice tone and facial expression data are passed to an analysis engine to classify emotions.

[0762] Step 7: Integrate the data

[0763] The server integrates the analyzed audio, video, and emotional data to understand the user's health status.

[0764] Input: Analyzed audio data, video data, emotion data

[0765] Output: Integrated health status (e.g., "Breakfast: Oatmeal", "Exercise: Walking", "Emotion: Tired")

[0766] Specifically, it integrates multiple data sources and compiles them into a single health status.

[0767] Step 8: Generate advice and reminders

[0768] The server uses generative AI (e.g., OpenAI's GPT-3) to generate custom advice and reminders based on the integrated data.

[0769] Input: Integrated Health Status

[0770] Output: Generated advice or reminders (e.g., "Oatmeal is a good choice. I recommend some light exercise today. Consider fatigue and do some stretching.")

[0771] Specifically, the prompt sentence is passed to the generation AI, which then generates advice.

[0772] Step 9: Advice and reminder notifications

[0773] The device receives advice and reminders generated by the server and notifies the user using a screen display or voice synthesis technology (e.g., Google Text-to-Speech).

[0774] Input: Advice and reminders sent from the server

[0775] Output: Advice and reminders sent to the user

[0776] Specifically, the system displays a message to the user via the smartphone's notification function.

[0777] (Application example 2)

[0778] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0779] In conventional factories, it was difficult to grasp the health status and emotions of workers in real time and provide appropriate advice. As a result, there were issues such as reduced work efficiency and safety and increased health risks for workers. In particular, the inability to properly manage workers' fatigue and stress levels led to problems such as long working hours and an increased risk of workplace accidents.

[0780] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0781] In this invention, the server includes means for collecting audio data and video data, means for analyzing the collected audio data and performing text conversion and semantic analysis, means for analyzing user motion information from the collected video data, means for ascertaining the user's health status by integrating the analyzed audio data and video data, means for generating advice and reminders based on the ascertained health status and emotional data, means for notifying the user of the generated advice and reminders, means for storing past data in a database and providing customized support based on the stored data, and means for generating advice for the user using a generative AI model. This makes it possible to ascertain the health status and emotions of workers in real time and provide appropriate advice and reminders.

[0782] "Audio data" refers to acoustic information collected by a sound recording device such as a microphone.

[0783] "Video data" refers to visual information collected by a video recording device such as a camera.

[0784] "Text conversion" is the process of converting voice data into text using voice recognition technology.

[0785] "Semantic analysis" is a natural language processing technique for extracting meaning from text data and understanding its content.

[0786] "Movement information" is information about the movements and postures of people detected from video data.

[0787] "Health status" refers to an individual's health condition as determined based on analyzed audio and video data.

[0788] "Emotion data" is information about an individual's emotional state that is analyzed from audio data and video data.

[0789] "Advice" refers to advice that supports a user's actions or choices based on analyzed data.

[0790] A "reminder" is a notification that reminds the user of a particular action or event.

[0791] A "database" is an information system for systematically storing and managing various data.

[0792] "Custom-made support" refers to personalized assistance based on the individual user's characteristics and past data.

[0793] A "generative AI model" is an artificial intelligence algorithm that is trained on large amounts of data and generates documents and data.

[0794] A "prompt" is an instruction given to a generative AI model to obtain a specific output.

[0795] This invention is a system in which a factory robot monitors the health and emotional state of workers in real time and provides appropriate advice and reminders, thereby improving work efficiency and safety and reducing health risks to workers.

[0796] System Configuration

[0797] This system is mainly composed of a server and terminals. Details of each element are described below.

[0798] Data processing by the server

[0799] The server processes data using the following software and hardware:

[0800] Speech Recognition Technology: We use technologies such as the Google Speech-to-Text API to convert collected voice data into text.

[0801] Video analysis technology: Using technologies such as OpenCV, motion information is extracted from collected video data.

[0802] Emotion recognition technology: Using technologies such as the Microsoft Azure Emotion API, emotional information is analyzed from audio and video data.

[0803] Natural language processing technology: Using technologies such as spaCy, text data can be analyzed to understand the health and emotional state of workers.

[0804] Database management system: Uses MySQL, PostgreSQL, etc. to store and manage collected data for the long term.

[0805] Generative AI models: Use pre-trained generative AI models (e.g., OpenAI's GPT-3) to generate advice and reminders for users.

[0806] Device-based data collection and notification

[0807] The device uses the following hardware and software to collect data and send notifications.

[0808] Microphone: Collects the voice of the worker.

[0809] Camera: Collects footage of workers.

[0810] Display and speaker: Notifies workers of advice and reminders generated by the server.

[0811] Operating procedure

[0812] 1. Data collection: The device collects the worker's voice and video and sends the data to the server.

[0813] 2. Data analysis: The server converts the voice data into text and analyzes the meaning using natural language processing technology. It also extracts motion information from the video data and analyzes the emotional data using emotion recognition technology.

[0814] 3. Integration and understanding: The server integrates the analyzed audio data, video data, and emotion data to understand the worker's health status and emotional state.

[0815] 4. Advice generation: Use generative AI models to generate custom advice and reminders.

[0816] 5. Notification: The device notifies the worker of generated advice and reminders via a display or voice announcement.

[0817] Specific examples

[0818] The worker reports, "I'm a little tired, but my morning work is going well," and the camera captures this. The device collects this information and sends it to the server. The server analyzes the audio data using speech recognition and natural language processing technology, and then checks the level of fatigue from the video data. Based on this information, the generative AI model generates advice such as "Take a 10-minute break to relieve fatigue," and notifies the worker via the device's display and speaker.

[0819] Prompt Sentence Examples

[0820] Today's worker reports, "I'm a little tired, but the morning's work went well." Video analysis shows that the worker appears fatigued. Please take this situation into consideration and provide appropriate rest advice.

[0821] The above is an embodiment of the present invention, which makes it possible to grasp the health condition and emotions of workers in real time and provide appropriate support.

[0822] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0823] Step 1:

[0824] The terminal collects audio and video of the workers.

[0825] Input: Audio and video data of the worker.

[0826] How it works: The microphone collects audio data and the camera collects video data. These data are collected in real time and sent to a server.

[0827] Output: The collected audio and video data is sent to a server.

[0828] Step 2:

[0829] The server converts the audio data into text.

[0830] Input: Collected audio data.

[0831] What it does: Uses speech recognition technology (e.g., Google Speech-to-Text API) to convert voice data into text.

[0832] Output: The converted text data.

[0833] Step 3:

[0834] The server analyzes the text data and extracts meaning.

[0835] Input: The converted text data.

[0836] How it works: Natural language processing techniques (e.g., spaCy) are used to analyze text data and extract information about the worker's health and emotional state.

[0837] Output: Parsed health and emotional state data.

[0838] Step 4:

[0839] The server analyzes the video data and extracts motion information and emotion data.

[0840] Input: Collected video data.

[0841] Movement: Video analysis technology (e.g., OpenCV) is used to extract movement information from video data, and emotion recognition technology (e.g., Microsoft Azure Emotion API) is used to analyze emotional states.

[0842] Output: Extracted motion information and emotion data.

[0843] Step 5:

[0844] The server integrates health status data, movement information, and emotional data to understand the overall health status.

[0845] Input: Analyzed health status data, motion information, and emotion data.

[0846] How it works: Integrates various data to assess the overall health status of workers.

[0847] Output: Consolidated health status data.

[0848] Step 6:

[0849] The server uses generative AI models to generate custom advice and reminders.

[0850] Input: Consolidated health status data.

[0851] How it works: It uses a generative AI model (e.g., OpenAI's GPT-3) to generate specific advice and reminders for workers based on a prompt.

[0852] Output: The generated advice or reminder.

[0853] Step 7:

[0854] The server sends advice and reminders to the device.

[0855] Input: Generated advice or reminder.

[0856] Operation: The server sends generated advice and reminders to the device.

[0857] Output: Advice or reminder sent.

[0858] Step 8:

[0859] The device will provide workers with advice and reminders.

[0860] Input: Advice or reminder sent.

[0861] How it works: Provides advice and reminders to workers through display and audio announcements.

[0862] Output: Workers receive advice and reminders.

[0863] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0864] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0865] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0866] [Third embodiment]

[0867] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0868] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0869] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0870] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0871] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0872] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0873] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0874] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0875] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0876] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0877] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0878] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0879] The present invention is a system for supporting patients with lifestyle-related diseases to efficiently manage their daily diet, exercise, mental and physical condition, and medication, and specific embodiments thereof will be described below.

[0880] Data processing by the server

[0881] The server receives audio and video data sent from the user's device. The received audio data is converted into text using speech recognition technology. This text is then analyzed using natural language processing technology to extract meanings such as the user's health status and dietary details. Meanwhile, the video data is analyzed using video recognition technology to extract information about the user's movements.

[0882] The analyzed audio and video data are integrated to understand the user's health status. For example, if a user reports "chicken salad" as their diet and sends a video of themselves walking, the server analyzes this information and evaluates whether the user ate an appropriate diet or how much exercise they performed. Based on this evaluation, the generative AI generates customized advice and reminders.

[0883] The generated advice and reminders are stored in a database and managed as a long-term health record for the user, enabling more accurate personalized support based on the user's past data.

[0884] Device-based data collection and notification

[0885] The device has a means for collecting the user's voice and video. When the user reports verbally, "I feel fine this morning. I had chicken salad for dinner," the device records the voice via a microphone and sends it to the server. Similarly, when the user takes a video of themselves walking, the device records the video via a camera and sends it to the server.

[0886] When the device receives advice or reminders generated by the server, it notifies the user. Notifications are made using screen displays or voice synthesis technology, and specific instructions or reminders are presented to the user. For example, advice such as "We recommend you walk 30 minutes today" or a reminder such as "It's time to take your medication" is provided to the user.

[0887] User Interaction

[0888] Users can interact with the system simply by verbally reporting their health status. They can also record their daily health information easily by using the camera function on their smartphone or tablet to record their meals and exercise. This eliminates the need to record information and helps encourage continuous self-management.

[0889] As a concrete example, a user may report by voice, "I had oatmeal for breakfast today. I feel a little tired," and then film and send a video of their walking for the day. The server analyzes this information and generates advice based on their health condition and diet, such as, "Oatmeal is a good food choice. I recommend some light exercise today." The device notifies the user of this advice, and the user adjusts their lifestyle accordingly.

[0890] In this way, the present invention provides a system for supporting the user's daily lifestyle habits and for managing health efficiently and effectively.

[0891] The processing flow will be explained below.

[0892] Step 1:

[0893] The user reports their health condition and dietary habits via voice and records their exercise using a smartphone or tablet camera. For example, they can report, "I had oatmeal and fruit for breakfast today. I feel great," and record themselves walking.

[0894] Step 2:

[0895] The device records the user's voice through a microphone and records video through a camera, thereby collecting audio and video data.

[0896] Step 3:

[0897] The terminal transmits the collected audio and video data to the server, and the transmitted data arrives at the server via a communication line.

[0898] Step 4:

[0899] The server analyzes the received voice data. First, it converts the voice data into text using voice recognition technology. Next, it uses natural language processing technology to analyze the meaning of the text and extract the user's health condition and dietary details.

[0900] Step 5:

[0901] The server analyzes the received video data and uses video recognition technology to extract information about the user's movements and analyze the amount and type of exercise.

[0902] Step 6:

[0903] The server combines the analyzed audio and video data to understand the user's health status, and provides a comprehensive assessment of the user's reported diet, physical condition, and exercise volume.

[0904] Step 7:

[0905] The AI ​​generates custom advice and reminders based on the user's health status, such as "Oatmeal is nutritious. Continue walking for 30 minutes today."

[0906] Step 8:

[0907] The server stores the generated advice and reminders in a database and manages them as long-term health data for the user, allowing for future data analysis and trend analysis.

[0908] Step 9:

[0909] The device receives advice and reminders from the server, and this information is obtained using the device's communication functions.

[0910] Step 10:

[0911] The device will notify the user of advice and reminders. Notifications are made using screen displays and voice synthesis technology, and provide specific instructions and reminders to the user. For example, a notification such as "We recommend walking for 30 minutes today" will be displayed.

[0912] Step 11:

[0913] Users receive advice and reminders to adjust their daily habits, for example by making time for walking and eating healthier.

[0914] Through the above steps, the system of the present invention supports patients with lifestyle-related diseases in efficiently managing their health.

[0915] Example 1

[0916] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0917] When patients with lifestyle-related diseases manage their daily health, they need a system that efficiently and effectively supports the management of diet, exercise, physical and mental condition, and medication. However, conventional systems have difficulty efficiently linking and using audio and video data, and lack the functionality to provide personalized advice and reminders. There is a need to resolve these issues and provide an environment in which users can sustainably manage their health.

[0918] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0919] In this invention, the server includes means for collecting audio data and video data, means for analyzing the collected audio data and performing text conversion and semantic analysis, means for analyzing user motion information from the collected video data, means for integrating the analyzed audio data and video data to grasp the user's health status, means for generating customized advice and reminders using a generative artificial intelligence model based on the grasped health status, and means for notifying the user of the generated advice and reminders, thereby enabling the user to receive efficient and personalized support based on their own health data.

[0920] "Audio data" refers to audio signals recorded and stored as digital data, and is generally collected using an input device such as a microphone.

[0921] "Video data" refers to visual information recorded using an input device such as a video camera or a smartphone camera and stored in digital format.

[0922] "Text conversion" refers to the process of analyzing audio data and converting its content into corresponding written information.

[0923] "Semantic analysis" refers to analysis performed using natural language processing techniques to extract important information and intent from text data.

[0924] "Movement information" is data relating to the physical movements of the user extracted from the video data, and includes walking, eating, exercise, and the like.

[0925] "Health status" refers to a set of data and information about a user's health condition, including diet, exercise, physical condition, etc.

[0926] A "generative artificial intelligence model" is an algorithm or system that generates responses in natural language based on given prompts, primarily using machine learning and deep learning techniques.

[0927] "Customized advice" means instructions or suggestions that are specially created based on a user's individual health data and patterns.

[0928] A "reminder" is a notification or message that prompts a user to take a specific action, such as taking medication or recommended exercise times.

[0929] MODE FOR CARRYING OUT THE INVENTION

[0930] The present invention is a system for supporting patients with lifestyle-related diseases to efficiently manage their daily health, and specific embodiments thereof will be described below.

[0931] Data processing by the server

[0932] The server receives audio and video data sent from the user's device. First, the received audio data is converted into text using speech recognition technology. This process uses technologies such as Google Cloud Speech-to-Text and IBM Watson Speech to Text. Next, this text is analyzed using natural language processing technology. Specifically, natural language processing technology using Hugging Face's Transformers library is used to extract meanings such as the user's health status and dietary details.

[0933] Meanwhile, video data is analyzed using video recognition technology, such as OpenCV and TensorFlow. User movement information is extracted from the video data, and the amount and pattern of movement can be identified from a video of someone walking, for example.

[0934] The analyzed audio and video data are integrated to understand the user's health status. For example, if a user reports "chicken salad" as their diet and sends a video of themselves walking, the server analyzes this information to evaluate whether the user ate a proper diet or how much exercise they performed. Based on this evaluation, a generative AI model (e.g., OpenAI GPT-3) generates customized advice and reminders.

[0935] The generated advice and reminders are stored in a database, typically using MongoDB or PostgreSQL. This data is managed as the user's long-term health data, enabling more accurate personalized support based on the user's past data.

[0936] Device-based data collection and notification

[0937] A user uses a device to collect daily health information. The device is equipped with a microphone and a camera, and records audio and video data. For example, if a user reports in voice, "I feel good this morning. I had chicken salad for dinner," and records a video of themselves walking, the device sends this data to a server.

[0938] Furthermore, the device receives advice and reminders generated by the server and notifies the user. Notifications are displayed on the device screen or using voice synthesis technology such as Google Text-to-Speech. For example, advice such as "We recommend walking 30 minutes today" or a reminder such as "It's time to take your medication" are provided.

[0939] User Interaction

[0940] Users can interact with the system simply by reporting their health status through voice input. They can also use the camera function on their smartphone or tablet to record their meals and exercise. This eliminates the need to record information and the risk of forgetting, enabling continuous self-management.

[0941] As a concrete example, a user may report by voice, "I had oatmeal for breakfast today. I feel a little tired," and then film and send a video of their daily walking. The server analyzes this information and generates advice based on their health condition and dietary details, such as, "Oatmeal is a good food choice. I recommend some light exercise today." The device then notifies the user of this advice, and the user adjusts their lifestyle accordingly.

[0942] Example prompts for generative AI models

[0943] Below are some example prompts to input to a generative AI model:

[0944] "The user's diet is oatmeal. Their current health condition is a little tired. They have sent a video of their exercise today. Please provide appropriate advice based on this."

[0945] In this way, the present invention provides a system for supporting the user's daily lifestyle habits and for managing health efficiently and effectively.

[0946] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0947] Step 1:

[0948] The user reports in voice, "I feel fine this morning. I had chicken salad for dinner," and films the walking video.

[0949] Specific operation: The user opens the voice recorder app on their smartphone and speaks into the microphone about what they ate and how they feel, then opens the camera app and records a video of their walking.

[0950] Input: User voice input, video recording

[0951] Output: Audio data, video data

[0952] Step 2:

[0953] The terminal records the user's voice via a microphone and records video data with a camera.

[0954] Specific operation: The device's recording function is activated and the recorded audio data is temporarily saved. At the same time, video data taken with the camera app is also saved.

[0955] Input: User's audio and video data

[0956] Output: Audio and video files temporarily saved on the device

[0957] Step 3:

[0958] The terminal transmits the collected audio and video data to the server.

[0959] Specific operation: The device uploads audio and video files to the specified endpoint on the server via Wi-Fi or mobile data communication using an HTTP / HTTPS request.

[0960] Input: Audio files, video files

[0961] Output: Audio and video files transferred to the server

[0962] Step 4:

[0963] The server converts the received voice data into text using the Google Cloud Speech-to-Text API.

[0964] Specific operation: The processing program on the server sends audio data to the API endpoint and receives the returned text data.

[0965] Input: Audio file

[0966] Output: Text data

[0967] Step 5:

[0968] The server uses Hugging Face's Transformers library to perform natural language processing on the text data, extracting meaning such as the user's health status and dietary details.

[0969] Specific operation: The server inputs the text data into a natural language processing model and extracts health status labels and dietary keywords as analysis results.

[0970] Input: Text data

[0971] Output: Health status label, dietary keywords

[0972] Step 6:

[0973] The server analyzes the video data using OpenCV and TensorFlow and extracts information about the user's movements.

[0974] Specific operation: Video data stored on the server is processed frame by frame, and walking movements are detected using a motion recognition algorithm.

[0975] Input: Video data

[0976] Output: Operation information

[0977] Step 7:

[0978] The server combines the analyzed audio and video data to determine the user's overall health status.

[0979] Specific operation: Combine the food information from the audio data and the exercise information from the video data into one dataset.

[0980] Input: Health status label, dietary keywords, behavior information

[0981] Output: Health status data

[0982] Step 8:

[0983] Based on the analysis results, the server inputs prompts into a generative AI model to generate custom advice and reminders.

[0984] Specific operation: Send a prompt to the generative AI model: "The user's meal is chicken salad. We have analyzed the walking video. Please provide appropriate advice." The generated result is returned and obtained as advice.

[0985] Input: Health status data, prompt text

[0986] Output: Custom advice and reminders

[0987] Step 9:

[0988] The generated advice and reminders are stored in a database.

[0989] Specific operation: The database connection program on the server inserts the advice data into the database.

[0990] Input: Custom advice, reminders

[0991] Output: Advice data stored in database, reminders

[0992] Step 10:

[0993] The server sends the generated advice and reminders to the device.

[0994] Specific operation: The server makes an HTTP / HTTPS request to send advice data to the device.

[0995] Input: Advice data stored in the database, reminders

[0996] Output: Advice data and reminders transferred to the device

[0997] Step 11:

[0998] The device notifies the user of advice and reminders received from the server.

[0999] Specific operation: Using the device's notification function, a message such as "We recommend walking for 30 minutes today" is displayed on the screen, and a voice notification is also given using voice synthesis technology such as Google Text-to-Speech.

[1000] Input: Advice data and reminders transferred to the device

[1001] Output: Advice and reminders sent to the user

[1002] The above are the specific processing steps of this system.

[1003] (Application example 1)

[1004] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1005] Currently, there are support systems that help patients with lifestyle-related diseases efficiently manage their daily diet, exercise, mental and physical condition, and medication. However, systems that collect and analyze users' health information in real time and provide appropriate advice and reminders while in a vehicle are not yet widespread. Furthermore, these systems are generally intended for use in fixed locations, making it difficult to receive the necessary support while traveling. Therefore, there is a need to provide a new system that provides appropriate advice and reminders by continuously managing health while in a vehicle and understanding the user's health condition in real time.

[1006] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1007] In this invention, the server includes means for collecting audio data and video data, means for analyzing the collected audio data and performing text conversion and semantic analysis, means for analyzing user motion information from the collected video data, means for ascertaining the user's health status by integrating the analyzed audio data and video data, means for generating advice and reminders based on the ascertained health status, means for notifying the user of the generated advice and reminders, and means for monitoring the user's health status using sensors and displays in the vehicle and providing feedback in real time, thereby enabling the user to effectively monitor their own health status even while in the vehicle and receive appropriate instructions and reminders in real time.

[1008] "Audio data" refers to data that is a digital recording of a user's voice or other acoustic information.

[1009] "Video data" refers to moving image information captured using a camera or other imaging device and recorded in digital format.

[1010] The term "collection means" refers to a device or method used to acquire data, and in the present invention, particularly refers to a device or method for acquiring audio data and video data.

[1011] "Means for analysis" refers to the algorithms or devices used to analyze the collected data, such as converting audio data into text or conducting semantic analysis, or performing motion analysis on video data.

[1012] "Text conversion" is the process of converting audio data into written data, and is accomplished using speech recognition technology.

[1013] "Semantic analysis" is the process of extracting meaning from data converted into text and understanding its content. It is done using natural language processing techniques.

[1014] "Motion information" refers to information about movements and actions extracted from video data.

[1015] "Health status" refers to information about the user's health condition and physical condition, and is understood from analyzed audio and video data.

[1016] "Advice and reminders" are instructions and warnings provided based on the user's health condition.

[1017] "In-vehicle sensors" refers to various types of sensors installed in the vehicle, including cameras and microphones.

[1018] A "display" is a device for visually displaying digital information, and is installed in a vehicle to provide information to a user.

[1019] "Means for providing feedback in real time" refers to technologies and devices that instantly analyze collected data and instantly provide appropriate information and instructions to users.

[1020] The present invention relates to a system for supporting a user's health management by utilizing audio data and video data. Specific embodiments of the system will be described below.

[1021] System Configuration

[1022] The system of the present invention includes means for collecting audio and video data, means for analyzing, means for generating advice and reminders, means for notifying, and means for providing feedback using sensors and displays within the vehicle.

[1023] Hardware and software used

[1024] Hardware: microphones, cameras, displays, speakers, and sensors in the car

[1025] Software: Speech recognition technology (Google Speech-to-Text), image recognition technology (OpenCV), natural language processing technology (GPT-3), generative AI (OpenAI)

[1026] Data Collection and Transmission

[1027] A microphone and a camera installed in the car collect the user's voice and video data. For example, if a user says, "I had yogurt this morning. I'm a little tired," the camera records the user's speech.

[1028] Data analysis and health status generation

[1029] The collected voice data is converted into text using voice recognition technology. The meaning of the text data is then analyzed using natural language processing technology. In parallel, the video data is analyzed using video recognition technology to extract information about the user's movements. These data are then integrated to generate the user's health status.

[1030] Generate advice and reminders

[1031] The server uses generative AI to generate custom advice and reminders based on the analyzed health status, for example, "Yogurt is a good choice. I recommend relaxed driving today."

[1032] Advice and reminder notifications

[1033] The generated advice and reminders are then sent to the user via the in-car display and speaker. For example, the display might say, "Yogurt is a good choice. We recommend a relaxed driving experience today."

[1034] Examples of concrete examples and prompts

[1035] Specific examples

[1036] The user reports in the car, "I had yogurt this morning. I'm a little tired," and the car's camera captures this information. The server analyzes this information and generates advice such as, "Yogurt is a good choice. I recommend you drive relaxed today," which is displayed on the in-car display.

[1037] Prompt Sentence Examples

[1038] User:

[1039] I had yogurt this morning. I'm a little tired.

[1040] Video data:

[1041] [Video of user relaxing in a lounge chair]

[1042] Analysis results:

[1043] The user's breakfast is healthy, but the user is feeling tired, so they should be encouraged to drive in a relaxed manner.

[1044] Generative AI advice:

[1045] Yogurt is a good choice. I recommend a relaxed drive today.

[1046] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1047] Step 1:

[1048] Audio and video data collection

[1049] The device uses the in-car microphone and camera to collect the user's voice and video data. When a user says, "I had yogurt this morning. I'm a little tired," the microphone records the voice, and the camera records the user's face and movements as video data. This data is sent to the server in real time.

[1050] Input: Audio data, video data

[1051] Output: Raw data transferred to the server

[1052] Step 2:

[1053] Speech data text conversion and semantic analysis

[1054] The server converts the received voice data into text using speech recognition technology (Google Speech-to-Text), then uses natural language processing technology (GPT-3) to analyze the meaning of the text data and extract the user's health condition and dietary details.

[1055] Specifically, the voice data is converted into text data such as "I had yogurt this morning. I feel a little tired," and then semantic analysis is used to identify "I had yogurt for breakfast" and "a complaint of fatigue" from the text.

[1056] Input: Audio data

[1057] Output: Text data and analysis results

[1058] Step 3:

[1059] Motion information analysis of video data

[1060] The server analyzes the received video data using video recognition technology (OpenCV) to extract information about the user's movements. For example, it analyzes the user's posture and facial expressions from the video data to determine whether the user is relaxed.

[1061] As for specific actions, the system analyzes the user's movements in the video and extracts motion information such as the user sitting in a relaxing chair.

[1062] Input: Video data

[1063] Output: Operation information

[1064] Step 4:

[1065] Integrated generation of health status

[1066] The server combines the analyzed text data and motion information to generate a health status for the user. For example, by combining the information "yogurt for breakfast," "complaint of fatigue," and "sitting in a lounge chair," the server determines that the user had a healthy breakfast but feels fatigued.

[1067] Input: Text data, operation information

[1068] Output: Health status

[1069] Step 5:

[1070] Generate advice and reminders

[1071] The server uses a generative AI model (OpenAI) to generate custom advice and reminders based on the generated health status, such as "Yogurt is a good choice. I recommend relaxed driving today."

[1072] Input: Health Status

[1073] Output: Advice, reminder

[1074] Step 6:

[1075] Advice and reminder notifications

[1076] The device then notifies the user of the generated advice and reminders through the display and speaker in the car. For example, the display might say, "Yogurt is a good choice. We recommend a relaxed driving experience today."

[1077] Input: Advice, Reminder

[1078] Output: Information notified to the user

[1079] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1080] The present invention is a system that supports patients with lifestyle-related diseases in efficiently managing their daily diet, exercise, mental and physical condition, and medication, and provides customized support by combining it with an emotion engine that recognizes the user's emotions. Specific embodiments of the system are described below.

[1081] Data processing by the server

[1082] The server receives audio and video data sent from the user's device. The received audio data is converted into text using speech recognition technology. This text is then analyzed using natural language processing technology to extract meanings such as the user's health condition and dietary details. Meanwhile, the video data is analyzed using video recognition technology to extract information about the user's movements. An emotion engine then recognizes the user's emotions from the audio and video data.

[1083] The analyzed voice, video, and emotional data are integrated to understand the user's health status. For example, if a user reports "chicken salad" as their diet, sends a video of themselves walking, and displays a "happy" expression, the server analyzes this information and evaluates that the user is eating a healthy diet, exercising, and feeling positive. Based on this evaluation, the generative AI generates customized advice and reminders.

[1084] The generated advice and reminders are stored in a database and managed as a long-term health record for the user, enabling more accurate personalized support based on past data, including the user's emotional state.

[1085] Device-based data collection and notification

[1086] The device has a means for collecting the user's voice and video. When the user reports in a voice message, "I feel fine this morning. I had chicken salad for dinner," and films himself walking and smiles, the device records the voice via a microphone and the video via a camera, and transmits them to a server.

[1087] When the device receives advice or reminders generated by the server, it notifies the user. Notifications are made using screen displays or voice synthesis technology, and specific instructions or reminders are presented to the user. For example, advice such as "I see you're enjoying your exercise. Keep up the same exercise," or a reminder such as "It's time to take your medication" is provided to the user.

[1088] User Interaction

[1089] Users can interact with the system simply by verbally reporting their health status. To easily record daily health information, users can also use the camera function on their smartphone or tablet to record videos of their meals and exercise. This eliminates the need to record information and the risk of forgetting, enabling continuous self-management. Additionally, the system takes the user's emotions into consideration, providing motivation and psychological support.

[1090] As a concrete example, a user may report by voice, "I had oatmeal for breakfast today. I feel a little tired," and then take and send a video of their daily walking. The server analyzes this information and generates advice based on their health condition, dietary content, and emotional data, such as, "Oatmeal is a good choice. I recommend some light exercise today. Considering your fatigue, you should do some stretching." The device then notifies the user of this advice, and the user adjusts their lifestyle accordingly.

[1091] In this way, the system of the present invention, combined with the emotion engine, supports the user's daily lifestyle habits and provides comprehensive and personalized health management.

[1092] The processing flow will be explained below.

[1093] Step 1:

[1094] The user reports their health condition and dietary habits via voice and records their exercise using a smartphone or tablet camera. For example, they can report, "I had oatmeal and fruit for breakfast today. I feel great," and record themselves walking.

[1095] Step 2:

[1096] The device records the user's voice through a microphone and records video through a camera, thereby collecting audio and video data.

[1097] Step 3:

[1098] The terminal transmits the collected audio and video data to the server, and the transmitted data arrives at the server via a communication line.

[1099] Step 4:

[1100] The server analyzes the received voice data. First, it converts the voice data into text using voice recognition technology. Next, it uses natural language processing technology to analyze the meaning of the text and extract the user's health condition and dietary details.

[1101] Step 5:

[1102] The server analyzes the received video data and uses video recognition technology to extract information about the user's movements and analyze the amount and type of exercise.

[1103] Step 6:

[1104] The emotion engine recognizes the user's emotions from the voice data, for example, determining emotions such as "joy" or "fatigue" from the tone of voice and speaking style.

[1105] Step 7:

[1106] The emotion engine recognizes the user's emotions from video data. For example, it uses facial expression recognition technology to analyze smiles and the presence or absence of wrinkles to determine emotions.

[1107] Step 8:

[1108] The server integrates the analyzed audio, video, and emotional data to understand the user's health status, and comprehensively evaluates the user's reported diet, physical condition, exercise, and emotions.

[1109] Step 9:

[1110] The AI ​​generates custom advice and reminders based on the user's health status, such as "Oatmeal is nutritious. Continue walking for 30 minutes today. Also, you seem tired, so take some time to relax."

[1111] Step 10:

[1112] The server stores the generated advice and reminders in a database and manages them as long-term health data for the user, allowing for future data analysis and trend analysis.

[1113] Step 11:

[1114] The device receives advice and reminders from the server, and this information is obtained using the device's communication functions.

[1115] Step 12:

[1116] The device will notify the user with advice and reminders. Notifications are made using screen displays and voice synthesis technology, and provide the user with specific instructions and reminders. For example, a notification might say, "I noticed you were enjoying your exercise. Please continue exercising. Also, you seem tired, so please get plenty of rest."

[1117] Step 13:

[1118] Users receive advice and reminders to adjust their daily habits, for example, by making time for walking, relaxing, or eating healthier.

[1119] Through these steps, the system of the present invention provides comprehensive and personalized health management that also takes into account the user's emotions.

[1120] Example 2

[1121] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1122] Patients with lifestyle-related diseases face the challenge of efficiently managing their daily diet, exercise, physical and mental health, and medication. Furthermore, there is a need to provide support tailored to each individual patient, taking into account their emotions and psychological state. However, conventional systems have not yet fully realized the integration and analysis of this complex data to provide highly accurate, personalized support.

[1123] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting audio data and video data, means for analyzing the collected audio data to perform text conversion and semantic analysis, and means for analyzing user movement information from the collected video data. This makes it possible to analyze the user's emotions from the collected audio data and video data. Furthermore, means for integrating the analyzed audio data, video data, and emotion data to grasp the user's health status is provided, and advice or notifications can be generated based on the grasped health status and notified to the user. This enables more accurate personalized support based on complex data including the user's emotions.

[1124] 1. "Voice data" means data that is a digital recording of voice information uttered by a user.

[1125] 2. "Video data" refers to data that captures a user's movements and facial expressions using a camera or other device and records them in digital format.

[1126] 3. "User movement information" refers to information about a user's movements and actions obtained by analyzing video data.

[1127] 4. "Emotional Data" means information about a user's emotional state obtained by analyzing audio and video data.

[1128] 5. "Health Status" refers to information about the user's physical and mental health status obtained by integrating audio data, video data, and emotional data.

[1129] 6. "Advice" means a health care suggestion or recommendation generated based on the analyzed health status.

[1130] 7. "Notification" means a server-generated display or audio output that conveys advice or reminders to the user.

[1131] 8. "Collection means" means the devices or methods used to capture audio and video data.

[1132] 9. "Analytical means" are techniques and methods used to process collected data and extract meaning from it.

[1133] 10. "Integration means" refers to techniques or methods for combining individually analyzed data to understand a single health status.

[1134] 11. "Means of understanding" refers to the technologies and methods used to understand a user's current health status from the integrated data.

[1135] 12. "Generating means" refers to techniques or methods for generating appropriate advice or reminders based on the analysis results.

[1136] The present invention is a system that supports patients with lifestyle-related diseases in efficiently managing their daily diet, exercise, mental and physical condition, and medication, and provides customized support by combining it with an emotion engine that recognizes the user's emotions. Specific embodiments of the system are described below.

[1137] Data processing by the server

[1138] The server receives audio and video data sent from the user's device. The received audio data is converted into text using Google Cloud Speech-to-Text, a speech recognition technology. This text is then analyzed using natural language processing technologies such as spaCy and Google Cloud Natural Language to extract meanings such as the user's health condition and dietary details. Meanwhile, the video data is analyzed using OpenCV and Google Cloud Vision to extract information about the user's movements. Furthermore, the server uses emotion engines such as Affectiva and Microsoft Azure's Emotion API to recognize the user's emotions from the audio and video data.

[1139] The analyzed audio, video, and emotional data are integrated and the server understands the user's health status. For example, if a user reports "chicken salad" as their diet, sends a video of themselves walking, and displays a "happy" expression, the server analyzes this information and evaluates that the user is eating a healthy diet, exercising, and feeling positive. Based on this evaluation, a generative AI (e.g., OpenAI's GPT-3) is used to generate customized advice and reminders.

[1140] The generated advice and reminders are stored in a database using MySQL or PostgreSQL and managed as a long-term health record for the user, allowing the server to provide more accurate personalized support based on past data, including the user's emotional state.

[1141] Device-based data collection and notification

[1142] The device has a means for collecting the user's voice and video. When the user uses the device to report, "I feel fine this morning. I had chicken salad for dinner," and takes a video of their day's walking and smiles, the device records the user's voice via a microphone and the video via a camera, and sends them to the server.

[1143] When the device receives advice or reminders generated by the server, it notifies the user. Notifications are displayed on the screen or using Google Text-to-Speech as speech synthesis technology, and specific instructions or reminders are presented to the user. For example, advice such as "I see you're enjoying your exercise. Keep doing it," or a reminder such as "It's time to take your medication" is provided to the user.

[1144] User Interaction

[1145] Users can interact with the system simply by verbally reporting their health status. To easily record daily health information, users can also use the camera function on their smartphone or tablet to record videos of their meals and exercise. This eliminates the need to record information and the risk of forgetting, enabling continuous self-management. Additionally, the system takes the user's emotions into consideration, providing motivation and psychological support.

[1146] Examples of concrete examples and prompts

[1147] As a concrete example, a user may report by voice, "I had oatmeal for breakfast today. I feel a little tired," and then take and send a video of their daily walking. The server analyzes this information and generates advice based on their health condition, dietary content, and emotional data, such as, "Oatmeal is a good choice. I recommend some light exercise today. Considering your fatigue, you should do some stretching." The device then notifies the user of this advice, and the user adjusts their lifestyle accordingly.

[1148] An example of a prompt for a generative AI model is:

[1149] "A user sends a video of themselves walking, saying, 'I had oatmeal for breakfast today. I'm feeling a little tired.' Please provide appropriate advice to the user based on their health condition, dietary habits, and emotional data."

[1150] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1151] Step 1: Collect data

[1152] The device collects audio and video data. The user uses a smartphone to report in audio, "I had oatmeal for breakfast today. I'm feeling a little tired," and records their walking.

[1153] Input: Voice spoken by the user into the device and captured video

[1154] Output: Collected audio and video data

[1155] Specifically, the device's built-in microphone records audio and the camera records video.

[1156] Step 2: Sending data

[1157] The device sends the collected audio and video data to a server over an internet connection using the HTTPS protocol.

[1158] Input: Audio and video data stored on the device

[1159] Output: Audio and video data sent to the server

[1160] Specifically, the device encrypts the collected data using HTTPS and sends it to the server.

[1161] Step 3: Convert audio data to text

[1162] The server receives the audio data and converts it into text using the Google Cloud Speech-to-Text API.

[1163] Input: Audio data sent to the server

[1164] Output: Text data (e.g., "I had oatmeal for breakfast today. I feel a little tired.")

[1165] Specifically, the audio file is sent to the API and the returned text data is retrieved.

[1166] Step 4: Semantic analysis of text data

[1167] The server analyzes the text data using natural language processing technology (e.g., spaCy or Google Cloud Natural Language API) to extract meanings such as the user's health status and dietary details.

[1168] Input: Converted text data

[1169] Output: Extracted information (e.g., "Breakfast: Oatmeal", "Health condition: Tired")

[1170] Specifically, the text data is passed to an analysis engine, and the target keywords and phrases are extracted.

[1171] Step 5: Analyze the video data

[1172] The server receives the video data, analyzes it using OpenCV and Google Cloud Vision, and extracts user behavior information.

[1173] Input: Video data sent to the server

[1174] Output: Extracted motion information (e.g., "Exercise: Walking")

[1175] To determine specific movements, the video data is analyzed frame by frame to detect movement characteristics.

[1176] Step 6: Sentiment Analysis

[1177] The server uses an emotion engine (such as Affectiva or Microsoft Azure's Emotion API) to analyze the user's emotions from the received audio and video data.

[1178] Input: Audio and video data

[1179] Output: Extracted emotion data (e.g., "Emotion: Tired")

[1180] Specifically, voice tone and facial expression data are passed to an analysis engine to classify emotions.

[1181] Step 7: Integrate the data

[1182] The server integrates the analyzed audio, video, and emotional data to understand the user's health status.

[1183] Input: Analyzed audio data, video data, emotion data

[1184] Output: Integrated health status (e.g., "Breakfast: Oatmeal", "Exercise: Walking", "Emotion: Tired")

[1185] Specifically, it integrates multiple data sources and compiles them into a single health status.

[1186] Step 8: Generate advice and reminders

[1187] The server uses generative AI (e.g., OpenAI's GPT-3) to generate custom advice and reminders based on the integrated data.

[1188] Input: Integrated Health Status

[1189] Output: Generated advice or reminders (e.g., "Oatmeal is a good choice. I recommend some light exercise today. Consider fatigue and do some stretching.")

[1190] Specifically, the prompt sentence is passed to the generation AI, which then generates advice.

[1191] Step 9: Advice and reminder notifications

[1192] The device receives advice and reminders generated by the server and notifies the user using a screen display or voice synthesis technology (e.g., Google Text-to-Speech).

[1193] Input: Advice and reminders sent from the server

[1194] Output: Advice and reminders sent to the user

[1195] Specifically, the system displays a message to the user via the smartphone's notification function.

[1196] (Application example 2)

[1197] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1198] In conventional factories, it was difficult to grasp the health status and emotions of workers in real time and provide appropriate advice. As a result, there were issues such as reduced work efficiency and safety and increased health risks for workers. In particular, the inability to properly manage workers' fatigue and stress levels led to problems such as long working hours and an increased risk of workplace accidents.

[1199] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1200] In this invention, the server includes means for collecting audio data and video data, means for analyzing the collected audio data and performing text conversion and semantic analysis, means for analyzing user motion information from the collected video data, means for ascertaining the user's health status by integrating the analyzed audio data and video data, means for generating advice and reminders based on the ascertained health status and emotional data, means for notifying the user of the generated advice and reminders, means for storing past data in a database and providing customized support based on the stored data, and means for generating advice for the user using a generative AI model. This makes it possible to ascertain the health status and emotions of workers in real time and provide appropriate advice and reminders.

[1201] "Audio data" refers to acoustic information collected by a sound recording device such as a microphone.

[1202] "Video data" refers to visual information collected by a video recording device such as a camera.

[1203] "Text conversion" is the process of converting voice data into text using voice recognition technology.

[1204] "Semantic analysis" is a natural language processing technique for extracting meaning from text data and understanding its content.

[1205] "Movement information" is information about the movements and postures of people detected from video data.

[1206] "Health status" refers to an individual's health condition as determined based on analyzed audio and video data.

[1207] "Emotion data" is information about an individual's emotional state that is analyzed from audio data and video data.

[1208] "Advice" refers to advice that supports a user's actions or choices based on analyzed data.

[1209] A "reminder" is a notification that reminds the user of a particular action or event.

[1210] A "database" is an information system for systematically storing and managing various data.

[1211] "Custom-made support" refers to personalized assistance based on the individual user's characteristics and past data.

[1212] A "generative AI model" is an artificial intelligence algorithm that is trained on large amounts of data and generates documents and data.

[1213] A "prompt" is an instruction given to a generative AI model to obtain a specific output.

[1214] This invention is a system in which a factory robot monitors the health and emotional state of workers in real time and provides appropriate advice and reminders, thereby improving work efficiency and safety and reducing health risks to workers.

[1215] System Configuration

[1216] This system is mainly composed of a server and terminals. Details of each element are described below.

[1217] Data processing by the server

[1218] The server processes data using the following software and hardware:

[1219] Speech Recognition Technology: We use technologies such as the Google Speech-to-Text API to convert collected voice data into text.

[1220] Video analysis technology: Using technologies such as OpenCV, motion information is extracted from collected video data.

[1221] Emotion recognition technology: Using technologies such as the Microsoft Azure Emotion API, emotional information is analyzed from audio and video data.

[1222] Natural language processing technology: Using technologies such as spaCy, text data can be analyzed to understand the health and emotional state of workers.

[1223] Database management system: Uses MySQL, PostgreSQL, etc. to store and manage collected data for the long term.

[1224] Generative AI models: Use pre-trained generative AI models (e.g., OpenAI's GPT-3) to generate advice and reminders for users.

[1225] Device-based data collection and notification

[1226] The device uses the following hardware and software to collect data and send notifications.

[1227] Microphone: Collects the voice of the worker.

[1228] Camera: Collects footage of workers.

[1229] Display and speaker: Notifies workers of advice and reminders generated by the server.

[1230] Operating procedure

[1231] 1. Data collection: The device collects the worker's voice and video and sends the data to the server.

[1232] 2. Data analysis: The server converts the voice data into text and analyzes the meaning using natural language processing technology. It also extracts motion information from the video data and analyzes the emotional data using emotion recognition technology.

[1233] 3. Integration and understanding: The server integrates the analyzed audio data, video data, and emotion data to understand the worker's health status and emotional state.

[1234] 4. Advice generation: Use generative AI models to generate custom advice and reminders.

[1235] 5. Notification: The device notifies the worker of generated advice and reminders via a display or voice announcement.

[1236] Specific examples

[1237] The worker reports, "I'm a little tired, but my morning work is going well," and the camera captures this. The device collects this information and sends it to the server. The server analyzes the audio data using speech recognition and natural language processing technology, and then checks the level of fatigue from the video data. Based on this information, the generative AI model generates advice such as "Take a 10-minute break to relieve fatigue," and notifies the worker via the device's display and speaker.

[1238] Prompt Sentence Examples

[1239] Today's worker reports, "I'm a little tired, but the morning's work went well." Video analysis shows that the worker appears fatigued. Please take this situation into consideration and provide appropriate rest advice.

[1240] The above is an embodiment of the present invention, which makes it possible to grasp the health condition and emotions of workers in real time and provide appropriate support.

[1241] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1242] Step 1:

[1243] The terminal collects audio and video of the workers.

[1244] Input: Audio and video data of the worker.

[1245] How it works: The microphone collects audio data and the camera collects video data. These data are collected in real time and sent to a server.

[1246] Output: The collected audio and video data is sent to a server.

[1247] Step 2:

[1248] The server converts the audio data into text.

[1249] Input: Collected audio data.

[1250] What it does: Uses speech recognition technology (e.g., Google Speech-to-Text API) to convert voice data into text.

[1251] Output: The converted text data.

[1252] Step 3:

[1253] The server analyzes the text data and extracts meaning.

[1254] Input: The converted text data.

[1255] How it works: Natural language processing techniques (e.g., spaCy) are used to analyze text data and extract information about the worker's health and emotional state.

[1256] Output: Parsed health and emotional state data.

[1257] Step 4:

[1258] The server analyzes the video data and extracts motion information and emotion data.

[1259] Input: Collected video data.

[1260] Movement: Video analysis technology (e.g., OpenCV) is used to extract movement information from video data, and emotion recognition technology (e.g., Microsoft Azure Emotion API) is used to analyze emotional states.

[1261] Output: Extracted motion information and emotion data.

[1262] Step 5:

[1263] The server integrates health status data, movement information, and emotional data to understand the overall health status.

[1264] Input: Analyzed health status data, motion information, and emotion data.

[1265] How it works: Integrates various data to assess the overall health status of workers.

[1266] Output: Consolidated health status data.

[1267] Step 6:

[1268] The server uses generative AI models to generate custom advice and reminders.

[1269] Input: Consolidated health status data.

[1270] How it works: It uses a generative AI model (e.g., OpenAI's GPT-3) to generate specific advice and reminders for workers based on a prompt.

[1271] Output: The generated advice or reminder.

[1272] Step 7:

[1273] The server sends advice and reminders to the device.

[1274] Input: Generated advice or reminder.

[1275] Operation: The server sends generated advice and reminders to the device.

[1276] Output: Advice or reminder sent.

[1277] Step 8:

[1278] The device will provide workers with advice and reminders.

[1279] Input: Advice or reminder sent.

[1280] How it works: Provides advice and reminders to workers through display and audio announcements.

[1281] Output: Workers receive advice and reminders.

[1282] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1283] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1284] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1285] [Fourth embodiment]

[1286] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1287] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1288] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1289] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1290] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1291] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1292] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1293] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1294] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1295] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1296] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1297] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1298] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1299] The present invention is a system for supporting patients with lifestyle-related diseases to efficiently manage their daily diet, exercise, mental and physical condition, and medication, and specific embodiments thereof will be described below.

[1300] Data processing by the server

[1301] The server receives audio and video data sent from the user's device. The received audio data is converted into text using speech recognition technology. This text is then analyzed using natural language processing technology to extract meanings such as the user's health status and dietary details. Meanwhile, the video data is analyzed using video recognition technology to extract information about the user's movements.

[1302] The analyzed audio and video data are integrated to understand the user's health status. For example, if a user reports "chicken salad" as their diet and sends a video of themselves walking, the server analyzes this information and evaluates whether the user ate an appropriate diet or how much exercise they performed. Based on this evaluation, the generative AI generates customized advice and reminders.

[1303] The generated advice and reminders are stored in a database and managed as a long-term health record for the user, enabling more accurate personalized support based on the user's past data.

[1304] Device-based data collection and notification

[1305] The device has a means for collecting the user's voice and video. When the user reports verbally, "I feel fine this morning. I had chicken salad for dinner," the device records the voice via a microphone and sends it to the server. Similarly, when the user takes a video of themselves walking, the device records the video via a camera and sends it to the server.

[1306] When the device receives advice or reminders generated by the server, it notifies the user. Notifications are made using screen displays or voice synthesis technology, and specific instructions or reminders are presented to the user. For example, advice such as "We recommend you walk 30 minutes today" or a reminder such as "It's time to take your medication" is provided to the user.

[1307] User Interaction

[1308] Users can interact with the system simply by verbally reporting their health status. They can also record their daily health information easily by using the camera function on their smartphone or tablet to record their meals and exercise. This eliminates the need to record information and helps encourage continuous self-management.

[1309] As a concrete example, a user may report by voice, "I had oatmeal for breakfast today. I feel a little tired," and then film and send a video of their walking for the day. The server analyzes this information and generates advice based on their health condition and diet, such as, "Oatmeal is a good food choice. I recommend some light exercise today." The device notifies the user of this advice, and the user adjusts their lifestyle accordingly.

[1310] In this way, the present invention provides a system for supporting the user's daily lifestyle habits and for managing health efficiently and effectively.

[1311] The processing flow will be explained below.

[1312] Step 1:

[1313] The user reports their health condition and dietary habits via voice and records their exercise using a smartphone or tablet camera. For example, they can report, "I had oatmeal and fruit for breakfast today. I feel great," and record themselves walking.

[1314] Step 2:

[1315] The device records the user's voice through a microphone and records video through a camera, thereby collecting audio and video data.

[1316] Step 3:

[1317] The terminal transmits the collected audio and video data to the server, and the transmitted data arrives at the server via a communication line.

[1318] Step 4:

[1319] The server analyzes the received voice data. First, it converts the voice data into text using voice recognition technology. Next, it uses natural language processing technology to analyze the meaning of the text and extract the user's health condition and dietary details.

[1320] Step 5:

[1321] The server analyzes the received video data and uses video recognition technology to extract information about the user's movements and analyze the amount and type of exercise.

[1322] Step 6:

[1323] The server combines the analyzed audio and video data to understand the user's health status, and provides a comprehensive assessment of the user's reported diet, physical condition, and exercise volume.

[1324] Step 7:

[1325] The AI ​​generates custom advice and reminders based on the user's health status, such as "Oatmeal is nutritious. Continue walking for 30 minutes today."

[1326] Step 8:

[1327] The server stores the generated advice and reminders in a database and manages them as long-term health data for the user, allowing for future data analysis and trend analysis.

[1328] Step 9:

[1329] The device receives advice and reminders from the server, and this information is obtained using the device's communication functions.

[1330] Step 10:

[1331] The device will notify the user of advice and reminders. Notifications are made using screen displays and voice synthesis technology, and provide specific instructions and reminders to the user. For example, a notification such as "We recommend walking for 30 minutes today" will be displayed.

[1332] Step 11:

[1333] Users receive advice and reminders to adjust their daily habits, for example by making time for walking and eating healthier.

[1334] Through the above steps, the system of the present invention supports patients with lifestyle-related diseases in efficiently managing their health.

[1335] Example 1

[1336] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1337] When patients with lifestyle-related diseases manage their daily health, they need a system that efficiently and effectively supports the management of diet, exercise, physical and mental condition, and medication. However, conventional systems have difficulty efficiently linking and using audio and video data, and lack the functionality to provide personalized advice and reminders. There is a need to resolve these issues and provide an environment in which users can sustainably manage their health.

[1338] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1339] In this invention, the server includes means for collecting audio data and video data, means for analyzing the collected audio data and performing text conversion and semantic analysis, means for analyzing user motion information from the collected video data, means for integrating the analyzed audio data and video data to grasp the user's health status, means for generating customized advice and reminders using a generative artificial intelligence model based on the grasped health status, and means for notifying the user of the generated advice and reminders, thereby enabling the user to receive efficient and personalized support based on their own health data.

[1340] "Audio data" refers to audio signals recorded and stored as digital data, and is generally collected using an input device such as a microphone.

[1341] "Video data" refers to visual information recorded using an input device such as a video camera or a smartphone camera and stored in digital format.

[1342] "Text conversion" refers to the process of analyzing audio data and converting its content into corresponding written information.

[1343] "Semantic analysis" refers to analysis performed using natural language processing techniques to extract important information and intent from text data.

[1344] "Movement information" is data relating to the physical movements of the user extracted from the video data, and includes walking, eating, exercise, and the like.

[1345] "Health status" refers to a set of data and information about a user's health condition, including diet, exercise, physical condition, etc.

[1346] A "generative artificial intelligence model" is an algorithm or system that generates responses in natural language based on given prompts, primarily using machine learning and deep learning techniques.

[1347] "Customized advice" means instructions or suggestions that are specially created based on a user's individual health data and patterns.

[1348] A "reminder" is a notification or message that prompts a user to take a specific action, such as taking medication or recommended exercise times.

[1349] MODE FOR CARRYING OUT THE INVENTION

[1350] The present invention is a system for supporting patients with lifestyle-related diseases to efficiently manage their daily health, and specific embodiments thereof will be described below.

[1351] Data processing by the server

[1352] The server receives audio and video data sent from the user's device. First, the received audio data is converted into text using speech recognition technology. This process uses technologies such as Google Cloud Speech-to-Text and IBM Watson Speech to Text. Next, this text is analyzed using natural language processing technology. Specifically, natural language processing technology using Hugging Face's Transformers library is used to extract meanings such as the user's health status and dietary details.

[1353] Meanwhile, video data is analyzed using video recognition technology, such as OpenCV and TensorFlow. User movement information is extracted from the video data, and the amount and pattern of movement can be identified from a video of someone walking, for example.

[1354] The analyzed audio and video data are integrated to understand the user's health status. For example, if a user reports "chicken salad" as their diet and sends a video of themselves walking, the server analyzes this information to evaluate whether the user ate a proper diet or how much exercise they performed. Based on this evaluation, a generative AI model (e.g., OpenAI GPT-3) generates customized advice and reminders.

[1355] The generated advice and reminders are stored in a database, typically using MongoDB or PostgreSQL. This data is managed as the user's long-term health data, enabling more accurate personalized support based on the user's past data.

[1356] Device-based data collection and notification

[1357] A user uses a device to collect daily health information. The device is equipped with a microphone and a camera, and records audio and video data. For example, if a user reports in voice, "I feel good this morning. I had chicken salad for dinner," and records a video of themselves walking, the device sends this data to a server.

[1358] Furthermore, the device receives advice and reminders generated by the server and notifies the user. Notifications are displayed on the device screen or using voice synthesis technology such as Google Text-to-Speech. For example, advice such as "We recommend walking 30 minutes today" or a reminder such as "It's time to take your medication" are provided.

[1359] User Interaction

[1360] Users can interact with the system simply by reporting their health status through voice input. They can also use the camera function on their smartphone or tablet to record their meals and exercise. This eliminates the need to record information and the risk of forgetting, enabling continuous self-management.

[1361] As a concrete example, a user may report by voice, "I had oatmeal for breakfast today. I feel a little tired," and then film and send a video of their daily walking. The server analyzes this information and generates advice based on their health condition and dietary details, such as, "Oatmeal is a good food choice. I recommend some light exercise today." The device then notifies the user of this advice, and the user adjusts their lifestyle accordingly.

[1362] Example prompts for generative AI models

[1363] Below are some example prompts to input to a generative AI model:

[1364] "The user's diet is oatmeal. Their current health condition is a little tired. They have sent a video of their exercise today. Please provide appropriate advice based on this."

[1365] In this way, the present invention provides a system for supporting the user's daily lifestyle habits and for managing health efficiently and effectively.

[1366] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1367] Step 1:

[1368] The user reports in voice, "I feel fine this morning. I had chicken salad for dinner," and films the walking video.

[1369] Specific operation: The user opens the voice recorder app on their smartphone and speaks into the microphone about what they ate and how they feel, then opens the camera app and records a video of their walking.

[1370] Input: User voice input, video recording

[1371] Output: Audio data, video data

[1372] Step 2:

[1373] The terminal records the user's voice via a microphone and records video data with a camera.

[1374] Specific operation: The device's recording function is activated and the recorded audio data is temporarily saved. At the same time, video data taken with the camera app is also saved.

[1375] Input: User's audio and video data

[1376] Output: Audio and video files temporarily saved on the device

[1377] Step 3:

[1378] The terminal transmits the collected audio and video data to the server.

[1379] Specific operation: The device uploads audio and video files to the specified endpoint on the server via Wi-Fi or mobile data communication using an HTTP / HTTPS request.

[1380] Input: Audio files, video files

[1381] Output: Audio and video files transferred to the server

[1382] Step 4:

[1383] The server converts the received voice data into text using the Google Cloud Speech-to-Text API.

[1384] Specific operation: The processing program on the server sends audio data to the API endpoint and receives the returned text data.

[1385] Input: Audio file

[1386] Output: Text data

[1387] Step 5:

[1388] The server uses Hugging Face's Transformers library to perform natural language processing on the text data, extracting meaning such as the user's health status and dietary details.

[1389] Specific operation: The server inputs the text data into a natural language processing model and extracts health status labels and dietary keywords as analysis results.

[1390] Input: Text data

[1391] Output: Health status label, dietary keywords

[1392] Step 6:

[1393] The server analyzes the video data using OpenCV and TensorFlow and extracts information about the user's movements.

[1394] Specific operation: Video data stored on the server is processed frame by frame, and walking movements are detected using a motion recognition algorithm.

[1395] Input: Video data

[1396] Output: Operation information

[1397] Step 7:

[1398] The server combines the analyzed audio and video data to determine the user's overall health status.

[1399] Specific operation: Combine the food information from the audio data and the exercise information from the video data into one dataset.

[1400] Input: Health status label, dietary keywords, behavior information

[1401] Output: Health status data

[1402] Step 8:

[1403] Based on the analysis results, the server inputs prompts into a generative AI model to generate custom advice and reminders.

[1404] Specific operation: Send a prompt to the generative AI model: "The user's meal is chicken salad. We have analyzed the walking video. Please provide appropriate advice." The generated result is returned and obtained as advice.

[1405] Input: Health status data, prompt text

[1406] Output: Custom advice and reminders

[1407] Step 9:

[1408] The generated advice and reminders are stored in a database.

[1409] Specific operation: The database connection program on the server inserts the advice data into the database.

[1410] Input: Custom advice, reminders

[1411] Output: Advice data stored in database, reminders

[1412] Step 10:

[1413] The server sends the generated advice and reminders to the device.

[1414] Specific operation: The server makes an HTTP / HTTPS request to send advice data to the device.

[1415] Input: Advice data stored in the database, reminders

[1416] Output: Advice data and reminders transferred to the device

[1417] Step 11:

[1418] The device notifies the user of advice and reminders received from the server.

[1419] Specific operation: Using the device's notification function, a message such as "We recommend walking for 30 minutes today" is displayed on the screen, and a voice notification is also given using voice synthesis technology such as Google Text-to-Speech.

[1420] Input: Advice data and reminders transferred to the device

[1421] Output: Advice and reminders sent to the user

[1422] The above are the specific processing steps of this system.

[1423] (Application example 1)

[1424] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1425] Currently, there are support systems that help patients with lifestyle-related diseases efficiently manage their daily diet, exercise, mental and physical condition, and medication. However, systems that collect and analyze users' health information in real time and provide appropriate advice and reminders while in a vehicle are not yet widespread. Furthermore, these systems are generally intended for use in fixed locations, making it difficult to receive the necessary support while traveling. Therefore, there is a need to provide a new system that provides appropriate advice and reminders by continuously managing health while in a vehicle and understanding the user's health condition in real time.

[1426] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1427] In this invention, the server includes means for collecting audio data and video data, means for analyzing the collected audio data and performing text conversion and semantic analysis, means for analyzing user motion information from the collected video data, means for ascertaining the user's health status by integrating the analyzed audio data and video data, means for generating advice and reminders based on the ascertained health status, means for notifying the user of the generated advice and reminders, and means for monitoring the user's health status using sensors and displays in the vehicle and providing feedback in real time, thereby enabling the user to effectively monitor their own health status even while in the vehicle and receive appropriate instructions and reminders in real time.

[1428] "Audio data" refers to data that is a digital recording of a user's voice or other acoustic information.

[1429] "Video data" refers to moving image information captured using a camera or other imaging device and recorded in digital format.

[1430] The term "collection means" refers to a device or method used to acquire data, and in the present invention, particularly refers to a device or method for acquiring audio data and video data.

[1431] "Means for analysis" refers to the algorithms or devices used to analyze the collected data, such as converting audio data into text or conducting semantic analysis, or performing motion analysis on video data.

[1432] "Text conversion" is the process of converting audio data into written data, and is accomplished using speech recognition technology.

[1433] "Semantic analysis" is the process of extracting meaning from data converted into text and understanding its content. It is done using natural language processing techniques.

[1434] "Motion information" refers to information about movements and actions extracted from video data.

[1435] "Health status" refers to information about the user's health condition and physical condition, and is understood from analyzed audio and video data.

[1436] "Advice and reminders" are instructions and warnings provided based on the user's health condition.

[1437] "In-vehicle sensors" refers to various types of sensors installed in the vehicle, including cameras and microphones.

[1438] A "display" is a device for visually displaying digital information, and is installed in a vehicle to provide information to a user.

[1439] "Means for providing feedback in real time" refers to technologies and devices that instantly analyze collected data and instantly provide appropriate information and instructions to users.

[1440] The present invention relates to a system for supporting a user's health management by utilizing audio data and video data. Specific embodiments of the system will be described below.

[1441] System Configuration

[1442] The system of the present invention includes means for collecting audio and video data, means for analyzing, means for generating advice and reminders, means for notifying, and means for providing feedback using sensors and displays within the vehicle.

[1443] Hardware and software used

[1444] Hardware: microphones, cameras, displays, speakers, and sensors in the car

[1445] Software: Speech recognition technology (Google Speech-to-Text), image recognition technology (OpenCV), natural language processing technology (GPT-3), generative AI (OpenAI)

[1446] Data Collection and Transmission

[1447] A microphone and a camera installed in the car collect the user's voice and video data. For example, if a user says, "I had yogurt this morning. I'm a little tired," the camera records the user's speech.

[1448] Data analysis and health status generation

[1449] The collected voice data is converted into text using voice recognition technology. The meaning of the text data is then analyzed using natural language processing technology. In parallel, the video data is analyzed using video recognition technology to extract information about the user's movements. These data are then integrated to generate the user's health status.

[1450] Generate advice and reminders

[1451] The server uses generative AI to generate custom advice and reminders based on the analyzed health status, for example, "Yogurt is a good choice. I recommend relaxed driving today."

[1452] Advice and reminder notifications

[1453] The generated advice and reminders are then sent to the user via the in-car display and speaker. For example, the display might say, "Yogurt is a good choice. We recommend a relaxed driving experience today."

[1454] Examples of concrete examples and prompts

[1455] Specific examples

[1456] The user reports in the car, "I had yogurt this morning. I'm a little tired," and the car's camera captures this information. The server analyzes this information and generates advice such as, "Yogurt is a good choice. I recommend you drive relaxed today," which is displayed on the in-car display.

[1457] Prompt Sentence Examples

[1458] User:

[1459] I had yogurt this morning. I'm a little tired.

[1460] Video data:

[1461] [Video of user relaxing in a lounge chair]

[1462] Analysis results:

[1463] The user's breakfast is healthy, but the user is feeling tired, so they should be encouraged to drive in a relaxed manner.

[1464] Generative AI advice:

[1465] Yogurt is a good choice. I recommend a relaxed drive today.

[1466] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1467] Step 1:

[1468] Audio and video data collection

[1469] The device uses the in-car microphone and camera to collect the user's voice and video data. When a user says, "I had yogurt this morning. I'm a little tired," the microphone records the voice, and the camera records the user's face and movements as video data. This data is sent to the server in real time.

[1470] Input: Audio data, video data

[1471] Output: Raw data transferred to the server

[1472] Step 2:

[1473] Speech data text conversion and semantic analysis

[1474] The server converts the received voice data into text using speech recognition technology (Google Speech-to-Text), then uses natural language processing technology (GPT-3) to analyze the meaning of the text data and extract the user's health condition and dietary details.

[1475] Specifically, the voice data is converted into text data such as "I had yogurt this morning. I feel a little tired," and then semantic analysis is used to identify "I had yogurt for breakfast" and "a complaint of fatigue" from the text.

[1476] Input: Audio data

[1477] Output: Text data and analysis results

[1478] Step 3:

[1479] Motion information analysis of video data

[1480] The server analyzes the received video data using video recognition technology (OpenCV) to extract information about the user's movements. For example, it analyzes the user's posture and facial expressions from the video data to determine whether the user is relaxed.

[1481] As for specific actions, the system analyzes the user's movements in the video and extracts motion information such as the user sitting in a relaxing chair.

[1482] Input: Video data

[1483] Output: Operation information

[1484] Step 4:

[1485] Integrated generation of health status

[1486] The server combines the analyzed text data and motion information to generate a health status for the user. For example, by combining the information "yogurt for breakfast," "complaint of fatigue," and "sitting in a lounge chair," the server determines that the user had a healthy breakfast but feels fatigued.

[1487] Input: Text data, operation information

[1488] Output: Health status

[1489] Step 5:

[1490] Generate advice and reminders

[1491] The server uses a generative AI model (OpenAI) to generate custom advice and reminders based on the generated health status, such as "Yogurt is a good choice. I recommend relaxed driving today."

[1492] Input: Health Status

[1493] Output: Advice, reminder

[1494] Step 6:

[1495] Advice and reminder notifications

[1496] The device then notifies the user of the generated advice and reminders through the display and speaker in the car. For example, the display might say, "Yogurt is a good choice. We recommend a relaxed driving experience today."

[1497] Input: Advice, Reminder

[1498] Output: Information notified to the user

[1499] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1500] The present invention is a system that supports patients with lifestyle-related diseases in efficiently managing their daily diet, exercise, mental and physical condition, and medication, and provides customized support by combining it with an emotion engine that recognizes the user's emotions. Specific embodiments of the system are described below.

[1501] Data processing by the server

[1502] The server receives audio and video data sent from the user's device. The received audio data is converted into text using speech recognition technology. This text is then analyzed using natural language processing technology to extract meanings such as the user's health condition and dietary details. Meanwhile, the video data is analyzed using video recognition technology to extract information about the user's movements. An emotion engine then recognizes the user's emotions from the audio and video data.

[1503] The analyzed voice, video, and emotional data are integrated to understand the user's health status. For example, if a user reports "chicken salad" as their diet, sends a video of themselves walking, and displays a "happy" expression, the server analyzes this information and evaluates that the user is eating a healthy diet, exercising, and feeling positive. Based on this evaluation, the generative AI generates customized advice and reminders.

[1504] The generated advice and reminders are stored in a database and managed as a long-term health record for the user, enabling more accurate personalized support based on past data, including the user's emotional state.

[1505] Device-based data collection and notification

[1506] The device has a means for collecting the user's voice and video. When the user reports in a voice message, "I feel fine this morning. I had chicken salad for dinner," and films himself walking and smiles, the device records the voice via a microphone and the video via a camera, and transmits them to a server.

[1507] When the device receives advice or reminders generated by the server, it notifies the user. Notifications are made using screen displays or voice synthesis technology, and specific instructions or reminders are presented to the user. For example, advice such as "I see you're enjoying your exercise. Keep up the same exercise," or a reminder such as "It's time to take your medication" is provided to the user.

[1508] User Interaction

[1509] Users can interact with the system simply by verbally reporting their health status. To easily record daily health information, users can also use the camera function on their smartphone or tablet to record videos of their meals and exercise. This eliminates the need to record information and the risk of forgetting, enabling continuous self-management. Additionally, the system takes the user's emotions into consideration, providing motivation and psychological support.

[1510] As a concrete example, a user may report by voice, "I had oatmeal for breakfast today. I feel a little tired," and then take and send a video of their daily walking. The server analyzes this information and generates advice based on their health condition, dietary content, and emotional data, such as, "Oatmeal is a good choice. I recommend some light exercise today. Considering your fatigue, you should do some stretching." The device then notifies the user of this advice, and the user adjusts their lifestyle accordingly.

[1511] In this way, the system of the present invention, combined with the emotion engine, supports the user's daily lifestyle habits and provides comprehensive and personalized health management.

[1512] The processing flow will be explained below.

[1513] Step 1:

[1514] The user reports their health condition and dietary habits via voice and records their exercise using a smartphone or tablet camera. For example, they can report, "I had oatmeal and fruit for breakfast today. I feel great," and record themselves walking.

[1515] Step 2:

[1516] The device records the user's voice through a microphone and records video through a camera, thereby collecting audio and video data.

[1517] Step 3:

[1518] The terminal transmits the collected audio and video data to the server, and the transmitted data arrives at the server via a communication line.

[1519] Step 4:

[1520] The server analyzes the received voice data. First, it converts the voice data into text using voice recognition technology. Next, it uses natural language processing technology to analyze the meaning of the text and extract the user's health condition and dietary details.

[1521] Step 5:

[1522] The server analyzes the received video data and uses video recognition technology to extract information about the user's movements and analyze the amount and type of exercise.

[1523] Step 6:

[1524] The emotion engine recognizes the user's emotions from the voice data, for example, determining emotions such as "joy" or "fatigue" from the tone of voice and speaking style.

[1525] Step 7:

[1526] The emotion engine recognizes the user's emotions from video data. For example, it uses facial expression recognition technology to analyze smiles and the presence or absence of wrinkles to determine emotions.

[1527] Step 8:

[1528] The server integrates the analyzed audio, video, and emotional data to understand the user's health status, and comprehensively evaluates the user's reported diet, physical condition, exercise, and emotions.

[1529] Step 9:

[1530] The AI ​​generates custom advice and reminders based on the user's health status, such as "Oatmeal is nutritious. Continue walking for 30 minutes today. Also, you seem tired, so take some time to relax."

[1531] Step 10:

[1532] The server stores the generated advice and reminders in a database and manages them as long-term health data for the user, allowing for future data analysis and trend analysis.

[1533] Step 11:

[1534] The device receives advice and reminders from the server, and this information is obtained using the device's communication functions.

[1535] Step 12:

[1536] The device will notify the user with advice and reminders. Notifications are made using screen displays and voice synthesis technology, and provide the user with specific instructions and reminders. For example, a notification might say, "I noticed you were enjoying your exercise. Please continue exercising. Also, you seem tired, so please get plenty of rest."

[1537] Step 13:

[1538] Users receive advice and reminders to adjust their daily habits, for example, by making time for walking, relaxing, or eating healthier.

[1539] Through these steps, the system of the present invention provides comprehensive and personalized health management that also takes into account the user's emotions.

[1540] Example 2

[1541] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1542] Patients with lifestyle-related diseases face the challenge of efficiently managing their daily diet, exercise, physical and mental health, and medication. Furthermore, there is a need to provide support tailored to each individual patient, taking into account their emotions and psychological state. However, conventional systems have not yet fully realized the integration and analysis of this complex data to provide highly accurate, personalized support.

[1543] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting audio data and video data, means for analyzing the collected audio data to perform text conversion and semantic analysis, and means for analyzing user movement information from the collected video data. This makes it possible to analyze the user's emotions from the collected audio data and video data. Furthermore, means for integrating the analyzed audio data, video data, and emotion data to grasp the user's health status is provided, and advice or notifications can be generated based on the grasped health status and notified to the user. This enables more accurate personalized support based on complex data including the user's emotions.

[1544] 1. "Voice data" means data that is a digital recording of voice information uttered by a user.

[1545] 2. "Video data" refers to data that captures a user's movements and facial expressions using a camera or other device and records them in digital format.

[1546] 3. "User movement information" refers to information about a user's movements and actions obtained by analyzing video data.

[1547] 4. "Emotional Data" means information about a user's emotional state obtained by analyzing audio and video data.

[1548] 5. "Health Status" refers to information about the user's physical and mental health status obtained by integrating audio data, video data, and emotional data.

[1549] 6. "Advice" means a health care suggestion or recommendation generated based on the analyzed health status.

[1550] 7. "Notification" means a server-generated display or audio output that conveys advice or reminders to the user.

[1551] 8. "Collection means" means the devices or methods used to capture audio and video data.

[1552] 9. "Analytical means" are techniques and methods used to process collected data and extract meaning from it.

[1553] 10. "Integration means" refers to techniques or methods for combining individually analyzed data to understand a single health status.

[1554] 11. "Means of understanding" refers to the technologies and methods used to understand a user's current health status from the integrated data.

[1555] 12. "Generating means" refers to techniques or methods for generating appropriate advice or reminders based on the analysis results.

[1556] The present invention is a system that supports patients with lifestyle-related diseases in efficiently managing their daily diet, exercise, mental and physical condition, and medication, and provides customized support by combining it with an emotion engine that recognizes the user's emotions. Specific embodiments of the system are described below.

[1557] Data processing by the server

[1558] The server receives audio and video data sent from the user's device. The received audio data is converted into text using Google Cloud Speech-to-Text, a speech recognition technology. This text is then analyzed using natural language processing technologies such as spaCy and Google Cloud Natural Language to extract meanings such as the user's health condition and dietary details. Meanwhile, the video data is analyzed using OpenCV and Google Cloud Vision to extract information about the user's movements. Furthermore, the server uses emotion engines such as Affectiva and Microsoft Azure's Emotion API to recognize the user's emotions from the audio and video data.

[1559] The analyzed audio, video, and emotional data are integrated and the server understands the user's health status. For example, if a user reports "chicken salad" as their diet, sends a video of themselves walking, and displays a "happy" expression, the server analyzes this information and evaluates that the user is eating a healthy diet, exercising, and feeling positive. Based on this evaluation, a generative AI (e.g., OpenAI's GPT-3) is used to generate customized advice and reminders.

[1560] The generated advice and reminders are stored in a database using MySQL or PostgreSQL and managed as a long-term health record for the user, allowing the server to provide more accurate personalized support based on past data, including the user's emotional state.

[1561] Device-based data collection and notification

[1562] The device has a means for collecting the user's voice and video. When the user uses the device to report, "I feel fine this morning. I had chicken salad for dinner," and takes a video of their day's walking and smiles, the device records the user's voice via a microphone and the video via a camera, and sends them to the server.

[1563] When the device receives advice or reminders generated by the server, it notifies the user. Notifications are displayed on the screen or using Google Text-to-Speech as speech synthesis technology, and specific instructions or reminders are presented to the user. For example, advice such as "I see you're enjoying your exercise. Keep doing it," or a reminder such as "It's time to take your medication" is provided to the user.

[1564] User Interaction

[1565] Users can interact with the system simply by verbally reporting their health status. To easily record daily health information, users can also use the camera function on their smartphone or tablet to record videos of their meals and exercise. This eliminates the need to record information and the risk of forgetting, enabling continuous self-management. Additionally, the system takes the user's emotions into consideration, providing motivation and psychological support.

[1566] Examples of concrete examples and prompts

[1567] As a concrete example, a user may report by voice, "I had oatmeal for breakfast today. I feel a little tired," and then take and send a video of their daily walking. The server analyzes this information and generates advice based on their health condition, dietary content, and emotional data, such as, "Oatmeal is a good choice. I recommend some light exercise today. Considering your fatigue, you should do some stretching." The device then notifies the user of this advice, and the user adjusts their lifestyle accordingly.

[1568] An example of a prompt for a generative AI model is:

[1569] "A user sends a video of themselves walking, saying, 'I had oatmeal for breakfast today. I'm feeling a little tired.' Please provide appropriate advice to the user based on their health condition, dietary habits, and emotional data."

[1570] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1571] Step 1: Collect data

[1572] The device collects audio and video data. The user uses a smartphone to report in audio, "I had oatmeal for breakfast today. I'm feeling a little tired," and records their walking.

[1573] Input: Voice spoken by the user into the device and captured video

[1574] Output: Collected audio and video data

[1575] Specifically, the device's built-in microphone records audio and the camera records video.

[1576] Step 2: Sending data

[1577] The device sends the collected audio and video data to a server over an internet connection using the HTTPS protocol.

[1578] Input: Audio and video data stored on the device

[1579] Output: Audio and video data sent to the server

[1580] Specifically, the device encrypts the collected data using HTTPS and sends it to the server.

[1581] Step 3: Convert audio data to text

[1582] The server receives the audio data and converts it into text using the Google Cloud Speech-to-Text API.

[1583] Input: Audio data sent to the server

[1584] Output: Text data (e.g., "I had oatmeal for breakfast today. I feel a little tired.")

[1585] Specifically, the audio file is sent to the API and the returned text data is retrieved.

[1586] Step 4: Semantic analysis of text data

[1587] The server analyzes the text data using natural language processing technology (e.g., spaCy or Google Cloud Natural Language API) to extract meanings such as the user's health status and dietary details.

[1588] Input: Converted text data

[1589] Output: Extracted information (e.g., "Breakfast: Oatmeal", "Health condition: Tired")

[1590] Specifically, the text data is passed to an analysis engine, and the target keywords and phrases are extracted.

[1591] Step 5: Analyze the video data

[1592] The server receives the video data, analyzes it using OpenCV and Google Cloud Vision, and extracts user behavior information.

[1593] Input: Video data sent to the server

[1594] Output: Extracted motion information (e.g., "Exercise: Walking")

[1595] To determine specific movements, the video data is analyzed frame by frame to detect movement characteristics.

[1596] Step 6: Sentiment Analysis

[1597] The server uses an emotion engine (such as Affectiva or Microsoft Azure's Emotion API) to analyze the user's emotions from the received audio and video data.

[1598] Input: Audio and video data

[1599] Output: Extracted emotion data (e.g., "Emotion: Tired")

[1600] Specifically, voice tone and facial expression data are passed to an analysis engine to classify emotions.

[1601] Step 7: Integrate the data

[1602] The server integrates the analyzed audio, video, and emotional data to understand the user's health status.

[1603] Input: Analyzed audio data, video data, emotion data

[1604] Output: Integrated health status (e.g., "Breakfast: Oatmeal", "Exercise: Walking", "Emotion: Tired")

[1605] Specifically, it integrates multiple data sources and compiles them into a single health status.

[1606] Step 8: Generate advice and reminders

[1607] The server uses generative AI (e.g., OpenAI's GPT-3) to generate custom advice and reminders based on the integrated data.

[1608] Input: Integrated Health Status

[1609] Output: Generated advice or reminders (e.g., "Oatmeal is a good choice. I recommend some light exercise today. Consider fatigue and do some stretching.")

[1610] Specifically, the prompt sentence is passed to the generation AI, which then generates advice.

[1611] Step 9: Advice and reminder notifications

[1612] The device receives advice and reminders generated by the server and notifies the user using a screen display or voice synthesis technology (e.g., Google Text-to-Speech).

[1613] Input: Advice and reminders sent from the server

[1614] Output: Advice and reminders sent to the user

[1615] Specifically, the system displays a message to the user via the smartphone's notification function.

[1616] (Application example 2)

[1617] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1618] In conventional factories, it was difficult to grasp the health status and emotions of workers in real time and provide appropriate advice. As a result, there were issues such as reduced work efficiency and safety and increased health risks for workers. In particular, the inability to properly manage workers' fatigue and stress levels led to problems such as long working hours and an increased risk of workplace accidents.

[1619] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1620] In this invention, the server includes means for collecting audio data and video data, means for analyzing the collected audio data and performing text conversion and semantic analysis, means for analyzing user motion information from the collected video data, means for ascertaining the user's health status by integrating the analyzed audio data and video data, means for generating advice and reminders based on the ascertained health status and emotional data, means for notifying the user of the generated advice and reminders, means for storing past data in a database and providing customized support based on the stored data, and means for generating advice for the user using a generative AI model. This makes it possible to ascertain the health status and emotions of workers in real time and provide appropriate advice and reminders.

[1621] "Audio data" refers to acoustic information collected by a sound recording device such as a microphone.

[1622] "Video data" refers to visual information collected by a video recording device such as a camera.

[1623] "Text conversion" is the process of converting voice data into text using voice recognition technology.

[1624] "Semantic analysis" is a natural language processing technique for extracting meaning from text data and understanding its content.

[1625] "Movement information" is information about the movements and postures of people detected from video data.

[1626] "Health status" refers to an individual's health condition as determined based on analyzed audio and video data.

[1627] "Emotion data" is information about an individual's emotional state that is analyzed from audio data and video data.

[1628] "Advice" refers to advice that supports a user's actions or choices based on analyzed data.

[1629] A "reminder" is a notification that reminds the user of a particular action or event.

[1630] A "database" is an information system for systematically storing and managing various data.

[1631] "Custom-made support" refers to personalized assistance based on the individual user's characteristics and past data.

[1632] A "generative AI model" is an artificial intelligence algorithm that is trained on large amounts of data and generates documents and data.

[1633] A "prompt" is an instruction given to a generative AI model to obtain a specific output.

[1634] This invention is a system in which a factory robot monitors the health and emotional state of workers in real time and provides appropriate advice and reminders, thereby improving work efficiency and safety and reducing health risks to workers.

[1635] System Configuration

[1636] This system is mainly composed of a server and terminals. Details of each element are described below.

[1637] Data processing by the server

[1638] The server processes data using the following software and hardware:

[1639] Speech Recognition Technology: We use technologies such as the Google Speech-to-Text API to convert collected voice data into text.

[1640] Video analysis technology: Using technologies such as OpenCV, motion information is extracted from collected video data.

[1641] Emotion recognition technology: Using technologies such as the Microsoft Azure Emotion API, emotional information is analyzed from audio and video data.

[1642] Natural language processing technology: Using technologies such as spaCy, text data can be analyzed to understand the health and emotional state of workers.

[1643] Database management system: Uses MySQL, PostgreSQL, etc. to store and manage collected data for the long term.

[1644] Generative AI models: Use pre-trained generative AI models (e.g., OpenAI's GPT-3) to generate advice and reminders for users.

[1645] Device-based data collection and notification

[1646] The device uses the following hardware and software to collect data and send notifications.

[1647] Microphone: Collects the voice of the worker.

[1648] Camera: Collects footage of workers.

[1649] Display and speaker: Notifies workers of advice and reminders generated by the server.

[1650] Operating procedure

[1651] 1. Data collection: The device collects the worker's voice and video and sends the data to the server.

[1652] 2. Data analysis: The server converts the voice data into text and analyzes the meaning using natural language processing technology. It also extracts motion information from the video data and analyzes the emotional data using emotion recognition technology.

[1653] 3. Integration and understanding: The server integrates the analyzed audio data, video data, and emotion data to understand the worker's health status and emotional state.

[1654] 4. Advice generation: Use generative AI models to generate custom advice and reminders.

[1655] 5. Notification: The device notifies the worker of generated advice and reminders via a display or voice announcement.

[1656] Specific examples

[1657] The worker reports, "I'm a little tired, but my morning work is going well," and the camera captures this. The device collects this information and sends it to the server. The server analyzes the audio data using speech recognition and natural language processing technology, and then checks the level of fatigue from the video data. Based on this information, the generative AI model generates advice such as "Take a 10-minute break to relieve fatigue," and notifies the worker via the device's display and speaker.

[1658] Prompt Sentence Examples

[1659] Today's worker reports, "I'm a little tired, but the morning's work went well." Video analysis shows that the worker appears fatigued. Please take this situation into consideration and provide appropriate rest advice.

[1660] The above is an embodiment of the present invention, which makes it possible to grasp the health condition and emotions of workers in real time and provide appropriate support.

[1661] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1662] Step 1:

[1663] The terminal collects audio and video of the workers.

[1664] Input: Audio and video data of the worker.

[1665] How it works: The microphone collects audio data and the camera collects video data. These data are collected in real time and sent to a server.

[1666] Output: The collected audio and video data is sent to a server.

[1667] Step 2:

[1668] The server converts the audio data into text.

[1669] Input: Collected audio data.

[1670] What it does: Uses speech recognition technology (e.g., Google Speech-to-Text API) to convert voice data into text.

[1671] Output: The converted text data.

[1672] Step 3:

[1673] The server analyzes the text data and extracts meaning.

[1674] Input: The converted text data.

[1675] How it works: Natural language processing techniques (e.g., spaCy) are used to analyze text data and extract information about the worker's health and emotional state.

[1676] Output: Parsed health and emotional state data.

[1677] Step 4:

[1678] The server analyzes the video data and extracts motion information and emotion data.

[1679] Input: Collected video data.

[1680] Movement: Video analysis technology (e.g., OpenCV) is used to extract movement information from video data, and emotion recognition technology (e.g., Microsoft Azure Emotion API) is used to analyze emotional states.

[1681] Output: Extracted motion information and emotion data.

[1682] Step 5:

[1683] The server integrates health status data, movement information, and emotional data to understand the overall health status.

[1684] Input: Analyzed health status data, motion information, and emotion data.

[1685] How it works: Integrates various data to assess the overall health status of workers.

[1686] Output: Consolidated health status data.

[1687] Step 6:

[1688] The server uses generative AI models to generate custom advice and reminders.

[1689] Input: Consolidated health status data.

[1690] How it works: It uses a generative AI model (e.g., OpenAI's GPT-3) to generate specific advice and reminders for workers based on a prompt.

[1691] Output: The generated advice or reminder.

[1692] Step 7:

[1693] The server sends advice and reminders to the device.

[1694] Input: Generated advice or reminder.

[1695] Operation: The server sends generated advice and reminders to the device.

[1696] Output: Advice or reminder sent.

[1697] Step 8:

[1698] The device will provide workers with advice and reminders.

[1699] Input: Advice or reminder sent.

[1700] How it works: Provides advice and reminders to workers through display and audio announcements.

[1701] Output: Workers receive advice and reminders.

[1702] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1703] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1704] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1705] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1706] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1707] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1708] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1709] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1710] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1711] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1712] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1713] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1714] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1715] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1716] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1717] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1718] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1719] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1720] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1721] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1722] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1723] The following is further disclosed regarding the above embodiment.

[1724] (Claim 1)

[1725] means for collecting audio and video data;

[1726] A means for analyzing the collected voice data to convert it into text and perform semantic analysis;

[1727] A means for analyzing user motion information from the collected video data;

[1728] A means for integrating the analyzed audio data and video data to grasp the user's health status;

[1729] A means for generating advice and reminders based on the perceived health status;

[1730] a means for notifying the user of the generated advice or reminder;

[1731] A system including:

[1732] (Claim 2)

[1733] 2. The system according to claim 1, further comprising means for storing the user's health status from the audio data and video data in a database, enabling long-term analysis.

[1734] (Claim 3)

[1735] 10. The system of claim 1, further comprising means for customizing the advice and reminders based on the user's past health data.

[1736]

[1737] "Example 1"

[1738] (Claim 1)

[1739] means for collecting audio and video data;

[1740] A means for analyzing the collected voice data to convert it into text and perform semantic analysis;

[1741] A means for analyzing user motion information from the collected video data;

[1742] A means for integrating the analyzed audio data and video data to grasp the user's health status;

[1743] A means for generating customized advice and reminders using a generative artificial intelligence model based on the perceived health status; and

[1744] a means for notifying the user of the generated advice or reminder;

[1745] A system including:

[1746] (Claim 2)

[1747] 2. The system according to claim 1, further comprising means for storing the user's health status from the audio and video data in a database to enable long-term analysis.

[1748] (Claim 3)

[1749] 10. The system of claim 1, further comprising means for customizing advice and reminders based on the user's past health data.

[1750] "Application Example 1"

[1751] (Claim 1)

[1752] means for collecting audio and video data;

[1753] A means for analyzing the collected voice data to convert it into text and perform semantic analysis;

[1754] A means for analyzing user motion information from the collected video data;

[1755] A means for integrating the analyzed audio data and video data to grasp the user's health status;

[1756] A means for generating advice and reminders based on the perceived health status;

[1757] a means for notifying the user of the generated advice or reminder;

[1758] a means for monitoring the user's health status using sensors and displays within the vehicle and providing real-time feedback;

[1759] A system including:

[1760] (Claim 2)

[1761] 2. The system according to claim 1, further comprising means for storing the user's health status from the audio and video data in a database to enable long-term analysis.

[1762] (Claim 3)

[1763] 10. The system of claim 1, further comprising means for customizing advice and reminders based on the user's past health data.

[1764] "Example 2: Combining Emotion Engines"

[1765] (Claim 1)

[1766] means for collecting audio and video data;

[1767] A means for analyzing the collected voice data to convert it into text and perform semantic analysis;

[1768] A means for analyzing user movement information from the collected video data;

[1769] A means for analyzing user emotions from the collected audio data and video data;

[1770] A means for integrating the analyzed voice data, video data, and emotion data to grasp the health status of the user;

[1771] a means for generating advice or notifications based on the perceived health status;

[1772] a means for notifying the user of the generated advice or notice;

[1773] A system including:

[1774] (Claim 2)

[1775] 2. The system according to claim 1, further comprising means for storing the health status of the user from the audio data, video data and emotion data in a storage device, and enabling long-term analysis.

[1776] (Claim 3)

[1777] The system of claim 1, further comprising means for customizing the advice and notifications based on the user's past health data.

[1778] "Application example 2 when combining emotion engines"

[1779] (Claim 1)

[1780] means for collecting audio and video data;

[1781] A means for analyzing the collected voice data to convert it into text and perform semantic analysis;

[1782] A means for analyzing user motion information from the collected video data;

[1783] A means for integrating the analyzed audio data and video data to grasp the user's health status;

[1784] means for generating advice and reminders based on the captured health status and emotion data;

[1785] a means for notifying the user of the generated advice or reminder;

[1786] A means of storing historical data in a database and providing customised support based on that data;

[1787] A means for generating advice for a user using a generative AI model;

[1788] A system including:

[1789] (Claim 2)

[1790] 2. The system according to claim 1, further comprising means for storing the user's health status and emotional data from the audio and video data in a database for long-term analysis.

[1791] (Claim 3)

[1792] 10. The system of claim 1, further comprising means for customizing the advice and reminders based on the user's past health and emotional data. [Explanation of symbols]

[1793] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for collecting audio and video data; A means for analyzing the collected voice data to convert it into text and perform semantic analysis; A means for analyzing user motion information from the collected video data; A means for integrating the analyzed audio data and video data to grasp the user's health status; A means for generating advice and reminders based on the perceived health status; a means for notifying the user of the generated advice or reminder; A system including:

2. 2. The system according to claim 1, further comprising means for storing the health status of the user from the audio data and video data in a database, enabling long-term analysis.

3. 10. The system of claim 1, further comprising means for customizing the advice and reminders based on the user's past health data.

Images