System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting voice data of a patient and converting the voice data into text data; means for collecting video data of a patient and using the video data for personal identification; means for inputting, integrating, and analyzing health data; means for displaying analysis results on a dashboard; means for generating and presenting an optimal lifestyle improvement proposal for an individual patient; and means for presenting possible diseases and recommended treatments to assist a doctor's diagnosis.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, medical facilities have faced serious problems such as labor shortages and healthcare disparities. In rural areas in particular, there is a severe lack of medical information and services, and there are many situations where accurate diagnoses and rapid responses are required. Under these circumstances, the burden on medical professionals increases, making it difficult to provide appropriate medical services to patients. This invention aims to eliminate healthcare disparities by utilizing multimodal AI technology to reduce the burden on medical professionals, improve diagnostic accuracy, and provide lifestyle improvement proposals tailored to individual patient needs. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by the following means. The system includes a means for collecting patient voice data and converting the voice data into text data. This means automatically converts voice to text. The system then provides a means for collecting patient video data and using the video data for personal identification. This allows for accurate identification of individual patients and maintains data consistency. The system also includes a means for inputting, integrating, and analyzing health data. This allows for comprehensive analysis of multiple biometric data to derive highly accurate results. The system also includes a means for displaying the analysis results on a dashboard, allowing users and medical professionals to intuitively understand their health status. The system also includes a means for generating and presenting optimal lifestyle improvement plans for each patient, enabling personalized health management. The system also includes a means for presenting possible diseases and recommended treatments to assist doctors in their diagnosis. These means provide a system that reduces the burden on medical professionals, improves diagnostic accuracy, and eliminates healthcare disparities.

[0006] "Voice data" refers to voice information collected from patients, and is information obtained in the form of greetings, reports of symptoms, questions, etc.

[0007] "Text data" refers to the text information of voice data converted using voice recognition technology.

[0008] "Video data" refers to video information of patients collected using cameras and other imaging equipment, and is used for purposes such as facial recognition to identify individuals.

[0009] "Personal Identification" refers to the process of identifying a specific individual from collected data.

[0010] "Health data" refers to a patient's biometric information, such as body temperature, blood pressure, heart rate, and blood sugar level, which is used to assess their health condition.

[0011] "Integration" refers to combining multiple input data into one and analyzing it.

[0012] "Analysis" refers to the process of detecting patterns and anomalies based on collected data and making diagnoses and recommendations.

[0013] A "dashboard" refers to an interface that visually displays analysis results and allows users to intuitively understand their health status.

[0014] "Lifestyle improvement suggestions" refer to specific suggestions for improving a patient's lifestyle based on analyzed data, including meal plans and exercise methods.

[0015] "Possible disease" refers to a disease that the patient may be suffering from, as predicted based on the collected and analyzed data.

[0016] "Treatment recommendations" refer to the optimal treatment for a potential disease and are information presented to physicians to aid in diagnosis. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention is a medical management system that utilizes multimodal AI technology and is designed to resolve the labor shortage and medical disparities in medical settings. Specific embodiments of the present invention are described below.

[0039] 1. Biometric data collection and management

[0040] User:

[0041] Report your health and symptoms using a voice input device.

[0042] A camera device is used to collect facial images.

[0043] Health data is entered using devices such as blood pressure monitors and thermometers.

[0044] Device:

[0045] Voice data is collected and converted into text data using voice recognition technology.

[0046] Video data is collected and individuals are identified using facial recognition technology.

[0047] Health data is collected, associated with the user's profile, and sent to a cloud server.

[0048] server:

[0049] The transmitted audio data, video data, and health data are integrated and analyzed.

[0050] Calculates the user's health indicators and displays the results on a dashboard.

[0051] 2. Generate and propose health improvement plans

[0052] User:

[0053] Enter information about your daily habits (e.g., diet, exercise, etc.) into a dedicated app.

[0054] Provide genetic information as needed.

[0055] Device:

[0056] The entered habit information and genetic information is sent to a cloud server.

[0057] server:

[0058] The user's habits and genetic information are analyzed.

[0059] Generate optimal health improvement recommendations and suggest meal plans, exercise methods, etc.

[0060] Improvement suggestions are presented to users via dashboards and notification functions.

[0061] 3. Assisting doctors in diagnosis

[0062] User:

[0063] Enter your symptoms of illness by voice or text.

[0064] Provide historical medical data.

[0065] Device:

[0066] The entered symptom data and past medical data are sent to a cloud server.

[0067] server:

[0068] Analyzes voice and text data to predict possible diseases.

[0069] Compile supplementary information to improve diagnostic accuracy by referring to past medical data.

[0070] Present the diagnosis and recommended treatment to the physician.

[0071] Specific examples

[0072] Examples of biometric data self-management support:

[0073] 1. The user speaks into a voice input device, "I'd like to enter my blood pressure record for this week."

[0074] 2. The device collects the voice data and sends it to the cloud server.

[0075] 3. The server analyzes the data, generates a graph of blood pressure fluctuations for this week, and displays it on the device.

[0076] Examples of health improvement suggestions:

[0077] 1. The user types "Enter recent food and exercise records" into the dedicated app.

[0078] 2. The device sends the input information to the cloud server.

[0079] 3. The server proposes the optimal meal plan and exercise method based on the user's data and displays it on the device.

[0080] Examples of diagnostic assistance for doctors:

[0081] 1. The user enters the following into the dedicated app: "I have persistent abdominal pain. I have already uploaded my past medical data."

[0082] 2. The device sends the entered data to the cloud server.

[0083] 3. The server analyzes the symptoms, lists possible illnesses, and recommends treatments for the doctor.

[0084] By automating and streamlining these processes, the system of the present invention reduces the burden on medical professionals, improves diagnostic accuracy, and significantly contributes to eliminating disparities in medical care.

[0085] The processing flow will be explained below.

[0086] (Biometric data self-management support)

[0087] Audio data recording and conversion

[0088] Step 1:

[0089] The user speaks about their physical condition and symptoms into a voice input device.

[0090] Step 2:

[0091] The terminal receives the audio signal and temporarily stores it as audio data.

[0092] Step 3:

[0093] The terminal uses voice recognition technology to convert the voice data into text data.

[0094] Step 4:

[0095] The terminal transmits the converted text data to the cloud server.

[0096] Video data recording and facial recognition

[0097] Step 1:

[0098] The user faces the camera.

[0099] Step 2:

[0100] The device collects and temporarily stores video data.

[0101] Step 3:

[0102] The device runs a facial recognition algorithm to identify the individual.

[0103] Step 4:

[0104] The device transmits the video data and identification information to the cloud server.

[0105] Health Data Collection and Transmission

[0106] Step 1:

[0107] Users enter health data such as blood pressure and body temperature into a dedicated app or device.

[0108] Step 2:

[0109] The terminal associates the entered health data with the user's profile and temporarily stores it.

[0110] Step 3:

[0111] The device sends the health data to a cloud server.

[0112] Data integration and analysis

[0113] Step 1:

[0114] The server integrates the received audio data, video data, and health data.

[0115] Step 2:

[0116] The server calculates health indicators using a dedicated algorithm.

[0117] Step 3:

[0118] The server converts health indicators into graphs and statistical information and generates a dashboard.

[0119] Step 4:

[0120] The server sends the dashboard and result display data to the terminal.

[0121] (Providing health improvement ideas)

[0122] Entering daily habits

[0123] Step 1:

[0124] The user inputs information about their daily habits, such as what they eat and how much exercise they do, into a dedicated app.

[0125] Step 2:

[0126] The device organizes the input habit information by associating it with the user's profile.

[0127] Step 3:

[0128] The device sends the organized habit information to a cloud server.

[0129] Analysis of habit information and genetic information

[0130] Step 1:

[0131] The server integrates the received habit information with the genetic information provided by the user.

[0132] Step 2:

[0133] The server uses statistical and genetic techniques to generate optimal health improvement recommendations.

[0134] Proposal of improvement proposals

[0135] Step 1:

[0136] The server then compiles the generated health improvement suggestions into a display format on a dashboard for the user.

[0137] Step 2:

[0138] The server sends the improvement proposal data to the terminal.

[0139] Step 3:

[0140] The improvement suggestions received by the device are displayed on the user's dashboard and an alert is sent via a notification function.

[0141] (Assisting doctors in diagnosis)

[0142] Enter symptoms

[0143] Step 1:

[0144] The user enters their symptoms of illness into a dedicated app by voice or text.

[0145] Step 2:

[0146] The terminal converts the input symptom data into text data and temporarily stores it.

[0147] Step 3:

[0148] The device sends the text data to the cloud server.

[0149] Referencing past medical data

[0150] Step 1:

[0151] The server looks up past medical data from the user's profile.

[0152] Step 2:

[0153] The server analyzes the historical data and the new symptom data.

[0154] Diagnosis and treatment suggestions

[0155] Step 1:

[0156] The server uses AI algorithms to predict possible diseases.

[0157] Step 2:

[0158] The server compares the data with past medical data and organizes supplementary information to improve diagnostic accuracy.

[0159] Step 3:

[0160] The server compiles the diagnosis and recommended treatment in the form of a report for the physician.

[0161] Step 4:

[0162] The server sends the report to the device.

[0163] Step 5:

[0164] The device receives a diagnosis and displays treatment suggestions on a screen for the doctor.

[0165] Example 1

[0166] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0167] In today's medical field, increasing patient numbers and limited medical resources have led to serious problems such as labor shortages and disparities in medical care. In conventional medical systems, the collection and management of patients' health data, the generation of lifestyle improvement recommendations, and diagnostic assistance for doctors are not fully automated, and these processes require time and effort. Therefore, there is a need to build a system that can efficiently collect and analyze data and provide diagnostic assistance.

[0168] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0169] In this invention, the server includes means for collecting voice data and converting the voice data into text data, means for collecting video data and using the video data for personal identification, and means for inputting health data and integrating and analyzing the health data, thereby enabling efficient collection and analysis of patient data and assisting doctors in making diagnoses.

[0170] "Voice data" refers to data that records a user's spoken words or audio information and is stored in digital form.

[0171] "Text data" is voice data converted into text information, and is data used for natural language processing, etc.

[0172] "Video data" refers to image and video data captured by a camera or other video collection device.

[0173] "Personal identification" is the act of distinguishing a specific person from other people and identifying that person's characteristics and attributes.

[0174] "Health data" is information that indicates the user's health condition, and includes physiological data such as blood pressure and body temperature.

[0175] "Integration" is the act of bringing together multiple different data sets and managing and analyzing them as consistent information.

[0176] "Analysis" is the act of examining collected data in detail and drawing meaningful conclusions or results from that information.

[0177] A "display device" is a device for visually presenting analytical results or other information to a user or medical professional.

[0178] The "lifestyle improvement plan" includes specific suggestions and plans for improving the user's health condition and lifestyle habits.

[0179] "Diagnostic assistance" refers to the act of providing additional information or analytical results to assist medical professionals in making a diagnosis.

[0180] "Possible diseases" refers to diseases that are suspected to be possible based on the user's symptoms and data.

[0181] "Recommended treatments" include the treatment methods that are considered most appropriate for a particular disease or condition.

[0182] "User" refers to an individual who uses this system to provide health data.

[0183] "Health professionals" refers to people who work in the medical profession, such as doctors and nurses.

[0184] This invention is a medical management system that utilizes multimodal AI technology, designed to resolve the shortage of medical staff and healthcare disparities in the medical field. This system integrates and analyzes audio, video, and health data to generate optimal lifestyle improvement recommendations for users and provide information to assist medical professionals in making diagnoses.

[0185] Biometric data collection and management

[0186] User:

[0187] Use your voice input device (e.g., a microphone on your voice input device) to report that you are "feeling unwell."

[0188] Use your smartphone camera to collect facial footage.

[0189] Health data is measured using a blood pressure monitor and thermometer and entered into a dedicated app.

[0190] Device:

[0191] Collect audio data and convert it to text using the Google Cloud Speech-to-Text API.

[0192] Video data is collected using a smartphone camera and facial recognition and personal identification are performed using the Azure Face API.

[0193] Health data is entered into a dedicated app and sent to the Google Firebase cloud server.

[0194] server:

[0195] A Python data analysis script is run to integrate and analyze the transmitted audio, video, and health data.

[0196] Calculates user health indicators and displays the results on a dashboard created with Microsoft Power BI.

[0197] The results are sent to the user's dedicated app and displayed.

[0198] Generate and propose health improvement plans

[0199] User:

[0200] Enter your diet and exercise amount into a dedicated mobile app.

[0201] If necessary, genetic information (e.g., data from a genetic information service) will also be uploaded.

[0202] Device:

[0203] Habit information and genetic information entered into a dedicated app is sent to the Google Firebase cloud server.

[0204] server:

[0205] To analyze the user's data, the data is processed using a Hadoop cluster.

[0206] Based on user data, health improvement recommendations are generated using generative AI models (e.g., large-scale language models).

[0207] Suggested meal plans and exercise methods are sent to a dedicated app and presented to the user.

[0208] Assisting doctors in diagnosis

[0209] User:

[0210] Enter the following into the dedicated app: "I have persistent abdominal pain. I have already uploaded my past medical data."

[0211] Past medical data will be provided via a dedicated app (e.g., a medical data management app).

[0212] Device:

[0213] The entered symptom data and past medical data are sent to the Google Firebase cloud server.

[0214] server:

[0215] The received data is analyzed using natural language processing technology (e.g., natural language processing model) to predict possible diseases.

[0216] Compile supplementary information to improve diagnostic accuracy by referring to past medical data.

[0217] The diagnosis and recommended treatments are sent to medical professionals via an information communication service and displayed on the medical professional's dashboard.

[0218] Examples of specific examples and prompts

[0219] In the "Biometric Data Self-Management Support Example," a user speaks to a voice input device, saying, "I'm going to enter my blood pressure record for this week." The device collects the voice data and sends it to a cloud server using the Google Cloud Speech-to-Text API. The server analyzes the data and generates a graph of blood pressure fluctuations for this week, which is then displayed on the device.

[0220] Example prompt sentence:

[0221] "I have entered this week's blood pressure data. Please display the analysis results."

[0222] In the "Health Improvement Suggestion" example, the user types "Enter my recent meal and exercise records" into the dedicated app. The device then sends the input information to a cloud server. The server uses a generative AI model to propose optimal meal plans and exercise methods based on the user's data, and displays them on the device.

[0223] Example prompt sentence:

[0224] "Please suggest the best improvement plan based on my recent diet and exercise data."

[0225] In the "Doctor's Diagnosis Assistance Example," the user enters "I have persistent abdominal pain. I have already uploaded my past medical data" into a dedicated app. The device then sends the entered data to a cloud server. The server then analyzes the symptoms using a natural language processing model, lists possible illnesses, and presents recommended treatments to the medical professional.

[0226] Example prompt sentence:

[0227] "I would like some diagnostic support based on my abdominal pain symptoms and past medical data."

[0228] The system aims to integrate multiple data sources and reduce the burden on medical facilities through automation and efficiency.

[0229] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0230] Biometric data collection and management

[0231] Step 1:

[0232] A user speaks "I'm not feeling well" into a voice input device, which collects voice data.

[0233] Input: User's voice

[0234] Output: Audio data

[0235] Step 2:

[0236] The device sends the collected voice data to the Google Cloud Speech-to-Text API, converts it into text data, and sends the converted text data to a cloud server.

[0237] Input: Audio data

[0238] Data processing: Converting voice data into text

[0239] Output: Text data

[0240] Step 3:

[0241] A user collects a video of their face using the camera on their smartphone, thereby acquiring video data.

[0242] Input: User video

[0243] Output: Video data

[0244] Step 4:

[0245] The device uses the Azure Face API to identify individuals from the video data collected, and sends the results to a cloud server.

[0246] Input: Video data

[0247] Data processing: Personal identification of video data

[0248] Output: Personally Identifiable Data

[0249] Step 5:

[0250] Users measure their health data using a blood pressure monitor or thermometer and enter it into a dedicated app, which then registers the health data in the app.

[0251] Input: Health data

[0252] Output: Input health data

[0253] Step 6:

[0254] The device sends the health data entered into a dedicated app to the Google Firebase cloud server.

[0255] Input: Health data

[0256] Output: Health data sent to the cloud server

[0257] Step 7:

[0258] The server integrates the received audio, video, and health data and analyzes it using a Python data analysis script. The analysis results are displayed on a dashboard.

[0259] Input: Audio data, video data, health data

[0260] Data Computing: Data Integration and Analysis

[0261] Output: Analysis results displayed on a dashboard

[0262] Generate and propose health improvement plans

[0263] Step 1:

[0264] Users enter their diet and exercise habits into a dedicated mobile app.

[0265] Input: Dietary information, exercise amount

[0266] Output: Data entered into the dedicated app

[0267] Step 2:

[0268] The device sends the habit information and genetic information entered into a dedicated app to the Google Firebase cloud server.

[0269] Input: Habit information, genetic information

[0270] Output: Data sent to the cloud server

[0271] Step 3:

[0272] The server processes the data using a Hadoop cluster to analyze the user's data, and then generates health improvement recommendations using a generative AI model based on the processed data.

[0273] Input: Habit information, genetic information

[0274] Data calculation: Data analysis and generation of improvement proposals using generative AI models

[0275] Output: Health improvement plan

[0276] Step 4:

[0277] The health improvement suggestions generated by the server are sent to a dedicated app and presented to the user.

[0278] Input: Health Improvement Suggestion

[0279] Output: Improvement suggestions displayed in the dedicated app

[0280] Assisting doctors in diagnosis

[0281] Step 1:

[0282] The user enters into the dedicated app, "I have persistent abdominal pain. I have already uploaded my past medical data."

[0283] Input: Symptom data, past medical data

[0284] Output: Data entered into the dedicated app

[0285] Step 2:

[0286] The device sends the entered symptom data and past medical data to the Google Firebase cloud server.

[0287] Input: Symptom data, past medical data

[0288] Output: Data sent to the cloud server

[0289] Step 3:

[0290] The server analyzes the received data using natural language processing technology to predict possible diseases, and then organizes the predicted diseases and recommended treatments.

[0291] Input: Symptom data, past medical data

[0292] Data Computing: Analysis and Disease Prediction with Natural Language Processing

[0293] Output: predicted disease, recommended treatment

[0294] Step 4:

[0295] The server sends the diagnosis results and recommended treatments to medical professionals via an information delivery service and displays them on a dashboard.

[0296] Input: predicted disease, recommended treatment

[0297] Output: Diagnosis and treatment presented to medical professionals

[0298] (Application example 1)

[0299] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0300] In modern factories, there is a lack of means to monitor the health status and work safety of workers in real time and immediately report any abnormalities. This increases the risk of worker health risks and the likelihood of accidents. Furthermore, there is a lack of effective real-time data collection and analysis to support doctors' diagnoses. There is a need to provide a system that can resolve this situation and efficiently manage the safety of the work environment and the health of workers.

[0301] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0302] In this invention, the server includes means for collecting patient voice data and converting the voice data into text data, means for collecting patient video data and using the video data for personal identification, means for inputting, integrating, and analyzing health data, means for displaying the analysis results on a dashboard, means for generating and presenting optimal lifestyle improvement plans for each patient, means for presenting possible diseases and recommended treatments to assist doctors in making diagnoses, means for collecting worker health data in real time, means for immediately issuing an alert if an abnormality is detected, means for collecting and analyzing work environment data, and means for monitoring work safety based on the analyzed data. This enables real-time monitoring of worker health conditions and work environment safety and rapid notification of abnormalities.

[0303] The "means for collecting patient voice data and converting the voice data into text data" is a technology for acquiring the patient's voice and converting the voice into text information.

[0304] The "means for collecting patient image data and using the image data for personal identification" refers to a technology for acquiring patient images and identifying individuals from the images.

[0305] "Means for inputting, integrating, and analyzing health data" refers to technology for inputting health information such as blood pressure and body temperature into a system and integrating and analyzing it.

[0306] The "means for displaying the analysis results on a dashboard" is an interface for visually displaying the analyzed health information.

[0307] "Means for generating and presenting optimal lifestyle improvement proposals for individual patients" refers to technology for automatically creating and presenting health improvement proposals tailored to each patient.

[0308] "Means for presenting possible diseases and recommended treatments to assist doctors in diagnosis" refers to technology that uses AI and data analysis to provide possible diseases and recommended treatments to assist in diagnosis.

[0309] "Means for collecting worker health data in real time" refers to technology for obtaining health information on workers working in factories in real time.

[0310] "Means for immediately issuing an alert in the event of an abnormality" refers to technology that immediately issues a warning when an abnormality is detected in health data or work environment data.

[0311] "Means for collecting and analyzing work environment data" refers to technology for acquiring data related to the work environment within a factory and analyzing it.

[0312] "Means for monitoring work safety based on analytical data" refers to technology for constantly monitoring work safety based on analyzed data.

[0313] This invention is a system for improving the health management and work safety of factory workers. This system uses technology to collect, analyze, and display audio, video, and health data in real time.

[0314] Specifically, the system includes the following means:

[0315] Sensor data collection and management

[0316] User:

[0317] Report your health and symptoms using a voice input device.

[0318] A camera device is used to collect facial images.

[0319] Health data is entered using devices such as blood pressure monitors and thermometers.

[0320] Device:

[0321] Voice data is collected and converted into text data using voice recognition technology.

[0322] Video data is collected and workers are identified using personal identification technology.

[0323] Health data is collected, associated with the worker's profile, and sent to a cloud server.

[0324] server:

[0325] The transmitted data is consolidated and analyzed.

[0326] Calculates the worker's health index and issues an alert if any abnormalities are detected.

[0327] The results are displayed on a dashboard and notified to the work team leader as needed.

[0328] Improving health management and work safety

[0329] User:

[0330] Enter information about your daily habits (e.g., diet, exercise, etc.) into a dedicated app.

[0331] Provide work environment data as needed.

[0332] Device:

[0333] The input habit information and work environment data are sent to the cloud server.

[0334] server:

[0335] Analyze habit information and work environment data.

[0336] Generate optimal health improvement proposals and safety measures.

[0337] Improvement suggestions are presented to users via dashboards and notification functions.

[0338] Specific examples

[0339] Examples of biometric data self-management support:

[0340] 1. The user speaks into a voice input device, "I'd like to enter my blood pressure record for this week."

[0341] 2. The device collects the voice data and sends it to the cloud server.

[0342] 3. The server analyzes the data, generates a graph of blood pressure fluctuations for this week, and displays it on the device.

[0343] Examples of health improvement suggestions:

[0344] 1. The user types "Enter recent food and exercise records" into the dedicated app.

[0345] 2. The device sends the input information to the cloud server.

[0346] 3. The server proposes the optimal meal plan and exercise method based on the user's data and displays it on the device.

[0347] Hardware and software used

[0348] Hardware:

[0349] Smart glasses camera: Used to collect video data.

[0350] Microphone in smart glasses: Used to collect voice data.

[0351] Health monitoring devices: thermometers, blood pressure monitors, etc.

[0352] software:

[0353] Python: The main programming language.

[0354] OpenCV: A library for video data analysis.

[0355] SpeechRecognition: A library for analyzing speech data.

[0356] Cloud server: Stores and analyzes health data.

[0357] Example prompt for a generative AI model:

[0358] A worker speaks to the smart glasses and says, "Please check my health." The smart glasses monitor the worker's body temperature, blood pressure, complexion, and other conditions in real time, sending the data to the cloud. Cloud AI analyzes the data, and if any abnormalities are detected, an alert is sent to the work team leader's smartphone.

[0359] By using this system, it is expected that health management and safety assurance for workers in factories will be automated and carried out efficiently.

[0360] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0361] Program processing flow of the system that realizes the application example

[0362] Step 1:

[0363] The user speaks to the smart glasses, saying, "Check your health."

[0364] Specific behavior:

[0365] Input: Audio data

[0366] Output: Raw audio data

[0367] The user speaks into the smart glasses, and the microphone in the glasses collects the voice data, which is then converted into text data using voice recognition technology.

[0368] Step 2:

[0369] The device sends the voice data to the cloud server.

[0370] Specific behavior:

[0371] Input: Audio data

[0372] Output: Text data

[0373] The device (smart glasses) sends the collected voice data to a cloud server, where it is converted into text data using voice recognition technology and then sent.

[0374] Step 3:

[0375] Collect user health data (body temperature, blood pressure).

[0376] Specific behavior:

[0377] Input: Data from health monitoring devices

[0378] Output: Health data (body temperature, blood pressure)

[0379] A user uses a health monitoring device to measure their body temperature and blood pressure, and the data is sent to a cloud server via smart glasses.

[0380] Step 4:

[0381] The server integrates and analyzes the transmitted audio data, health data, and video data.

[0382] Specific behavior:

[0383] Input: Audio data, health data, video data

[0384] Output: Analysis results

[0385] The server integrates the various data sent and analyzes it using machine learning models or data analysis algorithms, using Python or dedicated analysis libraries.

[0386] Step 5:

[0387] The server displays the analysis results on a dashboard and issues an alert if an abnormality is detected.

[0388] Specific behavior:

[0389] Input: Analysis results

[0390] Output: Dashboard display, alert notification

[0391] The server updates the dashboard based on the analysis results and displays them visually. If an abnormality is detected, an alert is sent to the team leader's smartphone in real time.

[0392] Step 6:

[0393] The server generates health improvement suggestions based on the worker's habit information and presents them on the terminal.

[0394] Specific behavior:

[0395] Input: Habit information, analysis results

[0396] Output: Health improvement plan

[0397] The server analyzes the user's habit information and health data and generates health improvement suggestions, which are presented as suggestions to the user's smart glasses or associated devices.

[0398] In this way, the system collects, analyzes, displays, and notifies data at each step, monitoring worker health and the safety of the work environment in real time, enabling rapid response when an abnormality occurs.

[0399] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0400] The present invention aims to incorporate an emotion engine into a medical management system that utilizes multimodal AI technology, thereby making diagnoses and suggestions that reflect the patient's emotions, reducing the burden on medical professionals, improving the accuracy of diagnoses, and providing lifestyle improvement suggestions tailored to the needs of each individual patient.

[0401] 1. Collection and management of biometric and emotional data

[0402] User:

[0403] Report your health and symptoms using a voice input device.

[0404] A camera device is used to collect facial images.

[0405] Health data is entered using devices such as blood pressure monitors and thermometers.

[0406] Device:

[0407] Voice data is collected and converted into text data using voice recognition technology.

[0408] Video data is collected and individuals are identified using facial recognition technology.

[0409] Health data is collected, associated with the user's profile, and sent to a cloud server.

[0410] Emotions are recognized from audio data and video data, and emotion data is generated.

[0411] server:

[0412] The transmitted audio data, video data, health data, and emotional data are integrated and analyzed.

[0413] Calculates the user's health and emotional indicators and displays the results on a dashboard.

[0414] 2. Generating and Proposing Health Improvement Plans Using Emotional Data

[0415] User:

[0416] Enter information about your daily habits (e.g., diet, exercise, etc.) into a dedicated app.

[0417] Provide genetic information as needed.

[0418] Device:

[0419] The entered habit information and genetic information is sent to a cloud server.

[0420] server:

[0421] The user's habits, genetic information, and emotional data are analyzed.

[0422] Based on emotional data, optimal health improvement proposals are generated, including meal plans and exercise methods.

[0423] Improvement suggestions are presented to users via dashboards and notification functions.

[0424] 3. Assisting doctors in diagnosis and utilizing emotion data

[0425] User:

[0426] Enter your symptoms of illness by voice or text.

[0427] Provide historical medical data.

[0428] Device:

[0429] The entered symptom data and past medical data are sent to a cloud server.

[0430] server:

[0431] Analyzes voice and text data to predict possible diseases.

[0432] Compilation of supplementary information to improve diagnostic accuracy is carried out by comparing it with past medical data and referring to emotional data.

[0433] Present the diagnosis and recommended treatment to the physician.

[0434] Specific examples

[0435] Examples of biometric and emotional data self-management support:

[0436] 1. The user speaks into a voice input device, "I'd like to enter my blood pressure record for this week."

[0437] 2. The device collects the voice data and sends it to the cloud server.

[0438] 3. The device recognizes emotions from the voice data and generates emotion data.

[0439] 4. The server analyzes the data, generates a graph of blood pressure fluctuations for this week, and displays it on the device along with emotional data.

[0440] Example of using emotion data to provide health improvement suggestions:

[0441] 1. The user types "Enter recent food and exercise records" into the dedicated app.

[0442] 2. The device sends the input information to the cloud server.

[0443] 3. The server proposes optimal meal plans and exercise methods based on the user's data, and displays them on the device, reflecting the user's emotional data.

[0444] Examples of using emotion data to assist doctors in diagnosis:

[0445] 1. The user enters the following into the dedicated app: "I have persistent abdominal pain. I have already uploaded my past medical data."

[0446] 2. The device sends the entered data to the cloud server.

[0447] 3. The server analyzes the symptoms, lists possible illnesses based on the emotional data, and presents recommended treatments to the doctor.

[0448] By automating and streamlining these processes, the system of the present invention reduces the burden on medical professionals, improves diagnostic accuracy, and significantly contributes to eliminating disparities in medical care. Furthermore, by using emotion data, comprehensive health management that takes into account the patient's psychological state becomes possible.

[0449] The processing flow will be explained below.

[0450] (Collection and management of biometric and emotional data)

[0451] Audio data recording and conversion

[0452] Step 1:

[0453] The user speaks about their physical condition and symptoms into a voice input device.

[0454] Step 2:

[0455] The terminal receives the audio signal and temporarily stores it as audio data.

[0456] Step 3:

[0457] The terminal uses voice recognition technology to convert the voice data into text data.

[0458] Step 4:

[0459] The terminal transmits the converted text data to the cloud server.

[0460] Video data recording and facial recognition

[0461] Step 1:

[0462] The user faces the camera.

[0463] Step 2:

[0464] The device collects and temporarily stores video data.

[0465] Step 3:

[0466] The device runs a facial recognition algorithm to identify the individual.

[0467] Step 4:

[0468] The device transmits the video data and identification information to the cloud server.

[0469] Generating emotion data

[0470] Step 1:

[0471] The terminal recognizes emotions from audio data and video data.

[0472] Step 2:

[0473] The device generates the recognized emotion data and temporarily stores it.

[0474] Step 3:

[0475] The device transmits the emotion data to the cloud server.

[0476] Health Data Collection and Transmission

[0477] Step 1:

[0478] Users enter health data such as blood pressure and body temperature into a dedicated app or device.

[0479] Step 2:

[0480] The terminal associates the entered health data with the user's profile and temporarily stores it.

[0481] Step 3:

[0482] The device sends the health data to a cloud server.

[0483] Data integration and analysis

[0484] Step 1:

[0485] The server integrates the received audio data, video data, health data, and emotional data.

[0486] Step 2:

[0487] The server calculates health and emotional indicators using a proprietary algorithm.

[0488] Step 3:

[0489] The server converts health and emotional indicators into graphs and statistical information to generate a dashboard.

[0490] Step 4:

[0491] The server sends the dashboard and result display data to the terminal.

[0492] (Providing health improvement ideas)

[0493] Entering daily habits

[0494] Step 1:

[0495] The user inputs information about their daily habits, such as what they eat and how much exercise they do, into a dedicated app.

[0496] Step 2:

[0497] The device organizes the input habit information by associating it with the user's profile.

[0498] Step 3:

[0499] The device sends the organized habit information to a cloud server.

[0500] Analysis of habit information and genetic information

[0501] Step 1:

[0502] The server integrates the received habit information with the genetic information provided by the user.

[0503] Step 2:

[0504] The server uses statistical and genetic techniques to generate optimal health improvement recommendations.

[0505] Customization using emotional data

[0506] Step 1:

[0507] The server customizes the generated health improvement proposals by reflecting emotional data.

[0508] Proposal of improvement proposals

[0509] Step 1:

[0510] The server then compiles the generated health improvement suggestions into a display format on a dashboard for the user.

[0511] Step 2:

[0512] The server sends the improvement proposal data to the terminal.

[0513] Step 3:

[0514] The improvement suggestions received by the device are displayed on the user's dashboard and an alert is sent via a notification function.

[0515] (Assisting doctors in diagnosis)

[0516] Enter symptoms

[0517] Step 1:

[0518] The user enters their symptoms of illness into a dedicated app by voice or text.

[0519] Step 2:

[0520] The terminal converts the input symptom data into text data and temporarily stores it.

[0521] Step 3:

[0522] The device sends the text data to the cloud server.

[0523] Referencing past medical data

[0524] Step 1:

[0525] The server looks up past medical data from the user's profile.

[0526] Step 2:

[0527] The server analyzes the historical data and the new symptom data.

[0528] Diagnosis and treatment suggestions

[0529] Step 1:

[0530] The server uses AI algorithms to predict possible diseases.

[0531] Step 2:

[0532] The server compares the data with past medical data and uses emotional data to organize supplementary information to improve diagnostic accuracy.

[0533] Step 3:

[0534] The server compiles the diagnosis and recommended treatment in the form of a report for the physician.

[0535] Step 4:

[0536] The server sends the report to the device.

[0537] Step 5:

[0538] The device receives a diagnosis and displays treatment suggestions on a screen for the doctor.

[0539] Example 2

[0540] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0541] The current medical system does not take patients' emotions into consideration when making diagnoses or recommendations, which increases the burden on medical professionals and results in insufficient diagnostic accuracy. Furthermore, there is a lack of systems that provide lifestyle improvement suggestions tailored to individual patients' needs, making it difficult for patients to manage their own health. Furthermore, there is a need for improved data analysis and disease prediction accuracy to assist doctors in their diagnoses.

[0542] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0543] In this invention, the server includes: means for collecting a patient's voice data and converting the voice data into text data; means for collecting a patient's video data and using the video data for personal identification; means for inputting health and emotional data and transmitting it to a cloud server; means for recognizing emotions from the voice and video data and generating emotional data; means for integrating and analyzing the transmitted voice, video, health, and emotional data; means for displaying the analysis results on a dashboard; means for generating and presenting optimal lifestyle improvement proposals for each patient; and means for presenting possible diseases and recommended treatments using a generative AI model to assist physicians in their diagnosis. This enables diagnoses and proposals that take the patient's emotions into account, reducing the burden on medical professionals and improving diagnostic accuracy. Furthermore, providing lifestyle improvement proposals tailored to each patient's needs makes it easier for patients to manage their own health.

[0544] The "means for collecting patient voice data and converting the voice data into text data" is a mechanism for collecting voice data reported by the patient through a voice input device and converting the voice data into text format.

[0545] The "means for collecting patient image data and using the image data for personal identification" is a mechanism for acquiring an image of the patient's face and using the image data to identify the individual.

[0546] "Means for inputting health data and emotional data and transmitting it to a cloud server" refers to a mechanism for inputting information about a patient's health and emotions and transmitting that data to a server on the cloud.

[0547] The "means for recognizing emotions from audio data and video data and generating emotional data" refers to a mechanism for analyzing collected audio data and video data to recognize the patient's emotions and generate data related to emotions.

[0548] "Means for integrating and analyzing transmitted audio data, video data, health data, and emotional data" refers to a mechanism for integrating and comprehensively analyzing various data transmitted to the cloud server.

[0549] "Means for displaying analysis results on a dashboard" refers to a mechanism for outputting the results of the analyzed data to a dashboard for visually displaying them.

[0550] "A means for generating and presenting optimal lifestyle improvement proposals for individual patients" is a system that generates optimal lifestyle improvement proposals based on data on each patient and presents them to the patient.

[0551] "Means of using generative AI models to suggest possible diseases and recommended treatments to assist doctors in their diagnoses" refers to a system that uses generative AI technology to predict possible diseases from collected data and provide doctors with optimal treatments.

[0552] This invention is a medical management system that utilizes multimodal AI technology and incorporates an emotion engine to make diagnoses and recommendations that reflect the patient's emotions, reducing the burden on medical professionals. The system also aims to improve the accuracy of diagnoses and provide lifestyle improvement proposals tailored to the individual needs of each patient.

[0553] The system consists of three main components: users, terminals, and servers.

[0554] User

[0555] A user reports their physical condition and symptoms using a voice input device. A camera device is also used to collect facial images, and health data is input using devices such as a blood pressure monitor and a thermometer. As a specific example, data collection begins when a user speaks into the voice input device, saying, "I'd like to enter this week's blood pressure record."

[0556] Terminal

[0557] The terminal uses multiple pieces of hardware and software to collect and process data.

[0558] The audio data is converted into text data using the Google Cloud Speech-to-Text service.

[0559] Video data is collected and individuals are identified using the Microsoft Azure Face API.

[0560] Health data is sent to AWS cloud servers.

[0561] Emotions are recognized from audio and video data using IBM Watson's Natural Language Understanding and Visual Recognition, and emotion data is generated.

[0562] For example, when a user types "I would like to enter my recent meal and exercise records" into a dedicated app, that information is sent to the cloud server.

[0563] server

[0564] The server is the central component that processes data on the cloud.

[0565] The audio data, video data, health data, and emotion data sent to the cloud server are integrated and analyzed using data processing libraries such as Python's Pandas and NumPy.

[0566] Calculates user health and emotional indicators and displays the results on a dashboard using Tableau or Power BI.

[0567] For example, it can recognize emotions based on voice data and evaluate stress levels to provide comprehensive health management. It also uses GPT-4 as a generative AI model to generate optimal health improvement proposals and suggest optimal diet plans and exercise methods for users.

[0568] Specific examples

[0569] Examples of biometric and emotional data self-management support:

[0570] The user speaks to the voice input device, saying, "I'd like to enter my blood pressure record for this week." The device then collects the voice data and sends it to a cloud server using the Google Cloud Speech-to-Text service. The device then uses IBM Watson's Natural Language Understanding to recognize emotions from the voice data and generate emotion data. The server then analyzes this data, generates a graph of blood pressure fluctuations for this week, and displays it on a Tableau dashboard along with the emotion data.

[0571] Example of using emotion data to provide health improvement suggestions:

[0572] The user enters text data into the dedicated app, saying, "I'd like to enter my recent meal and exercise records." The device then sends the input information to an AWS cloud server. The server generates an optimal meal plan and exercise method based on the user's data, and uses GPT-4 to create suggestions that also reflect emotional data, which are then displayed on the device.

[0573] Examples of using emotion data to assist doctors in diagnosis:

[0574] The user enters into the dedicated app, "I have persistent stomach pain. I have already uploaded my past medical data." The device then sends the entered data to an AWS cloud server. The server analyzes the symptoms, lists possible illnesses based on the emotional data, and generates a recommended treatment using GPT-4 to present to the doctor.

[0575] By automating and streamlining these processes, this system reduces the burden on medical professionals and improves the accuracy of diagnoses. It also enables comprehensive health management based on emotional data, providing patients with optimal treatments and lifestyle improvement suggestions.

[0576] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0577] Step 1:

[0578] The user reports their physical condition and symptoms using a voice input device.

[0579] Input: User voice input (e.g., "Enter this week's blood pressure record.")

[0580] Data processing: The device collects voice data and converts it into text data using the Google Cloud Speech-to-Text service.

[0581] Output: Report content as text data (e.g., "Enter your blood pressure record for this week")

[0582] Step 2:

[0583] A user uses a camera device to collect facial images.

[0584] Input: User's face image

[0585] Data processing: The device collects video data and uses the Microsoft Azure Face API to perform facial recognition and recognize emotions from the video data.

[0586] Output: Identified person ID and emotion data

[0587] Step 3:

[0588] A user inputs health data using devices such as a blood pressure monitor and a thermometer.

[0589] Input: Health data (e.g. blood pressure, body temperature)

[0590] Data processing: The device collects health data and sends it to AWS cloud servers.

[0591] Output: Health data stored on a cloud server

[0592] Step 4:

[0593] Sending audio and video data to the cloud

[0594] Input: Text data, identified personal ID, emotion data

[0595] Data processing: The device integrates the audio and video data to generate emotion data, which is then sent to the AWS cloud server.

[0596] Output: Integrated data sent to cloud server

[0597] Step 5:

[0598] The audio data, video data, health data, and emotional data sent to the cloud server are integrated and analyzed.

[0599] Input: Audio data, video data, health data, and emotion data stored on a cloud server

[0600] Data processing: The server integrates this data and analyzes it using data processing libraries such as Python's Pandas and NumPy.

[0601] Output: Integrated data analysis results (e.g., health index, emotional index)

[0602] Step 6:

[0603] Display the analysis results on a dashboard.

[0604] Input: Integrated data analysis results

[0605] Data processing: The server visualizes and displays the analysis results on a dashboard using Tableau or Power BI.

[0606] Output: Dashboard screen viewable by users and doctors

[0607] Step 7:

[0608] The user enters information about their daily habits into a dedicated app.

[0609] Input: User's habit information (e.g., dietary content, amount of exercise)

[0610] Data processing: The terminal sends the input data to the AWS cloud server.

[0611] Output: Habit information stored on the cloud server

[0612] Step 8:

[0613] The server generates and presents health improvement suggestions based on the user's habit information, health data, and emotional data.

[0614] Input: Habit information, health data, emotional data

[0615] Data processing: The server analyzes these data and generates optimal health improvement recommendations using a generative AI model (GPT-4).

[0616] Output: Generated health improvement suggestions, including recommendations displayed in notifications and on the dashboard

[0617] Step 9:

[0618] Providing data to assist doctors in diagnosis

[0619] Input: User's symptom information, past medical data

[0620] Data processing: The device sends the input data to the AWS cloud server, which analyzes the data and uses GPT-4 to list possible diseases based on the emotional data.

[0621] Output: A list of diagnoses and treatment recommendations presented to the doctor

[0622] At each step, the system processes and analyzes the collected data to provide optimal improvement suggestions for each individual user and diagnostic support information for doctors, thereby realizing comprehensive health management that takes the patient's emotions into consideration.

[0623] (Application example 2)

[0624] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0625] Conventional medical management systems provide diagnoses and treatment recommendations based on biometric data and past medical data, but they do not take into account the patient's emotions or psychological state, making it difficult to fully meet the needs of individual patients. Furthermore, food delivery services, in particular, do not propose optimal meal plans that reflect the user's health status and emotions, making it difficult to provide individually customized services. Therefore, new methods are needed to increase patient and user satisfaction, reduce the burden on medical professionals, and improve diagnostic accuracy.

[0626] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0627] In this invention, the server includes means for collecting patient voice data and converting the voice data into text data, means for collecting patient video data and using the video data for personal identification, means for inputting, integrating, and analyzing health data, means for displaying the analysis results on a dashboard, means for generating and presenting optimal lifestyle improvement plans for each patient, means for presenting possible diseases and recommended treatments to assist doctors in making diagnoses, means for collecting user emotion data and analyzing the emotion data to reflect it in health improvement plans, and means for inputting health reports by voice or text and proposing and delivering optimal meal plans. This enables personalized medical management and food delivery services that comprehensively consider the emotions and health status of patients and users.

[0628] "Voice data" refers to data for collecting voice information from patients and converting the content into text data.

[0629] "Video data" refers to data that collects images of a patient's face or body and is used to identify the individual.

[0630] "Health data" refers to physiological information such as a patient's blood pressure, body temperature, and heart rate, and is data that can be integrated and analyzed.

[0631] A "dashboard" is an interface for visually displaying information such as analysis results and proposals.

[0632] A "lifestyle improvement plan" is a specific action plan proposed to improve the quality of daily life of patients or users.

[0633] "Aiding diagnosis" means providing supplementary information or recommendations to help a doctor make an appropriate diagnosis of a patient's symptoms.

[0634] "Disease" refers to an illness or condition that adversely affects a patient's health.

[0635] A "therapy" is a specific medical procedure or method used to treat a disease.

[0636] "Emotional data" refers to data about the user's psychological state and emotions extracted from facial expressions, etc.

[0637] "Health report" refers to the act of a patient or user recording or reporting their current health condition or symptoms.

[0638] "Meal Plan" refers to a meal menu and plan suggested based on the user's health condition and mood.

[0639] "Delivery" refers to the act of delivering cooked meals based on the proposed meal plan to the user.

[0640] This invention is a system that collects and analyzes biometric and emotional data of patients and users, and provides optimal lifestyle improvement proposals and health management plans based on the collected data. The system aims to provide individually customized services that comprehensively consider the health status and emotions of patients and users.

[0641] 1. Collection and management of biometric and emotional data

[0642] User:

[0643] Users report their physical condition and symptoms using a voice input device, which is collected by the terminal as voice data. A camera device is also used to collect facial images. Furthermore, health data is input using devices such as a blood pressure monitor and a thermometer.

[0644] Device:

[0645] The device collects voice data and converts it into text data using voice recognition technology. It also collects video data and identifies individuals using facial recognition technology. The collected health data is associated with the user's profile and sent to a cloud server. At the same time, emotions are recognized from the voice and video data, and emotion data is generated.

[0646] server:

[0647] The server integrates and analyzes the transmitted audio, video, health, and emotional data, calculates the user's health and emotional indices, and displays the results on a dashboard.

[0648] 2. Generating and Proposing Health Improvement Plans Using Emotional Data

[0649] User:

[0650] Users input information about their daily habits (e.g., dietary habits, amount of exercise, etc.) into a dedicated app, and also provide genetic information if necessary.

[0651] Device:

[0652] The terminal transmits the input habit information and genetic information to a cloud server.

[0653] server:

[0654] The server analyzes the user's habits, genetic information, and emotional data. Based on the emotional data, it generates optimal health improvement proposals, suggesting meal plans and exercise methods to the user. The proposals are presented to the user via a dashboard and notification function.

[0655] 3. Assisting doctors in diagnosis and utilizing emotion data

[0656] User:

[0657] Users can input their symptoms of illness by voice or text, and also provide past medical data.

[0658] Device:

[0659] The terminal transmits the input symptom data and past medical data to a cloud server.

[0660] server:

[0661] The server analyzes the voice and text data to predict possible illnesses, compares it with past medical data, and uses emotional data to organize supplementary information to improve diagnostic accuracy. The diagnosis results and recommended treatments are then presented to the doctor.

[0662] Specific examples

[0663] Examples of biometric and emotional data self-management support:

[0664] 1. The user speaks into a voice input device, "I'd like to enter my blood pressure record for this week."

[0665] 2. The device collects the voice data and sends it to the cloud server.

[0666] 3. The device recognizes emotions from the voice data and generates emotion data.

[0667] 4. The server analyzes the data, generates a graph of blood pressure fluctuations for this week, and displays it on the device along with emotional data.

[0668] Example of using emotion data to provide health improvement suggestions:

[0669] 1. The user types "Enter recent food and exercise records" into the dedicated app.

[0670] 2. The device sends the input information to the cloud server.

[0671] 3. The server proposes optimal meal plans and exercise methods based on the user's data, and displays them on the device, reflecting the user's emotional data.

[0672] Examples of using emotion data to assist doctors in diagnosis:

[0673] 1. The user enters the following into the dedicated app: "I have persistent abdominal pain. I have already uploaded my past medical data."

[0674] 2. The device sends the entered data to the cloud server.

[0675] 3. The server analyzes the symptoms, lists possible illnesses based on the emotional data, and presents recommended treatments to the doctor.

[0676] Prompt Sentence Examples

[0677] "Please suggest personalized food delivery based on the user's emotional and health data. Here's the user's data for this week:

[0678] Blood pressure: 110

[0679] Temperature: 36.5

[0680] Emotion: neutral

[0681] Please suggest a suitable meal plan and why.”

[0682] By building a system in this way, it is possible to comprehensively manage a user's health condition and emotions, and realize personalized medical services and food delivery.

[0683] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0684] Step 1:

[0685] A user reports their physical condition and symptoms using a voice input device. This voice data is collected and input data from the voice input device is generated.

[0686] Step 2:

[0687] The device receives the voice data and converts it into text data using speech recognition technology (e.g., Google Speech Recognition API). The input is voice data, and the output is text data.

[0688] Step 3:

[0689] A user uses a camera device to capture a video of his or her face, generating video data that is provided to a terminal as a video input.

[0690] Step 4:

[0691] The device receives the video data and identifies individuals using facial recognition technology (e.g., using OpenCV and dlib). The input is the video data, and the output is the data of the identified individuals.

[0692] Step 5:

[0693] A user inputs health data using a device such as a blood pressure monitor or a thermometer, and the data is sent to a terminal. The input is health data (blood pressure, body temperature, etc.).

[0694] Step 6:

[0695] The device associates the collected health data with the user's profile and transmits it to the cloud server. The input is the health data, and the output is the integrated health data transmitted to the cloud server.

[0696] Step 7:

[0697] The device recognizes emotions from audio and video data and generates emotion data (e.g., extracts emotions using the FER library). The input is audio and video data, and the output is emotion data.

[0698] Step 8:

[0699] The server integrates and analyzes the audio data, video data, health data, and emotion data sent to the cloud server. These data are the input, and the output is the health index and emotion index as the analysis results.

[0700] Step 9:

[0701] The server displays the analysis results on a dashboard, which includes graphs and indicators to visualize the user's health and emotional state. The input is the analysis results, and the output is the dashboard display.

[0702] Step 10:

[0703] The user inputs information about their daily habits (such as dietary habits and exercise amount) into a dedicated app. The input is habit information, which is then provided to the device.

[0704] Step 11:

[0705] The device sends habit information to the cloud server. The input is the habit information, and the output is the data sent to the cloud server.

[0706] Step 12:

[0707] The server generates optimal health improvement proposals based on emotional data and habit information. It also suggests meal plans and exercise methods. The input is emotional data and habit information, and the output is improvement proposals.

[0708] Step 13:

[0709] The server generates improvement proposals and presents them to the user via a dashboard or notification function. The input is the improvement proposal, and the output is a notification to the user.

[0710] Step 14:

[0711] The user inputs symptoms of illness by voice or text and provides this to a dedicated app. The input is symptom data, which is then sent to the device.

[0712] Step 15:

[0713] The device sends symptom data and past medical data to the cloud server. The input is the symptom data and medical data, and the output is the data sent to the cloud server.

[0714] Step 16:

[0715] The server analyzes the symptom data, predicts possible diseases based on the emotion data, and then organizes recommended treatments. The input is the symptom data and emotion data, and the output is a list of diseases and recommended treatments.

[0716] Step 17:

[0717] The server presents the diagnosis results and recommended treatment to the doctor. The input is the analysis results, and the output is a notification to the doctor.

[0718] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0719] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0720] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0721] [Second embodiment]

[0722] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0723] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0724] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0725] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0726] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0727] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0728] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0729] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0730] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0731] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0732] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0733] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0734] The present invention is a medical management system that utilizes multimodal AI technology and is designed to resolve the labor shortage and medical disparities in medical settings. Specific embodiments of the present invention are described below.

[0735] 1. Biometric data collection and management

[0736] User:

[0737] Report your health and symptoms using a voice input device.

[0738] A camera device is used to collect facial images.

[0739] Health data is entered using devices such as blood pressure monitors and thermometers.

[0740] Device:

[0741] Voice data is collected and converted into text data using voice recognition technology.

[0742] Video data is collected and individuals are identified using facial recognition technology.

[0743] Health data is collected, associated with the user's profile, and sent to a cloud server.

[0744] server:

[0745] The transmitted audio data, video data, and health data are integrated and analyzed.

[0746] Calculates the user's health indicators and displays the results on a dashboard.

[0747] 2. Generate and propose health improvement plans

[0748] User:

[0749] Enter information about your daily habits (e.g., diet, exercise, etc.) into a dedicated app.

[0750] Provide genetic information as needed.

[0751] Device:

[0752] The entered habit information and genetic information is sent to a cloud server.

[0753] server:

[0754] The user's habits and genetic information are analyzed.

[0755] Generate optimal health improvement recommendations and suggest meal plans, exercise methods, etc.

[0756] Improvement suggestions are presented to users via dashboards and notification functions.

[0757] 3. Assisting doctors in diagnosis

[0758] User:

[0759] Enter your symptoms of illness by voice or text.

[0760] Provide historical medical data.

[0761] Device:

[0762] The entered symptom data and past medical data are sent to a cloud server.

[0763] server:

[0764] Analyzes voice and text data to predict possible diseases.

[0765] Compile supplementary information to improve diagnostic accuracy by referring to past medical data.

[0766] Present the diagnosis and recommended treatment to the physician.

[0767] Specific examples

[0768] Examples of biometric data self-management support:

[0769] 1. The user speaks into a voice input device, "I'd like to enter my blood pressure record for this week."

[0770] 2. The device collects the voice data and sends it to the cloud server.

[0771] 3. The server analyzes the data, generates a graph of blood pressure fluctuations for this week, and displays it on the device.

[0772] Examples of health improvement suggestions:

[0773] 1. The user types "Enter recent food and exercise records" into the dedicated app.

[0774] 2. The device sends the input information to the cloud server.

[0775] 3. The server proposes the optimal meal plan and exercise method based on the user's data and displays it on the device.

[0776] Examples of diagnostic assistance for doctors:

[0777] 1. The user enters the following into the dedicated app: "I have persistent abdominal pain. I have already uploaded my past medical data."

[0778] 2. The device sends the entered data to the cloud server.

[0779] 3. The server analyzes the symptoms, lists possible illnesses, and recommends treatments for the doctor.

[0780] By automating and streamlining these processes, the system of the present invention reduces the burden on medical professionals, improves diagnostic accuracy, and significantly contributes to eliminating disparities in medical care.

[0781] The processing flow will be explained below.

[0782] (Biometric data self-management support)

[0783] Audio data recording and conversion

[0784] Step 1:

[0785] The user speaks about their physical condition and symptoms into a voice input device.

[0786] Step 2:

[0787] The terminal receives the audio signal and temporarily stores it as audio data.

[0788] Step 3:

[0789] The terminal uses voice recognition technology to convert the voice data into text data.

[0790] Step 4:

[0791] The terminal transmits the converted text data to the cloud server.

[0792] Video data recording and facial recognition

[0793] Step 1:

[0794] The user faces the camera.

[0795] Step 2:

[0796] The device collects and temporarily stores video data.

[0797] Step 3:

[0798] The device runs a facial recognition algorithm to identify the individual.

[0799] Step 4:

[0800] The device transmits the video data and identification information to the cloud server.

[0801] Health Data Collection and Transmission

[0802] Step 1:

[0803] Users enter health data such as blood pressure and body temperature into a dedicated app or device.

[0804] Step 2:

[0805] The terminal associates the entered health data with the user's profile and temporarily stores it.

[0806] Step 3:

[0807] The device sends the health data to a cloud server.

[0808] Data integration and analysis

[0809] Step 1:

[0810] The server integrates the received audio data, video data, and health data.

[0811] Step 2:

[0812] The server calculates health indicators using a dedicated algorithm.

[0813] Step 3:

[0814] The server converts health indicators into graphs and statistical information and generates a dashboard.

[0815] Step 4:

[0816] The server sends the dashboard and result display data to the terminal.

[0817] (Providing health improvement ideas)

[0818] Entering daily habits

[0819] Step 1:

[0820] The user inputs information about their daily habits, such as what they eat and how much exercise they do, into a dedicated app.

[0821] Step 2:

[0822] The device organizes the input habit information by associating it with the user's profile.

[0823] Step 3:

[0824] The device sends the organized habit information to a cloud server.

[0825] Analysis of habit information and genetic information

[0826] Step 1:

[0827] The server integrates the received habit information with the genetic information provided by the user.

[0828] Step 2:

[0829] The server uses statistical and genetic techniques to generate optimal health improvement recommendations.

[0830] Proposal of improvement proposals

[0831] Step 1:

[0832] The server then compiles the generated health improvement suggestions into a display format on a dashboard for the user.

[0833] Step 2:

[0834] The server sends the improvement proposal data to the terminal.

[0835] Step 3:

[0836] The improvement suggestions received by the device are displayed on the user's dashboard and an alert is sent via a notification function.

[0837] (Assisting doctors in diagnosis)

[0838] Enter symptoms

[0839] Step 1:

[0840] The user enters their symptoms of illness into a dedicated app by voice or text.

[0841] Step 2:

[0842] The terminal converts the input symptom data into text data and temporarily stores it.

[0843] Step 3:

[0844] The device sends the text data to the cloud server.

[0845] Referencing past medical data

[0846] Step 1:

[0847] The server looks up past medical data from the user's profile.

[0848] Step 2:

[0849] The server analyzes the historical data and the new symptom data.

[0850] Diagnosis and treatment suggestions

[0851] Step 1:

[0852] The server uses AI algorithms to predict possible diseases.

[0853] Step 2:

[0854] The server compares the data with past medical data and organizes supplementary information to improve diagnostic accuracy.

[0855] Step 3:

[0856] The server compiles the diagnosis and recommended treatment in the form of a report for the physician.

[0857] Step 4:

[0858] The server sends the report to the device.

[0859] Step 5:

[0860] The device receives a diagnosis and displays treatment suggestions on a screen for the doctor.

[0861] Example 1

[0862] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0863] In today's medical field, increasing patient numbers and limited medical resources have led to serious problems such as labor shortages and disparities in medical care. In conventional medical systems, the collection and management of patients' health data, the generation of lifestyle improvement recommendations, and diagnostic assistance for doctors are not fully automated, and these processes require time and effort. Therefore, there is a need to build a system that can efficiently collect and analyze data and provide diagnostic assistance.

[0864] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0865] In this invention, the server includes means for collecting voice data and converting the voice data into text data, means for collecting video data and using the video data for personal identification, and means for inputting health data and integrating and analyzing the health data, thereby enabling efficient collection and analysis of patient data and assisting doctors in making diagnoses.

[0866] "Voice data" refers to data that records a user's spoken words or audio information and is stored in digital form.

[0867] "Text data" is voice data converted into text information, and is data used for natural language processing, etc.

[0868] "Video data" refers to image and video data captured by a camera or other video collection device.

[0869] "Personal identification" is the act of distinguishing a specific person from other people and identifying that person's characteristics and attributes.

[0870] "Health data" is information that indicates the user's health condition, and includes physiological data such as blood pressure and body temperature.

[0871] "Integration" is the act of bringing together multiple different data sets and managing and analyzing them as consistent information.

[0872] "Analysis" is the act of examining collected data in detail and drawing meaningful conclusions or results from that information.

[0873] A "display device" is a device for visually presenting analytical results or other information to a user or medical professional.

[0874] The "lifestyle improvement plan" includes specific suggestions and plans for improving the user's health condition and lifestyle habits.

[0875] "Diagnostic assistance" refers to the act of providing additional information or analytical results to assist medical professionals in making a diagnosis.

[0876] "Possible diseases" refers to diseases that are suspected to be possible based on the user's symptoms and data.

[0877] "Recommended treatments" include the treatment methods that are considered most appropriate for a particular disease or condition.

[0878] "User" refers to an individual who uses this system to provide health data.

[0879] "Health professionals" refers to people who work in the medical profession, such as doctors and nurses.

[0880] This invention is a medical management system that utilizes multimodal AI technology, designed to resolve the shortage of medical staff and healthcare disparities in the medical field. This system integrates and analyzes audio, video, and health data to generate optimal lifestyle improvement recommendations for users and provide information to assist medical professionals in making diagnoses.

[0881] Biometric data collection and management

[0882] User:

[0883] Use your voice input device (e.g., a microphone on your voice input device) to report that you are "feeling unwell."

[0884] Use your smartphone camera to collect facial footage.

[0885] Health data is measured using a blood pressure monitor and thermometer and entered into a dedicated app.

[0886] Device:

[0887] Collect audio data and convert it to text using the Google Cloud Speech-to-Text API.

[0888] Video data is collected using a smartphone camera and facial recognition and personal identification are performed using the Azure Face API.

[0889] Health data is entered into a dedicated app and sent to the Google Firebase cloud server.

[0890] server:

[0891] A Python data analysis script is run to integrate and analyze the transmitted audio, video, and health data.

[0892] Calculates user health indicators and displays the results on a dashboard created with Microsoft Power BI.

[0893] The results are sent to the user's dedicated app and displayed.

[0894] Generate and propose health improvement plans

[0895] User:

[0896] Enter your diet and exercise amount into a dedicated mobile app.

[0897] If necessary, genetic information (e.g., data from a genetic information service) will also be uploaded.

[0898] Device:

[0899] Habit information and genetic information entered into a dedicated app is sent to the Google Firebase cloud server.

[0900] server:

[0901] To analyze the user's data, the data is processed using a Hadoop cluster.

[0902] Based on user data, health improvement recommendations are generated using generative AI models (e.g., large-scale language models).

[0903] Suggested meal plans and exercise methods are sent to a dedicated app and presented to the user.

[0904] Assisting doctors in diagnosis

[0905] User:

[0906] Enter the following into the dedicated app: "I have persistent abdominal pain. I have already uploaded my past medical data."

[0907] Past medical data will be provided via a dedicated app (e.g., a medical data management app).

[0908] Device:

[0909] The entered symptom data and past medical data are sent to the Google Firebase cloud server.

[0910] server:

[0911] The received data is analyzed using natural language processing technology (e.g., natural language processing model) to predict possible diseases.

[0912] Compile supplementary information to improve diagnostic accuracy by referring to past medical data.

[0913] The diagnosis and recommended treatments are sent to medical professionals via an information communication service and displayed on the medical professional's dashboard.

[0914] Examples of specific examples and prompts

[0915] In the "Biometric Data Self-Management Support Example," a user speaks to a voice input device, saying, "I'm going to enter my blood pressure record for this week." The device collects the voice data and sends it to a cloud server using the Google Cloud Speech-to-Text API. The server analyzes the data and generates a graph of blood pressure fluctuations for this week, which is then displayed on the device.

[0916] Example prompt sentence:

[0917] "I have entered this week's blood pressure data. Please display the analysis results."

[0918] In the "Health Improvement Suggestion" example, the user types "Enter my recent meal and exercise records" into the dedicated app. The device then sends the input information to a cloud server. The server uses a generative AI model to propose optimal meal plans and exercise methods based on the user's data, and displays them on the device.

[0919] Example prompt sentence:

[0920] "Please suggest the best improvement plan based on my recent diet and exercise data."

[0921] In the "Doctor's Diagnosis Assistance Example," the user enters "I have persistent abdominal pain. I have already uploaded my past medical data" into a dedicated app. The device then sends the entered data to a cloud server. The server then analyzes the symptoms using a natural language processing model, lists possible illnesses, and presents recommended treatments to the medical professional.

[0922] Example prompt sentence:

[0923] "I would like some diagnostic support based on my abdominal pain symptoms and past medical data."

[0924] The system aims to integrate multiple data sources and reduce the burden on medical facilities through automation and efficiency.

[0925] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0926] Biometric data collection and management

[0927] Step 1:

[0928] A user speaks "I'm not feeling well" into a voice input device, which collects voice data.

[0929] Input: User's voice

[0930] Output: Audio data

[0931] Step 2:

[0932] The device sends the collected voice data to the Google Cloud Speech-to-Text API, converts it into text data, and sends the converted text data to a cloud server.

[0933] Input: Audio data

[0934] Data processing: Converting voice data into text

[0935] Output: Text data

[0936] Step 3:

[0937] A user collects a video of their face using the camera on their smartphone, thereby acquiring video data.

[0938] Input: User video

[0939] Output: Video data

[0940] Step 4:

[0941] The device uses the Azure Face API to identify individuals from the video data collected, and sends the results to a cloud server.

[0942] Input: Video data

[0943] Data processing: Personal identification of video data

[0944] Output: Personally Identifiable Data

[0945] Step 5:

[0946] Users measure their health data using a blood pressure monitor or thermometer and enter it into a dedicated app, which then registers the health data in the app.

[0947] Input: Health data

[0948] Output: Input health data

[0949] Step 6:

[0950] The device sends the health data entered into a dedicated app to the Google Firebase cloud server.

[0951] Input: Health data

[0952] Output: Health data sent to the cloud server

[0953] Step 7:

[0954] The server integrates the received audio, video, and health data and analyzes it using a Python data analysis script. The analysis results are displayed on a dashboard.

[0955] Input: Audio data, video data, health data

[0956] Data Computing: Data Integration and Analysis

[0957] Output: Analysis results displayed on a dashboard

[0958] Generate and propose health improvement plans

[0959] Step 1:

[0960] Users enter their diet and exercise habits into a dedicated mobile app.

[0961] Input: Dietary information, exercise amount

[0962] Output: Data entered into the dedicated app

[0963] Step 2:

[0964] The device sends the habit information and genetic information entered into a dedicated app to the Google Firebase cloud server.

[0965] Input: Habit information, genetic information

[0966] Output: Data sent to the cloud server

[0967] Step 3:

[0968] The server processes the data using a Hadoop cluster to analyze the user's data, and then generates health improvement recommendations using a generative AI model based on the processed data.

[0969] Input: Habit information, genetic information

[0970] Data calculation: Data analysis and generation of improvement proposals using generative AI models

[0971] Output: Health improvement plan

[0972] Step 4:

[0973] The health improvement suggestions generated by the server are sent to a dedicated app and presented to the user.

[0974] Input: Health Improvement Suggestion

[0975] Output: Improvement suggestions displayed in the dedicated app

[0976] Assisting doctors in diagnosis

[0977] Step 1:

[0978] The user enters into the dedicated app, "I have persistent abdominal pain. I have already uploaded my past medical data."

[0979] Input: Symptom data, past medical data

[0980] Output: Data entered into the dedicated app

[0981] Step 2:

[0982] The device sends the entered symptom data and past medical data to the Google Firebase cloud server.

[0983] Input: Symptom data, past medical data

[0984] Output: Data sent to the cloud server

[0985] Step 3:

[0986] The server analyzes the received data using natural language processing technology to predict possible diseases, and then organizes the predicted diseases and recommended treatments.

[0987] Input: Symptom data, past medical data

[0988] Data Computing: Analysis and Disease Prediction with Natural Language Processing

[0989] Output: predicted disease, recommended treatment

[0990] Step 4:

[0991] The server sends the diagnosis results and recommended treatments to medical professionals via an information delivery service and displays them on a dashboard.

[0992] Input: predicted disease, recommended treatment

[0993] Output: Diagnosis and treatment presented to medical professionals

[0994] (Application example 1)

[0995] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0996] In modern factories, there is a lack of means to monitor the health status and work safety of workers in real time and immediately report any abnormalities. This increases the risk of worker health risks and the likelihood of accidents. Furthermore, there is a lack of effective real-time data collection and analysis to support doctors' diagnoses. There is a need to provide a system that can resolve this situation and efficiently manage the safety of the work environment and the health of workers.

[0997] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0998] In this invention, the server includes means for collecting patient voice data and converting the voice data into text data, means for collecting patient video data and using the video data for personal identification, means for inputting, integrating, and analyzing health data, means for displaying the analysis results on a dashboard, means for generating and presenting optimal lifestyle improvement plans for each patient, means for presenting possible diseases and recommended treatments to assist doctors in making diagnoses, means for collecting worker health data in real time, means for immediately issuing an alert if an abnormality is detected, means for collecting and analyzing work environment data, and means for monitoring work safety based on the analyzed data. This enables real-time monitoring of worker health conditions and work environment safety and rapid notification of abnormalities.

[0999] The "means for collecting patient voice data and converting the voice data into text data" is a technology for acquiring the patient's voice and converting the voice into text information.

[1000] The "means for collecting patient image data and using the image data for personal identification" refers to a technology for acquiring patient images and identifying individuals from the images.

[1001] "Means for inputting, integrating, and analyzing health data" refers to technology for inputting health information such as blood pressure and body temperature into a system and integrating and analyzing it.

[1002] The "means for displaying the analysis results on a dashboard" is an interface for visually displaying the analyzed health information.

[1003] "Means for generating and presenting optimal lifestyle improvement proposals for individual patients" refers to technology for automatically creating and presenting health improvement proposals tailored to each patient.

[1004] "Means for presenting possible diseases and recommended treatments to assist doctors in diagnosis" refers to technology that uses AI and data analysis to provide possible diseases and recommended treatments to assist in diagnosis.

[1005] "Means for collecting worker health data in real time" refers to technology for obtaining health information on workers working in factories in real time.

[1006] "Means for immediately issuing an alert in the event of an abnormality" refers to technology that immediately issues a warning when an abnormality is detected in health data or work environment data.

[1007] "Means for collecting and analyzing work environment data" refers to technology for acquiring data related to the work environment within a factory and analyzing it.

[1008] "Means for monitoring work safety based on analytical data" refers to technology for constantly monitoring work safety based on analyzed data.

[1009] This invention is a system for improving the health management and work safety of factory workers. This system uses technology to collect, analyze, and display audio, video, and health data in real time.

[1010] Specifically, the system includes the following means:

[1011] Sensor data collection and management

[1012] User:

[1013] Report your health and symptoms using a voice input device.

[1014] A camera device is used to collect facial images.

[1015] Health data is entered using devices such as blood pressure monitors and thermometers.

[1016] Device:

[1017] Voice data is collected and converted into text data using voice recognition technology.

[1018] Video data is collected and workers are identified using personal identification technology.

[1019] Health data is collected, associated with the worker's profile, and sent to a cloud server.

[1020] server:

[1021] The transmitted data is consolidated and analyzed.

[1022] Calculates the worker's health index and issues an alert if any abnormalities are detected.

[1023] The results are displayed on a dashboard and notified to the work team leader as needed.

[1024] Improving health management and work safety

[1025] User:

[1026] Enter information about your daily habits (e.g., diet, exercise, etc.) into a dedicated app.

[1027] Provide work environment data as needed.

[1028] Device:

[1029] The input habit information and work environment data are sent to the cloud server.

[1030] server:

[1031] Analyze habit information and work environment data.

[1032] Generate optimal health improvement proposals and safety measures.

[1033] Improvement suggestions are presented to users via dashboards and notification functions.

[1034] Specific examples

[1035] Examples of biometric data self-management support:

[1036] 1. The user speaks into a voice input device, "I'd like to enter my blood pressure record for this week."

[1037] 2. The device collects the voice data and sends it to the cloud server.

[1038] 3. The server analyzes the data, generates a graph of blood pressure fluctuations for this week, and displays it on the device.

[1039] Examples of health improvement suggestions:

[1040] 1. The user types "Enter recent food and exercise records" into the dedicated app.

[1041] 2. The device sends the input information to the cloud server.

[1042] 3. The server proposes the optimal meal plan and exercise method based on the user's data and displays it on the device.

[1043] Hardware and software used

[1044] Hardware:

[1045] Smart glasses camera: Used to collect video data.

[1046] Microphone in smart glasses: Used to collect voice data.

[1047] Health monitoring devices: thermometers, blood pressure monitors, etc.

[1048] software:

[1049] Python: The main programming language.

[1050] OpenCV: A library for video data analysis.

[1051] SpeechRecognition: A library for analyzing speech data.

[1052] Cloud server: Stores and analyzes health data.

[1053] Example prompt for a generative AI model:

[1054] A worker speaks to the smart glasses and says, "Please check my health." The smart glasses monitor the worker's body temperature, blood pressure, complexion, and other conditions in real time, sending the data to the cloud. Cloud AI analyzes the data, and if any abnormalities are detected, an alert is sent to the work team leader's smartphone.

[1055] By using this system, it is expected that health management and safety assurance for workers in factories will be automated and carried out efficiently.

[1056] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1057] Program processing flow of the system that realizes the application example

[1058] Step 1:

[1059] The user speaks to the smart glasses, saying, "Check your health."

[1060] Specific behavior:

[1061] Input: Audio data

[1062] Output: Raw audio data

[1063] The user speaks into the smart glasses, and the microphone in the glasses collects the voice data, which is then converted into text data using voice recognition technology.

[1064] Step 2:

[1065] The device sends the voice data to the cloud server.

[1066] Specific behavior:

[1067] Input: Audio data

[1068] Output: Text data

[1069] The device (smart glasses) sends the collected voice data to a cloud server, where it is converted into text data using voice recognition technology and then sent.

[1070] Step 3:

[1071] Collect user health data (body temperature, blood pressure).

[1072] Specific behavior:

[1073] Input: Data from health monitoring devices

[1074] Output: Health data (body temperature, blood pressure)

[1075] A user uses a health monitoring device to measure their body temperature and blood pressure, and the data is sent to a cloud server via smart glasses.

[1076] Step 4:

[1077] The server integrates and analyzes the transmitted audio data, health data, and video data.

[1078] Specific behavior:

[1079] Input: Audio data, health data, video data

[1080] Output: Analysis results

[1081] The server integrates the various data sent and analyzes it using machine learning models or data analysis algorithms, using Python or dedicated analysis libraries.

[1082] Step 5:

[1083] The server displays the analysis results on a dashboard and issues an alert if an abnormality is detected.

[1084] Specific behavior:

[1085] Input: Analysis results

[1086] Output: Dashboard display, alert notification

[1087] The server updates the dashboard based on the analysis results and displays them visually. If an abnormality is detected, an alert is sent to the team leader's smartphone in real time.

[1088] Step 6:

[1089] The server generates health improvement suggestions based on the worker's habit information and presents them on the terminal.

[1090] Specific behavior:

[1091] Input: Habit information, analysis results

[1092] Output: Health improvement plan

[1093] The server analyzes the user's habit information and health data and generates health improvement suggestions, which are presented as suggestions to the user's smart glasses or associated devices.

[1094] In this way, the system collects, analyzes, displays, and notifies data at each step, monitoring worker health and the safety of the work environment in real time, enabling rapid response when an abnormality occurs.

[1095] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1096] The present invention aims to incorporate an emotion engine into a medical management system that utilizes multimodal AI technology, thereby making diagnoses and suggestions that reflect the patient's emotions, reducing the burden on medical professionals, improving the accuracy of diagnoses, and providing lifestyle improvement suggestions tailored to the needs of each individual patient.

[1097] 1. Collection and management of biometric and emotional data

[1098] User:

[1099] Report your health and symptoms using a voice input device.

[1100] A camera device is used to collect facial images.

[1101] Health data is entered using devices such as blood pressure monitors and thermometers.

[1102] Device:

[1103] Voice data is collected and converted into text data using voice recognition technology.

[1104] Video data is collected and individuals are identified using facial recognition technology.

[1105] Health data is collected, associated with the user's profile, and sent to a cloud server.

[1106] Emotions are recognized from audio data and video data, and emotion data is generated.

[1107] server:

[1108] The transmitted audio data, video data, health data, and emotional data are integrated and analyzed.

[1109] Calculates the user's health and emotional indicators and displays the results on a dashboard.

[1110] 2. Generating and Proposing Health Improvement Plans Using Emotional Data

[1111] User:

[1112] Enter information about your daily habits (e.g., diet, exercise, etc.) into a dedicated app.

[1113] Provide genetic information as needed.

[1114] Device:

[1115] The entered habit information and genetic information is sent to a cloud server.

[1116] server:

[1117] The user's habits, genetic information, and emotional data are analyzed.

[1118] Based on emotional data, optimal health improvement proposals are generated, including meal plans and exercise methods.

[1119] Improvement suggestions are presented to users via dashboards and notification functions.

[1120] 3. Assisting doctors in diagnosis and utilizing emotion data

[1121] User:

[1122] Enter your symptoms of illness by voice or text.

[1123] Provide historical medical data.

[1124] Device:

[1125] The entered symptom data and past medical data are sent to a cloud server.

[1126] server:

[1127] Analyzes voice and text data to predict possible diseases.

[1128] Compilation of supplementary information to improve diagnostic accuracy is carried out by comparing it with past medical data and referring to emotional data.

[1129] Present the diagnosis and recommended treatment to the physician.

[1130] Specific examples

[1131] Examples of biometric and emotional data self-management support:

[1132] 1. The user speaks into a voice input device, "I'd like to enter my blood pressure record for this week."

[1133] 2. The device collects the voice data and sends it to the cloud server.

[1134] 3. The device recognizes emotions from the voice data and generates emotion data.

[1135] 4. The server analyzes the data, generates a graph of blood pressure fluctuations for this week, and displays it on the device along with emotional data.

[1136] Example of using emotion data to provide health improvement suggestions:

[1137] 1. The user types "Enter recent food and exercise records" into the dedicated app.

[1138] 2. The device sends the input information to the cloud server.

[1139] 3. The server proposes optimal meal plans and exercise methods based on the user's data, and displays them on the device, reflecting the user's emotional data.

[1140] Examples of using emotion data to assist doctors in diagnosis:

[1141] 1. The user enters the following into the dedicated app: "I have persistent abdominal pain. I have already uploaded my past medical data."

[1142] 2. The device sends the entered data to the cloud server.

[1143] 3. The server analyzes the symptoms, lists possible illnesses based on the emotional data, and presents recommended treatments to the doctor.

[1144] By automating and streamlining these processes, the system of the present invention reduces the burden on medical professionals, improves diagnostic accuracy, and significantly contributes to eliminating disparities in medical care. Furthermore, by using emotion data, comprehensive health management that takes into account the patient's psychological state becomes possible.

[1145] The processing flow will be explained below.

[1146] (Collection and management of biometric and emotional data)

[1147] Audio data recording and conversion

[1148] Step 1:

[1149] The user speaks about their physical condition and symptoms into a voice input device.

[1150] Step 2:

[1151] The terminal receives the audio signal and temporarily stores it as audio data.

[1152] Step 3:

[1153] The terminal uses voice recognition technology to convert the voice data into text data.

[1154] Step 4:

[1155] The terminal transmits the converted text data to the cloud server.

[1156] Video data recording and facial recognition

[1157] Step 1:

[1158] The user faces the camera.

[1159] Step 2:

[1160] The device collects and temporarily stores video data.

[1161] Step 3:

[1162] The device runs a facial recognition algorithm to identify the individual.

[1163] Step 4:

[1164] The device transmits the video data and identification information to the cloud server.

[1165] Generating emotion data

[1166] Step 1:

[1167] The terminal recognizes emotions from audio data and video data.

[1168] Step 2:

[1169] The device generates the recognized emotion data and temporarily stores it.

[1170] Step 3:

[1171] The device transmits the emotion data to the cloud server.

[1172] Health Data Collection and Transmission

[1173] Step 1:

[1174] Users enter health data such as blood pressure and body temperature into a dedicated app or device.

[1175] Step 2:

[1176] The terminal associates the entered health data with the user's profile and temporarily stores it.

[1177] Step 3:

[1178] The device sends the health data to a cloud server.

[1179] Data integration and analysis

[1180] Step 1:

[1181] The server integrates the received audio data, video data, health data, and emotional data.

[1182] Step 2:

[1183] The server calculates health and emotional indicators using a proprietary algorithm.

[1184] Step 3:

[1185] The server converts health and emotional indicators into graphs and statistical information to generate a dashboard.

[1186] Step 4:

[1187] The server sends the dashboard and result display data to the terminal.

[1188] (Providing health improvement ideas)

[1189] Entering daily habits

[1190] Step 1:

[1191] The user inputs information about their daily habits, such as what they eat and how much exercise they do, into a dedicated app.

[1192] Step 2:

[1193] The device organizes the input habit information by associating it with the user's profile.

[1194] Step 3:

[1195] The device sends the organized habit information to a cloud server.

[1196] Analysis of habit information and genetic information

[1197] Step 1:

[1198] The server integrates the received habit information with the genetic information provided by the user.

[1199] Step 2:

[1200] The server uses statistical and genetic techniques to generate optimal health improvement recommendations.

[1201] Customization using emotional data

[1202] Step 1:

[1203] The server customizes the generated health improvement proposals by reflecting emotional data.

[1204] Proposal of improvement proposals

[1205] Step 1:

[1206] The server then compiles the generated health improvement suggestions into a display format on a dashboard for the user.

[1207] Step 2:

[1208] The server sends the improvement proposal data to the terminal.

[1209] Step 3:

[1210] The improvement suggestions received by the device are displayed on the user's dashboard and an alert is sent via a notification function.

[1211] (Assisting doctors in diagnosis)

[1212] Enter symptoms

[1213] Step 1:

[1214] The user enters their symptoms of illness into a dedicated app by voice or text.

[1215] Step 2:

[1216] The terminal converts the input symptom data into text data and temporarily stores it.

[1217] Step 3:

[1218] The device sends the text data to the cloud server.

[1219] Referencing past medical data

[1220] Step 1:

[1221] The server looks up past medical data from the user's profile.

[1222] Step 2:

[1223] The server analyzes the historical data and the new symptom data.

[1224] Diagnosis and treatment suggestions

[1225] Step 1:

[1226] The server uses AI algorithms to predict possible diseases.

[1227] Step 2:

[1228] The server compares the data with past medical data and uses emotional data to organize supplementary information to improve diagnostic accuracy.

[1229] Step 3:

[1230] The server compiles the diagnosis and recommended treatment in the form of a report for the physician.

[1231] Step 4:

[1232] The server sends the report to the device.

[1233] Step 5:

[1234] The device receives a diagnosis and displays treatment suggestions on a screen for the doctor.

[1235] Example 2

[1236] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1237] The current medical system does not take patients' emotions into consideration when making diagnoses or recommendations, which increases the burden on medical professionals and results in insufficient diagnostic accuracy. Furthermore, there is a lack of systems that provide lifestyle improvement suggestions tailored to individual patients' needs, making it difficult for patients to manage their own health. Furthermore, there is a need for improved data analysis and disease prediction accuracy to assist doctors in their diagnoses.

[1238] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1239] In this invention, the server includes: means for collecting a patient's voice data and converting the voice data into text data; means for collecting a patient's video data and using the video data for personal identification; means for inputting health and emotional data and transmitting it to a cloud server; means for recognizing emotions from the voice and video data and generating emotional data; means for integrating and analyzing the transmitted voice, video, health, and emotional data; means for displaying the analysis results on a dashboard; means for generating and presenting optimal lifestyle improvement proposals for each patient; and means for presenting possible diseases and recommended treatments using a generative AI model to assist physicians in their diagnosis. This enables diagnoses and proposals that take the patient's emotions into account, reducing the burden on medical professionals and improving diagnostic accuracy. Furthermore, providing lifestyle improvement proposals tailored to each patient's needs makes it easier for patients to manage their own health.

[1240] The "means for collecting patient voice data and converting the voice data into text data" is a mechanism for collecting voice data reported by the patient through a voice input device and converting the voice data into text format.

[1241] The "means for collecting patient image data and using the image data for personal identification" is a mechanism for acquiring an image of the patient's face and using the image data to identify the individual.

[1242] "Means for inputting health data and emotional data and transmitting it to a cloud server" refers to a mechanism for inputting information about a patient's health and emotions and transmitting that data to a server on the cloud.

[1243] The "means for recognizing emotions from audio data and video data and generating emotional data" refers to a mechanism for analyzing collected audio data and video data to recognize the patient's emotions and generate data related to emotions.

[1244] "Means for integrating and analyzing transmitted audio data, video data, health data, and emotional data" refers to a mechanism for integrating and comprehensively analyzing various data transmitted to the cloud server.

[1245] "Means for displaying analysis results on a dashboard" refers to a mechanism for outputting the results of the analyzed data to a dashboard for visually displaying them.

[1246] "A means for generating and presenting optimal lifestyle improvement proposals for individual patients" is a system that generates optimal lifestyle improvement proposals based on data on each patient and presents them to the patient.

[1247] "Means of using generative AI models to suggest possible diseases and recommended treatments to assist doctors in their diagnoses" refers to a system that uses generative AI technology to predict possible diseases from collected data and provide doctors with optimal treatments.

[1248] This invention is a medical management system that utilizes multimodal AI technology and incorporates an emotion engine to make diagnoses and recommendations that reflect the patient's emotions, reducing the burden on medical professionals. The system also aims to improve the accuracy of diagnoses and provide lifestyle improvement proposals tailored to the individual needs of each patient.

[1249] The system consists of three main components: users, terminals, and servers.

[1250] User

[1251] A user reports their physical condition and symptoms using a voice input device. A camera device is also used to collect facial images, and health data is input using devices such as a blood pressure monitor and a thermometer. As a specific example, data collection begins when a user speaks into the voice input device, saying, "I'd like to enter this week's blood pressure record."

[1252] Terminal

[1253] The terminal uses multiple pieces of hardware and software to collect and process data.

[1254] The audio data is converted into text data using the Google Cloud Speech-to-Text service.

[1255] Video data is collected and individuals are identified using the Microsoft Azure Face API.

[1256] Health data is sent to AWS cloud servers.

[1257] Emotions are recognized from audio and video data using IBM Watson's Natural Language Understanding and Visual Recognition, and emotion data is generated.

[1258] For example, when a user types "I would like to enter my recent meal and exercise records" into a dedicated app, that information is sent to the cloud server.

[1259] server

[1260] The server is the central component that processes data on the cloud.

[1261] The audio data, video data, health data, and emotion data sent to the cloud server are integrated and analyzed using data processing libraries such as Python's Pandas and NumPy.

[1262] Calculates user health and emotional indicators and displays the results on a dashboard using Tableau or Power BI.

[1263] For example, it can recognize emotions based on voice data and evaluate stress levels to provide comprehensive health management. It also uses GPT-4 as a generative AI model to generate optimal health improvement proposals and suggest optimal diet plans and exercise methods for users.

[1264] Specific examples

[1265] Examples of biometric and emotional data self-management support:

[1266] The user speaks to the voice input device, saying, "I'd like to enter my blood pressure record for this week." The device then collects the voice data and sends it to a cloud server using the Google Cloud Speech-to-Text service. The device then uses IBM Watson's Natural Language Understanding to recognize emotions from the voice data and generate emotion data. The server then analyzes this data, generates a graph of blood pressure fluctuations for this week, and displays it on a Tableau dashboard along with the emotion data.

[1267] Example of using emotion data to provide health improvement suggestions:

[1268] The user enters text data into the dedicated app, saying, "I'd like to enter my recent meal and exercise records." The device then sends the input information to an AWS cloud server. The server generates an optimal meal plan and exercise method based on the user's data, and uses GPT-4 to create suggestions that also reflect emotional data, which are then displayed on the device.

[1269] Examples of using emotion data to assist doctors in diagnosis:

[1270] The user enters into the dedicated app, "I have persistent stomach pain. I have already uploaded my past medical data." The device then sends the entered data to an AWS cloud server. The server analyzes the symptoms, lists possible illnesses based on the emotional data, and generates a recommended treatment using GPT-4 to present to the doctor.

[1271] By automating and streamlining these processes, this system reduces the burden on medical professionals and improves the accuracy of diagnoses. It also enables comprehensive health management based on emotional data, providing patients with optimal treatments and lifestyle improvement suggestions.

[1272] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1273] Step 1:

[1274] The user reports their physical condition and symptoms using a voice input device.

[1275] Input: User voice input (e.g., "Enter this week's blood pressure record.")

[1276] Data processing: The device collects voice data and converts it into text data using the Google Cloud Speech-to-Text service.

[1277] Output: Report content as text data (e.g., "Enter your blood pressure record for this week")

[1278] Step 2:

[1279] A user uses a camera device to collect facial images.

[1280] Input: User's face image

[1281] Data processing: The device collects video data and uses the Microsoft Azure Face API to perform facial recognition and recognize emotions from the video data.

[1282] Output: Identified person ID and emotion data

[1283] Step 3:

[1284] A user inputs health data using devices such as a blood pressure monitor and a thermometer.

[1285] Input: Health data (e.g. blood pressure, body temperature)

[1286] Data processing: The device collects health data and sends it to AWS cloud servers.

[1287] Output: Health data stored on a cloud server

[1288] Step 4:

[1289] Sending audio and video data to the cloud

[1290] Input: Text data, identified personal ID, emotion data

[1291] Data processing: The device integrates the audio and video data to generate emotion data, which is then sent to the AWS cloud server.

[1292] Output: Integrated data sent to cloud server

[1293] Step 5:

[1294] The audio data, video data, health data, and emotional data sent to the cloud server are integrated and analyzed.

[1295] Input: Audio data, video data, health data, and emotion data stored on a cloud server

[1296] Data processing: The server integrates this data and analyzes it using data processing libraries such as Python's Pandas and NumPy.

[1297] Output: Integrated data analysis results (e.g., health index, emotional index)

[1298] Step 6:

[1299] Display the analysis results on a dashboard.

[1300] Input: Integrated data analysis results

[1301] Data processing: The server visualizes and displays the analysis results on a dashboard using Tableau or Power BI.

[1302] Output: Dashboard screen viewable by users and doctors

[1303] Step 7:

[1304] The user enters information about their daily habits into a dedicated app.

[1305] Input: User's habit information (e.g., dietary content, amount of exercise)

[1306] Data processing: The terminal sends the input data to the AWS cloud server.

[1307] Output: Habit information stored on the cloud server

[1308] Step 8:

[1309] The server generates and presents health improvement suggestions based on the user's habit information, health data, and emotional data.

[1310] Input: Habit information, health data, emotional data

[1311] Data processing: The server analyzes these data and generates optimal health improvement recommendations using a generative AI model (GPT-4).

[1312] Output: Generated health improvement suggestions, including recommendations displayed in notifications and on the dashboard

[1313] Step 9:

[1314] Providing data to assist doctors in diagnosis

[1315] Input: User's symptom information, past medical data

[1316] Data processing: The device sends the input data to the AWS cloud server, which analyzes the data and uses GPT-4 to list possible diseases based on the emotional data.

[1317] Output: A list of diagnoses and treatment recommendations presented to the doctor

[1318] At each step, the system processes and analyzes the collected data to provide optimal improvement suggestions for each individual user and diagnostic support information for doctors, thereby realizing comprehensive health management that takes the patient's emotions into consideration.

[1319] (Application example 2)

[1320] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1321] Conventional medical management systems provide diagnoses and treatment recommendations based on biometric data and past medical data, but they do not take into account the patient's emotions or psychological state, making it difficult to fully meet the needs of individual patients. Furthermore, food delivery services, in particular, do not propose optimal meal plans that reflect the user's health status and emotions, making it difficult to provide individually customized services. Therefore, new methods are needed to increase patient and user satisfaction, reduce the burden on medical professionals, and improve diagnostic accuracy.

[1322] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1323] In this invention, the server includes means for collecting patient voice data and converting the voice data into text data, means for collecting patient video data and using the video data for personal identification, means for inputting, integrating, and analyzing health data, means for displaying the analysis results on a dashboard, means for generating and presenting optimal lifestyle improvement plans for each patient, means for presenting possible diseases and recommended treatments to assist doctors in making diagnoses, means for collecting user emotion data and analyzing the emotion data to reflect it in health improvement plans, and means for inputting health reports by voice or text and proposing and delivering optimal meal plans. This enables personalized medical management and food delivery services that comprehensively consider the emotions and health status of patients and users.

[1324] "Voice data" refers to data for collecting voice information from patients and converting the content into text data.

[1325] "Video data" refers to data that collects images of a patient's face or body and is used to identify the individual.

[1326] "Health data" refers to physiological information such as a patient's blood pressure, body temperature, and heart rate, and is data that can be integrated and analyzed.

[1327] A "dashboard" is an interface for visually displaying information such as analysis results and proposals.

[1328] A "lifestyle improvement plan" is a specific action plan proposed to improve the quality of daily life of patients or users.

[1329] "Aiding diagnosis" means providing supplementary information or recommendations to help a doctor make an appropriate diagnosis of a patient's symptoms.

[1330] "Disease" refers to an illness or condition that adversely affects a patient's health.

[1331] A "therapy" is a specific medical procedure or method used to treat a disease.

[1332] "Emotional data" refers to data about the user's psychological state and emotions extracted from facial expressions, etc.

[1333] "Health report" refers to the act of a patient or user recording or reporting their current health condition or symptoms.

[1334] "Meal Plan" refers to a meal menu and plan suggested based on the user's health condition and mood.

[1335] "Delivery" refers to the act of delivering cooked meals based on the proposed meal plan to the user.

[1336] This invention is a system that collects and analyzes biometric and emotional data of patients and users, and provides optimal lifestyle improvement proposals and health management plans based on the collected data. The system aims to provide individually customized services that comprehensively consider the health status and emotions of patients and users.

[1337] 1. Collection and management of biometric and emotional data

[1338] User:

[1339] Users report their physical condition and symptoms using a voice input device, which is collected by the terminal as voice data. A camera device is also used to collect facial images. Furthermore, health data is input using devices such as a blood pressure monitor and a thermometer.

[1340] Device:

[1341] The device collects voice data and converts it into text data using voice recognition technology. It also collects video data and identifies individuals using facial recognition technology. The collected health data is associated with the user's profile and sent to a cloud server. At the same time, emotions are recognized from the voice and video data, and emotion data is generated.

[1342] server:

[1343] The server integrates and analyzes the transmitted audio, video, health, and emotional data, calculates the user's health and emotional indices, and displays the results on a dashboard.

[1344] 2. Generating and Proposing Health Improvement Plans Using Emotional Data

[1345] User:

[1346] Users input information about their daily habits (e.g., dietary habits, amount of exercise, etc.) into a dedicated app, and also provide genetic information if necessary.

[1347] Device:

[1348] The terminal transmits the input habit information and genetic information to a cloud server.

[1349] server:

[1350] The server analyzes the user's habits, genetic information, and emotional data. Based on the emotional data, it generates optimal health improvement proposals, suggesting meal plans and exercise methods to the user. The proposals are presented to the user via a dashboard and notification function.

[1351] 3. Assisting doctors in diagnosis and utilizing emotion data

[1352] User:

[1353] Users can input their symptoms of illness by voice or text, and also provide past medical data.

[1354] Device:

[1355] The terminal transmits the input symptom data and past medical data to a cloud server.

[1356] server:

[1357] The server analyzes the voice and text data to predict possible illnesses, compares it with past medical data, and uses emotional data to organize supplementary information to improve diagnostic accuracy. The diagnosis results and recommended treatments are then presented to the doctor.

[1358] Specific examples

[1359] Examples of biometric and emotional data self-management support:

[1360] 1. The user speaks into a voice input device, "I'd like to enter my blood pressure record for this week."

[1361] 2. The device collects the voice data and sends it to the cloud server.

[1362] 3. The device recognizes emotions from the voice data and generates emotion data.

[1363] 4. The server analyzes the data, generates a graph of blood pressure fluctuations for this week, and displays it on the device along with emotional data.

[1364] Example of using emotion data to provide health improvement suggestions:

[1365] 1. The user types "Enter recent food and exercise records" into the dedicated app.

[1366] 2. The device sends the input information to the cloud server.

[1367] 3. The server proposes optimal meal plans and exercise methods based on the user's data, and displays them on the device, reflecting the user's emotional data.

[1368] Examples of using emotion data to assist doctors in diagnosis:

[1369] 1. The user enters the following into the dedicated app: "I have persistent abdominal pain. I have already uploaded my past medical data."

[1370] 2. The device sends the entered data to the cloud server.

[1371] 3. The server analyzes the symptoms, lists possible illnesses based on the emotional data, and presents recommended treatments to the doctor.

[1372] Prompt Sentence Examples

[1373] "Please suggest personalized food delivery based on the user's emotional and health data. Here's the user's data for this week:

[1374] Blood pressure: 110

[1375] Temperature: 36.5

[1376] Emotion: neutral

[1377] Please suggest a suitable meal plan and why.”

[1378] By building a system in this way, it is possible to comprehensively manage a user's health condition and emotions, and realize personalized medical services and food delivery.

[1379] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1380] Step 1:

[1381] A user reports their physical condition and symptoms using a voice input device. This voice data is collected and input data from the voice input device is generated.

[1382] Step 2:

[1383] The device receives the voice data and converts it into text data using speech recognition technology (e.g., Google Speech Recognition API). The input is voice data, and the output is text data.

[1384] Step 3:

[1385] A user uses a camera device to capture a video of his or her face, generating video data that is provided to a terminal as a video input.

[1386] Step 4:

[1387] The device receives the video data and identifies individuals using facial recognition technology (e.g., using OpenCV and dlib). The input is the video data, and the output is the data of the identified individuals.

[1388] Step 5:

[1389] A user inputs health data using a device such as a blood pressure monitor or a thermometer, and the data is sent to a terminal. The input is health data (blood pressure, body temperature, etc.).

[1390] Step 6:

[1391] The device associates the collected health data with the user's profile and transmits it to the cloud server. The input is the health data, and the output is the integrated health data transmitted to the cloud server.

[1392] Step 7:

[1393] The device recognizes emotions from audio and video data and generates emotion data (e.g., extracts emotions using the FER library). The input is audio and video data, and the output is emotion data.

[1394] Step 8:

[1395] The server integrates and analyzes the audio data, video data, health data, and emotion data sent to the cloud server. These data are the input, and the output is the health index and emotion index as the analysis results.

[1396] Step 9:

[1397] The server displays the analysis results on a dashboard, which includes graphs and indicators to visualize the user's health and emotional state. The input is the analysis results, and the output is the dashboard display.

[1398] Step 10:

[1399] The user inputs information about their daily habits (such as dietary habits and exercise amount) into a dedicated app. The input is habit information, which is then provided to the device.

[1400] Step 11:

[1401] The device sends habit information to the cloud server. The input is the habit information, and the output is the data sent to the cloud server.

[1402] Step 12:

[1403] The server generates optimal health improvement proposals based on emotional data and habit information. It also suggests meal plans and exercise methods. The input is emotional data and habit information, and the output is improvement proposals.

[1404] Step 13:

[1405] The server generates improvement proposals and presents them to the user via a dashboard or notification function. The input is the improvement proposal, and the output is a notification to the user.

[1406] Step 14:

[1407] The user inputs symptoms of illness by voice or text and provides this to a dedicated app. The input is symptom data, which is then sent to the device.

[1408] Step 15:

[1409] The device sends symptom data and past medical data to the cloud server. The input is the symptom data and medical data, and the output is the data sent to the cloud server.

[1410] Step 16:

[1411] The server analyzes the symptom data, predicts possible diseases based on the emotion data, and then organizes recommended treatments. The input is the symptom data and emotion data, and the output is a list of diseases and recommended treatments.

[1412] Step 17:

[1413] The server presents the diagnosis results and recommended treatment to the doctor. The input is the analysis results, and the output is a notification to the doctor.

[1414] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1415] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1416] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1417] [Third embodiment]

[1418] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1419] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1420] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1421] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1422] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1423] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1424] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1425] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1426] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1427] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1428] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1429] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1430] The present invention is a medical management system that utilizes multimodal AI technology and is designed to resolve the labor shortage and medical disparities in medical settings. Specific embodiments of the present invention are described below.

[1431] 1. Biometric data collection and management

[1432] User:

[1433] Report your health and symptoms using a voice input device.

[1434] A camera device is used to collect facial images.

[1435] Health data is entered using devices such as blood pressure monitors and thermometers.

[1436] Device:

[1437] Voice data is collected and converted into text data using voice recognition technology.

[1438] Video data is collected and individuals are identified using facial recognition technology.

[1439] Health data is collected, associated with the user's profile, and sent to a cloud server.

[1440] server:

[1441] The transmitted audio data, video data, and health data are integrated and analyzed.

[1442] Calculates the user's health indicators and displays the results on a dashboard.

[1443] 2. Generate and propose health improvement plans

[1444] User:

[1445] Enter information about your daily habits (e.g., diet, exercise, etc.) into a dedicated app.

[1446] Provide genetic information as needed.

[1447] Device:

[1448] The entered habit information and genetic information is sent to a cloud server.

[1449] server:

[1450] The user's habits and genetic information are analyzed.

[1451] Generate optimal health improvement recommendations and suggest meal plans, exercise methods, etc.

[1452] Improvement suggestions are presented to users via dashboards and notification functions.

[1453] 3. Assisting doctors in diagnosis

[1454] User:

[1455] Enter your symptoms of illness by voice or text.

[1456] Provide historical medical data.

[1457] Device:

[1458] The entered symptom data and past medical data are sent to a cloud server.

[1459] server:

[1460] Analyzes voice and text data to predict possible diseases.

[1461] Compile supplementary information to improve diagnostic accuracy by referring to past medical data.

[1462] Present the diagnosis and recommended treatment to the physician.

[1463] Specific examples

[1464] Examples of biometric data self-management support:

[1465] 1. The user speaks into a voice input device, "I'd like to enter my blood pressure record for this week."

[1466] 2. The device collects the voice data and sends it to the cloud server.

[1467] 3. The server analyzes the data, generates a graph of blood pressure fluctuations for this week, and displays it on the device.

[1468] Examples of health improvement suggestions:

[1469] 1. The user types "Enter recent food and exercise records" into the dedicated app.

[1470] 2. The device sends the input information to the cloud server.

[1471] 3. The server proposes the optimal meal plan and exercise method based on the user's data and displays it on the device.

[1472] Examples of diagnostic assistance for doctors:

[1473] 1. The user enters the following into the dedicated app: "I have persistent abdominal pain. I have already uploaded my past medical data."

[1474] 2. The device sends the entered data to the cloud server.

[1475] 3. The server analyzes the symptoms, lists possible illnesses, and recommends treatments for the doctor.

[1476] By automating and streamlining these processes, the system of the present invention reduces the burden on medical professionals, improves diagnostic accuracy, and significantly contributes to eliminating disparities in medical care.

[1477] The processing flow will be explained below.

[1478] (Biometric data self-management support)

[1479] Audio data recording and conversion

[1480] Step 1:

[1481] The user speaks about their physical condition and symptoms into a voice input device.

[1482] Step 2:

[1483] The terminal receives the audio signal and temporarily stores it as audio data.

[1484] Step 3:

[1485] The terminal uses voice recognition technology to convert the voice data into text data.

[1486] Step 4:

[1487] The terminal transmits the converted text data to the cloud server.

[1488] Video data recording and facial recognition

[1489] Step 1:

[1490] The user faces the camera.

[1491] Step 2:

[1492] The device collects and temporarily stores video data.

[1493] Step 3:

[1494] The device runs a facial recognition algorithm to identify the individual.

[1495] Step 4:

[1496] The device transmits the video data and identification information to the cloud server.

[1497] Health Data Collection and Transmission

[1498] Step 1:

[1499] Users enter health data such as blood pressure and body temperature into a dedicated app or device.

[1500] Step 2:

[1501] The terminal associates the entered health data with the user's profile and temporarily stores it.

[1502] Step 3:

[1503] The device sends the health data to a cloud server.

[1504] Data integration and analysis

[1505] Step 1:

[1506] The server integrates the received audio data, video data, and health data.

[1507] Step 2:

[1508] The server calculates health indicators using a dedicated algorithm.

[1509] Step 3:

[1510] The server converts health indicators into graphs and statistical information and generates a dashboard.

[1511] Step 4:

[1512] The server sends the dashboard and result display data to the terminal.

[1513] (Providing health improvement ideas)

[1514] Entering daily habits

[1515] Step 1:

[1516] The user inputs information about their daily habits, such as what they eat and how much exercise they do, into a dedicated app.

[1517] Step 2:

[1518] The device organizes the input habit information by associating it with the user's profile.

[1519] Step 3:

[1520] The device sends the organized habit information to a cloud server.

[1521] Analysis of habit information and genetic information

[1522] Step 1:

[1523] The server integrates the received habit information with the genetic information provided by the user.

[1524] Step 2:

[1525] The server uses statistical and genetic techniques to generate optimal health improvement recommendations.

[1526] Proposal of improvement proposals

[1527] Step 1:

[1528] The server then compiles the generated health improvement suggestions into a display format on a dashboard for the user.

[1529] Step 2:

[1530] The server sends the improvement proposal data to the terminal.

[1531] Step 3:

[1532] The improvement suggestions received by the device are displayed on the user's dashboard and an alert is sent via a notification function.

[1533] (Assisting doctors in diagnosis)

[1534] Enter symptoms

[1535] Step 1:

[1536] The user enters their symptoms of illness into a dedicated app by voice or text.

[1537] Step 2:

[1538] The terminal converts the input symptom data into text data and temporarily stores it.

[1539] Step 3:

[1540] The device sends the text data to the cloud server.

[1541] Referencing past medical data

[1542] Step 1:

[1543] The server looks up past medical data from the user's profile.

[1544] Step 2:

[1545] The server analyzes the historical data and the new symptom data.

[1546] Diagnosis and treatment suggestions

[1547] Step 1:

[1548] The server uses AI algorithms to predict possible diseases.

[1549] Step 2:

[1550] The server compares the data with past medical data and organizes supplementary information to improve diagnostic accuracy.

[1551] Step 3:

[1552] The server compiles the diagnosis and recommended treatment in the form of a report for the physician.

[1553] Step 4:

[1554] The server sends the report to the device.

[1555] Step 5:

[1556] The device receives a diagnosis and displays treatment suggestions on a screen for the doctor.

[1557] Example 1

[1558] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1559] In today's medical field, increasing patient numbers and limited medical resources have led to serious problems such as labor shortages and disparities in medical care. In conventional medical systems, the collection and management of patients' health data, the generation of lifestyle improvement recommendations, and diagnostic assistance for doctors are not fully automated, and these processes require time and effort. Therefore, there is a need to build a system that can efficiently collect and analyze data and provide diagnostic assistance.

[1560] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1561] In this invention, the server includes means for collecting voice data and converting the voice data into text data, means for collecting video data and using the video data for personal identification, and means for inputting health data and integrating and analyzing the health data, thereby enabling efficient collection and analysis of patient data and assisting doctors in making diagnoses.

[1562] "Voice data" refers to data that records a user's spoken words or audio information and is stored in digital form.

[1563] "Text data" is voice data converted into text information, and is data used for natural language processing, etc.

[1564] "Video data" refers to image and video data captured by a camera or other video collection device.

[1565] "Personal identification" is the act of distinguishing a specific person from other people and identifying that person's characteristics and attributes.

[1566] "Health data" is information that indicates the user's health condition, and includes physiological data such as blood pressure and body temperature.

[1567] "Integration" is the act of bringing together multiple different data sets and managing and analyzing them as consistent information.

[1568] "Analysis" is the act of examining collected data in detail and drawing meaningful conclusions or results from that information.

[1569] A "display device" is a device for visually presenting analytical results or other information to a user or medical professional.

[1570] The "lifestyle improvement plan" includes specific suggestions and plans for improving the user's health condition and lifestyle habits.

[1571] "Diagnostic assistance" refers to the act of providing additional information or analytical results to assist medical professionals in making a diagnosis.

[1572] "Possible diseases" refers to diseases that are suspected to be possible based on the user's symptoms and data.

[1573] "Recommended treatments" include the treatment methods that are considered most appropriate for a particular disease or condition.

[1574] "User" refers to an individual who uses this system to provide health data.

[1575] "Health professionals" refers to people who work in the medical profession, such as doctors and nurses.

[1576] This invention is a medical management system that utilizes multimodal AI technology, designed to resolve the shortage of medical staff and healthcare disparities in the medical field. This system integrates and analyzes audio, video, and health data to generate optimal lifestyle improvement recommendations for users and provide information to assist medical professionals in making diagnoses.

[1577] Biometric data collection and management

[1578] User:

[1579] Use your voice input device (e.g., a microphone on your voice input device) to report that you are "feeling unwell."

[1580] Use your smartphone camera to collect facial footage.

[1581] Health data is measured using a blood pressure monitor and thermometer and entered into a dedicated app.

[1582] Device:

[1583] Collect audio data and convert it to text using the Google Cloud Speech-to-Text API.

[1584] Video data is collected using a smartphone camera and facial recognition and personal identification are performed using the Azure Face API.

[1585] Health data is entered into a dedicated app and sent to the Google Firebase cloud server.

[1586] server:

[1587] A Python data analysis script is run to integrate and analyze the transmitted audio, video, and health data.

[1588] Calculates user health indicators and displays the results on a dashboard created with Microsoft Power BI.

[1589] The results are sent to the user's dedicated app and displayed.

[1590] Generate and propose health improvement plans

[1591] User:

[1592] Enter your diet and exercise amount into a dedicated mobile app.

[1593] If necessary, genetic information (e.g., data from a genetic information service) will also be uploaded.

[1594] Device:

[1595] Habit information and genetic information entered into a dedicated app is sent to the Google Firebase cloud server.

[1596] server:

[1597] To analyze the user's data, the data is processed using a Hadoop cluster.

[1598] Based on user data, health improvement recommendations are generated using generative AI models (e.g., large-scale language models).

[1599] Suggested meal plans and exercise methods are sent to a dedicated app and presented to the user.

[1600] Assisting doctors in diagnosis

[1601] User:

[1602] Enter the following into the dedicated app: "I have persistent abdominal pain. I have already uploaded my past medical data."

[1603] Past medical data will be provided via a dedicated app (e.g., a medical data management app).

[1604] Device:

[1605] The entered symptom data and past medical data are sent to the Google Firebase cloud server.

[1606] server:

[1607] The received data is analyzed using natural language processing technology (e.g., natural language processing model) to predict possible diseases.

[1608] Compile supplementary information to improve diagnostic accuracy by referring to past medical data.

[1609] The diagnosis and recommended treatments are sent to medical professionals via an information communication service and displayed on the medical professional's dashboard.

[1610] Examples of specific examples and prompts

[1611] In the "Biometric Data Self-Management Support Example," a user speaks to a voice input device, saying, "I'm going to enter my blood pressure record for this week." The device collects the voice data and sends it to a cloud server using the Google Cloud Speech-to-Text API. The server analyzes the data and generates a graph of blood pressure fluctuations for this week, which is then displayed on the device.

[1612] Example prompt sentence:

[1613] "I have entered this week's blood pressure data. Please display the analysis results."

[1614] In the "Health Improvement Suggestion" example, the user types "Enter my recent meal and exercise records" into the dedicated app. The device then sends the input information to a cloud server. The server uses a generative AI model to propose optimal meal plans and exercise methods based on the user's data, and displays them on the device.

[1615] Example prompt sentence:

[1616] "Please suggest the best improvement plan based on my recent diet and exercise data."

[1617] In the "Doctor's Diagnosis Assistance Example," the user enters "I have persistent abdominal pain. I have already uploaded my past medical data" into a dedicated app. The device then sends the entered data to a cloud server. The server then analyzes the symptoms using a natural language processing model, lists possible illnesses, and presents recommended treatments to the medical professional.

[1618] Example prompt sentence:

[1619] "I would like some diagnostic support based on my abdominal pain symptoms and past medical data."

[1620] The system aims to integrate multiple data sources and reduce the burden on medical facilities through automation and efficiency.

[1621] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1622] Biometric data collection and management

[1623] Step 1:

[1624] A user speaks "I'm not feeling well" into a voice input device, which collects voice data.

[1625] Input: User's voice

[1626] Output: Audio data

[1627] Step 2:

[1628] The device sends the collected voice data to the Google Cloud Speech-to-Text API, converts it into text data, and sends the converted text data to a cloud server.

[1629] Input: Audio data

[1630] Data processing: Converting voice data into text

[1631] Output: Text data

[1632] Step 3:

[1633] A user collects a video of their face using the camera on their smartphone, thereby acquiring video data.

[1634] Input: User video

[1635] Output: Video data

[1636] Step 4:

[1637] The device uses the Azure Face API to identify individuals from the video data collected, and sends the results to a cloud server.

[1638] Input: Video data

[1639] Data processing: Personal identification of video data

[1640] Output: Personally Identifiable Data

[1641] Step 5:

[1642] Users measure their health data using a blood pressure monitor or thermometer and enter it into a dedicated app, which then registers the health data in the app.

[1643] Input: Health data

[1644] Output: Input health data

[1645] Step 6:

[1646] The device sends the health data entered into a dedicated app to the Google Firebase cloud server.

[1647] Input: Health data

[1648] Output: Health data sent to the cloud server

[1649] Step 7:

[1650] The server integrates the received audio, video, and health data and analyzes it using a Python data analysis script. The analysis results are displayed on a dashboard.

[1651] Input: Audio data, video data, health data

[1652] Data Computing: Data Integration and Analysis

[1653] Output: Analysis results displayed on a dashboard

[1654] Generate and propose health improvement plans

[1655] Step 1:

[1656] Users enter their diet and exercise habits into a dedicated mobile app.

[1657] Input: Dietary information, exercise amount

[1658] Output: Data entered into the dedicated app

[1659] Step 2:

[1660] The device sends the habit information and genetic information entered into a dedicated app to the Google Firebase cloud server.

[1661] Input: Habit information, genetic information

[1662] Output: Data sent to the cloud server

[1663] Step 3:

[1664] The server processes the data using a Hadoop cluster to analyze the user's data, and then generates health improvement recommendations using a generative AI model based on the processed data.

[1665] Input: Habit information, genetic information

[1666] Data calculation: Data analysis and generation of improvement proposals using generative AI models

[1667] Output: Health improvement plan

[1668] Step 4:

[1669] The health improvement suggestions generated by the server are sent to a dedicated app and presented to the user.

[1670] Input: Health Improvement Suggestion

[1671] Output: Improvement suggestions displayed in the dedicated app

[1672] Assisting doctors in diagnosis

[1673] Step 1:

[1674] The user enters into the dedicated app, "I have persistent abdominal pain. I have already uploaded my past medical data."

[1675] Input: Symptom data, past medical data

[1676] Output: Data entered into the dedicated app

[1677] Step 2:

[1678] The device sends the entered symptom data and past medical data to the Google Firebase cloud server.

[1679] Input: Symptom data, past medical data

[1680] Output: Data sent to the cloud server

[1681] Step 3:

[1682] The server analyzes the received data using natural language processing technology to predict possible diseases, and then organizes the predicted diseases and recommended treatments.

[1683] Input: Symptom data, past medical data

[1684] Data Computing: Analysis and Disease Prediction with Natural Language Processing

[1685] Output: predicted disease, recommended treatment

[1686] Step 4:

[1687] The server sends the diagnosis results and recommended treatments to medical professionals via an information delivery service and displays them on a dashboard.

[1688] Input: predicted disease, recommended treatment

[1689] Output: Diagnosis and treatment presented to medical professionals

[1690] (Application example 1)

[1691] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1692] In modern factories, there is a lack of means to monitor the health status and work safety of workers in real time and immediately report any abnormalities. This increases the risk of worker health risks and the likelihood of accidents. Furthermore, there is a lack of effective real-time data collection and analysis to support doctors' diagnoses. There is a need to provide a system that can resolve this situation and efficiently manage the safety of the work environment and the health of workers.

[1693] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1694] In this invention, the server includes means for collecting patient voice data and converting the voice data into text data, means for collecting patient video data and using the video data for personal identification, means for inputting, integrating, and analyzing health data, means for displaying the analysis results on a dashboard, means for generating and presenting optimal lifestyle improvement plans for each patient, means for presenting possible diseases and recommended treatments to assist doctors in making diagnoses, means for collecting worker health data in real time, means for immediately issuing an alert if an abnormality is detected, means for collecting and analyzing work environment data, and means for monitoring work safety based on the analyzed data. This enables real-time monitoring of worker health conditions and work environment safety and rapid notification of abnormalities.

[1695] The "means for collecting patient voice data and converting the voice data into text data" is a technology for acquiring the patient's voice and converting the voice into text information.

[1696] The "means for collecting patient image data and using the image data for personal identification" refers to a technology for acquiring patient images and identifying individuals from the images.

[1697] "Means for inputting, integrating, and analyzing health data" refers to technology for inputting health information such as blood pressure and body temperature into a system and integrating and analyzing it.

[1698] The "means for displaying the analysis results on a dashboard" is an interface for visually displaying the analyzed health information.

[1699] "Means for generating and presenting optimal lifestyle improvement proposals for individual patients" refers to technology for automatically creating and presenting health improvement proposals tailored to each patient.

[1700] "Means for presenting possible diseases and recommended treatments to assist doctors in diagnosis" refers to technology that uses AI and data analysis to provide possible diseases and recommended treatments to assist in diagnosis.

[1701] "Means for collecting worker health data in real time" refers to technology for obtaining health information on workers working in factories in real time.

[1702] "Means for immediately issuing an alert in the event of an abnormality" refers to technology that immediately issues a warning when an abnormality is detected in health data or work environment data.

[1703] "Means for collecting and analyzing work environment data" refers to technology for acquiring data related to the work environment within a factory and analyzing it.

[1704] "Means for monitoring work safety based on analytical data" refers to technology for constantly monitoring work safety based on analyzed data.

[1705] This invention is a system for improving the health management and work safety of factory workers. This system uses technology to collect, analyze, and display audio, video, and health data in real time.

[1706] Specifically, the system includes the following means:

[1707] Sensor data collection and management

[1708] User:

[1709] Report your health and symptoms using a voice input device.

[1710] A camera device is used to collect facial images.

[1711] Health data is entered using devices such as blood pressure monitors and thermometers.

[1712] Device:

[1713] Voice data is collected and converted into text data using voice recognition technology.

[1714] Video data is collected and workers are identified using personal identification technology.

[1715] Health data is collected, associated with the worker's profile, and sent to a cloud server.

[1716] server:

[1717] The transmitted data is consolidated and analyzed.

[1718] Calculates the worker's health index and issues an alert if any abnormalities are detected.

[1719] The results are displayed on a dashboard and notified to the work team leader as needed.

[1720] Improving health management and work safety

[1721] User:

[1722] Enter information about your daily habits (e.g., diet, exercise, etc.) into a dedicated app.

[1723] Provide work environment data as needed.

[1724] Device:

[1725] The input habit information and work environment data are sent to the cloud server.

[1726] server:

[1727] Analyze habit information and work environment data.

[1728] Generate optimal health improvement proposals and safety measures.

[1729] Improvement suggestions are presented to users via dashboards and notification functions.

[1730] Specific examples

[1731] Examples of biometric data self-management support:

[1732] 1. The user speaks into a voice input device, "I'd like to enter my blood pressure record for this week."

[1733] 2. The device collects the voice data and sends it to the cloud server.

[1734] 3. The server analyzes the data, generates a graph of blood pressure fluctuations for this week, and displays it on the device.

[1735] Examples of health improvement suggestions:

[1736] 1. The user types "Enter recent food and exercise records" into the dedicated app.

[1737] 2. The device sends the input information to the cloud server.

[1738] 3. The server proposes the optimal meal plan and exercise method based on the user's data and displays it on the device.

[1739] Hardware and software used

[1740] Hardware:

[1741] Smart glasses camera: Used to collect video data.

[1742] Microphone in smart glasses: Used to collect voice data.

[1743] Health monitoring devices: thermometers, blood pressure monitors, etc.

[1744] software:

[1745] Python: The main programming language.

[1746] OpenCV: A library for video data analysis.

[1747] SpeechRecognition: A library for analyzing speech data.

[1748] Cloud server: Stores and analyzes health data.

[1749] Example prompt for a generative AI model:

[1750] A worker speaks to the smart glasses and says, "Please check my health." The smart glasses monitor the worker's body temperature, blood pressure, complexion, and other conditions in real time, sending the data to the cloud. Cloud AI analyzes the data, and if any abnormalities are detected, an alert is sent to the work team leader's smartphone.

[1751] By using this system, it is expected that health management and safety assurance for workers in factories will be automated and carried out efficiently.

[1752] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1753] Program processing flow of the system that realizes the application example

[1754] Step 1:

[1755] The user speaks to the smart glasses, saying, "Check your health."

[1756] Specific behavior:

[1757] Input: Audio data

[1758] Output: Raw audio data

[1759] The user speaks into the smart glasses, and the microphone in the glasses collects the voice data, which is then converted into text data using voice recognition technology.

[1760] Step 2:

[1761] The device sends the voice data to the cloud server.

[1762] Specific behavior:

[1763] Input: Audio data

[1764] Output: Text data

[1765] The device (smart glasses) sends the collected voice data to a cloud server, where it is converted into text data using voice recognition technology and then sent.

[1766] Step 3:

[1767] Collect user health data (body temperature, blood pressure).

[1768] Specific behavior:

[1769] Input: Data from health monitoring devices

[1770] Output: Health data (body temperature, blood pressure)

[1771] A user uses a health monitoring device to measure their body temperature and blood pressure, and the data is sent to a cloud server via smart glasses.

[1772] Step 4:

[1773] The server integrates and analyzes the transmitted audio data, health data, and video data.

[1774] Specific behavior:

[1775] Input: Audio data, health data, video data

[1776] Output: Analysis results

[1777] The server integrates the various data sent and analyzes it using machine learning models or data analysis algorithms, using Python or dedicated analysis libraries.

[1778] Step 5:

[1779] The server displays the analysis results on a dashboard and issues an alert if an abnormality is detected.

[1780] Specific behavior:

[1781] Input: Analysis results

[1782] Output: Dashboard display, alert notification

[1783] The server updates the dashboard based on the analysis results and displays them visually. If an abnormality is detected, an alert is sent to the team leader's smartphone in real time.

[1784] Step 6:

[1785] The server generates health improvement suggestions based on the worker's habit information and presents them on the terminal.

[1786] Specific behavior:

[1787] Input: Habit information, analysis results

[1788] Output: Health improvement plan

[1789] The server analyzes the user's habit information and health data and generates health improvement suggestions, which are presented as suggestions to the user's smart glasses or associated devices.

[1790] In this way, the system collects, analyzes, displays, and notifies data at each step, monitoring worker health and the safety of the work environment in real time, enabling rapid response when an abnormality occurs.

[1791] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1792] The present invention aims to incorporate an emotion engine into a medical management system that utilizes multimodal AI technology, thereby making diagnoses and suggestions that reflect the patient's emotions, reducing the burden on medical professionals, improving the accuracy of diagnoses, and providing lifestyle improvement suggestions tailored to the needs of each individual patient.

[1793] 1. Collection and management of biometric and emotional data

[1794] User:

[1795] Report your health and symptoms using a voice input device.

[1796] A camera device is used to collect facial images.

[1797] Health data is entered using devices such as blood pressure monitors and thermometers.

[1798] Device:

[1799] Voice data is collected and converted into text data using voice recognition technology.

[1800] Video data is collected and individuals are identified using facial recognition technology.

[1801] Health data is collected, associated with the user's profile, and sent to a cloud server.

[1802] Emotions are recognized from audio data and video data, and emotion data is generated.

[1803] server:

[1804] The transmitted audio data, video data, health data, and emotional data are integrated and analyzed.

[1805] Calculates the user's health and emotional indicators and displays the results on a dashboard.

[1806] 2. Generating and Proposing Health Improvement Plans Using Emotional Data

[1807] User:

[1808] Enter information about your daily habits (e.g., diet, exercise, etc.) into a dedicated app.

[1809] Provide genetic information as needed.

[1810] Device:

[1811] The entered habit information and genetic information is sent to a cloud server.

[1812] server:

[1813] The user's habits, genetic information, and emotional data are analyzed.

[1814] Based on emotional data, optimal health improvement proposals are generated, including meal plans and exercise methods.

[1815] Improvement suggestions are presented to users via dashboards and notification functions.

[1816] 3. Assisting doctors in diagnosis and utilizing emotion data

[1817] User:

[1818] Enter your symptoms of illness by voice or text.

[1819] Provide historical medical data.

[1820] Device:

[1821] The entered symptom data and past medical data are sent to a cloud server.

[1822] server:

[1823] Analyzes voice and text data to predict possible diseases.

[1824] Compilation of supplementary information to improve diagnostic accuracy is carried out by comparing it with past medical data and referring to emotional data.

[1825] Present the diagnosis and recommended treatment to the physician.

[1826] Specific examples

[1827] Examples of biometric and emotional data self-management support:

[1828] 1. The user speaks into a voice input device, "I'd like to enter my blood pressure record for this week."

[1829] 2. The device collects the voice data and sends it to the cloud server.

[1830] 3. The device recognizes emotions from the voice data and generates emotion data.

[1831] 4. The server analyzes the data, generates a graph of blood pressure fluctuations for this week, and displays it on the device along with emotional data.

[1832] Example of using emotion data to provide health improvement suggestions:

[1833] 1. The user types "Enter recent food and exercise records" into the dedicated app.

[1834] 2. The device sends the input information to the cloud server.

[1835] 3. The server proposes optimal meal plans and exercise methods based on the user's data, and displays them on the device, reflecting the user's emotional data.

[1836] Examples of using emotion data to assist doctors in diagnosis:

[1837] 1. The user enters the following into the dedicated app: "I have persistent abdominal pain. I have already uploaded my past medical data."

[1838] 2. The device sends the entered data to the cloud server.

[1839] 3. The server analyzes the symptoms, lists possible illnesses based on the emotional data, and presents recommended treatments to the doctor.

[1840] By automating and streamlining these processes, the system of the present invention reduces the burden on medical professionals, improves diagnostic accuracy, and significantly contributes to eliminating disparities in medical care. Furthermore, by using emotion data, comprehensive health management that takes into account the patient's psychological state becomes possible.

[1841] The processing flow will be explained below.

[1842] (Collection and management of biometric and emotional data)

[1843] Audio data recording and conversion

[1844] Step 1:

[1845] The user speaks about their physical condition and symptoms into a voice input device.

[1846] Step 2:

[1847] The terminal receives the audio signal and temporarily stores it as audio data.

[1848] Step 3:

[1849] The terminal uses voice recognition technology to convert the voice data into text data.

[1850] Step 4:

[1851] The terminal transmits the converted text data to the cloud server.

[1852] Video data recording and facial recognition

[1853] Step 1:

[1854] The user faces the camera.

[1855] Step 2:

[1856] The device collects and temporarily stores video data.

[1857] Step 3:

[1858] The device runs a facial recognition algorithm to identify the individual.

[1859] Step 4:

[1860] The device transmits the video data and identification information to the cloud server.

[1861] Generating emotion data

[1862] Step 1:

[1863] The terminal recognizes emotions from audio data and video data.

[1864] Step 2:

[1865] The device generates the recognized emotion data and temporarily stores it.

[1866] Step 3:

[1867] The device transmits the emotion data to the cloud server.

[1868] Health Data Collection and Transmission

[1869] Step 1:

[1870] Users enter health data such as blood pressure and body temperature into a dedicated app or device.

[1871] Step 2:

[1872] The terminal associates the entered health data with the user's profile and temporarily stores it.

[1873] Step 3:

[1874] The device sends the health data to a cloud server.

[1875] Data integration and analysis

[1876] Step 1:

[1877] The server integrates the received audio data, video data, health data, and emotional data.

[1878] Step 2:

[1879] The server calculates health and emotional indicators using a proprietary algorithm.

[1880] Step 3:

[1881] The server converts health and emotional indicators into graphs and statistical information to generate a dashboard.

[1882] Step 4:

[1883] The server sends the dashboard and result display data to the terminal.

[1884] (Providing health improvement ideas)

[1885] Entering daily habits

[1886] Step 1:

[1887] The user inputs information about their daily habits, such as what they eat and how much exercise they do, into a dedicated app.

[1888] Step 2:

[1889] The device organizes the input habit information by associating it with the user's profile.

[1890] Step 3:

[1891] The device sends the organized habit information to a cloud server.

[1892] Analysis of habit information and genetic information

[1893] Step 1:

[1894] The server integrates the received habit information with the genetic information provided by the user.

[1895] Step 2:

[1896] The server uses statistical and genetic techniques to generate optimal health improvement recommendations.

[1897] Customization using emotional data

[1898] Step 1:

[1899] The server customizes the generated health improvement proposals by reflecting emotional data.

[1900] Proposal of improvement proposals

[1901] Step 1:

[1902] The server then compiles the generated health improvement suggestions into a display format on a dashboard for the user.

[1903] Step 2:

[1904] The server sends the improvement proposal data to the terminal.

[1905] Step 3:

[1906] The improvement suggestions received by the device are displayed on the user's dashboard and an alert is sent via a notification function.

[1907] (Assisting doctors in diagnosis)

[1908] Enter symptoms

[1909] Step 1:

[1910] The user enters their symptoms of illness into a dedicated app by voice or text.

[1911] Step 2:

[1912] The terminal converts the input symptom data into text data and temporarily stores it.

[1913] Step 3:

[1914] The device sends the text data to the cloud server.

[1915] Referencing past medical data

[1916] Step 1:

[1917] The server looks up past medical data from the user's profile.

[1918] Step 2:

[1919] The server analyzes the historical data and the new symptom data.

[1920] Diagnosis and treatment suggestions

[1921] Step 1:

[1922] The server uses AI algorithms to predict possible diseases.

[1923] Step 2:

[1924] The server compares the data with past medical data and uses emotional data to organize supplementary information to improve diagnostic accuracy.

[1925] Step 3:

[1926] The server compiles the diagnosis and recommended treatment in the form of a report for the physician.

[1927] Step 4:

[1928] The server sends the report to the device.

[1929] Step 5:

[1930] The device receives a diagnosis and displays treatment suggestions on a screen for the doctor.

[1931] Example 2

[1932] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1933] The current medical system does not take patients' emotions into consideration when making diagnoses or recommendations, which increases the burden on medical professionals and results in insufficient diagnostic accuracy. Furthermore, there is a lack of systems that provide lifestyle improvement suggestions tailored to individual patients' needs, making it difficult for patients to manage their own health. Furthermore, there is a need for improved data analysis and disease prediction accuracy to assist doctors in their diagnoses.

[1934] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1935] In this invention, the server includes: means for collecting a patient's voice data and converting the voice data into text data; means for collecting a patient's video data and using the video data for personal identification; means for inputting health and emotional data and transmitting it to a cloud server; means for recognizing emotions from the voice and video data and generating emotional data; means for integrating and analyzing the transmitted voice, video, health, and emotional data; means for displaying the analysis results on a dashboard; means for generating and presenting optimal lifestyle improvement proposals for each patient; and means for presenting possible diseases and recommended treatments using a generative AI model to assist physicians in their diagnosis. This enables diagnoses and proposals that take the patient's emotions into account, reducing the burden on medical professionals and improving diagnostic accuracy. Furthermore, providing lifestyle improvement proposals tailored to each patient's needs makes it easier for patients to manage their own health.

[1936] The "means for collecting patient voice data and converting the voice data into text data" is a mechanism for collecting voice data reported by the patient through a voice input device and converting the voice data into text format.

[1937] The "means for collecting patient image data and using the image data for personal identification" is a mechanism for acquiring an image of the patient's face and using the image data to identify the individual.

[1938] "Means for inputting health data and emotional data and transmitting it to a cloud server" refers to a mechanism for inputting information about a patient's health and emotions and transmitting that data to a server on the cloud.

[1939] The "means for recognizing emotions from audio data and video data and generating emotional data" refers to a mechanism for analyzing collected audio data and video data to recognize the patient's emotions and generate data related to emotions.

[1940] "Means for integrating and analyzing transmitted audio data, video data, health data, and emotional data" refers to a mechanism for integrating and comprehensively analyzing various data transmitted to the cloud server.

[1941] "Means for displaying analysis results on a dashboard" refers to a mechanism for outputting the results of the analyzed data to a dashboard for visually displaying them.

[1942] "A means for generating and presenting optimal lifestyle improvement proposals for individual patients" is a system that generates optimal lifestyle improvement proposals based on data on each patient and presents them to the patient.

[1943] "Means of using generative AI models to suggest possible diseases and recommended treatments to assist doctors in their diagnoses" refers to a system that uses generative AI technology to predict possible diseases from collected data and provide doctors with optimal treatments.

[1944] This invention is a medical management system that utilizes multimodal AI technology and incorporates an emotion engine to make diagnoses and recommendations that reflect the patient's emotions, reducing the burden on medical professionals. The system also aims to improve the accuracy of diagnoses and provide lifestyle improvement proposals tailored to the individual needs of each patient.

[1945] The system consists of three main components: users, terminals, and servers.

[1946] User

[1947] A user reports their physical condition and symptoms using a voice input device. A camera device is also used to collect facial images, and health data is input using devices such as a blood pressure monitor and a thermometer. As a specific example, data collection begins when a user speaks into the voice input device, saying, "I'd like to enter this week's blood pressure record."

[1948] Terminal

[1949] The terminal uses multiple pieces of hardware and software to collect and process data.

[1950] The audio data is converted into text data using the Google Cloud Speech-to-Text service.

[1951] Video data is collected and individuals are identified using the Microsoft Azure Face API.

[1952] Health data is sent to AWS cloud servers.

[1953] Emotions are recognized from audio and video data using IBM Watson's Natural Language Understanding and Visual Recognition, and emotion data is generated.

[1954] For example, when a user types "I would like to enter my recent meal and exercise records" into a dedicated app, that information is sent to the cloud server.

[1955] server

[1956] The server is the central component that processes data on the cloud.

[1957] The audio data, video data, health data, and emotion data sent to the cloud server are integrated and analyzed using data processing libraries such as Python's Pandas and NumPy.

[1958] Calculates user health and emotional indicators and displays the results on a dashboard using Tableau or Power BI.

[1959] For example, it can recognize emotions based on voice data and evaluate stress levels to provide comprehensive health management. It also uses GPT-4 as a generative AI model to generate optimal health improvement proposals and suggest optimal diet plans and exercise methods for users.

[1960] Specific examples

[1961] Examples of biometric and emotional data self-management support:

[1962] The user speaks to the voice input device, saying, "I'd like to enter my blood pressure record for this week." The device then collects the voice data and sends it to a cloud server using the Google Cloud Speech-to-Text service. The device then uses IBM Watson's Natural Language Understanding to recognize emotions from the voice data and generate emotion data. The server then analyzes this data, generates a graph of blood pressure fluctuations for this week, and displays it on a Tableau dashboard along with the emotion data.

[1963] Example of using emotion data to provide health improvement suggestions:

[1964] The user enters text data into the dedicated app, saying, "I'd like to enter my recent meal and exercise records." The device then sends the input information to an AWS cloud server. The server generates an optimal meal plan and exercise method based on the user's data, and uses GPT-4 to create suggestions that also reflect emotional data, which are then displayed on the device.

[1965] Examples of using emotion data to assist doctors in diagnosis:

[1966] The user enters into the dedicated app, "I have persistent stomach pain. I have already uploaded my past medical data." The device then sends the entered data to an AWS cloud server. The server analyzes the symptoms, lists possible illnesses based on the emotional data, and generates a recommended treatment using GPT-4 to present to the doctor.

[1967] By automating and streamlining these processes, this system reduces the burden on medical professionals and improves the accuracy of diagnoses. It also enables comprehensive health management based on emotional data, providing patients with optimal treatments and lifestyle improvement suggestions.

[1968] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1969] Step 1:

[1970] The user reports their physical condition and symptoms using a voice input device.

[1971] Input: User voice input (e.g., "Enter this week's blood pressure record.")

[1972] Data processing: The device collects voice data and converts it into text data using the Google Cloud Speech-to-Text service.

[1973] Output: Report content as text data (e.g., "Enter your blood pressure record for this week")

[1974] Step 2:

[1975] A user uses a camera device to collect facial images.

[1976] Input: User's face image

[1977] Data processing: The device collects video data and uses the Microsoft Azure Face API to perform facial recognition and recognize emotions from the video data.

[1978] Output: Identified person ID and emotion data

[1979] Step 3:

[1980] A user inputs health data using devices such as a blood pressure monitor and a thermometer.

[1981] Input: Health data (e.g. blood pressure, body temperature)

[1982] Data processing: The device collects health data and sends it to AWS cloud servers.

[1983] Output: Health data stored on a cloud server

[1984] Step 4:

[1985] Sending audio and video data to the cloud

[1986] Input: Text data, identified personal ID, emotion data

[1987] Data processing: The device integrates the audio and video data to generate emotion data, which is then sent to the AWS cloud server.

[1988] Output: Integrated data sent to cloud server

[1989] Step 5:

[1990] The audio data, video data, health data, and emotional data sent to the cloud server are integrated and analyzed.

[1991] Input: Audio data, video data, health data, and emotion data stored on a cloud server

[1992] Data processing: The server integrates this data and analyzes it using data processing libraries such as Python's Pandas and NumPy.

[1993] Output: Integrated data analysis results (e.g., health index, emotional index)

[1994] Step 6:

[1995] Display the analysis results on a dashboard.

[1996] Input: Integrated data analysis results

[1997] Data processing: The server visualizes and displays the analysis results on a dashboard using Tableau or Power BI.

[1998] Output: Dashboard screen viewable by users and doctors

[1999] Step 7:

[2000] The user enters information about their daily habits into a dedicated app.

[2001] Input: User's habit information (e.g., dietary content, amount of exercise)

[2002] Data processing: The terminal sends the input data to the AWS cloud server.

[2003] Output: Habit information stored on the cloud server

[2004] Step 8:

[2005] The server generates and presents health improvement suggestions based on the user's habit information, health data, and emotional data.

[2006] Input: Habit information, health data, emotional data

[2007] Data processing: The server analyzes these data and generates optimal health improvement recommendations using a generative AI model (GPT-4).

[2008] Output: Generated health improvement suggestions, including recommendations displayed in notifications and on the dashboard

[2009] Step 9:

[2010] Providing data to assist doctors in diagnosis

[2011] Input: User's symptom information, past medical data

[2012] Data processing: The device sends the input data to the AWS cloud server, which analyzes the data and uses GPT-4 to list possible diseases based on the emotional data.

[2013] Output: A list of diagnoses and treatment recommendations presented to the doctor

[2014] At each step, the system processes and analyzes the collected data to provide optimal improvement suggestions for each individual user and diagnostic support information for doctors, thereby realizing comprehensive health management that takes the patient's emotions into consideration.

[2015] (Application example 2)

[2016] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2017] Conventional medical management systems provide diagnoses and treatment recommendations based on biometric data and past medical data, but they do not take into account the patient's emotions or psychological state, making it difficult to fully meet the needs of individual patients. Furthermore, food delivery services, in particular, do not propose optimal meal plans that reflect the user's health status and emotions, making it difficult to provide individually customized services. Therefore, new methods are needed to increase patient and user satisfaction, reduce the burden on medical professionals, and improve diagnostic accuracy.

[2018] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2019] In this invention, the server includes means for collecting patient voice data and converting the voice data into text data, means for collecting patient video data and using the video data for personal identification, means for inputting, integrating, and analyzing health data, means for displaying the analysis results on a dashboard, means for generating and presenting optimal lifestyle improvement plans for each patient, means for presenting possible diseases and recommended treatments to assist doctors in making diagnoses, means for collecting user emotion data and analyzing the emotion data to reflect it in health improvement plans, and means for inputting health reports by voice or text and proposing and delivering optimal meal plans. This enables personalized medical management and food delivery services that comprehensively consider the emotions and health status of patients and users.

[2020] "Voice data" refers to data for collecting voice information from patients and converting the content into text data.

[2021] "Video data" refers to data that collects images of a patient's face or body and is used to identify the individual.

[2022] "Health data" refers to physiological information such as a patient's blood pressure, body temperature, and heart rate, and is data that can be integrated and analyzed.

[2023] A "dashboard" is an interface for visually displaying information such as analysis results and proposals.

[2024] A "lifestyle improvement plan" is a specific action plan proposed to improve the quality of daily life of patients or users.

[2025] "Aiding diagnosis" means providing supplementary information or recommendations to help a doctor make an appropriate diagnosis of a patient's symptoms.

[2026] "Disease" refers to an illness or condition that adversely affects a patient's health.

[2027] A "therapy" is a specific medical procedure or method used to treat a disease.

[2028] "Emotional data" refers to data about the user's psychological state and emotions extracted from facial expressions, etc.

[2029] "Health report" refers to the act of a patient or user recording or reporting their current health condition or symptoms.

[2030] "Meal Plan" refers to a meal menu and plan suggested based on the user's health condition and mood.

[2031] "Delivery" refers to the act of delivering cooked meals based on the proposed meal plan to the user.

[2032] This invention is a system that collects and analyzes biometric and emotional data of patients and users, and provides optimal lifestyle improvement proposals and health management plans based on the collected data. The system aims to provide individually customized services that comprehensively consider the health status and emotions of patients and users.

[2033] 1. Collection and management of biometric and emotional data

[2034] User:

[2035] Users report their physical condition and symptoms using a voice input device, which is collected by the terminal as voice data. A camera device is also used to collect facial images. Furthermore, health data is input using devices such as a blood pressure monitor and a thermometer.

[2036] Device:

[2037] The device collects voice data and converts it into text data using voice recognition technology. It also collects video data and identifies individuals using facial recognition technology. The collected health data is associated with the user's profile and sent to a cloud server. At the same time, emotions are recognized from the voice and video data, and emotion data is generated.

[2038] server:

[2039] The server integrates and analyzes the transmitted audio, video, health, and emotional data, calculates the user's health and emotional indices, and displays the results on a dashboard.

[2040] 2. Generating and Proposing Health Improvement Plans Using Emotional Data

[2041] User:

[2042] Users input information about their daily habits (e.g., dietary habits, amount of exercise, etc.) into a dedicated app, and also provide genetic information if necessary.

[2043] Device:

[2044] The terminal transmits the input habit information and genetic information to a cloud server.

[2045] server:

[2046] The server analyzes the user's habits, genetic information, and emotional data. Based on the emotional data, it generates optimal health improvement proposals, suggesting meal plans and exercise methods to the user. The proposals are presented to the user via a dashboard and notification function.

[2047] 3. Assisting doctors in diagnosis and utilizing emotion data

[2048] User:

[2049] Users can input their symptoms of illness by voice or text, and also provide past medical data.

[2050] Device:

[2051] The terminal transmits the input symptom data and past medical data to a cloud server.

[2052] server:

[2053] The server analyzes the voice and text data to predict possible illnesses, compares it with past medical data, and uses emotional data to organize supplementary information to improve diagnostic accuracy. The diagnosis results and recommended treatments are then presented to the doctor.

[2054] Specific examples

[2055] Examples of biometric and emotional data self-management support:

[2056] 1. The user speaks into a voice input device, "I'd like to enter my blood pressure record for this week."

[2057] 2. The device collects the voice data and sends it to the cloud server.

[2058] 3. The device recognizes emotions from the voice data and generates emotion data.

[2059] 4. The server analyzes the data, generates a graph of blood pressure fluctuations for this week, and displays it on the device along with emotional data.

[2060] Example of using emotion data to provide health improvement suggestions:

[2061] 1. The user types "Enter recent food and exercise records" into the dedicated app.

[2062] 2. The device sends the input information to the cloud server.

[2063] 3. The server proposes optimal meal plans and exercise methods based on the user's data, and displays them on the device, reflecting the user's emotional data.

[2064] Examples of using emotion data to assist doctors in diagnosis:

[2065] 1. The user enters the following into the dedicated app: "I have persistent abdominal pain. I have already uploaded my past medical data."

[2066] 2. The device sends the entered data to the cloud server.

[2067] 3. The server analyzes the symptoms, lists possible illnesses based on the emotional data, and presents recommended treatments to the doctor.

[2068] Prompt Sentence Examples

[2069] "Please suggest personalized food delivery based on the user's emotional and health data. Here's the user's data for this week:

[2070] Blood pressure: 110

[2071] Temperature: 36.5

[2072] Emotion: neutral

[2073] Please suggest a suitable meal plan and why.”

[2074] By building a system in this way, it is possible to comprehensively manage a user's health condition and emotions, and realize personalized medical services and food delivery.

[2075] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2076] Step 1:

[2077] A user reports their physical condition and symptoms using a voice input device. This voice data is collected and input data from the voice input device is generated.

[2078] Step 2:

[2079] The device receives the voice data and converts it into text data using speech recognition technology (e.g., Google Speech Recognition API). The input is voice data, and the output is text data.

[2080] Step 3:

[2081] A user uses a camera device to capture a video of his or her face, generating video data that is provided to a terminal as a video input.

[2082] Step 4:

[2083] The device receives the video data and identifies individuals using facial recognition technology (e.g., using OpenCV and dlib). The input is the video data, and the output is the data of the identified individuals.

[2084] Step 5:

[2085] A user inputs health data using a device such as a blood pressure monitor or a thermometer, and the data is sent to a terminal. The input is health data (blood pressure, body temperature, etc.).

[2086] Step 6:

[2087] The device associates the collected health data with the user's profile and transmits it to the cloud server. The input is the health data, and the output is the integrated health data transmitted to the cloud server.

[2088] Step 7:

[2089] The device recognizes emotions from audio and video data and generates emotion data (e.g., extracts emotions using the FER library). The input is audio and video data, and the output is emotion data.

[2090] Step 8:

[2091] The server integrates and analyzes the audio data, video data, health data, and emotion data sent to the cloud server. These data are the input, and the output is the health index and emotion index as the analysis results.

[2092] Step 9:

[2093] The server displays the analysis results on a dashboard, which includes graphs and indicators to visualize the user's health and emotional state. The input is the analysis results, and the output is the dashboard display.

[2094] Step 10:

[2095] The user inputs information about their daily habits (such as dietary habits and exercise amount) into a dedicated app. The input is habit information, which is then provided to the device.

[2096] Step 11:

[2097] The device sends habit information to the cloud server. The input is the habit information, and the output is the data sent to the cloud server.

[2098] Step 12:

[2099] The server generates optimal health improvement proposals based on emotional data and habit information. It also suggests meal plans and exercise methods. The input is emotional data and habit information, and the output is improvement proposals.

[2100] Step 13:

[2101] The server generates improvement proposals and presents them to the user via a dashboard or notification function. The input is the improvement proposal, and the output is a notification to the user.

[2102] Step 14:

[2103] The user inputs symptoms of illness by voice or text and provides this to a dedicated app. The input is symptom data, which is then sent to the device.

[2104] Step 15:

[2105] The device sends symptom data and past medical data to the cloud server. The input is the symptom data and medical data, and the output is the data sent to the cloud server.

[2106] Step 16:

[2107] The server analyzes the symptom data, predicts possible diseases based on the emotion data, and then organizes recommended treatments. The input is the symptom data and emotion data, and the output is a list of diseases and recommended treatments.

[2108] Step 17:

[2109] The server presents the diagnosis results and recommended treatment to the doctor. The input is the analysis results, and the output is a notification to the doctor.

[2110] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2111] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2112] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2113] [Fourth embodiment]

[2114] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2115] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2116] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2117] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2118] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2119] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2120] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2121] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2122] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2123] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2124] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2125] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2126] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2127] The present invention is a medical management system that utilizes multimodal AI technology and is designed to resolve the labor shortage and medical disparities in medical settings. Specific embodiments of the present invention are described below.

[2128] 1. Biometric data collection and management

[2129] User:

[2130] Report your health and symptoms using a voice input device.

[2131] A camera device is used to collect facial images.

[2132] Health data is entered using devices such as blood pressure monitors and thermometers.

[2133] Device:

[2134] Voice data is collected and converted into text data using voice recognition technology.

[2135] Video data is collected and individuals are identified using facial recognition technology.

[2136] Health data is collected, associated with the user's profile, and sent to a cloud server.

[2137] server:

[2138] The transmitted audio data, video data, and health data are integrated and analyzed.

[2139] Calculates the user's health indicators and displays the results on a dashboard.

[2140] 2. Generate and propose health improvement plans

[2141] User:

[2142] Enter information about your daily habits (e.g., diet, exercise, etc.) into a dedicated app.

[2143] Provide genetic information as needed.

[2144] Device:

[2145] The entered habit information and genetic information is sent to a cloud server.

[2146] server:

[2147] The user's habits and genetic information are analyzed.

[2148] Generate optimal health improvement recommendations and suggest meal plans, exercise methods, etc.

[2149] Improvement suggestions are presented to users via dashboards and notification functions.

[2150] 3. Assisting doctors in diagnosis

[2151] User:

[2152] Enter your symptoms of illness by voice or text.

[2153] Provide historical medical data.

[2154] Device:

[2155] The entered symptom data and past medical data are sent to a cloud server.

[2156] server:

[2157] Analyzes voice and text data to predict possible diseases.

[2158] Compile supplementary information to improve diagnostic accuracy by referring to past medical data.

[2159] Present the diagnosis and recommended treatment to the physician.

[2160] Specific examples

[2161] Examples of biometric data self-management support:

[2162] 1. The user speaks into a voice input device, "I'd like to enter my blood pressure record for this week."

[2163] 2. The device collects the voice data and sends it to the cloud server.

[2164] 3. The server analyzes the data, generates a graph of blood pressure fluctuations for this week, and displays it on the device.

[2165] Examples of health improvement suggestions:

[2166] 1. The user types "Enter recent food and exercise records" into the dedicated app.

[2167] 2. The device sends the input information to the cloud server.

[2168] 3. The server proposes the optimal meal plan and exercise method based on the user's data and displays it on the device.

[2169] Examples of diagnostic assistance for doctors:

[2170] 1. The user enters the following into the dedicated app: "I have persistent abdominal pain. I have already uploaded my past medical data."

[2171] 2. The device sends the entered data to the cloud server.

[2172] 3. The server analyzes the symptoms, lists possible illnesses, and recommends treatments for the doctor.

[2173] By automating and streamlining these processes, the system of the present invention reduces the burden on medical professionals, improves diagnostic accuracy, and significantly contributes to eliminating disparities in medical care.

[2174] The processing flow will be explained below.

[2175] (Biometric data self-management support)

[2176] Audio data recording and conversion

[2177] Step 1:

[2178] The user speaks about their physical condition and symptoms into a voice input device.

[2179] Step 2:

[2180] The terminal receives the audio signal and temporarily stores it as audio data.

[2181] Step 3:

[2182] The terminal uses voice recognition technology to convert the voice data into text data.

[2183] Step 4:

[2184] The terminal transmits the converted text data to the cloud server.

[2185] Video data recording and facial recognition

[2186] Step 1:

[2187] The user faces the camera.

[2188] Step 2:

[2189] The device collects and temporarily stores video data.

[2190] Step 3:

[2191] The device runs a facial recognition algorithm to identify the individual.

[2192] Step 4:

[2193] The device transmits the video data and identification information to the cloud server.

[2194] Health Data Collection and Transmission

[2195] Step 1:

[2196] Users enter health data such as blood pressure and body temperature into a dedicated app or device.

[2197] Step 2:

[2198] The terminal associates the entered health data with the user's profile and temporarily stores it.

[2199] Step 3:

[2200] The device sends the health data to a cloud server.

[2201] Data integration and analysis

[2202] Step 1:

[2203] The server integrates the received audio data, video data, and health data.

[2204] Step 2:

[2205] The server calculates health indicators using a dedicated algorithm.

[2206] Step 3:

[2207] The server converts health indicators into graphs and statistical information and generates a dashboard.

[2208] Step 4:

[2209] The server sends the dashboard and result display data to the terminal.

[2210] (Providing health improvement ideas)

[2211] Entering daily habits

[2212] Step 1:

[2213] The user inputs information about their daily habits, such as what they eat and how much exercise they do, into a dedicated app.

[2214] Step 2:

[2215] The device organizes the input habit information by associating it with the user's profile.

[2216] Step 3:

[2217] The device sends the organized habit information to a cloud server.

[2218] Analysis of habit information and genetic information

[2219] Step 1:

[2220] The server integrates the received habit information with the genetic information provided by the user.

[2221] Step 2:

[2222] The server uses statistical and genetic techniques to generate optimal health improvement recommendations.

[2223] Proposal of improvement proposals

[2224] Step 1:

[2225] The server then compiles the generated health improvement suggestions into a display format on a dashboard for the user.

[2226] Step 2:

[2227] The server sends the improvement proposal data to the terminal.

[2228] Step 3:

[2229] The improvement suggestions received by the device are displayed on the user's dashboard and an alert is sent via a notification function.

[2230] (Assisting doctors in diagnosis)

[2231] Enter symptoms

[2232] Step 1:

[2233] The user enters their symptoms of illness into a dedicated app by voice or text.

[2234] Step 2:

[2235] The terminal converts the input symptom data into text data and temporarily stores it.

[2236] Step 3:

[2237] The device sends the text data to the cloud server.

[2238] Referencing past medical data

[2239] Step 1:

[2240] The server looks up past medical data from the user's profile.

[2241] Step 2:

[2242] The server analyzes the historical data and the new symptom data.

[2243] Diagnosis and treatment suggestions

[2244] Step 1:

[2245] The server uses AI algorithms to predict possible diseases.

[2246] Step 2:

[2247] The server compares the data with past medical data and organizes supplementary information to improve diagnostic accuracy.

[2248] Step 3:

[2249] The server compiles the diagnosis and recommended treatment in the form of a report for the physician.

[2250] Step 4:

[2251] The server sends the report to the device.

[2252] Step 5:

[2253] The device receives a diagnosis and displays treatment suggestions on a screen for the doctor.

[2254] Example 1

[2255] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2256] In today's medical field, increasing patient numbers and limited medical resources have led to serious problems such as labor shortages and disparities in medical care. In conventional medical systems, the collection and management of patients' health data, the generation of lifestyle improvement recommendations, and diagnostic assistance for doctors are not fully automated, and these processes require time and effort. Therefore, there is a need to build a system that can efficiently collect and analyze data and provide diagnostic assistance.

[2257] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2258] In this invention, the server includes means for collecting voice data and converting the voice data into text data, means for collecting video data and using the video data for personal identification, and means for inputting health data and integrating and analyzing the health data, thereby enabling efficient collection and analysis of patient data and assisting doctors in making diagnoses.

[2259] "Voice data" refers to data that records a user's spoken words or audio information and is stored in digital form.

[2260] "Text data" is voice data converted into text information, and is data used for natural language processing, etc.

[2261] "Video data" refers to image and video data captured by a camera or other video collection device.

[2262] "Personal identification" is the act of distinguishing a specific person from other people and identifying that person's characteristics and attributes.

[2263] "Health data" is information that indicates the user's health condition, and includes physiological data such as blood pressure and body temperature.

[2264] "Integration" is the act of bringing together multiple different data sets and managing and analyzing them as consistent information.

[2265] "Analysis" is the act of examining collected data in detail and drawing meaningful conclusions or results from that information.

[2266] A "display device" is a device for visually presenting analytical results or other information to a user or medical professional.

[2267] The "lifestyle improvement plan" includes specific suggestions and plans for improving the user's health condition and lifestyle habits.

[2268] "Diagnostic assistance" refers to the act of providing additional information or analytical results to assist medical professionals in making a diagnosis.

[2269] "Possible diseases" refers to diseases that are suspected to be possible based on the user's symptoms and data.

[2270] "Recommended treatments" include the treatment methods that are considered most appropriate for a particular disease or condition.

[2271] "User" refers to an individual who uses this system to provide health data.

[2272] "Health professionals" refers to people who work in the medical profession, such as doctors and nurses.

[2273] This invention is a medical management system that utilizes multimodal AI technology, designed to resolve the shortage of medical staff and healthcare disparities in the medical field. This system integrates and analyzes audio, video, and health data to generate optimal lifestyle improvement recommendations for users and provide information to assist medical professionals in making diagnoses.

[2274] Biometric data collection and management

[2275] User:

[2276] Use your voice input device (e.g., a microphone on your voice input device) to report that you are "feeling unwell."

[2277] Use your smartphone camera to collect facial footage.

[2278] Health data is measured using a blood pressure monitor and thermometer and entered into a dedicated app.

[2279] Device:

[2280] Collect audio data and convert it to text using the Google Cloud Speech-to-Text API.

[2281] Video data is collected using a smartphone camera and facial recognition and personal identification are performed using the Azure Face API.

[2282] Health data is entered into a dedicated app and sent to the Google Firebase cloud server.

[2283] server:

[2284] A Python data analysis script is run to integrate and analyze the transmitted audio, video, and health data.

[2285] Calculates user health indicators and displays the results on a dashboard created with Microsoft Power BI.

[2286] The results are sent to the user's dedicated app and displayed.

[2287] Generate and propose health improvement plans

[2288] User:

[2289] Enter your diet and exercise amount into a dedicated mobile app.

[2290] If necessary, genetic information (e.g., data from a genetic information service) will also be uploaded.

[2291] Device:

[2292] Habit information and genetic information entered into a dedicated app is sent to the Google Firebase cloud server.

[2293] server:

[2294] To analyze the user's data, the data is processed using a Hadoop cluster.

[2295] Based on user data, health improvement recommendations are generated using generative AI models (e.g., large-scale language models).

[2296] Suggested meal plans and exercise methods are sent to a dedicated app and presented to the user.

[2297] Assisting doctors in diagnosis

[2298] User:

[2299] Enter the following into the dedicated app: "I have persistent abdominal pain. I have already uploaded my past medical data."

[2300] Past medical data will be provided via a dedicated app (e.g., a medical data management app).

[2301] Device:

[2302] The entered symptom data and past medical data are sent to the Google Firebase cloud server.

[2303] server:

[2304] The received data is analyzed using natural language processing technology (e.g., natural language processing model) to predict possible diseases.

[2305] Compile supplementary information to improve diagnostic accuracy by referring to past medical data.

[2306] The diagnosis and recommended treatments are sent to medical professionals via an information communication service and displayed on the medical professional's dashboard.

[2307] Examples of specific examples and prompts

[2308] In the "Biometric Data Self-Management Support Example," a user speaks to a voice input device, saying, "I'm going to enter my blood pressure record for this week." The device collects the voice data and sends it to a cloud server using the Google Cloud Speech-to-Text API. The server analyzes the data and generates a graph of blood pressure fluctuations for this week, which is then displayed on the device.

[2309] Example prompt sentence:

[2310] "I have entered this week's blood pressure data. Please display the analysis results."

[2311] In the "Health Improvement Suggestion" example, the user types "Enter my recent meal and exercise records" into the dedicated app. The device then sends the input information to a cloud server. The server uses a generative AI model to propose optimal meal plans and exercise methods based on the user's data, and displays them on the device.

[2312] Example prompt sentence:

[2313] "Please suggest the best improvement plan based on my recent diet and exercise data."

[2314] In the "Doctor's Diagnosis Assistance Example," the user enters "I have persistent abdominal pain. I have already uploaded my past medical data" into a dedicated app. The device then sends the entered data to a cloud server. The server then analyzes the symptoms using a natural language processing model, lists possible illnesses, and presents recommended treatments to the medical professional.

[2315] Example prompt sentence:

[2316] "I would like some diagnostic support based on my abdominal pain symptoms and past medical data."

[2317] The system aims to integrate multiple data sources and reduce the burden on medical facilities through automation and efficiency.

[2318] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2319] Biometric data collection and management

[2320] Step 1:

[2321] A user speaks "I'm not feeling well" into a voice input device, which collects voice data.

[2322] Input: User's voice

[2323] Output: Audio data

[2324] Step 2:

[2325] The device sends the collected voice data to the Google Cloud Speech-to-Text API, converts it into text data, and sends the converted text data to a cloud server.

[2326] Input: Audio data

[2327] Data processing: Converting voice data into text

[2328] Output: Text data

[2329] Step 3:

[2330] A user collects a video of their face using the camera on their smartphone, thereby acquiring video data.

[2331] Input: User video

[2332] Output: Video data

[2333] Step 4:

[2334] The device uses the Azure Face API to identify individuals from the video data collected, and sends the results to a cloud server.

[2335] Input: Video data

[2336] Data processing: Personal identification of video data

[2337] Output: Personally Identifiable Data

[2338] Step 5:

[2339] Users measure their health data using a blood pressure monitor or thermometer and enter it into a dedicated app, which then registers the health data in the app.

[2340] Input: Health data

[2341] Output: Input health data

[2342] Step 6:

[2343] The device sends the health data entered into a dedicated app to the Google Firebase cloud server.

[2344] Input: Health data

[2345] Output: Health data sent to the cloud server

[2346] Step 7:

[2347] The server integrates the received audio, video, and health data and analyzes it using a Python data analysis script. The analysis results are displayed on a dashboard.

[2348] Input: Audio data, video data, health data

[2349] Data Computing: Data Integration and Analysis

[2350] Output: Analysis results displayed on a dashboard

[2351] Generate and propose health improvement plans

[2352] Step 1:

[2353] Users enter their diet and exercise habits into a dedicated mobile app.

[2354] Input: Dietary information, exercise amount

[2355] Output: Data entered into the dedicated app

[2356] Step 2:

[2357] The device sends the habit information and genetic information entered into a dedicated app to the Google Firebase cloud server.

[2358] Input: Habit information, genetic information

[2359] Output: Data sent to the cloud server

[2360] Step 3:

[2361] The server processes the data using a Hadoop cluster to analyze the user's data, and then generates health improvement recommendations using a generative AI model based on the processed data.

[2362] Input: Habit information, genetic information

[2363] Data calculation: Data analysis and generation of improvement proposals using generative AI models

[2364] Output: Health improvement plan

[2365] Step 4:

[2366] The health improvement suggestions generated by the server are sent to a dedicated app and presented to the user.

[2367] Input: Health Improvement Suggestion

[2368] Output: Improvement suggestions displayed in the dedicated app

[2369] Assisting doctors in diagnosis

[2370] Step 1:

[2371] The user enters into the dedicated app, "I have persistent abdominal pain. I have already uploaded my past medical data."

[2372] Input: Symptom data, past medical data

[2373] Output: Data entered into the dedicated app

[2374] Step 2:

[2375] The device sends the entered symptom data and past medical data to the Google Firebase cloud server.

[2376] Input: Symptom data, past medical data

[2377] Output: Data sent to the cloud server

[2378] Step 3:

[2379] The server analyzes the received data using natural language processing technology to predict possible diseases, and then organizes the predicted diseases and recommended treatments.

[2380] Input: Symptom data, past medical data

[2381] Data Computing: Analysis and Disease Prediction with Natural Language Processing

[2382] Output: predicted disease, recommended treatment

[2383] Step 4:

[2384] The server sends the diagnosis results and recommended treatments to medical professionals via an information delivery service and displays them on a dashboard.

[2385] Input: predicted disease, recommended treatment

[2386] Output: Diagnosis and treatment presented to medical professionals

[2387] (Application example 1)

[2388] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2389] In modern factories, there is a lack of means to monitor the health status and work safety of workers in real time and immediately report any abnormalities. This increases the risk of worker health risks and the likelihood of accidents. Furthermore, there is a lack of effective real-time data collection and analysis to support doctors' diagnoses. There is a need to provide a system that can resolve this situation and efficiently manage the safety of the work environment and the health of workers.

[2390] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2391] In this invention, the server includes means for collecting patient voice data and converting the voice data into text data, means for collecting patient video data and using the video data for personal identification, means for inputting, integrating, and analyzing health data, means for displaying the analysis results on a dashboard, means for generating and presenting optimal lifestyle improvement plans for each patient, means for presenting possible diseases and recommended treatments to assist doctors in making diagnoses, means for collecting worker health data in real time, means for immediately issuing an alert if an abnormality is detected, means for collecting and analyzing work environment data, and means for monitoring work safety based on the analyzed data. This enables real-time monitoring of worker health conditions and work environment safety and rapid notification of abnormalities.

[2392] The "means for collecting patient voice data and converting the voice data into text data" is a technology for acquiring the patient's voice and converting the voice into text information.

[2393] The "means for collecting patient image data and using the image data for personal identification" refers to a technology for acquiring patient images and identifying individuals from the images.

[2394] "Means for inputting, integrating, and analyzing health data" refers to technology for inputting health information such as blood pressure and body temperature into a system and integrating and analyzing it.

[2395] The "means for displaying the analysis results on a dashboard" is an interface for visually displaying the analyzed health information.

[2396] "Means for generating and presenting optimal lifestyle improvement proposals for individual patients" refers to technology for automatically creating and presenting health improvement proposals tailored to each patient.

[2397] "Means for presenting possible diseases and recommended treatments to assist doctors in diagnosis" refers to technology that uses AI and data analysis to provide possible diseases and recommended treatments to assist in diagnosis.

[2398] "Means for collecting worker health data in real time" refers to technology for obtaining health information on workers working in factories in real time.

[2399] "Means for immediately issuing an alert in the event of an abnormality" refers to technology that immediately issues a warning when an abnormality is detected in health data or work environment data.

[2400] "Means for collecting and analyzing work environment data" refers to technology for acquiring data related to the work environment within a factory and analyzing it.

[2401] "Means for monitoring work safety based on analytical data" refers to technology for constantly monitoring work safety based on analyzed data.

[2402] This invention is a system for improving the health management and work safety of factory workers. This system uses technology to collect, analyze, and display audio, video, and health data in real time.

[2403] Specifically, the system includes the following means:

[2404] Sensor data collection and management

[2405] User:

[2406] Report your health and symptoms using a voice input device.

[2407] A camera device is used to collect facial images.

[2408] Health data is entered using devices such as blood pressure monitors and thermometers.

[2409] Device:

[2410] Voice data is collected and converted into text data using voice recognition technology.

[2411] Video data is collected and workers are identified using personal identification technology.

[2412] Health data is collected, associated with the worker's profile, and sent to a cloud server.

[2413] server:

[2414] The transmitted data is consolidated and analyzed.

[2415] Calculates the worker's health index and issues an alert if any abnormalities are detected.

[2416] The results are displayed on a dashboard and notified to the work team leader as needed.

[2417] Improving health management and work safety

[2418] User:

[2419] Enter information about your daily habits (e.g., diet, exercise, etc.) into a dedicated app.

[2420] Provide work environment data as needed.

[2421] Device:

[2422] The input habit information and work environment data are sent to the cloud server.

[2423] server:

[2424] Analyze habit information and work environment data.

[2425] Generate optimal health improvement proposals and safety measures.

[2426] Improvement suggestions are presented to users via dashboards and notification functions.

[2427] Specific examples

[2428] Examples of biometric data self-management support:

[2429] 1. The user speaks into a voice input device, "I'd like to enter my blood pressure record for this week."

[2430] 2. The device collects the voice data and sends it to the cloud server.

[2431] 3. The server analyzes the data, generates a graph of blood pressure fluctuations for this week, and displays it on the device.

[2432] Examples of health improvement suggestions:

[2433] 1. The user types "Enter recent food and exercise records" into the dedicated app.

[2434] 2. The device sends the input information to the cloud server.

[2435] 3. The server proposes the optimal meal plan and exercise method based on the user's data and displays it on the device.

[2436] Hardware and software used

[2437] Hardware:

[2438] Smart glasses camera: Used to collect video data.

[2439] Microphone in smart glasses: Used to collect voice data.

[2440] Health monitoring devices: thermometers, blood pressure monitors, etc.

[2441] software:

[2442] Python: The main programming language.

[2443] OpenCV: A library for video data analysis.

[2444] SpeechRecognition: A library for analyzing speech data.

[2445] Cloud server: Stores and analyzes health data.

[2446] Example prompt for a generative AI model:

[2447] A worker speaks to the smart glasses and says, "Please check my health." The smart glasses monitor the worker's body temperature, blood pressure, complexion, and other conditions in real time, sending the data to the cloud. Cloud AI analyzes the data, and if any abnormalities are detected, an alert is sent to the work team leader's smartphone.

[2448] By using this system, it is expected that health management and safety assurance for workers in factories will be automated and carried out efficiently.

[2449] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2450] Program processing flow of the system that realizes the application example

[2451] Step 1:

[2452] The user speaks to the smart glasses, saying, "Check your health."

[2453] Specific behavior:

[2454] Input: Audio data

[2455] Output: Raw audio data

[2456] The user speaks into the smart glasses, and the microphone in the glasses collects the voice data, which is then converted into text data using voice recognition technology.

[2457] Step 2:

[2458] The device sends the voice data to the cloud server.

[2459] Specific behavior:

[2460] Input: Audio data

[2461] Output: Text data

[2462] The device (smart glasses) sends the collected voice data to a cloud server, where it is converted into text data using voice recognition technology and then sent.

[2463] Step 3:

[2464] Collect user health data (body temperature, blood pressure).

[2465] Specific behavior:

[2466] Input: Data from health monitoring devices

[2467] Output: Health data (body temperature, blood pressure)

[2468] A user uses a health monitoring device to measure their body temperature and blood pressure, and the data is sent to a cloud server via smart glasses.

[2469] Step 4:

[2470] The server integrates and analyzes the transmitted audio data, health data, and video data.

[2471] Specific behavior:

[2472] Input: Audio data, health data, video data

[2473] Output: Analysis results

[2474] The server integrates the various data sent and analyzes it using machine learning models or data analysis algorithms, using Python or dedicated analysis libraries.

[2475] Step 5:

[2476] The server displays the analysis results on a dashboard and issues an alert if an abnormality is detected.

[2477] Specific behavior:

[2478] Input: Analysis results

[2479] Output: Dashboard display, alert notification

[2480] The server updates the dashboard based on the analysis results and displays them visually. If an abnormality is detected, an alert is sent to the team leader's smartphone in real time.

[2481] Step 6:

[2482] The server generates health improvement suggestions based on the worker's habit information and presents them on the terminal.

[2483] Specific behavior:

[2484] Input: Habit information, analysis results

[2485] Output: Health improvement plan

[2486] The server analyzes the user's habit information and health data and generates health improvement suggestions, which are presented as suggestions to the user's smart glasses or associated devices.

[2487] In this way, the system collects, analyzes, displays, and notifies data at each step, monitoring worker health and the safety of the work environment in real time, enabling rapid response when an abnormality occurs.

[2488] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2489] The present invention aims to incorporate an emotion engine into a medical management system that utilizes multimodal AI technology, thereby making diagnoses and suggestions that reflect the patient's emotions, reducing the burden on medical professionals, improving the accuracy of diagnoses, and providing lifestyle improvement suggestions tailored to the needs of each individual patient.

[2490] 1. Collection and management of biometric and emotional data

[2491] User:

[2492] Report your health and symptoms using a voice input device.

[2493] A camera device is used to collect facial images.

[2494] Health data is entered using devices such as blood pressure monitors and thermometers.

[2495] Device:

[2496] Voice data is collected and converted into text data using voice recognition technology.

[2497] Video data is collected and individuals are identified using facial recognition technology.

[2498] Health data is collected, associated with the user's profile, and sent to a cloud server.

[2499] Emotions are recognized from audio data and video data, and emotion data is generated.

[2500] server:

[2501] The transmitted audio data, video data, health data, and emotional data are integrated and analyzed.

[2502] Calculates the user's health and emotional indicators and displays the results on a dashboard.

[2503] 2. Generating and Proposing Health Improvement Plans Using Emotional Data

[2504] User:

[2505] Enter information about your daily habits (e.g., diet, exercise, etc.) into a dedicated app.

[2506] Provide genetic information as needed.

[2507] Device:

[2508] The entered habit information and genetic information is sent to a cloud server.

[2509] server:

[2510] The user's habits, genetic information, and emotional data are analyzed.

[2511] Based on emotional data, optimal health improvement proposals are generated, including meal plans and exercise methods.

[2512] Improvement suggestions are presented to users via dashboards and notification functions.

[2513] 3. Assisting doctors in diagnosis and utilizing emotion data

[2514] User:

[2515] Enter your symptoms of illness by voice or text.

[2516] Provide historical medical data.

[2517] Device:

[2518] The entered symptom data and past medical data are sent to a cloud server.

[2519] server:

[2520] Analyzes voice and text data to predict possible diseases.

[2521] Compilation of supplementary information to improve diagnostic accuracy is carried out by comparing it with past medical data and referring to emotional data.

[2522] Present the diagnosis and recommended treatment to the physician.

[2523] Specific examples

[2524] Examples of biometric and emotional data self-management support:

[2525] 1. The user speaks into a voice input device, "I'd like to enter my blood pressure record for this week."

[2526] 2. The device collects the voice data and sends it to the cloud server.

[2527] 3. The device recognizes emotions from the voice data and generates emotion data.

[2528] 4. The server analyzes the data, generates a graph of blood pressure fluctuations for this week, and displays it on the device along with emotional data.

[2529] Example of using emotion data to provide health improvement suggestions:

[2530] 1. The user types "Enter recent food and exercise records" into the dedicated app.

[2531] 2. The device sends the input information to the cloud server.

[2532] 3. The server proposes optimal meal plans and exercise methods based on the user's data, and displays them on the device, reflecting the user's emotional data.

[2533] Examples of using emotion data to assist doctors in diagnosis:

[2534] 1. The user enters the following into the dedicated app: "I have persistent abdominal pain. I have already uploaded my past medical data."

[2535] 2. The device sends the entered data to the cloud server.

[2536] 3. The server analyzes the symptoms, lists possible illnesses based on the emotional data, and presents recommended treatments to the doctor.

[2537] By automating and streamlining these processes, the system of the present invention reduces the burden on medical professionals, improves diagnostic accuracy, and significantly contributes to eliminating disparities in medical care. Furthermore, by using emotion data, comprehensive health management that takes into account the patient's psychological state becomes possible.

[2538] The processing flow will be explained below.

[2539] (Collection and management of biometric and emotional data)

[2540] Audio data recording and conversion

[2541] Step 1:

[2542] The user speaks about their physical condition and symptoms into a voice input device.

[2543] Step 2:

[2544] The terminal receives the audio signal and temporarily stores it as audio data.

[2545] Step 3:

[2546] The terminal uses voice recognition technology to convert the voice data into text data.

[2547] Step 4:

[2548] The terminal transmits the converted text data to the cloud server.

[2549] Video data recording and facial recognition

[2550] Step 1:

[2551] The user faces the camera.

[2552] Step 2:

[2553] The device collects and temporarily stores video data.

[2554] Step 3:

[2555] The device runs a facial recognition algorithm to identify the individual.

[2556] Step 4:

[2557] The device transmits the video data and identification information to the cloud server.

[2558] Generating emotion data

[2559] Step 1:

[2560] The terminal recognizes emotions from audio data and video data.

[2561] Step 2:

[2562] The device generates the recognized emotion data and temporarily stores it.

[2563] Step 3:

[2564] The device transmits the emotion data to the cloud server.

[2565] Health Data Collection and Transmission

[2566] Step 1:

[2567] Users enter health data such as blood pressure and body temperature into a dedicated app or device.

[2568] Step 2:

[2569] The terminal associates the entered health data with the user's profile and temporarily stores it.

[2570] Step 3:

[2571] The device sends the health data to a cloud server.

[2572] Data integration and analysis

[2573] Step 1:

[2574] The server integrates the received audio data, video data, health data, and emotional data.

[2575] Step 2:

[2576] The server calculates health and emotional indicators using a proprietary algorithm.

[2577] Step 3:

[2578] The server converts health and emotional indicators into graphs and statistical information to generate a dashboard.

[2579] Step 4:

[2580] The server sends the dashboard and result display data to the terminal.

[2581] (Providing health improvement ideas)

[2582] Entering daily habits

[2583] Step 1:

[2584] The user inputs information about their daily habits, such as what they eat and how much exercise they do, into a dedicated app.

[2585] Step 2:

[2586] The device organizes the input habit information by associating it with the user's profile.

[2587] Step 3:

[2588] The device sends the organized habit information to a cloud server.

[2589] Analysis of habit information and genetic information

[2590] Step 1:

[2591] The server integrates the received habit information with the genetic information provided by the user.

[2592] Step 2:

[2593] The server uses statistical and genetic techniques to generate optimal health improvement recommendations.

[2594] Customization using emotional data

[2595] Step 1:

[2596] The server customizes the generated health improvement proposals by reflecting emotional data.

[2597] Proposal of improvement proposals

[2598] Step 1:

[2599] The server then compiles the generated health improvement suggestions into a display format on a dashboard for the user.

[2600] Step 2:

[2601] The server sends the improvement proposal data to the terminal.

[2602] Step 3:

[2603] The improvement suggestions received by the device are displayed on the user's dashboard and an alert is sent via a notification function.

[2604] (Assisting doctors in diagnosis)

[2605] Enter symptoms

[2606] Step 1:

[2607] The user enters their symptoms of illness into a dedicated app by voice or text.

[2608] Step 2:

[2609] The terminal converts the input symptom data into text data and temporarily stores it.

[2610] Step 3:

[2611] The device sends the text data to the cloud server.

[2612] Referencing past medical data

[2613] Step 1:

[2614] The server looks up past medical data from the user's profile.

[2615] Step 2:

[2616] The server analyzes the historical data and the new symptom data.

[2617] Diagnosis and treatment suggestions

[2618] Step 1:

[2619] The server uses AI algorithms to predict possible diseases.

[2620] Step 2:

[2621] The server compares the data with past medical data and uses emotional data to organize supplementary information to improve diagnostic accuracy.

[2622] Step 3:

[2623] The server compiles the diagnosis and recommended treatment in the form of a report for the physician.

[2624] Step 4:

[2625] The server sends the report to the device.

[2626] Step 5:

[2627] The device receives a diagnosis and displays treatment suggestions on a screen for the doctor.

[2628] Example 2

[2629] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2630] The current medical system does not take patients' emotions into consideration when making diagnoses or recommendations, which increases the burden on medical professionals and results in insufficient diagnostic accuracy. Furthermore, there is a lack of systems that provide lifestyle improvement suggestions tailored to individual patients' needs, making it difficult for patients to manage their own health. Furthermore, there is a need for improved data analysis and disease prediction accuracy to assist doctors in their diagnoses.

[2631] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2632] In this invention, the server includes: means for collecting a patient's voice data and converting the voice data into text data; means for collecting a patient's video data and using the video data for personal identification; means for inputting health and emotional data and transmitting it to a cloud server; means for recognizing emotions from the voice and video data and generating emotional data; means for integrating and analyzing the transmitted voice, video, health, and emotional data; means for displaying the analysis results on a dashboard; means for generating and presenting optimal lifestyle improvement proposals for each patient; and means for presenting possible diseases and recommended treatments using a generative AI model to assist physicians in their diagnosis. This enables diagnoses and proposals that take the patient's emotions into account, reducing the burden on medical professionals and improving diagnostic accuracy. Furthermore, providing lifestyle improvement proposals tailored to each patient's needs makes it easier for patients to manage their own health.

[2633] The "means for collecting patient voice data and converting the voice data into text data" is a mechanism for collecting voice data reported by the patient through a voice input device and converting the voice data into text format.

[2634] The "means for collecting patient image data and using the image data for personal identification" is a mechanism for acquiring an image of the patient's face and using the image data to identify the individual.

[2635] "Means for inputting health data and emotional data and transmitting it to a cloud server" refers to a mechanism for inputting information about a patient's health and emotions and transmitting that data to a server on the cloud.

[2636] The "means for recognizing emotions from audio data and video data and generating emotional data" refers to a mechanism for analyzing collected audio data and video data to recognize the patient's emotions and generate data related to emotions.

[2637] "Means for integrating and analyzing transmitted audio data, video data, health data, and emotional data" refers to a mechanism for integrating and comprehensively analyzing various data transmitted to the cloud server.

[2638] "Means for displaying analysis results on a dashboard" refers to a mechanism for outputting the results of the analyzed data to a dashboard for visually displaying them.

[2639] "A means for generating and presenting optimal lifestyle improvement proposals for individual patients" is a system that generates optimal lifestyle improvement proposals based on data on each patient and presents them to the patient.

[2640] "Means of using generative AI models to suggest possible diseases and recommended treatments to assist doctors in their diagnoses" refers to a system that uses generative AI technology to predict possible diseases from collected data and provide doctors with optimal treatments.

[2641] This invention is a medical management system that utilizes multimodal AI technology and incorporates an emotion engine to make diagnoses and recommendations that reflect the patient's emotions, reducing the burden on medical professionals. The system also aims to improve the accuracy of diagnoses and provide lifestyle improvement proposals tailored to the individual needs of each patient.

[2642] The system consists of three main components: users, terminals, and servers.

[2643] User

[2644] A user reports their physical condition and symptoms using a voice input device. A camera device is also used to collect facial images, and health data is input using devices such as a blood pressure monitor and a thermometer. As a specific example, data collection begins when a user speaks into the voice input device, saying, "I'd like to enter this week's blood pressure record."

[2645] Terminal

[2646] The terminal uses multiple pieces of hardware and software to collect and process data.

[2647] The audio data is converted into text data using the Google Cloud Speech-to-Text service.

[2648] Video data is collected and individuals are identified using the Microsoft Azure Face API.

[2649] Health data is sent to AWS cloud servers.

[2650] Emotions are recognized from audio and video data using IBM Watson's Natural Language Understanding and Visual Recognition, and emotion data is generated.

[2651] For example, when a user types "I would like to enter my recent meal and exercise records" into a dedicated app, that information is sent to the cloud server.

[2652] server

[2653] The server is the central component that processes data on the cloud.

[2654] The audio data, video data, health data, and emotion data sent to the cloud server are integrated and analyzed using data processing libraries such as Python's Pandas and NumPy.

[2655] Calculates user health and emotional indicators and displays the results on a dashboard using Tableau or Power BI.

[2656] For example, it can recognize emotions based on voice data and evaluate stress levels to provide comprehensive health management. It also uses GPT-4 as a generative AI model to generate optimal health improvement proposals and suggest optimal diet plans and exercise methods for users.

[2657] Specific examples

[2658] Examples of biometric and emotional data self-management support:

[2659] The user speaks to the voice input device, saying, "I'd like to enter my blood pressure record for this week." The device then collects the voice data and sends it to a cloud server using the Google Cloud Speech-to-Text service. The device then uses IBM Watson's Natural Language Understanding to recognize emotions from the voice data and generate emotion data. The server then analyzes this data, generates a graph of blood pressure fluctuations for this week, and displays it on a Tableau dashboard along with the emotion data.

[2660] Example of using emotion data to provide health improvement suggestions:

[2661] The user enters text data into the dedicated app, saying, "I'd like to enter my recent meal and exercise records." The device then sends the input information to an AWS cloud server. The server generates an optimal meal plan and exercise method based on the user's data, and uses GPT-4 to create suggestions that also reflect emotional data, which are then displayed on the device.

[2662] Examples of using emotion data to assist doctors in diagnosis:

[2663] The user enters into the dedicated app, "I have persistent stomach pain. I have already uploaded my past medical data." The device then sends the entered data to an AWS cloud server. The server analyzes the symptoms, lists possible illnesses based on the emotional data, and generates a recommended treatment using GPT-4 to present to the doctor.

[2664] By automating and streamlining these processes, this system reduces the burden on medical professionals and improves the accuracy of diagnoses. It also enables comprehensive health management based on emotional data, providing patients with optimal treatments and lifestyle improvement suggestions.

[2665] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2666] Step 1:

[2667] The user reports their physical condition and symptoms using a voice input device.

[2668] Input: User voice input (e.g., "Enter this week's blood pressure record.")

[2669] Data processing: The device collects voice data and converts it into text data using the Google Cloud Speech-to-Text service.

[2670] Output: Report content as text data (e.g., "Enter your blood pressure record for this week")

[2671] Step 2:

[2672] A user uses a camera device to collect facial images.

[2673] Input: User's face image

[2674] Data processing: The device collects video data and uses the Microsoft Azure Face API to perform facial recognition and recognize emotions from the video data.

[2675] Output: Identified person ID and emotion data

[2676] Step 3:

[2677] A user inputs health data using devices such as a blood pressure monitor and a thermometer.

[2678] Input: Health data (e.g. blood pressure, body temperature)

[2679] Data processing: The device collects health data and sends it to AWS cloud servers.

[2680] Output: Health data stored on a cloud server

[2681] Step 4:

[2682] Sending audio and video data to the cloud

[2683] Input: Text data, identified personal ID, emotion data

[2684] Data processing: The device integrates the audio and video data to generate emotion data, which is then sent to the AWS cloud server.

[2685] Output: Integrated data sent to cloud server

[2686] Step 5:

[2687] The audio data, video data, health data, and emotional data sent to the cloud server are integrated and analyzed.

[2688] Input: Audio data, video data, health data, and emotion data stored on a cloud server

[2689] Data processing: The server integrates this data and analyzes it using data processing libraries such as Python's Pandas and NumPy.

[2690] Output: Integrated data analysis results (e.g., health index, emotional index)

[2691] Step 6:

[2692] Display the analysis results on a dashboard.

[2693] Input: Integrated data analysis results

[2694] Data processing: The server visualizes and displays the analysis results on a dashboard using Tableau or Power BI.

[2695] Output: Dashboard screen viewable by users and doctors

[2696] Step 7:

[2697] The user enters information about their daily habits into a dedicated app.

[2698] Input: User's habit information (e.g., dietary content, amount of exercise)

[2699] Data processing: The terminal sends the input data to the AWS cloud server.

[2700] Output: Habit information stored on the cloud server

[2701] Step 8:

[2702] The server generates and presents health improvement suggestions based on the user's habit information, health data, and emotional data.

[2703] Input: Habit information, health data, emotional data

[2704] Data processing: The server analyzes these data and generates optimal health improvement recommendations using a generative AI model (GPT-4).

[2705] Output: Generated health improvement suggestions, including recommendations displayed in notifications and on the dashboard

[2706] Step 9:

[2707] Providing data to assist doctors in diagnosis

[2708] Input: User's symptom information, past medical data

[2709] Data processing: The device sends the input data to the AWS cloud server, which analyzes the data and uses GPT-4 to list possible diseases based on the emotional data.

[2710] Output: A list of diagnoses and treatment recommendations presented to the doctor

[2711] At each step, the system processes and analyzes the collected data to provide optimal improvement suggestions for each individual user and diagnostic support information for doctors, thereby realizing comprehensive health management that takes the patient's emotions into consideration.

[2712] (Application example 2)

[2713] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2714] Conventional medical management systems provide diagnoses and treatment recommendations based on biometric data and past medical data, but they do not take into account the patient's emotions or psychological state, making it difficult to fully meet the needs of individual patients. Furthermore, food delivery services, in particular, do not propose optimal meal plans that reflect the user's health status and emotions, making it difficult to provide individually customized services. Therefore, new methods are needed to increase patient and user satisfaction, reduce the burden on medical professionals, and improve diagnostic accuracy.

[2715] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2716] In this invention, the server includes means for collecting patient voice data and converting the voice data into text data, means for collecting patient video data and using the video data for personal identification, means for inputting, integrating, and analyzing health data, means for displaying the analysis results on a dashboard, means for generating and presenting optimal lifestyle improvement plans for each patient, means for presenting possible diseases and recommended treatments to assist doctors in making diagnoses, means for collecting user emotion data and analyzing the emotion data to reflect it in health improvement plans, and means for inputting health reports by voice or text and proposing and delivering optimal meal plans. This enables personalized medical management and food delivery services that comprehensively consider the emotions and health status of patients and users.

[2717] "Voice data" refers to data for collecting voice information from patients and converting the content into text data.

[2718] "Video data" refers to data that collects images of a patient's face or body and is used to identify the individual.

[2719] "Health data" refers to physiological information such as a patient's blood pressure, body temperature, and heart rate, and is data that can be integrated and analyzed.

[2720] A "dashboard" is an interface for visually displaying information such as analysis results and proposals.

[2721] A "lifestyle improvement plan" is a specific action plan proposed to improve the quality of daily life of patients or users.

[2722] "Aiding diagnosis" means providing supplementary information or recommendations to help a doctor make an appropriate diagnosis of a patient's symptoms.

[2723] "Disease" refers to an illness or condition that adversely affects a patient's health.

[2724] A "therapy" is a specific medical procedure or method used to treat a disease.

[2725] "Emotional data" refers to data about the user's psychological state and emotions extracted from facial expressions, etc.

[2726] "Health report" refers to the act of a patient or user recording or reporting their current health condition or symptoms.

[2727] "Meal Plan" refers to a meal menu and plan suggested based on the user's health condition and mood.

[2728] "Delivery" refers to the act of delivering cooked meals based on the proposed meal plan to the user.

[2729] This invention is a system that collects and analyzes biometric and emotional data of patients and users, and provides optimal lifestyle improvement proposals and health management plans based on the collected data. The system aims to provide individually customized services that comprehensively consider the health status and emotions of patients and users.

[2730] 1. Collection and management of biometric and emotional data

[2731] User:

[2732] Users report their physical condition and symptoms using a voice input device, which is collected by the terminal as voice data. A camera device is also used to collect facial images. Furthermore, health data is input using devices such as a blood pressure monitor and a thermometer.

[2733] Device:

[2734] The device collects voice data and converts it into text data using voice recognition technology. It also collects video data and identifies individuals using facial recognition technology. The collected health data is associated with the user's profile and sent to a cloud server. At the same time, emotions are recognized from the voice and video data, and emotion data is generated.

[2735] server:

[2736] The server integrates and analyzes the transmitted audio, video, health, and emotional data, calculates the user's health and emotional indices, and displays the results on a dashboard.

[2737] 2. Generating and Proposing Health Improvement Plans Using Emotional Data

[2738] User:

[2739] Users input information about their daily habits (e.g., dietary habits, amount of exercise, etc.) into a dedicated app, and also provide genetic information if necessary.

[2740] Device:

[2741] The terminal transmits the input habit information and genetic information to a cloud server.

[2742] server:

[2743] The server analyzes the user's habits, genetic information, and emotional data. Based on the emotional data, it generates optimal health improvement proposals, suggesting meal plans and exercise methods to the user. The proposals are presented to the user via a dashboard and notification function.

[2744] 3. Assisting doctors in diagnosis and utilizing emotion data

[2745] User:

[2746] Users can input their symptoms of illness by voice or text, and also provide past medical data.

[2747] Device:

[2748] The terminal transmits the input symptom data and past medical data to a cloud server.

[2749] server:

[2750] The server analyzes the voice and text data to predict possible illnesses, compares it with past medical data, and uses emotional data to organize supplementary information to improve diagnostic accuracy. The diagnosis results and recommended treatments are then presented to the doctor.

[2751] Specific examples

[2752] Examples of biometric and emotional data self-management support:

[2753] 1. The user speaks into a voice input device, "I'd like to enter my blood pressure record for this week."

[2754] 2. The device collects the voice data and sends it to the cloud server.

[2755] 3. The device recognizes emotions from the voice data and generates emotion data.

[2756] 4. The server analyzes the data, generates a graph of blood pressure fluctuations for this week, and displays it on the device along with emotional data.

[2757] Example of using emotion data to provide health improvement suggestions:

[2758] 1. The user types "Enter recent food and exercise records" into the dedicated app.

[2759] 2. The device sends the input information to the cloud server.

[2760] 3. The server proposes optimal meal plans and exercise methods based on the user's data, and displays them on the device, reflecting the user's emotional data.

[2761] Examples of using emotion data to assist doctors in diagnosis:

[2762] 1. The user enters the following into the dedicated app: "I have persistent abdominal pain. I have already uploaded my past medical data."

[2763] 2. The device sends the entered data to the cloud server.

[2764] 3. The server analyzes the symptoms, lists possible illnesses based on the emotional data, and presents recommended treatments to the doctor.

[2765] Prompt Sentence Examples

[2766] "Please suggest personalized food delivery based on the user's emotional and health data. Here's the user's data for this week:

[2767] Blood pressure: 110

[2768] Temperature: 36.5

[2769] Emotion: neutral

[2770] Please suggest a suitable meal plan and why.”

[2771] By building a system in this way, it is possible to comprehensively manage a user's health condition and emotions, and realize personalized medical services and food delivery.

[2772] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2773] Step 1:

[2774] A user reports their physical condition and symptoms using a voice input device. This voice data is collected and input data from the voice input device is generated.

[2775] Step 2:

[2776] The device receives the voice data and converts it into text data using speech recognition technology (e.g., Google Speech Recognition API). The input is voice data, and the output is text data.

[2777] Step 3:

[2778] A user uses a camera device to capture a video of his or her face, generating video data that is provided to a terminal as a video input.

[2779] Step 4:

[2780] The device receives the video data and identifies individuals using facial recognition technology (e.g., using OpenCV and dlib). The input is the video data, and the output is the data of the identified individuals.

[2781] Step 5:

[2782] A user inputs health data using a device such as a blood pressure monitor or a thermometer, and the data is sent to a terminal. The input is health data (blood pressure, body temperature, etc.).

[2783] Step 6:

[2784] The device associates the collected health data with the user's profile and transmits it to the cloud server. The input is the health data, and the output is the integrated health data transmitted to the cloud server.

[2785] Step 7:

[2786] The device recognizes emotions from audio and video data and generates emotion data (e.g., extracts emotions using the FER library). The input is audio and video data, and the output is emotion data.

[2787] Step 8:

[2788] The server integrates and analyzes the audio data, video data, health data, and emotion data sent to the cloud server. These data are the input, and the output is the health index and emotion index as the analysis results.

[2789] Step 9:

[2790] The server displays the analysis results on a dashboard, which includes graphs and indicators to visualize the user's health and emotional state. The input is the analysis results, and the output is the dashboard display.

[2791] Step 10:

[2792] The user inputs information about their daily habits (such as dietary habits and exercise amount) into a dedicated app. The input is habit information, which is then provided to the device.

[2793] Step 11:

[2794] The device sends habit information to the cloud s...

Claims

1. means for collecting voice data of a patient and converting the voice data into text data; means for collecting patient image data and using the image data for personal identification; A means to input, integrate and analyze health data; A means of displaying the analysis results on a dashboard; A means for generating and presenting optimal lifestyle improvement proposals for individual patients; A system that includes a means of presenting possible diseases and recommended treatments to assist a physician in diagnosis.

2. 10. The system of claim 1, further comprising the step of obtaining voice data from a voice input device and converting the voice data into text data.

3. 2. The system according to claim 1, further comprising means for collecting habit information of a patient and generating a health improvement plan based on the habit information.

4. The system according to claim 1, further comprising means for transmitting the acquired data to a cloud server and displaying the analysis results on a user terminal.

5. 2. The system according to claim 1, further comprising means for displaying the user's health condition as graphs and statistical information based on the analyzed health data.

Images