System

Abstract

A system is provided.SOLUTION: A system including a means for acquiring past health check data, a means for analyzing the acquired health check data using an image analysis means, a means for storing the analyzed data in a database, a means for predicting a health risk using a generated AI based on the stored data, a means for generating a personalized health advice based on the predicted health risk, a means for notifying a user of the generated advice or a prediction result, and an information acquisition means for acquiring and integrating external tracking data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, many people undergo regular health checkups, but it is difficult to centrally manage the results and understand their health status by comparing them with past data. Furthermore, because health checkup data is not organized chronologically, it is difficult to predict individual health risks and take appropriate measures. Furthermore, there are few means to consistently manage health advice and medication information, making it difficult for users to comprehensively manage their own health status. These challenges have led to a demand for centralizing individuals' medical histories, predicting health risks, and providing personalized health advice. [Means for solving the problem]

[0005] The present invention provides a system that allows a user to acquire past health checkup data, analyze that data using image analysis means, and store it in a database. The analyzed data is used to predict health risks using generative AI, and personalized health advice is generated based on those risks. The system also includes a means for notifying the user of the generated advice and prediction results. The system further includes a means for integrating health checkup data with external tracking data and managing and notifying information about the user's medication, enabling comprehensive management of the user's health status. The system also includes a means for providing the generated health advice and prediction results to medical professionals and displaying a summary of the analysis results, facilitating smooth communication with medical professionals.

[0006] "Health checkup data" refers to various medical data obtained by users through regular health checkups.

[0007] "Image analysis means" means a system for extracting text data from an image using techniques such as optical character recognition (OCR).

[0008] "Generative AI" is a system that uses artificial intelligence to analyze data and generate health risk predictions and personalized advice.

[0009] A "database" is a system for storing and managing analyzed health checkup data.

[0010] "Health risk" refers to the predicted likelihood of illness or disease based on the user's health status.

[0011] "Personalized health advice" means health-related advice or instructions that are individually tailored to a user and are provided based on that user's individual health data.

[0012] "External tracking data" refers to data obtained from apps that track exercise, sleep, nutrition, etc.

[0013] "Drug information" refers to data such as the name, usage, dosage, and side effects of the medication the user is taking.

[0014] A "healthcare professional" is someone with specialized medical knowledge, such as a doctor or nurse.

[0015] "Notification tools" are features that inform users of health risk predictions and personalized health advice. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention is a system that effectively utilizes a user's health checkup data to provide personalized health management. This system has a wide range of functions, including predicting health risks using AI generated from image analysis of health checkup data, a medicine notebook function, and linking with external tracking data. Specific embodiments of the system are described below.

[0038] Acquisition and analysis of health checkup data

[0039] The user takes a photo of the health checkup result form with the smartphone camera. The system receives the data by uploading the image to the terminal. The server sends the received image to an image analysis means and extracts text data from the image using optical character recognition (OCR) technology.

[0040] Identifying and storing text data

[0041] The extracted text data is sent to a natural language processing (NLP) module on the server, which identifies health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.) and stores the identified data in a database.

[0042] Predicting health risks and generating personalized advice

[0043] The server then uses the AI ​​generator to analyze the stored health checkup data and predict the user's health risks. Based on the prediction results, the AI ​​generator generates personalized health advice, such as "increase exercise to prevent diabetes" or "reduce sugar in your diet."

[0044] Medicine notebook function

[0045] The server also has a medicine notebook function, automatically managing the medicine information entered by the user. The entered medicine information is analyzed and important information regarding medication is notified to the user.

[0046] External tracking data integration and integration

[0047] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app. The device then sends data obtained from the external apps to a server, which then integrates this data with health checkup data. For example, by adding exercise data, the AI ​​can provide detailed advice such as "running for 30 minutes three times a week is recommended."

[0048] Providing advice and notifications

[0049] Finally, the device displays the analysis results and health advice from the server to the user, and notifies the user of important health risks and medication information via push notifications.

[0050] Specific examples

[0051] For example, a user takes a photo of their health checkup results for October 2023 with their smartphone and uploads it to the app. This data is analyzed by the server, and the value "blood sugar level: 110 mg / dL" is extracted and stored in the database. The generation AI uses this data to predict that the user has a high risk of diabetes and generates personalized advice such as "increase exercise to prevent diabetes." Furthermore, the user sets up a link with an exercise tracking app, and the server analyzes the data to suggest specific amounts of exercise.

[0052] In this way, users can effectively utilize their own health checkup data and take appropriate measures. Data from other linked health apps can also be utilized, enabling more comprehensive health management. This system will help users extend their healthy lifespan and facilitate smooth communication with medical professionals.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] The user takes a photo of the past health checkup results form using the smartphone camera.

[0056] Step 2:

[0057] The device will upload the captured images to the app.

[0058] Step 3:

[0059] The server receives the uploaded images and transmits them to the image analysis means.

[0060] Step 4:

[0061] The server's image analysis means uses optical character recognition (OCR) technology to extract text data from the image.

[0062] Step 5:

[0063] The server sends the extracted text data to a natural language processing (NLP) module.

[0064] Step 6:

[0065] The server's NLP module analyzes the text data and identifies health check items (e.g., blood sugar level, cholesterol level, BMI, etc.).

[0066] Step 7:

[0067] The server stores the identified data in a database.

[0068] Step 8:

[0069] The server sends the health checkup data stored in the database to the generation AI, which analyzes the data.

[0070] Step 9:

[0071] The server's generated AI predicts the user's health risks.

[0072] Step 10:

[0073] The server's generative AI generates personalized health advice based on predicted health risks.

[0074] Step 11:

[0075] The server formats and prepares the generated health advice for the user.

[0076] Step 12:

[0077] The server analyzes the medication information entered by the user and manages that information.

[0078] Step 13:

[0079] The server initiates a notification mechanism to notify the user of important information about the medication.

[0080] Step 14:

[0081] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app.

[0082] Step 15:

[0083] The terminal transmits the data acquired from the external application to the server.

[0084] Step 16:

[0085] The server integrates the health check data with data from external apps and sends it to the generation AI.

[0086] Step 17:

[0087] The server's generative AI generates detailed personalized advice based on the integrated data.

[0088] Step 18:

[0089] The device displays the analysis results and health advice from the server to the user.

[0090] Step 19:

[0091] The device will send users push notifications about important health risks and medication information.

[0092] Example 1

[0093] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0094] Conventional health management systems require users to manually input data, which is time-consuming and laborious, and also carries a high risk of input errors. Furthermore, because they only reference individual health checkup data, it is difficult to comprehensively grasp a user's current health status. Furthermore, it is difficult to integrate and utilize external tracking data. As a result, it is not possible to effectively provide personalized advice to users, and they are unable to provide sufficient support for maintaining their health and preventing disease.

[0095] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0096] In this invention, the server includes a means for uploading images of health checkup data taken by a user using a smartphone, a means for analyzing the uploaded images using an image analysis means and extracting text data using optical character recognition technology, a means for analyzing the extracted text data using a natural language processing module to identify health checkup items and store them in a database, a means for analyzing the stored health checkup data using a generative AI model and predicting the user's health risks, a means for generating personalized health advice based on the predicted health risks, a means for notifying the user of the generated advice and prediction results, an information acquisition means for acquiring and integrating external tracking data, and a means for integrating the acquired external tracking data and health checkup data to generate detailed advice. This allows users to easily import health checkup data and perform comprehensive health management. Furthermore, the automatically generated personalized advice allows users to take appropriate measures to maintain their health and prevent disease.

[0097] "User" means an individual who uses the system to provide health checkup results and external tracking data.

[0098] A "smartphone" is a portable information terminal equipped with a camera and applications that allows users to take and upload health checkup results.

[0099] "Image analysis means" refers to optical character recognition technology for extracting text data from uploaded images of medical examination results.

[0100] "Optical character recognition technology" is a technology that recognizes and extracts character information from images, and is also known as OCR (Optical Character Recognition).

[0101] "Text data" refers to character information extracted using optical character recognition technology.

[0102] The "natural language processing module" is software that analyzes the extracted text data and identifies specific health checkup items.

[0103] "Health check items" refer to the various measurement data and diagnostic results included in the health check results, such as blood sugar levels, cholesterol levels, and BMI.

[0104] A "database" is a system for systematically storing analyzed text data and user information.

[0105] A "generative AI model" is an artificial intelligence model used to analyze stored health checkup data and predict health risks.

[0106] "Health risk" is information indicating the likelihood that the user will have a health problem in the future.

[0107] "Personalized health advice" means specific health measures and advice tailored to the individual user's health condition and lifestyle.

[0108] The "notification means" refers to a means for notifying the user of the generated advice or prediction results, and includes push notifications, etc.

[0109] "External tracking data" refers to health-related data obtained from other tracking applications, such as exercise, sleep, and nutrition.

[0110] "Information acquisition means" refers to the function for acquiring data from external tracking applications and utilizing it within the system.

[0111] "Integration" refers to compiling information obtained from multiple data sources into a single dataset and using it for analysis.

[0112] The system of the present invention allows users to easily input health checkup data, analyze the data, predict health risks, and provide appropriate health advice. It is also possible to integrate external tracking data to provide more detailed advice.

[0113] This system is realized using the following hardware and software.

[0114] Obtaining health checkup data

[0115] The user uses their smartphone to take a photo of the form containing the health checkup results. After taking the photo, the user launches a dedicated application and uploads the image to the application. This application runs on the user's smartphone and has an image upload function.

[0116] Image analysis and text data extraction

[0117] The uploaded image data is sent from the device to the server. The server analyzes the received image and extracts text data using optical character recognition (OCR) technology. Specifically, the server uses the Tesseract OCR engine to analyze the characters in the image. For example, information such as "Blood glucose level: 110 mg / dL" is extracted as text data.

[0118] Identifying and storing text data

[0119] The extracted text data is sent to a natural language processing (NLP) module on the server, which uses the spaCy library to identify health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.). The identified data is stored in a MySQL (registered trademark) database on the server.

[0120] Predicting health risks and generating advice

[0121] Based on the stored data, the server uses a generative AI model (specifically, an AI model such as OpenAI's GPT-3) to predict the user's health risks. Based on the prediction, the generative AI generates personalized health advice. For example, an example prompt could be, "Please predict the user's health risks and provide advice based on the following health checkup data: blood sugar level 110 mg / dL, cholesterol level 200 mg / dL, BMI 25."

[0122] Provision of a medicine notebook function

[0123] Users can enter medication information within the app, and the server automatically manages the information. Medication information can also be entered using barcode scanning technology (e.g., the ZBar library). The entered data is analyzed, and important medication information and reminders are sent to the user by the server.

[0124] External tracking data integration and integration

[0125] Within the app, users can set up integration with external exercise, sleep, and nutrition tracking apps (such as Fitbit and MyFitnessPal). The device collects tracking data from these linked applications and sends it to a server. The server then combines this data with health checkup data to generate more detailed health advice. For example, specific advice such as "We recommend running 30 minutes three times a week" may be provided based on the exercise data.

[0126] Providing advice and notifications

[0127] Finally, the server generates analysis results and health advice, which are displayed to the user via the device. Important health risks and medication information are also notified to the user via the device's push notification function (Firebase Cloud Messaging).

[0128] This allows users to comprehensively utilize their own health checkup data and external tracking data to take more specific and appropriate health measures.

[0129] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0130] Step 1:

[0131] The user takes a photo of the medical checkup result form using the smartphone camera. The captured image is the input and triggers the system to start processing. Next, the user launches a dedicated app and uploads the captured image to the app. This operation outputs the image of the medical checkup result and uses it as input for the next step.

[0132] Step 2:

[0133] The terminal sends the images of the health checkup results uploaded by the user to the server. The input is the image data of the health checkup, and by transferring this to the server, the server is ready to perform image analysis. The output is the image data received by the server.

[0134] Step 3:

[0135] The server sends the received image to an image analysis means and extracts text data from the image using optical character recognition (OCR) technology. Specifically, it uses the Tesseract OCR engine to recognize characters in the image and extract them as text data. The input is the image data of the health checkup, and the output is the extracted text data.

[0136] Step 4:

[0137] The server sends the extracted text data to a natural language processing (NLP) module to identify health checkup items. Specifically, it uses the spaCy library to analyze and identify health checkup items such as blood glucose level, cholesterol level, and BMI. The input is the text data extracted by OCR, and the output is the data of the identified health checkup items.

[0138] Step 5:

[0139] The server stores the data of the identified health check items in a database. Here, the input is the text data identified by NLP, and the output is the health check data stored in the database. For example, data such as "Blood glucose level: 110 mg / dL" is stored in a MySQL database.

[0140] Step 6:

[0141] The server uses a generative AI model to analyze the stored health checkup data and predict the user's health risks. The health checkup data stored in the database is used as input, and health risks are predicted by entering prompt statements into the generative AI. The output is the predicted health risks and personalized health advice. For example, advice such as "You should increase exercise to prevent diabetes" is generated.

[0142] Step 7:

[0143] The server notifies the user of predicted health risks and personalized advice. The input is the advice and prediction results obtained from the generative AI, which are sent to the terminal and notified to the user. The output is the predicted health risks and advice displayed on the user's smartphone.

[0144] Step 8:

[0145] The user enters medication information within the app, for example by scanning the medication barcode. The input is the medication information provided by the user, and the output is the medication data sent to the server.

[0146] Step 9:

[0147] The server analyzes the entered medication information and notifies the user of medication reminders and important information. The input is the medication information provided by the user, and the output is the analyzed medication information and notifications of reminders and precautions based on it.

[0148] Step 10:

[0149] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app. The input is the integration information for the external app selected and configured by the user, and the output is the data obtained from the tracking app.

[0150] Step 11:

[0151] The device acquires data from the linked external tracking app and sends it to the server. The input is the data from the tracking app, and the output is the external data sent to the server.

[0152] Step 12:

[0153] The server integrates external data with the original health checkup data to generate personalized, detailed health advice. The input is the external data and the health checkup data stored in the database, and the output is specific advice based on the integrated data. For example, advice such as "We recommend running for 30 minutes three times a week" is provided.

[0154] (Application example 1)

[0155] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0156] Health management systems typically predict health risks based on a user's health checkup data and provide personalized health advice. However, current systems lack the functionality to recommend products based on the user's health condition or to select appropriate products by linking with prescription record information. This makes it difficult for users to easily select products that suit their health condition, particularly when taking specific medications. Another issue is that the generated health advice and prediction results are not adequately communicated, providing insufficient support for users to take appropriate actions.

[0157] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0158] In this invention, the server includes means for acquiring past health checkup data, means for analyzing the acquired health checkup data using image analysis means, means for saving the analyzed data in a database, means for predicting health risks using a generation AI based on the saved data, means for generating personalized health advice based on the predicted health risks, means for notifying the user of the generated advice and prediction results, information acquisition means for acquiring and integrating external tracking data, means for recommending personalized products based on the analyzed health data, and means for selecting products in conjunction with information in the prescription notebook. This allows the user to receive product recommendations tailored to their health condition, enabling them to select appropriate products, and enables the generated health advice and prediction results to be efficiently notified.

[0159] "Past health checkup data" refers to the results data of health checkups that the user has previously undergone.

[0160] "Image analysis means" refers to optical character recognition (OCR) technology and other technologies for extracting text information from images of health checkup data.

[0161] "Database" refers to a storage device for storing analyzed medical examination data and other related information.

[0162] "Generative AI" refers to artificial intelligence algorithms that generate new information based on past data.

[0163] "Health risk" refers to the user's assessment of the likelihood of developing a particular health problem in the future.

[0164] "Health advice" refers to specific suggestions for actions or improvements provided based on the user's health risk predictions.

[0165] "Notification means" refers to the methods and technologies used to notify users of generated health advice and prediction results.

[0166] "Tracking Data" refers to a user's health-related data obtained from external sources, such as exercise, sleep, and nutrition.

[0167] "Product recommendation means" refers to a function that presents appropriate products based on the user's health checkup data and health risk predictions.

[0168] A "medicine notebook" is a tool that allows users to record and manage information about the medications they are taking.

[0169] "Product selection methods" refer to methods and techniques for selecting the most suitable product for a user in conjunction with information in the medicine notebook.

[0170] The present invention is a system that effectively utilizes a user's health checkup data to provide personalized health management and product recommendations. This system has a wide range of functions, including health risk prediction using AI generated from image analysis of health checkup data, product recommendations, a medicine notebook function, and linkage with external tracking data. Specific embodiments of the system are described below.

[0171] Acquisition and analysis of health checkup data

[0172] The user takes a photo of the health checkup result form using the smartphone camera. The image is uploaded to the device, and the system receives the data. The server sends the received image to an image analysis means, and optical character recognition (OCR) technology is used to extract text data from the image. The specific software used is "Google (registered trademark) Cloud Vision API."

[0173] Identifying and storing text data

[0174] The extracted text data is sent to a natural language processing (NLP) module on the server, which identifies health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.). The identified data is stored in a database. The specific software used is "SpaCy" and "PostgreSQL."

[0175] Predicting health risks and generating personalized advice

[0176] The server then uses the Generative AI to analyze the stored health checkup data and predict the user's health risks. Based on the prediction results, the Generative AI generates personalized health advice. For example, specific advice such as "You should exercise more to prevent diabetes" or "You should reduce sugar in your diet" is provided. The specific software used is "GPT-4 (registered trademark) (OpenAI)."

[0177] Personalized product recommendations

[0178] The server uses generative AI to make personalized product recommendations based on health risk predictions. For example, it recommends health-conscious product categories such as "low-sugar foods," "snacks without trans fats," and "high-fiber foods." This allows users to easily choose products that best suit their health condition. Below is an example of a prompt sentence to input to the generative AI model:

[0179] User's health data: { "Blood sugar": "110 mg / dL", "Cholesterol": "200 mg / dL", "BMI": "25.0"}. Provide targeted shopping advice.

[0180] Medicine notebook function and product selection

[0181] The server also has a medicine notebook function, automatically managing the medication information entered by the user. The entered medication information is analyzed and important information about medication is notified to the user. At the same time, advice is given on avoiding foods that are incompatible with the medication being taken.

[0182] External tracking data integration and integration

[0183] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app. The device then sends data obtained from the external apps to a server, which then integrates this data with health checkup data. For example, by adding exercise data, the AI ​​can provide detailed advice such as "running for 30 minutes three times a week is recommended."

[0184] Providing advice and notifications

[0185] Finally, the device displays the analysis results and health advice from the server to the user, and also notifies the user of important health risks and product information via push notifications.The notification system uses Firebase Cloud Messaging (FCM), which allows information to be delivered to users quickly and efficiently.

[0186] Specific examples

[0187] For example, a user takes a photo of their health checkup results for October 2023 with their smartphone and uploads it to the app. This data is analyzed by the server, and the value "blood sugar level: 110 mg / dL" is extracted and stored in the database. The generation AI uses this data to predict that the user is at high risk of diabetes and generates personalized advice such as "increase exercise to prevent diabetes." Furthermore, the user can set up a link with an exercise tracking app, and the server will analyze that data to suggest specific amounts of exercise. At the same time, product recommendations that take health risks into consideration are also made, and specific sales notifications are sent, such as "20% off low-sugar snacks."

[0188] In this way, users can effectively utilize their own health checkup data and take appropriate measures. Data from other linked health apps can also be utilized, enabling more comprehensive health management. This system will help users extend their healthy lifespan and facilitate smooth communication with medical professionals.

[0189] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0190] Step 1:

[0191] The user takes a photo of the medical examination result form with the smartphone camera. The captured image is uploaded to the device. The input of this step is the image of the medical examination result, and the output is an image file saved on the device.

[0192] Step 2:

[0193] The device sends the uploaded image to the server. The server then sends the image to an image analysis means, which extracts text data from the image using optical character recognition (OCR) technology. The input for this step is the image of the health check result, and the output is the extracted text data. The specific software used is "Google Cloud Vision API."

[0194] Step 3:

[0195] The server sends the extracted text data to a natural language processing (NLP) module to identify health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.). The input of this step is the text data obtained from OCR, and the output is the identified health checkup items. The specific software used is "SpaCy."

[0196] Step 4:

[0197] The server stores the identified health check items in a database. The input of this step is the data of the health check items, and the output is the information stored in the database. The specific software used is "PostgreSQL."

[0198] Step 5:

[0199] The server inputs the stored health checkup data into the generation AI to predict health risks. The input for this step is the health checkup data stored in the database, and the output is the predicted health risk results. The specific software used is "GPT-4 (OpenAI)."

[0200] Step 6:

[0201] The server generates personalized health advice using generative AI based on the health risk prediction. The input of this step is the health risk prediction result, and the output is personalized health advice. Next, we will show a specific example in which the following prompt sentence is input to the generative AI model:

[0202] User's health data: { "Blood sugar": "110 mg / dL", "Cholesterol": "200 mg / dL", "BMI": "25.0"}. Provide targeted shopping advice.

[0203] Step 7:

[0204] The server then recommends personalized products based on the generated health advice. The input of this step is the health advice provided by the generative AI, and the output is a list of product recommendations.

[0205] Step 8:

[0206] The server manages medication information in conjunction with the medication notebook information and notifies the user of important medication-related information. The input for this step is the medication notebook data, and the output is the notification content.

[0207] Step 9:

[0208] The user sets up a link with an external exercise, sleep, or nutrition tracking app within the app, and the device sends the data acquired from the external app to the server. The input of this step is the external tracking data, and the output is the data integrated by the server.

[0209] Step 10:

[0210] The server aggregates the tracking data and provides more specific advice. The input of this step is the aggregated tracking data, and the output is detailed health advice.

[0211] Step 11:

[0212] Finally, the device displays the analysis results and health advice from the server to the user and notifies them of important information via push notifications. The input for this step is the final health advice and product recommendation information, and the output is the notification content. The specific software used for this is Firebase Cloud Messaging (FCM).

[0213] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0214] This invention is a system that effectively utilizes a user's health checkup data to provide personalized health management. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, a more accurate health management service is provided. This system integrates health risk prediction using AI generated from image analysis of health checkup data, a medicine notebook function, linkage with external tracking data, and the emotion engine function.

[0215] Acquisition and analysis of health checkup data

[0216] Users take a photo of their past health checkup results using their smartphone camera. The system receives the data by uploading the image to the device. The server then sends the image to an image analysis tool and extracts text data from the image using optical character recognition (OCR) technology.

[0217] Identifying and storing text data

[0218] The extracted text data is sent to a natural language processing (NLP) module on the server, which identifies health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.) and stores the identified data in a database.

[0219] Predicting health risks and generating personalized advice

[0220] The server then uses the AI ​​generator to analyze the stored health checkup data and predict the user's health risks. Based on the prediction results, the AI ​​generator generates personalized health advice, such as "increase exercise to prevent diabetes" or "reduce sugar in your diet."

[0221] Use of emotion engine

[0222] The server then uses an emotion engine to recognize the user's emotions. For example, it uses a camera and microphone to analyze the user's facial expressions and tone of voice to identify their current emotional state (e.g., stress, joy, anger, sadness). The recognized emotion data is then incorporated into the health advice provided by the generative AI.

[0223] Medicine notebook function

[0224] The server also has a medicine notebook function, automatically managing the medicine information entered by the user. The entered medicine information is analyzed and important information regarding medication is notified to the user.

[0225] External tracking data integration and integration

[0226] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app. The device then sends data obtained from the external apps to a server, which then integrates this data with health checkup data. For example, by adding exercise data, the AI ​​can provide detailed advice such as "running for 30 minutes three times a week is recommended."

[0227] Providing advice and notifications

[0228] Finally, the device displays the analysis results and health advice from the server to the user, and sends push notifications to inform the user of important health risks and medication information. The emotion engine also recognizes emotions and adjusts the content and manner of notifications. For example, if the user is feeling stressed, health advice will be delivered in a gentler tone.

[0229] Specific examples

[0230] For example, a user takes a photo of their health checkup results with their smartphone and uploads them to the app. This data is analyzed on the server, and a value such as "blood sugar level: 110 mg / dL" is extracted and stored in a database. The generation AI uses this data to predict that the user has a high risk of diabetes and generates personalized advice such as "You should exercise more to prevent diabetes."

[0231] Furthermore, once the user has linked up with an exercise tracking app, the server analyzes the data and suggests specific exercise amounts. If the emotion engine detects the user's stress level, it will provide advice that takes their emotions into consideration, such as "We recommend relaxation exercises to reduce your stress."

[0232] In this way, users can effectively utilize their own health check data and emotional state to take appropriate measures. Furthermore, by utilizing data from other linked health apps and the emotion engine, more comprehensive and responsive health management becomes possible. This system will help users extend their healthy lifespan and facilitate smooth communication with medical professionals.

[0233] The processing flow will be explained below.

[0234] Step 1:

[0235] The user takes a photo of the past health checkup results form using the smartphone camera.

[0236] Step 2:

[0237] The device will upload the captured images to the app.

[0238] Step 3:

[0239] The server receives the uploaded images and transmits them to the image analysis means.

[0240] Step 4:

[0241] The server's image analysis means uses optical character recognition (OCR) technology to extract text data from the image.

[0242] Step 5:

[0243] The server sends the extracted text data to a natural language processing (NLP) module.

[0244] Step 6:

[0245] The server's NLP module analyzes the text data and identifies health check items (e.g., blood sugar level, cholesterol level, BMI, etc.).

[0246] Step 7:

[0247] The server stores the identified data in a database.

[0248] Step 8:

[0249] The server sends the health checkup data stored in the database to the generation AI, which analyzes the data.

[0250] Step 9:

[0251] The server's generated AI predicts the user's health risks.

[0252] Step 10:

[0253] The server's generative AI generates personalized health advice based on predicted health risks.

[0254] Step 11:

[0255] The server formats and prepares the generated health advice for the user.

[0256] Step 12:

[0257] The server analyzes the medication information entered by the user and manages that information.

[0258] Step 13:

[0259] The server initiates a notification mechanism to notify the user of important information about the medication.

[0260] Step 14:

[0261] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app.

[0262] Step 15:

[0263] The terminal transmits the data acquired from the external application to the server.

[0264] Step 16:

[0265] The server integrates the health check data with data from external apps and sends it to the generation AI.

[0266] Step 17:

[0267] The server's generative AI generates detailed personalized advice based on the integrated data.

[0268] Step 18:

[0269] The server uses an emotion engine to recognize the user's emotion.

[0270] Step 19:

[0271] The server's emotion engine analyzes the user's facial expressions and tone of voice to identify their emotional state (e.g., stress, joy, anger, sadness).

[0272] Step 20:

[0273] The server sends the emotional data recognized by the emotion engine to the generative AI, which then adjusts the health advice.

[0274] Step 21:

[0275] The device displays the adjusted analysis results and health advice from the server to the user.

[0276] Step 22:

[0277] The device will send users push notifications about important health risks and medication information.

[0278] (Example)

[0279] For example, a user takes a photo of their health checkup results with their smartphone and uploads them to the app. This data is analyzed on the server, and a value such as "blood sugar level: 110 mg / dL" is extracted and stored in a database. The generation AI uses this data to predict that the user has a high risk of diabetes and generates personalized advice such as "You should exercise more to prevent diabetes."

[0280] Furthermore, once the user has linked up with an exercise tracking app, the server analyzes the data and suggests specific exercise amounts. If the emotion engine detects the user's stress level, it will provide advice that takes their emotions into consideration, such as "We recommend relaxation exercises to reduce your stress."

[0281] In this way, users can effectively utilize their own health check data and emotional state to take appropriate measures. Furthermore, by utilizing data from other linked health apps and the emotion engine, more comprehensive and responsive health management becomes possible. This system will help users extend their healthy lifespan and facilitate smooth communication with medical professionals.

[0282] Example 2

[0283] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0284] It is important for users to predict health risks based on their own health checkup data and receive personalized health advice. However, most systems have complex data acquisition and analysis processes, and it is difficult to provide advice that takes into account the user's emotional state. Therefore, it is a challenge to utilize users' health checkup data and realize highly accurate health management based on their emotional state.

[0285] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring past health checkup data, means for analyzing the acquired health checkup data using image analysis means, means for storing the analyzed data in a database, means for predicting health risks using a generative AI model based on the stored data, means for generating personalized health advice based on the predicted health risks, a terminal for notifying the user of the generated advice and prediction results, emotion analysis means for recognizing the user's emotional state, information acquisition means for acquiring and integrating external tracking data, and means for adjusting the content and method of notification based on the emotion analysis means. This makes it possible to effectively utilize the user's health checkup data, provide specific health risk predictions and personalized advice, and provide responsive health management based on the user's emotional state.

[0286] "Past health checkup data" refers to data that indicates the results of health checkups that the user has taken in the past.

[0287] "Image analysis means" refers to technology used to analyze information within an image, and specifically includes optical character recognition (OCR) technology.

[0288] A "database" is a system for storing and managing acquired and analyzed data.

[0289] A "generative AI model" is an artificial intelligence model that performs analysis and predictions based on a user's health checkup data, and specifically includes a type of generative model.

[0290] "Health risk prediction" refers to the act of analyzing a user's health checkup data and predicting future health risks.

[0291] "Personalized health advice" means individualized health advice provided to an individual user based on their health status and risks.

[0292] A "terminal" is an electronic device such as a smartphone or tablet used by a user.

[0293] "Emotion analysis means" refers to technology for recognizing a user's emotional state, and includes data analysis using a camera or microphone.

[0294] "External tracking data" refers to data about exercise, sleep, nutrition, etc. obtained from external tracking devices or applications.

[0295] "Information acquisition means" refers to the technology or protocol used to acquire necessary data from external devices or applications.

[0296] The "means for adjusting the notification content and notification method" refers to a means for optimally adjusting the content and method of the information to be notified based on the emotional state of the user.

[0297] The present invention is a system for improving a user's health management, which integrates the user's health checkup data and external tracking data, and uses a generative AI model to provide health risk prediction and personalized health advice. Furthermore, it uses emotion analysis means to recognize the user's emotional state and adjust the content of advice and notification method based on that information.

[0298] First, the user takes a photo of the past medical checkup results using the smartphone camera. The system receives the data by uploading the image to the device. The device then uploads the image to cloud storage and notifies the server.

[0299] The server retrieves images from cloud storage and extracts text data from the images using optical character recognition (OCR) technology (e.g., Tesseract OCR). The extracted text data is then analyzed using a natural language processing (NLP) module (e.g., SpaCy) to identify health checkup items. The identified health checkup data (e.g., blood glucose level, cholesterol level, BMI) is stored in a database.

[0300] Based on the stored data, the server uses a generative AI model (e.g., OpenAI GPT-3) to predict the user's health risks. For example, based on the data "blood sugar level 110 mg / dL," it predicts "high risk of diabetes" and generates personalized health advice such as "increase exercise by 30 minutes."

[0301] Furthermore, the server recognizes the user's emotional state using emotion analysis methods (e.g., Microsoft® Azure® Emotion API). It analyzes the user's facial expressions and tone of voice using the device's camera and microphone to identify their emotional state (e.g., stress, joy, anger, sadness). This emotional data is reflected in the generative AI's advice; for example, if the user is feeling stressed, it will provide health advice in a gentler tone.

[0302] Users can set up integration with external exercise, sleep, and nutrition tracking apps. The device sends data obtained from the external apps to a server, and this data is integrated with health checkup data to provide more detailed advice (e.g., "We recommend running for 30 minutes three times a week").

[0303] It also has a medicine notebook function, and the server automatically manages the medicine information entered by the user. The entered medicine information is analyzed, and important information about taking the medicine (e.g., interactions, precautions, etc.) is notified from the server.

[0304] Finally, the device displays the analysis results and health advice from the server to the user, and notifies the user of important information via push notifications. The content and method of notifications are also adjusted according to the emotions recognized by the emotion analysis means.

[0305] For example, the following prompt sentence is input to the generative AI model:

[0306] Generate specific health risk predictions and personalized advice based on the user's health checkup data. For example, if a blood sugar level is 110 mg / dL, predict a high risk of diabetes and provide specific advice on exercise and dietary improvements. Also, take into account the user's emotional state (e.g., high stress).

[0307] This allows users to effectively utilize their own health checkup data, external tracking data linked to their own health checkup data, and their emotional state to optimally manage their health.

[0308] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0309] Step 1:

[0310] The user takes a photo of a past medical checkup result form using the smartphone camera. The system receives the data by uploading the image to the terminal. The input image data is passed to the terminal. This input image data is then converted into text data through subsequent optical character recognition (OCR) processing.

[0311] Step 2:

[0312] The device uploads the captured images to the cloud storage. A cloud storage URL is generated and notified to the server. The server then receives the cloud storage URL as input for retrieving the images.

[0313] Step 3:

[0314] The server retrieves images from cloud storage and extracts text data from the images using optical character recognition (OCR) technology (e.g., Tesseract OCR). Specifically, the server inputs the retrieved image data into the OCR engine and outputs text data. This text data is used to identify and store health checkup items.

[0315] Step 4:

[0316] The server sends the text data extracted by OCR to a natural language processing (NLP) module (e.g., SpaCy) to identify various health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.). The input is text data, and the output is the identified health checkup items. This identified data is stored in a database.

[0317] Step 5:

[0318] The server stores the data of the specified medical examination items in a database, receiving the specified data as input and obtaining the medical examination data in the format stored in the database as output.

[0319] Step 6:

[0320] The server uses a generative AI model (e.g., OpenAI GPT-3) based on the stored health checkup data to predict the user's health risk. The generative AI model receives the health checkup data as input, and outputs a predicted health risk. For example, a specific prediction such as "Your blood sugar level is 110 mg / dL, so you're at high risk for diabetes" is generated.

[0321] Step 7:

[0322] The server generates personalized health advice using a generative AI model based on the predicted health risks. The predicted health risks are input to the generative AI model, and specific health advice is output. For example, advice such as "Increase your exercise by 30 minutes" is generated.

[0323] Step 8:

[0324] The server recognizes the user's emotional state using emotion analysis means (e.g., Microsoft Azure Emotion API). Data acquired from the camera and microphone is sent as input to the emotion analysis engine, and the user's emotional state is obtained as output. For example, a specific emotional state such as "high stress level" is recognized.

[0325] Step 9:

[0326] The server incorporates the recognized emotional data into the health advice provided by the generative AI and adjusts the content of the advice and notification method. Emotional state and health advice are given as input, and emotion-conscious health advice and notifications are generated as output. For example, if the user is feeling stressed, advice will be provided in a calm tone.

[0327] Step 10:

[0328] Users configure the app to connect to external exercise, sleep, and nutrition tracking apps. The device receives data from the external apps and sends it to a server. The server receives data from the external apps as input and provides integrated health data as output.

[0329] Step 11:

[0330] The server then integrates this data with the health checkup data to generate detailed health advice. The integrated data is given as input to a generative AI model, which outputs detailed health advice. For example, specific advice such as "running for 30 minutes three times a week is recommended" is generated.

[0331] Step 12:

[0332] The server automatically manages medication information entered by the user using the medication notebook function. It analyzes the entered medication information and notifies the user of important medication information, such as drug interactions and precautions.

[0333] Step 13:

[0334] The device displays the analysis results and health advice from the server to the user, and generates push notifications and health advice to be sent to the user as output.

[0335] Step 14:

[0336] The device adjusts the content and method of notifications based on the emotions recognized by the emotion analysis method, ultimately providing appropriate health advice to the user based on their emotions.

[0337] (Application example 2)

[0338] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0339] Health management is becoming increasingly important in modern society, but personalized health support based on individual users' health status and emotions is still lacking. In particular, in-store customer service often fails to provide prompt and accurate health advice that takes into account individual health data and current emotions. Therefore, there is a need for a system that can accurately predict users' health risks, generate personalized health advice based on those risks, and provide health support in-store that takes into account the user's emotions.

[0340] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring past health checkup data, means for analyzing the acquired health checkup data using image analysis means, means for saving the analyzed data in a database, means for predicting health risks using a generation AI based on the saved data, means for generating personalized health advice based on the predicted health risks, means for notifying the user of the generated advice and prediction results, information acquisition means for acquiring and integrating external tracking data, means for recognizing the user's emotions and incorporating the emotion data into the analysis results, and means for providing emotion-based health support via an automated answering device in a physical store. This makes it possible to quickly and accurately provide personalized health advice based on the health condition and emotions of individual users even in physical stores.

[0341] "Past health checkup data" refers to data showing the results of health checkups the user has undergone in the past, and specifically includes numerical information for assessing health status, such as blood sugar levels, cholesterol levels, and BMI.

[0342] The "image analysis means" is a technology for extracting text data from acquired image data, specifically using optical character recognition (OCR) technology.

[0343] "Generative AI" is a type of artificial intelligence model used to generate predictions or advice from specific input data. In this case, it refers to an AI model for predicting health risks.

[0344] "Personalized health advice" means advice, including individualized health guidance and recommendations, generated based on a user's individual health data and predicted health risks.

[0345] "External tracking data" refers to health-related data, such as a user's exercise, sleep, and nutrition, obtained from external devices or applications.

[0346] "Means for recognizing emotions" refers to technology that analyzes a user's facial expressions and tone of voice to identify their emotional state at that time.

[0347] An "automatic answering device" is a device installed in a physical store to support users in managing their health, and is responsible for providing appropriate support and advice based on the user's emotional state.

[0348] "Information acquisition means" refers to a means for acquiring information such as tracking data from the outside and integrating it into the system.

[0349] "Means of notification" refers to the means for informing users of the generated advice and prediction results, and specifically includes push notifications on smartphones.

[0350] This invention relates to a personalized health management system for brick-and-mortar stores. This system analyzes past health checkup data, provides individualized health advice to users, and recognizes users' emotions and incorporates their emotional state into the analysis results to provide health support within the brick-and-mortar stores.

[0351] The server first provides a means for users to obtain past health checkup data. Specifically, the user obtains this data by taking a photo of the health checkup result form using the camera on their smartphone. The obtained image data is uploaded to the server and analyzed using image analysis means. Optical character recognition (OCR) technology is used to extract text data from the image.

[0352] The server then analyzes the extracted text data using a natural language processing (NLP) module to identify health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.), and stores the identified data in a database.

[0353] The server then uses the stored data to predict the user's health risks using generative AI. The generative AI model comprehensively evaluates the user's health status and generates personalized health advice. For example, specific advice such as "You should increase exercise to prevent diabetes" or "You should reduce sugar in your diet" is generated.

[0354] The system also includes a means to recognize the user's emotions. The server uses a camera and microphone to analyze the user's facial expressions and tone of voice to identify their current emotional state (e.g., stress, joy, anger, sadness, etc.). This emotional data is incorporated into the generated health advice, and the content and delivery of notifications and advice are tailored accordingly.

[0355] Additionally, external tracking data such as exercise, sleep, and nutrition data from other tracking apps used by the user can be integrated using information acquisition methods, providing a more complete picture of the user's health and enabling more detailed health advice.

[0356] Based on the accumulated data, information about the user's medication is also managed. This allows information about the medications the user is taking and important notifications to be managed by the server and provided to the user. The generated health advice and prediction results are also provided to medical professionals, who can summarize and display the analysis results and provide appropriate health guidance.

[0357] For example, a user takes a photo of their health checkup results with their smartphone at a store counter and uploads it to the system. This data is analyzed on the server, and the value "Blood glucose level: 110 mg / dL" is extracted and saved. Based on this data, the generation AI predicts that the user is at high risk for diabetes and generates personalized advice such as "You should increase your exercise to prevent diabetes." If the AI ​​recognizes the user's emotional state and determines that stress is high, it provides advice that takes their emotions into consideration, such as "We recommend yoga to reduce stress."

[0358] An example of a prompt is as follows:

[0359] "User's health is at high risk. Dietary changes are required."

[0360] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0361] Step 1:

[0362] Users take photos of their past health checkup results with their smartphone camera and upload the image data to the device. At this stage, the input is image data of the health checkup results, and the output is image data sent to the server. The device converts the captured image into an appropriate format and sends it to the server via a stable communication path.

[0363] Step 2:

[0364] The server converts the received image data into text data using optical character recognition (OCR) technology. The input at this stage is the uploaded health check image data, and the output is text data extracted from the image. The server analyzes the image using an OCR engine such as "Tesseract" and extracts the necessary text information.

[0365] Step 3:

[0366] The server analyzes the extracted text data using a natural language processing (NLP) module to identify specific health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.). The input at this stage is the text data extracted by OCR, and the output is the analyzed health checkup item data. For example, formatted data such as "blood glucose level: 110 mg / dL" is obtained.

[0367] Step 4:

[0368] The server stores the analyzed health check item data in a database. The input at this stage is the data of the identified health check items, and the output is structured data stored in the database. The data will be used for later analysis and prediction by generative AI.

[0369] Step 5:

[0370] Next, the server uses the stored data to predict the user's health risk using a generative AI model. The input at this stage is the health checkup data stored in the database, and the output is the predicted health risk and its details. For example, the assessment may be "high risk of diabetes."

[0371] Step 6:

[0372] The server generates personalized health advice using a generative AI model based on the predicted health risks. The input at this stage is the predicted health risks, and the output is the generated specific health advice. An example of generated advice is "You should exercise more to prevent diabetes."

[0373] Step 7:

[0374] The server analyzes the user's emotional state using emotion recognition means. The user sends data of their facial expressions and tone of voice to the server via a camera or microphone. The input at this stage is data indicating the user's emotional state, and the output is data of the analyzed emotional state. For example, it can recognize that the user is feeling stressed.

[0375] Step 8:

[0376] The server then incorporates the recognized emotional state into the generated health advice. The input at this stage is the health advice and emotional state data, and the output is personalized health advice that takes the emotion into account. An example of generated advice is "We recommend relaxing exercises to reduce stress."

[0377] Step 9:

[0378] The server notifies the user of the generated health advice and prediction results. The device receives the notification from the server and displays it to the user via push notification. The input at this stage is the generated advice and prediction results, and the output is the notification displayed on the device. The notification may include suggested actions for the user or recommendations for device use.

[0379] Step 10:

[0380] The user can then put the advice into practice and provide feedback to the system. At this stage, the input is the user's action data, and the output is the feedback data sent to the system. This allows the system to provide continuous, personalized health management.

[0381] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0382] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0383] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0384] [Second embodiment]

[0385] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0386] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0387] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0388] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0389] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0390] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0391] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0392] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0393] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0394] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0395] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0396] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0397] The present invention is a system that effectively utilizes a user's health checkup data to provide personalized health management. This system has a wide range of functions, including predicting health risks using AI generated from image analysis of health checkup data, a medicine notebook function, and linking with external tracking data. Specific embodiments of the system are described below.

[0398] Acquisition and analysis of health checkup data

[0399] The user takes a photo of the health checkup result form with the smartphone camera. The system receives the data by uploading the image to the terminal. The server sends the received image to an image analysis means and extracts text data from the image using optical character recognition (OCR) technology.

[0400] Identifying and storing text data

[0401] The extracted text data is sent to a natural language processing (NLP) module on the server, which identifies health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.) and stores the identified data in a database.

[0402] Predicting health risks and generating personalized advice

[0403] The server then uses the AI ​​generator to analyze the stored health checkup data and predict the user's health risks. Based on the prediction results, the AI ​​generator generates personalized health advice, such as "increase exercise to prevent diabetes" or "reduce sugar in your diet."

[0404] Medicine notebook function

[0405] The server also has a medicine notebook function, automatically managing the medicine information entered by the user. The entered medicine information is analyzed and important information regarding medication is notified to the user.

[0406] External tracking data integration and integration

[0407] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app. The device then sends data obtained from the external apps to a server, which then integrates this data with health checkup data. For example, by adding exercise data, the AI ​​can provide detailed advice such as "running for 30 minutes three times a week is recommended."

[0408] Providing advice and notifications

[0409] Finally, the device displays the analysis results and health advice from the server to the user, and notifies the user of important health risks and medication information via push notifications.

[0410] Specific examples

[0411] For example, a user takes a photo of their health checkup results for October 2023 with their smartphone and uploads it to the app. This data is analyzed by the server, and the value "blood sugar level: 110 mg / dL" is extracted and stored in the database. The generation AI uses this data to predict that the user has a high risk of diabetes and generates personalized advice such as "increase exercise to prevent diabetes." Furthermore, the user sets up a link with an exercise tracking app, and the server analyzes the data to suggest specific amounts of exercise.

[0412] In this way, users can effectively utilize their own health checkup data and take appropriate measures. Data from other linked health apps can also be utilized, enabling more comprehensive health management. This system will help users extend their healthy lifespan and facilitate smooth communication with medical professionals.

[0413] The processing flow will be explained below.

[0414] Step 1:

[0415] The user takes a photo of the past health checkup results form using the smartphone camera.

[0416] Step 2:

[0417] The device will upload the captured images to the app.

[0418] Step 3:

[0419] The server receives the uploaded images and transmits them to the image analysis means.

[0420] Step 4:

[0421] The server's image analysis means uses optical character recognition (OCR) technology to extract text data from the image.

[0422] Step 5:

[0423] The server sends the extracted text data to a natural language processing (NLP) module.

[0424] Step 6:

[0425] The server's NLP module analyzes the text data and identifies health check items (e.g., blood sugar level, cholesterol level, BMI, etc.).

[0426] Step 7:

[0427] The server stores the identified data in a database.

[0428] Step 8:

[0429] The server sends the health checkup data stored in the database to the generation AI, which analyzes the data.

[0430] Step 9:

[0431] The server's generated AI predicts the user's health risks.

[0432] Step 10:

[0433] The server's generative AI generates personalized health advice based on predicted health risks.

[0434] Step 11:

[0435] The server formats and prepares the generated health advice for the user.

[0436] Step 12:

[0437] The server analyzes the medication information entered by the user and manages that information.

[0438] Step 13:

[0439] The server initiates a notification mechanism to notify the user of important information about the medication.

[0440] Step 14:

[0441] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app.

[0442] Step 15:

[0443] The terminal transmits the data acquired from the external application to the server.

[0444] Step 16:

[0445] The server integrates the health check data with data from external apps and sends it to the generation AI.

[0446] Step 17:

[0447] The server's generative AI generates detailed personalized advice based on the integrated data.

[0448] Step 18:

[0449] The device displays the analysis results and health advice from the server to the user.

[0450] Step 19:

[0451] The device will send users push notifications about important health risks and medication information.

[0452] Example 1

[0453] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0454] Conventional health management systems require users to manually input data, which is time-consuming and laborious, and also carries a high risk of input errors. Furthermore, because they only reference individual health checkup data, it is difficult to comprehensively grasp a user's current health status. Furthermore, it is difficult to integrate and utilize external tracking data. As a result, it is not possible to effectively provide personalized advice to users, and they are unable to provide sufficient support for maintaining their health and preventing disease.

[0455] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0456] In this invention, the server includes a means for uploading images of health checkup data taken by a user using a smartphone, a means for analyzing the uploaded images using an image analysis means and extracting text data using optical character recognition technology, a means for analyzing the extracted text data using a natural language processing module to identify health checkup items and store them in a database, a means for analyzing the stored health checkup data using a generative AI model and predicting the user's health risks, a means for generating personalized health advice based on the predicted health risks, a means for notifying the user of the generated advice and prediction results, an information acquisition means for acquiring and integrating external tracking data, and a means for integrating the acquired external tracking data and health checkup data to generate detailed advice. This allows users to easily import health checkup data and perform comprehensive health management. Furthermore, the automatically generated personalized advice allows users to take appropriate measures to maintain their health and prevent disease.

[0457] "User" means an individual who uses the system to provide health checkup results and external tracking data.

[0458] A "smartphone" is a portable information terminal equipped with a camera and applications that allows users to take and upload health checkup results.

[0459] "Image analysis means" refers to optical character recognition technology for extracting text data from uploaded images of medical examination results.

[0460] "Optical character recognition technology" is a technology that recognizes and extracts character information from images, and is also known as OCR (Optical Character Recognition).

[0461] "Text data" refers to character information extracted using optical character recognition technology.

[0462] The "natural language processing module" is software that analyzes the extracted text data and identifies specific health checkup items.

[0463] "Health check items" refer to the various measurement data and diagnostic results included in the health check results, such as blood sugar levels, cholesterol levels, and BMI.

[0464] A "database" is a system for systematically storing analyzed text data and user information.

[0465] A "generative AI model" is an artificial intelligence model used to analyze stored health checkup data and predict health risks.

[0466] "Health risk" is information indicating the likelihood that the user will have a health problem in the future.

[0467] "Personalized health advice" means specific health measures and advice tailored to the individual user's health condition and lifestyle.

[0468] The "notification means" refers to a means for notifying the user of the generated advice or prediction results, and includes push notifications, etc.

[0469] "External tracking data" refers to health-related data obtained from other tracking applications, such as exercise, sleep, and nutrition.

[0470] "Information acquisition means" refers to the function for acquiring data from external tracking applications and utilizing it within the system.

[0471] "Integration" refers to compiling information obtained from multiple data sources into a single dataset and using it for analysis.

[0472] The system of the present invention allows users to easily input health checkup data, analyze the data, predict health risks, and provide appropriate health advice. It is also possible to integrate external tracking data to provide more detailed advice.

[0473] This system is realized using the following hardware and software.

[0474] Obtaining health checkup data

[0475] The user uses their smartphone to take a photo of the form containing the health checkup results. After taking the photo, the user launches a dedicated application and uploads the image to the application. This application runs on the user's smartphone and has an image upload function.

[0476] Image analysis and text data extraction

[0477] The uploaded image data is sent from the device to the server. The server analyzes the received image and extracts text data using optical character recognition (OCR) technology. Specifically, the server uses the Tesseract OCR engine to analyze the characters in the image. For example, information such as "Blood glucose level: 110 mg / dL" is extracted as text data.

[0478] Identifying and storing text data

[0479] The extracted text data is sent to a natural language processing (NLP) module on the server, which uses the spaCy library to identify health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.) and stores the identified data in a MySQL database on the server.

[0480] Predicting health risks and generating advice

[0481] Based on the stored data, the server uses a generative AI model (specifically, an AI model such as OpenAI's GPT-3) to predict the user's health risks. Based on the prediction, the generative AI generates personalized health advice. For example, an example prompt could be, "Please predict the user's health risks and provide advice based on the following health checkup data: blood sugar level 110 mg / dL, cholesterol level 200 mg / dL, BMI 25."

[0482] Provision of a medicine notebook function

[0483] Users can enter medication information within the app, and the server automatically manages the information. Medication information can also be entered using barcode scanning technology (e.g., the ZBar library). The entered data is analyzed, and important medication information and reminders are sent to the user by the server.

[0484] External tracking data integration and integration

[0485] Within the app, users can set up integration with external exercise, sleep, and nutrition tracking apps (such as Fitbit and MyFitnessPal). The device collects tracking data from these linked applications and sends it to a server. The server then combines this data with health checkup data to generate more detailed health advice. For example, specific advice such as "We recommend running 30 minutes three times a week" may be provided based on the exercise data.

[0486] Providing advice and notifications

[0487] Finally, the server generates analysis results and health advice, which are displayed to the user via the device. Important health risks and medication information are also notified to the user via the device's push notification function (Firebase Cloud Messaging).

[0488] This allows users to comprehensively utilize their own health checkup data and external tracking data to take more specific and appropriate health measures.

[0489] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0490] Step 1:

[0491] The user takes a photo of the medical checkup result form using the smartphone camera. The captured image is the input and triggers the system to start processing. Next, the user launches a dedicated app and uploads the captured image to the app. This operation outputs the image of the medical checkup result and uses it as input for the next step.

[0492] Step 2:

[0493] The terminal sends the images of the health checkup results uploaded by the user to the server. The input is the image data of the health checkup, and by transferring this to the server, the server is ready to perform image analysis. The output is the image data received by the server.

[0494] Step 3:

[0495] The server sends the received image to an image analysis means and extracts text data from the image using optical character recognition (OCR) technology. Specifically, it uses the Tesseract OCR engine to recognize characters in the image and extract them as text data. The input is the image data of the health checkup, and the output is the extracted text data.

[0496] Step 4:

[0497] The server sends the extracted text data to a natural language processing (NLP) module to identify health checkup items. Specifically, it uses the spaCy library to analyze and identify health checkup items such as blood glucose level, cholesterol level, and BMI. The input is the text data extracted by OCR, and the output is the data of the identified health checkup items.

[0498] Step 5:

[0499] The server stores the data of the identified health check items in a database. Here, the input is the text data identified by NLP, and the output is the health check data stored in the database. For example, data such as "Blood glucose level: 110 mg / dL" is stored in a MySQL database.

[0500] Step 6:

[0501] The server uses a generative AI model to analyze the stored health checkup data and predict the user's health risks. The health checkup data stored in the database is used as input, and health risks are predicted by entering prompt statements into the generative AI. The output is the predicted health risks and personalized health advice. For example, advice such as "You should increase exercise to prevent diabetes" is generated.

[0502] Step 7:

[0503] The server notifies the user of predicted health risks and personalized advice. The input is the advice and prediction results obtained from the generative AI, which are sent to the terminal and notified to the user. The output is the predicted health risks and advice displayed on the user's smartphone.

[0504] Step 8:

[0505] The user enters medication information within the app, for example by scanning the medication barcode. The input is the medication information provided by the user, and the output is the medication data sent to the server.

[0506] Step 9:

[0507] The server analyzes the entered medication information and notifies the user of medication reminders and important information. The input is the medication information provided by the user, and the output is the analyzed medication information and notifications of reminders and precautions based on it.

[0508] Step 10:

[0509] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app. The input is the integration information for the external app selected and configured by the user, and the output is the data obtained from the tracking app.

[0510] Step 11:

[0511] The device acquires data from the linked external tracking app and sends it to the server. The input is the data from the tracking app, and the output is the external data sent to the server.

[0512] Step 12:

[0513] The server integrates external data with the original health checkup data to generate personalized, detailed health advice. The input is the external data and the health checkup data stored in the database, and the output is specific advice based on the integrated data. For example, advice such as "We recommend running for 30 minutes three times a week" is provided.

[0514] (Application example 1)

[0515] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0516] Health management systems typically predict health risks based on a user's health checkup data and provide personalized health advice. However, current systems lack the functionality to recommend products based on the user's health condition or to select appropriate products by linking with prescription record information. This makes it difficult for users to easily select products that suit their health condition, particularly when taking specific medications. Another issue is that the generated health advice and prediction results are not adequately communicated, providing insufficient support for users to take appropriate actions.

[0517] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0518] In this invention, the server includes means for acquiring past health checkup data, means for analyzing the acquired health checkup data using image analysis means, means for saving the analyzed data in a database, means for predicting health risks using a generation AI based on the saved data, means for generating personalized health advice based on the predicted health risks, means for notifying the user of the generated advice and prediction results, information acquisition means for acquiring and integrating external tracking data, means for recommending personalized products based on the analyzed health data, and means for selecting products in conjunction with information in the prescription notebook. This allows the user to receive product recommendations tailored to their health condition, enabling them to select appropriate products, and enables the generated health advice and prediction results to be efficiently notified.

[0519] "Past health checkup data" refers to the results data of health checkups that the user has previously undergone.

[0520] "Image analysis means" refers to optical character recognition (OCR) technology and other technologies for extracting text information from images of health checkup data.

[0521] "Database" refers to a storage device for storing analyzed medical examination data and other related information.

[0522] "Generative AI" refers to artificial intelligence algorithms that generate new information based on past data.

[0523] "Health risk" refers to the user's assessment of the likelihood of developing a particular health problem in the future.

[0524] "Health advice" refers to specific suggestions for actions or improvements provided based on the user's health risk predictions.

[0525] "Notification means" refers to the methods and technologies used to notify users of generated health advice and prediction results.

[0526] "Tracking Data" refers to a user's health-related data obtained from external sources, such as exercise, sleep, and nutrition.

[0527] "Product recommendation means" refers to a function that presents appropriate products based on the user's health checkup data and health risk predictions.

[0528] A "medicine notebook" is a tool that allows users to record and manage information about the medications they are taking.

[0529] "Product selection methods" refer to methods and techniques for selecting the most suitable product for a user in conjunction with information in the medicine notebook.

[0530] The present invention is a system that effectively utilizes a user's health checkup data to provide personalized health management and product recommendations. This system has a wide range of functions, including health risk prediction using AI generated from image analysis of health checkup data, product recommendations, a medicine notebook function, and linkage with external tracking data. Specific embodiments of the system are described below.

[0531] Acquisition and analysis of health checkup data

[0532] The user takes a photo of the health check result form using the smartphone camera. The image is uploaded to the device, and the system receives the data. The server then sends the received image to an image analysis tool, which extracts text data from the image using optical character recognition (OCR) technology. The specific software used is "Google Cloud Vision API."

[0533] Identifying and storing text data

[0534] The extracted text data is sent to a natural language processing (NLP) module on the server, which identifies health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.). The identified data is stored in a database. The specific software used is "SpaCy" and "PostgreSQL."

[0535] Predicting health risks and generating personalized advice

[0536] The server then uses the generative AI to analyze the stored health checkup data and predict the user's health risks. Based on the prediction results, the generative AI generates personalized health advice. For example, specific advice such as "You should increase exercise to prevent diabetes" or "You should reduce sugar in your diet" is provided. The specific software used is "GPT-4 (OpenAI)."

[0537] Personalized product recommendations

[0538] The server uses generative AI to make personalized product recommendations based on health risk predictions. For example, it recommends health-conscious product categories such as "low-sugar foods," "snacks without trans fats," and "high-fiber foods." This allows users to easily choose products that best suit their health condition. Below is an example of a prompt sentence to input to the generative AI model:

[0539] User's health data: { "Blood sugar": "110 mg / dL", "Cholesterol": "200 mg / dL", "BMI": "25.0"}. Provide targeted shopping advice.

[0540] Medicine notebook function and product selection

[0541] The server also has a medicine notebook function, automatically managing the medication information entered by the user. The entered medication information is analyzed and important information about medication is notified to the user. At the same time, advice is given on avoiding foods that are incompatible with the medication being taken.

[0542] External tracking data integration and integration

[0543] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app. The device then sends data obtained from the external apps to a server, which then integrates this data with health checkup data. For example, by adding exercise data, the AI ​​can provide detailed advice such as "running for 30 minutes three times a week is recommended."

[0544] Providing advice and notifications

[0545] Finally, the device displays the analysis results and health advice from the server to the user, and also notifies the user of important health risks and product information via push notifications.The notification system uses Firebase Cloud Messaging (FCM), which allows information to be delivered to users quickly and efficiently.

[0546] Specific examples

[0547] For example, a user takes a photo of their health checkup results for October 2023 with their smartphone and uploads it to the app. This data is analyzed by the server, and the value "blood sugar level: 110 mg / dL" is extracted and stored in the database. The generation AI uses this data to predict that the user is at high risk of diabetes and generates personalized advice such as "increase exercise to prevent diabetes." Furthermore, the user can set up a link with an exercise tracking app, and the server will analyze that data to suggest specific amounts of exercise. At the same time, product recommendations that take health risks into consideration are also made, and specific sales notifications are sent, such as "20% off low-sugar snacks."

[0548] In this way, users can effectively utilize their own health checkup data and take appropriate measures. Data from other linked health apps can also be utilized, enabling more comprehensive health management. This system will help users extend their healthy lifespan and facilitate smooth communication with medical professionals.

[0549] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0550] Step 1:

[0551] The user takes a photo of the medical examination result form with the smartphone camera. The captured image is uploaded to the device. The input of this step is the image of the medical examination result, and the output is an image file saved on the device.

[0552] Step 2:

[0553] The device sends the uploaded image to the server. The server then sends the image to an image analysis means, which extracts text data from the image using optical character recognition (OCR) technology. The input for this step is the image of the health check result, and the output is the extracted text data. The specific software used is "Google Cloud Vision API."

[0554] Step 3:

[0555] The server sends the extracted text data to a natural language processing (NLP) module to identify health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.). The input of this step is the text data obtained from OCR, and the output is the identified health checkup items. The specific software used is "SpaCy."

[0556] Step 4:

[0557] The server stores the identified health check items in a database. The input of this step is the data of the health check items, and the output is the information stored in the database. The specific software used is "PostgreSQL."

[0558] Step 5:

[0559] The server inputs the stored health checkup data into the generation AI to predict health risks. The input for this step is the health checkup data stored in the database, and the output is the predicted health risk results. The specific software used is "GPT-4 (OpenAI)."

[0560] Step 6:

[0561] The server generates personalized health advice using generative AI based on the health risk prediction. The input of this step is the health risk prediction result, and the output is personalized health advice. Next, we will show a specific example in which the following prompt sentence is input to the generative AI model:

[0562] User's health data: { "Blood sugar": "110 mg / dL", "Cholesterol": "200 mg / dL", "BMI": "25.0"}. Provide targeted shopping advice.

[0563] Step 7:

[0564] The server then recommends personalized products based on the generated health advice. The input of this step is the health advice provided by the generative AI, and the output is a list of product recommendations.

[0565] Step 8:

[0566] The server manages medication information in conjunction with the medication notebook information and notifies the user of important medication-related information. The input for this step is the medication notebook data, and the output is the notification content.

[0567] Step 9:

[0568] The user sets up a link with an external exercise, sleep, or nutrition tracking app within the app, and the device sends the data acquired from the external app to the server. The input of this step is the external tracking data, and the output is the data integrated by the server.

[0569] Step 10:

[0570] The server aggregates the tracking data and provides more specific advice. The input of this step is the aggregated tracking data, and the output is detailed health advice.

[0571] Step 11:

[0572] Finally, the device displays the analysis results and health advice from the server to the user and notifies them of important information via push notifications. The input for this step is the final health advice and product recommendation information, and the output is the notification content. The specific software used for this is Firebase Cloud Messaging (FCM).

[0573] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0574] This invention is a system that effectively utilizes a user's health checkup data to provide personalized health management. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, a more accurate health management service is provided. This system integrates health risk prediction using AI generated from image analysis of health checkup data, a medicine notebook function, linkage with external tracking data, and the emotion engine function.

[0575] Acquisition and analysis of health checkup data

[0576] Users take a photo of their past health checkup results using their smartphone camera. The system receives the data by uploading the image to the device. The server then sends the image to an image analysis tool and extracts text data from the image using optical character recognition (OCR) technology.

[0577] Identifying and storing text data

[0578] The extracted text data is sent to a natural language processing (NLP) module on the server, which identifies health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.) and stores the identified data in a database.

[0579] Predicting health risks and generating personalized advice

[0580] The server then uses the AI ​​generator to analyze the stored health checkup data and predict the user's health risks. Based on the prediction results, the AI ​​generator generates personalized health advice, such as "increase exercise to prevent diabetes" or "reduce sugar in your diet."

[0581] Use of emotion engine

[0582] The server then uses an emotion engine to recognize the user's emotions. For example, it uses a camera and microphone to analyze the user's facial expressions and tone of voice to identify their current emotional state (e.g., stress, joy, anger, sadness). The recognized emotion data is then incorporated into the health advice provided by the generative AI.

[0583] Medicine notebook function

[0584] The server also has a medicine notebook function, automatically managing the medicine information entered by the user. The entered medicine information is analyzed and important information regarding medication is notified to the user.

[0585] External tracking data integration and integration

[0586] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app. The device then sends data obtained from the external apps to a server, which then integrates this data with health checkup data. For example, by adding exercise data, the AI ​​can provide detailed advice such as "running for 30 minutes three times a week is recommended."

[0587] Providing advice and notifications

[0588] Finally, the device displays the analysis results and health advice from the server to the user, and sends push notifications to inform the user of important health risks and medication information. The emotion engine also recognizes emotions and adjusts the content and manner of notifications. For example, if the user is feeling stressed, health advice will be delivered in a gentler tone.

[0589] Specific examples

[0590] For example, a user takes a photo of their health checkup results with their smartphone and uploads them to the app. This data is analyzed on the server, and a value such as "blood sugar level: 110 mg / dL" is extracted and stored in a database. The generation AI uses this data to predict that the user has a high risk of diabetes and generates personalized advice such as "You should exercise more to prevent diabetes."

[0591] Furthermore, once the user has linked up with an exercise tracking app, the server analyzes the data and suggests specific exercise amounts. If the emotion engine detects the user's stress level, it will provide advice that takes their emotions into consideration, such as "We recommend relaxation exercises to reduce your stress."

[0592] In this way, users can effectively utilize their own health check data and emotional state to take appropriate measures. Furthermore, by utilizing data from other linked health apps and the emotion engine, more comprehensive and responsive health management becomes possible. This system will help users extend their healthy lifespan and facilitate smooth communication with medical professionals.

[0593] The processing flow will be explained below.

[0594] Step 1:

[0595] The user takes a photo of the past health checkup results form using the smartphone camera.

[0596] Step 2:

[0597] The device will upload the captured images to the app.

[0598] Step 3:

[0599] The server receives the uploaded images and transmits them to the image analysis means.

[0600] Step 4:

[0601] The server's image analysis means uses optical character recognition (OCR) technology to extract text data from the image.

[0602] Step 5:

[0603] The server sends the extracted text data to a natural language processing (NLP) module.

[0604] Step 6:

[0605] The server's NLP module analyzes the text data and identifies health check items (e.g., blood sugar level, cholesterol level, BMI, etc.).

[0606] Step 7:

[0607] The server stores the identified data in a database.

[0608] Step 8:

[0609] The server sends the health checkup data stored in the database to the generation AI, which analyzes the data.

[0610] Step 9:

[0611] The server's generated AI predicts the user's health risks.

[0612] Step 10:

[0613] The server's generative AI generates personalized health advice based on predicted health risks.

[0614] Step 11:

[0615] The server formats and prepares the generated health advice for the user.

[0616] Step 12:

[0617] The server analyzes the medication information entered by the user and manages that information.

[0618] Step 13:

[0619] The server initiates a notification mechanism to notify the user of important information about the medication.

[0620] Step 14:

[0621] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app.

[0622] Step 15:

[0623] The terminal transmits the data acquired from the external application to the server.

[0624] Step 16:

[0625] The server integrates the health check data with data from external apps and sends it to the generation AI.

[0626] Step 17:

[0627] The server's generative AI generates detailed personalized advice based on the integrated data.

[0628] Step 18:

[0629] The server uses an emotion engine to recognize the user's emotion.

[0630] Step 19:

[0631] The server's emotion engine analyzes the user's facial expressions and tone of voice to identify their emotional state (e.g., stress, joy, anger, sadness).

[0632] Step 20:

[0633] The server sends the emotional data recognized by the emotion engine to the generative AI, which then adjusts the health advice.

[0634] Step 21:

[0635] The device displays the adjusted analysis results and health advice from the server to the user.

[0636] Step 22:

[0637] The device will send users push notifications about important health risks and medication information.

[0638] (Example)

[0639] For example, a user takes a photo of their health checkup results with their smartphone and uploads them to the app. This data is analyzed on the server, and a value such as "blood sugar level: 110 mg / dL" is extracted and stored in a database. The generation AI uses this data to predict that the user has a high risk of diabetes and generates personalized advice such as "You should exercise more to prevent diabetes."

[0640] Furthermore, once the user has linked up with an exercise tracking app, the server analyzes the data and suggests specific exercise amounts. If the emotion engine detects the user's stress level, it will provide advice that takes their emotions into consideration, such as "We recommend relaxation exercises to reduce your stress."

[0641] In this way, users can effectively utilize their own health check data and emotional state to take appropriate measures. Furthermore, by utilizing data from other linked health apps and the emotion engine, more comprehensive and responsive health management becomes possible. This system will help users extend their healthy lifespan and facilitate smooth communication with medical professionals.

[0642] Example 2

[0643] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0644] It is important for users to predict health risks based on their own health checkup data and receive personalized health advice. However, most systems have complex data acquisition and analysis processes, and it is difficult to provide advice that takes into account the user's emotional state. Therefore, it is a challenge to utilize users' health checkup data and realize highly accurate health management based on their emotional state.

[0645] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring past health checkup data, means for analyzing the acquired health checkup data using image analysis means, means for storing the analyzed data in a database, means for predicting health risks using a generative AI model based on the stored data, means for generating personalized health advice based on the predicted health risks, a terminal for notifying the user of the generated advice and prediction results, emotion analysis means for recognizing the user's emotional state, information acquisition means for acquiring and integrating external tracking data, and means for adjusting the content and method of notification based on the emotion analysis means. This makes it possible to effectively utilize the user's health checkup data, provide specific health risk predictions and personalized advice, and provide responsive health management based on the user's emotional state.

[0646] "Past health checkup data" refers to data that indicates the results of health checkups that the user has taken in the past.

[0647] "Image analysis means" refers to technology used to analyze information within an image, and specifically includes optical character recognition (OCR) technology.

[0648] A "database" is a system for storing and managing acquired and analyzed data.

[0649] A "generative AI model" is an artificial intelligence model that performs analysis and predictions based on a user's health checkup data, and specifically includes a type of generative model.

[0650] "Health risk prediction" refers to the act of analyzing a user's health checkup data and predicting future health risks.

[0651] "Personalized health advice" means individualized health advice provided to an individual user based on their health status and risks.

[0652] A "terminal" is an electronic device such as a smartphone or tablet used by a user.

[0653] "Emotion analysis means" refers to technology for recognizing a user's emotional state, and includes data analysis using a camera or microphone.

[0654] "External tracking data" refers to data about exercise, sleep, nutrition, etc. obtained from external tracking devices or applications.

[0655] "Information acquisition means" refers to the technology or protocol used to acquire necessary data from external devices or applications.

[0656] The "means for adjusting the notification content and notification method" refers to a means for optimally adjusting the content and method of the information to be notified based on the emotional state of the user.

[0657] The present invention is a system for improving a user's health management, which integrates the user's health checkup data and external tracking data, and uses a generative AI model to provide health risk prediction and personalized health advice. Furthermore, it uses emotion analysis means to recognize the user's emotional state and adjust the content of advice and notification method based on that information.

[0658] First, the user takes a photo of the past medical checkup results using the smartphone camera. The system receives the data by uploading the image to the device. The device then uploads the image to cloud storage and notifies the server.

[0659] The server retrieves images from cloud storage and extracts text data from the images using optical character recognition (OCR) technology (e.g., Tesseract OCR). The extracted text data is then analyzed using a natural language processing (NLP) module (e.g., SpaCy) to identify health checkup items. The identified health checkup data (e.g., blood glucose level, cholesterol level, BMI) is stored in a database.

[0660] Based on the stored data, the server uses a generative AI model (e.g., OpenAI GPT-3) to predict the user's health risks. For example, based on the data "blood sugar level 110 mg / dL," it predicts "high risk of diabetes" and generates personalized health advice such as "increase exercise by 30 minutes."

[0661] Furthermore, the server recognizes the user's emotional state using emotion analysis methods (e.g., Microsoft Azure Emotion API). It analyzes the user's facial expressions and tone of voice using the device's camera and microphone to identify their emotional state (e.g., stress, joy, anger, sadness). This emotional data is reflected in the generative AI's advice. For example, if the user is feeling stressed, it will provide health advice in a gentler tone.

[0662] Users can set up integration with external exercise, sleep, and nutrition tracking apps. The device sends data obtained from the external apps to a server, and this data is integrated with health checkup data to provide more detailed advice (e.g., "We recommend running for 30 minutes three times a week").

[0663] It also has a medicine notebook function, and the server automatically manages the medicine information entered by the user. The entered medicine information is analyzed, and important information about taking the medicine (e.g., interactions, precautions, etc.) is notified from the server.

[0664] Finally, the device displays the analysis results and health advice from the server to the user, and notifies the user of important information via push notifications. The content and method of notifications are also adjusted according to the emotions recognized by the emotion analysis means.

[0665] For example, the following prompt sentence is input to the generative AI model:

[0666] Generate specific health risk predictions and personalized advice based on the user's health checkup data. For example, if a blood sugar level is 110 mg / dL, predict a high risk of diabetes and provide specific advice on exercise and dietary improvements. Also, take into account the user's emotional state (e.g., high stress).

[0667] This allows users to effectively utilize their own health checkup data, external tracking data linked to their own health checkup data, and their emotional state to optimally manage their health.

[0668] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0669] Step 1:

[0670] The user takes a photo of a past medical checkup result form using the smartphone camera. The system receives the data by uploading the image to the terminal. The input image data is passed to the terminal. This input image data is then converted into text data through subsequent optical character recognition (OCR) processing.

[0671] Step 2:

[0672] The device uploads the captured images to the cloud storage. A cloud storage URL is generated and notified to the server. The server then receives the cloud storage URL as input for retrieving the images.

[0673] Step 3:

[0674] The server retrieves images from cloud storage and extracts text data from the images using optical character recognition (OCR) technology (e.g., Tesseract OCR). Specifically, the server inputs the retrieved image data into the OCR engine and outputs text data. This text data is used to identify and store health checkup items.

[0675] Step 4:

[0676] The server sends the text data extracted by OCR to a natural language processing (NLP) module (e.g., SpaCy) to identify various health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.). The input is text data, and the output is the identified health checkup items. This identified data is stored in a database.

[0677] Step 5:

[0678] The server stores the data of the specified medical examination items in a database, receiving the specified data as input and obtaining the medical examination data in the format stored in the database as output.

[0679] Step 6:

[0680] The server uses a generative AI model (e.g., OpenAI GPT-3) based on the stored health checkup data to predict the user's health risk. The generative AI model receives the health checkup data as input, and outputs a predicted health risk. For example, a specific prediction such as "Your blood sugar level is 110 mg / dL, so you're at high risk for diabetes" is generated.

[0681] Step 7:

[0682] The server generates personalized health advice using a generative AI model based on the predicted health risks. The predicted health risks are input to the generative AI model, and specific health advice is output. For example, advice such as "Increase your exercise by 30 minutes" is generated.

[0683] Step 8:

[0684] The server recognizes the user's emotional state using emotion analysis means (e.g., Microsoft Azure Emotion API). Data acquired from the camera and microphone is sent as input to the emotion analysis engine, and the user's emotional state is obtained as output. For example, a specific emotional state such as "high stress level" is recognized.

[0685] Step 9:

[0686] The server incorporates the recognized emotional data into the health advice provided by the generative AI and adjusts the content of the advice and notification method. Emotional state and health advice are given as input, and emotion-conscious health advice and notifications are generated as output. For example, if the user is feeling stressed, advice will be provided in a calm tone.

[0687] Step 10:

[0688] Users configure the app to connect to external exercise, sleep, and nutrition tracking apps. The device receives data from the external apps and sends it to a server. The server receives data from the external apps as input and provides integrated health data as output.

[0689] Step 11:

[0690] The server then integrates this data with the health checkup data to generate detailed health advice. The integrated data is given as input to a generative AI model, which outputs detailed health advice. For example, specific advice such as "running for 30 minutes three times a week is recommended" is generated.

[0691] Step 12:

[0692] The server automatically manages medication information entered by the user using the medication notebook function. It analyzes the entered medication information and notifies the user of important medication information, such as drug interactions and precautions.

[0693] Step 13:

[0694] The device displays the analysis results and health advice from the server to the user, and generates push notifications and health advice to be sent to the user as output.

[0695] Step 14:

[0696] The device adjusts the content and method of notifications based on the emotions recognized by the emotion analysis method, ultimately providing appropriate health advice to the user based on their emotions.

[0697] (Application example 2)

[0698] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0699] Health management is becoming increasingly important in modern society, but personalized health support based on individual users' health status and emotions is still lacking. In particular, in-store customer service often fails to provide prompt and accurate health advice that takes into account individual health data and current emotions. Therefore, there is a need for a system that can accurately predict users' health risks, generate personalized health advice based on those risks, and provide health support in-store that takes into account the user's emotions.

[0700] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring past health checkup data, means for analyzing the acquired health checkup data using image analysis means, means for saving the analyzed data in a database, means for predicting health risks using a generation AI based on the saved data, means for generating personalized health advice based on the predicted health risks, means for notifying the user of the generated advice and prediction results, information acquisition means for acquiring and integrating external tracking data, means for recognizing the user's emotions and incorporating the emotion data into the analysis results, and means for providing emotion-based health support via an automated answering device in a physical store. This makes it possible to quickly and accurately provide personalized health advice based on the health condition and emotions of individual users even in physical stores.

[0701] "Past health checkup data" refers to data showing the results of health checkups the user has undergone in the past, and specifically includes numerical information for assessing health status, such as blood sugar levels, cholesterol levels, and BMI.

[0702] The "image analysis means" is a technology for extracting text data from acquired image data, specifically using optical character recognition (OCR) technology.

[0703] "Generative AI" is a type of artificial intelligence model used to generate predictions or advice from specific input data. In this case, it refers to an AI model for predicting health risks.

[0704] "Personalized health advice" means advice, including individualized health guidance and recommendations, generated based on a user's individual health data and predicted health risks.

[0705] "External tracking data" refers to health-related data, such as a user's exercise, sleep, and nutrition, obtained from external devices or applications.

[0706] "Means for recognizing emotions" refers to technology that analyzes a user's facial expressions and tone of voice to identify their emotional state at that time.

[0707] An "automatic answering device" is a device installed in a physical store to support users in managing their health, and is responsible for providing appropriate support and advice based on the user's emotional state.

[0708] "Information acquisition means" refers to a means for acquiring information such as tracking data from the outside and integrating it into the system.

[0709] "Means of notification" refers to the means for informing users of the generated advice and prediction results, and specifically includes push notifications on smartphones.

[0710] This invention relates to a personalized health management system for brick-and-mortar stores. This system analyzes past health checkup data, provides individualized health advice to users, and recognizes users' emotions and incorporates their emotional state into the analysis results to provide health support within the brick-and-mortar stores.

[0711] The server first provides a means for users to obtain past health checkup data. Specifically, the user obtains this data by taking a photo of the health checkup result form using the camera on their smartphone. The obtained image data is uploaded to the server and analyzed using image analysis means. Optical character recognition (OCR) technology is used to extract text data from the image.

[0712] The server then analyzes the extracted text data using a natural language processing (NLP) module to identify health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.), and stores the identified data in a database.

[0713] The server then uses the stored data to predict the user's health risks using generative AI. The generative AI model comprehensively evaluates the user's health status and generates personalized health advice. For example, specific advice such as "You should increase exercise to prevent diabetes" or "You should reduce sugar in your diet" is generated.

[0714] The system also includes a means to recognize the user's emotions. The server uses a camera and microphone to analyze the user's facial expressions and tone of voice to identify their current emotional state (e.g., stress, joy, anger, sadness, etc.). This emotional data is incorporated into the generated health advice, and the content and delivery of notifications and advice are tailored accordingly.

[0715] Additionally, external tracking data such as exercise, sleep, and nutrition data from other tracking apps used by the user can be integrated using information acquisition methods, providing a more complete picture of the user's health and enabling more detailed health advice.

[0716] Based on the accumulated data, information about the user's medication is also managed. This allows information about the medications the user is taking and important notifications to be managed by the server and provided to the user. The generated health advice and prediction results are also provided to medical professionals, who can summarize and display the analysis results and provide appropriate health guidance.

[0717] For example, a user takes a photo of their health checkup results with their smartphone at a store counter and uploads it to the system. This data is analyzed on the server, and the value "Blood glucose level: 110 mg / dL" is extracted and saved. Based on this data, the generation AI predicts that the user is at high risk for diabetes and generates personalized advice such as "You should increase your exercise to prevent diabetes." If the AI ​​recognizes the user's emotional state and determines that stress is high, it provides advice that takes their emotions into consideration, such as "We recommend yoga to reduce stress."

[0718] An example of a prompt is as follows:

[0719] "User's health is at high risk. Dietary changes are required."

[0720] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0721] Step 1:

[0722] Users take photos of their past health checkup results with their smartphone camera and upload the image data to the device. At this stage, the input is image data of the health checkup results, and the output is image data sent to the server. The device converts the captured image into an appropriate format and sends it to the server via a stable communication path.

[0723] Step 2:

[0724] The server converts the received image data into text data using optical character recognition (OCR) technology. The input at this stage is the uploaded health check image data, and the output is text data extracted from the image. The server analyzes the image using an OCR engine such as "Tesseract" and extracts the necessary text information.

[0725] Step 3:

[0726] The server analyzes the extracted text data using a natural language processing (NLP) module to identify specific health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.). The input at this stage is the text data extracted by OCR, and the output is the analyzed health checkup item data. For example, formatted data such as "blood glucose level: 110 mg / dL" is obtained.

[0727] Step 4:

[0728] The server stores the analyzed health check item data in a database. The input at this stage is the data of the identified health check items, and the output is structured data stored in the database. The data will be used for later analysis and prediction by generative AI.

[0729] Step 5:

[0730] Next, the server uses the stored data to predict the user's health risk using a generative AI model. The input at this stage is the health checkup data stored in the database, and the output is the predicted health risk and its details. For example, the assessment may be "high risk of diabetes."

[0731] Step 6:

[0732] The server generates personalized health advice using a generative AI model based on the predicted health risks. The input at this stage is the predicted health risks, and the output is the generated specific health advice. An example of generated advice is "You should exercise more to prevent diabetes."

[0733] Step 7:

[0734] The server analyzes the user's emotional state using emotion recognition means. The user sends data of their facial expressions and tone of voice to the server via a camera or microphone. The input at this stage is data indicating the user's emotional state, and the output is data of the analyzed emotional state. For example, it can recognize that the user is feeling stressed.

[0735] Step 8:

[0736] The server then incorporates the recognized emotional state into the generated health advice. The input at this stage is the health advice and emotional state data, and the output is personalized health advice that takes the emotion into account. An example of generated advice is "We recommend relaxing exercises to reduce stress."

[0737] Step 9:

[0738] The server notifies the user of the generated health advice and prediction results. The device receives the notification from the server and displays it to the user via push notification. The input at this stage is the generated advice and prediction results, and the output is the notification displayed on the device. The notification may include suggested actions for the user or recommendations for device use.

[0739] Step 10:

[0740] The user can then put the advice into practice and provide feedback to the system. At this stage, the input is the user's action data, and the output is the feedback data sent to the system. This allows the system to provide continuous, personalized health management.

[0741] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0742] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0743] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0744] [Third embodiment]

[0745] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0746] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0747] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0748] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0749] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0750] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0751] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0752] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0753] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0754] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0755] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0756] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0757] The present invention is a system that effectively utilizes a user's health checkup data to provide personalized health management. This system has a wide range of functions, including predicting health risks using AI generated from image analysis of health checkup data, a medicine notebook function, and linking with external tracking data. Specific embodiments of the system are described below.

[0758] Acquisition and analysis of health checkup data

[0759] The user takes a photo of the health checkup result form with the smartphone camera. The system receives the data by uploading the image to the terminal. The server sends the received image to an image analysis means and extracts text data from the image using optical character recognition (OCR) technology.

[0760] Identifying and storing text data

[0761] The extracted text data is sent to a natural language processing (NLP) module on the server, which identifies health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.) and stores the identified data in a database.

[0762] Predicting health risks and generating personalized advice

[0763] The server then uses the AI ​​generator to analyze the stored health checkup data and predict the user's health risks. Based on the prediction results, the AI ​​generator generates personalized health advice, such as "increase exercise to prevent diabetes" or "reduce sugar in your diet."

[0764] Medicine notebook function

[0765] The server also has a medicine notebook function, automatically managing the medicine information entered by the user. The entered medicine information is analyzed and important information regarding medication is notified to the user.

[0766] External tracking data integration and integration

[0767] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app. The device then sends data obtained from the external apps to a server, which then integrates this data with health checkup data. For example, by adding exercise data, the AI ​​can provide detailed advice such as "running for 30 minutes three times a week is recommended."

[0768] Providing advice and notifications

[0769] Finally, the device displays the analysis results and health advice from the server to the user, and notifies the user of important health risks and medication information via push notifications.

[0770] Specific examples

[0771] For example, a user takes a photo of their health checkup results for October 2023 with their smartphone and uploads it to the app. This data is analyzed by the server, and the value "blood sugar level: 110 mg / dL" is extracted and stored in the database. The generation AI uses this data to predict that the user has a high risk of diabetes and generates personalized advice such as "increase exercise to prevent diabetes." Furthermore, the user sets up a link with an exercise tracking app, and the server analyzes the data to suggest specific amounts of exercise.

[0772] In this way, users can effectively utilize their own health checkup data and take appropriate measures. Data from other linked health apps can also be utilized, enabling more comprehensive health management. This system will help users extend their healthy lifespan and facilitate smooth communication with medical professionals.

[0773] The processing flow will be explained below.

[0774] Step 1:

[0775] The user takes a photo of the past health checkup results form using the smartphone camera.

[0776] Step 2:

[0777] The device will upload the captured images to the app.

[0778] Step 3:

[0779] The server receives the uploaded images and transmits them to the image analysis means.

[0780] Step 4:

[0781] The server's image analysis means uses optical character recognition (OCR) technology to extract text data from the image.

[0782] Step 5:

[0783] The server sends the extracted text data to a natural language processing (NLP) module.

[0784] Step 6:

[0785] The server's NLP module analyzes the text data and identifies health check items (e.g., blood sugar level, cholesterol level, BMI, etc.).

[0786] Step 7:

[0787] The server stores the identified data in a database.

[0788] Step 8:

[0789] The server sends the health checkup data stored in the database to the generation AI, which analyzes the data.

[0790] Step 9:

[0791] The server's generated AI predicts the user's health risks.

[0792] Step 10:

[0793] The server's generative AI generates personalized health advice based on predicted health risks.

[0794] Step 11:

[0795] The server formats and prepares the generated health advice for the user.

[0796] Step 12:

[0797] The server analyzes the medication information entered by the user and manages that information.

[0798] Step 13:

[0799] The server initiates a notification mechanism to notify the user of important information about the medication.

[0800] Step 14:

[0801] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app.

[0802] Step 15:

[0803] The terminal transmits the data acquired from the external application to the server.

[0804] Step 16:

[0805] The server integrates the health check data with data from external apps and sends it to the generation AI.

[0806] Step 17:

[0807] The server's generative AI generates detailed personalized advice based on the integrated data.

[0808] Step 18:

[0809] The device displays the analysis results and health advice from the server to the user.

[0810] Step 19:

[0811] The device will send users push notifications about important health risks and medication information.

[0812] Example 1

[0813] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0814] Conventional health management systems require users to manually input data, which is time-consuming and laborious, and also carries a high risk of input errors. Furthermore, because they only reference individual health checkup data, it is difficult to comprehensively grasp a user's current health status. Furthermore, it is difficult to integrate and utilize external tracking data. As a result, it is not possible to effectively provide personalized advice to users, and they are unable to provide sufficient support for maintaining their health and preventing disease.

[0815] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0816] In this invention, the server includes a means for uploading images of health checkup data taken by a user using a smartphone, a means for analyzing the uploaded images using an image analysis means and extracting text data using optical character recognition technology, a means for analyzing the extracted text data using a natural language processing module to identify health checkup items and store them in a database, a means for analyzing the stored health checkup data using a generative AI model and predicting the user's health risks, a means for generating personalized health advice based on the predicted health risks, a means for notifying the user of the generated advice and prediction results, an information acquisition means for acquiring and integrating external tracking data, and a means for integrating the acquired external tracking data and health checkup data to generate detailed advice. This allows users to easily import health checkup data and perform comprehensive health management. Furthermore, the automatically generated personalized advice allows users to take appropriate measures to maintain their health and prevent disease.

[0817] "User" means an individual who uses the system to provide health checkup results and external tracking data.

[0818] A "smartphone" is a portable information terminal equipped with a camera and applications that allows users to take and upload health checkup results.

[0819] "Image analysis means" refers to optical character recognition technology for extracting text data from uploaded images of medical examination results.

[0820] "Optical character recognition technology" is a technology that recognizes and extracts character information from images, and is also known as OCR (Optical Character Recognition).

[0821] "Text data" refers to character information extracted using optical character recognition technology.

[0822] The "natural language processing module" is software that analyzes the extracted text data and identifies specific health checkup items.

[0823] "Health check items" refer to the various measurement data and diagnostic results included in the health check results, such as blood sugar levels, cholesterol levels, and BMI.

[0824] A "database" is a system for systematically storing analyzed text data and user information.

[0825] A "generative AI model" is an artificial intelligence model used to analyze stored health checkup data and predict health risks.

[0826] "Health risk" is information indicating the likelihood that the user will have a health problem in the future.

[0827] "Personalized health advice" means specific health measures and advice tailored to the individual user's health condition and lifestyle.

[0828] The "notification means" refers to a means for notifying the user of the generated advice or prediction results, and includes push notifications, etc.

[0829] "External tracking data" refers to health-related data obtained from other tracking applications, such as exercise, sleep, and nutrition.

[0830] "Information acquisition means" refers to the function for acquiring data from external tracking applications and utilizing it within the system.

[0831] "Integration" refers to compiling information obtained from multiple data sources into a single dataset and using it for analysis.

[0832] The system of the present invention allows users to easily input health checkup data, analyze the data, predict health risks, and provide appropriate health advice. It is also possible to integrate external tracking data to provide more detailed advice.

[0833] This system is realized using the following hardware and software.

[0834] Obtaining health checkup data

[0835] The user uses their smartphone to take a photo of the form containing the health checkup results. After taking the photo, the user launches a dedicated application and uploads the image to the application. This application runs on the user's smartphone and has an image upload function.

[0836] Image analysis and text data extraction

[0837] The uploaded image data is sent from the device to the server. The server analyzes the received image and extracts text data using optical character recognition (OCR) technology. Specifically, the server uses the Tesseract OCR engine to analyze the characters in the image. For example, information such as "Blood glucose level: 110 mg / dL" is extracted as text data.

[0838] Identifying and storing text data

[0839] The extracted text data is sent to a natural language processing (NLP) module on the server, which uses the spaCy library to identify health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.) and stores the identified data in a MySQL database on the server.

[0840] Predicting health risks and generating advice

[0841] Based on the stored data, the server uses a generative AI model (specifically, an AI model such as OpenAI's GPT-3) to predict the user's health risks. Based on the prediction, the generative AI generates personalized health advice. For example, an example prompt could be, "Please predict the user's health risks and provide advice based on the following health checkup data: blood sugar level 110 mg / dL, cholesterol level 200 mg / dL, BMI 25."

[0842] Provision of a medicine notebook function

[0843] Users can enter medication information within the app, and the server automatically manages the information. Medication information can also be entered using barcode scanning technology (e.g., the ZBar library). The entered data is analyzed, and important medication information and reminders are sent to the user by the server.

[0844] External tracking data integration and integration

[0845] Within the app, users can set up integration with external exercise, sleep, and nutrition tracking apps (such as Fitbit and MyFitnessPal). The device collects tracking data from these linked applications and sends it to a server. The server then combines this data with health checkup data to generate more detailed health advice. For example, specific advice such as "We recommend running 30 minutes three times a week" may be provided based on the exercise data.

[0846] Providing advice and notifications

[0847] Finally, the server generates analysis results and health advice, which are displayed to the user via the device. Important health risks and medication information are also notified to the user via the device's push notification function (Firebase Cloud Messaging).

[0848] This allows users to comprehensively utilize their own health checkup data and external tracking data to take more specific and appropriate health measures.

[0849] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0850] Step 1:

[0851] The user takes a photo of the medical checkup result form using the smartphone camera. The captured image is the input and triggers the system to start processing. Next, the user launches a dedicated app and uploads the captured image to the app. This operation outputs the image of the medical checkup result and uses it as input for the next step.

[0852] Step 2:

[0853] The terminal sends the images of the health checkup results uploaded by the user to the server. The input is the image data of the health checkup, and by transferring this to the server, the server is ready to perform image analysis. The output is the image data received by the server.

[0854] Step 3:

[0855] The server sends the received image to an image analysis means and extracts text data from the image using optical character recognition (OCR) technology. Specifically, it uses the Tesseract OCR engine to recognize characters in the image and extract them as text data. The input is the image data of the health checkup, and the output is the extracted text data.

[0856] Step 4:

[0857] The server sends the extracted text data to a natural language processing (NLP) module to identify health checkup items. Specifically, it uses the spaCy library to analyze and identify health checkup items such as blood glucose level, cholesterol level, and BMI. The input is the text data extracted by OCR, and the output is the data of the identified health checkup items.

[0858] Step 5:

[0859] The server stores the data of the identified health check items in a database. Here, the input is the text data identified by NLP, and the output is the health check data stored in the database. For example, data such as "Blood glucose level: 110 mg / dL" is stored in a MySQL database.

[0860] Step 6:

[0861] The server uses a generative AI model to analyze the stored health checkup data and predict the user's health risks. The health checkup data stored in the database is used as input, and health risks are predicted by entering prompt statements into the generative AI. The output is the predicted health risks and personalized health advice. For example, advice such as "You should increase exercise to prevent diabetes" is generated.

[0862] Step 7:

[0863] The server notifies the user of predicted health risks and personalized advice. The input is the advice and prediction results obtained from the generative AI, which are sent to the terminal and notified to the user. The output is the predicted health risks and advice displayed on the user's smartphone.

[0864] Step 8:

[0865] The user enters medication information within the app, for example by scanning the medication barcode. The input is the medication information provided by the user, and the output is the medication data sent to the server.

[0866] Step 9:

[0867] The server analyzes the entered medication information and notifies the user of medication reminders and important information. The input is the medication information provided by the user, and the output is the analyzed medication information and notifications of reminders and precautions based on it.

[0868] Step 10:

[0869] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app. The input is the integration information for the external app selected and configured by the user, and the output is the data obtained from the tracking app.

[0870] Step 11:

[0871] The device acquires data from the linked external tracking app and sends it to the server. The input is the data from the tracking app, and the output is the external data sent to the server.

[0872] Step 12:

[0873] The server integrates external data with the original health checkup data to generate personalized, detailed health advice. The input is the external data and the health checkup data stored in the database, and the output is specific advice based on the integrated data. For example, advice such as "We recommend running for 30 minutes three times a week" is provided.

[0874] (Application example 1)

[0875] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0876] Health management systems typically predict health risks based on a user's health checkup data and provide personalized health advice. However, current systems lack the functionality to recommend products based on the user's health condition or to select appropriate products by linking with prescription record information. This makes it difficult for users to easily select products that suit their health condition, particularly when taking specific medications. Another issue is that the generated health advice and prediction results are not adequately communicated, providing insufficient support for users to take appropriate actions.

[0877] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0878] In this invention, the server includes means for acquiring past health checkup data, means for analyzing the acquired health checkup data using image analysis means, means for saving the analyzed data in a database, means for predicting health risks using a generation AI based on the saved data, means for generating personalized health advice based on the predicted health risks, means for notifying the user of the generated advice and prediction results, information acquisition means for acquiring and integrating external tracking data, means for recommending personalized products based on the analyzed health data, and means for selecting products in conjunction with information in the prescription notebook. This allows the user to receive product recommendations tailored to their health condition, enabling them to select appropriate products, and enables the generated health advice and prediction results to be efficiently notified.

[0879] "Past health checkup data" refers to the results data of health checkups that the user has previously undergone.

[0880] "Image analysis means" refers to optical character recognition (OCR) technology and other technologies for extracting text information from images of health checkup data.

[0881] "Database" refers to a storage device for storing analyzed medical examination data and other related information.

[0882] "Generative AI" refers to artificial intelligence algorithms that generate new information based on past data.

[0883] "Health risk" refers to the user's assessment of the likelihood of developing a particular health problem in the future.

[0884] "Health advice" refers to specific suggestions for actions or improvements provided based on the user's health risk predictions.

[0885] "Notification means" refers to the methods and technologies used to notify users of generated health advice and prediction results.

[0886] "Tracking Data" refers to a user's health-related data obtained from external sources, such as exercise, sleep, and nutrition.

[0887] "Product recommendation means" refers to a function that presents appropriate products based on the user's health checkup data and health risk predictions.

[0888] A "medicine notebook" is a tool that allows users to record and manage information about the medications they are taking.

[0889] "Product selection methods" refer to methods and techniques for selecting the most suitable product for a user in conjunction with information in the medicine notebook.

[0890] The present invention is a system that effectively utilizes a user's health checkup data to provide personalized health management and product recommendations. This system has a wide range of functions, including health risk prediction using AI generated from image analysis of health checkup data, product recommendations, a medicine notebook function, and linkage with external tracking data. Specific embodiments of the system are described below.

[0891] Acquisition and analysis of health checkup data

[0892] The user takes a photo of the health check result form using the smartphone camera. The image is uploaded to the device, and the system receives the data. The server then sends the received image to an image analysis tool, which extracts text data from the image using optical character recognition (OCR) technology. The specific software used is "Google Cloud Vision API."

[0893] Identifying and storing text data

[0894] The extracted text data is sent to a natural language processing (NLP) module on the server, which identifies health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.). The identified data is stored in a database. The specific software used is "SpaCy" and "PostgreSQL."

[0895] Predicting health risks and generating personalized advice

[0896] The server then uses the generative AI to analyze the stored health checkup data and predict the user's health risks. Based on the prediction results, the generative AI generates personalized health advice. For example, specific advice such as "You should increase exercise to prevent diabetes" or "You should reduce sugar in your diet" is provided. The specific software used is "GPT-4 (OpenAI)."

[0897] Personalized product recommendations

[0898] The server uses generative AI to make personalized product recommendations based on health risk predictions. For example, it recommends health-conscious product categories such as "low-sugar foods," "snacks without trans fats," and "high-fiber foods." This allows users to easily choose products that best suit their health condition. Below is an example of a prompt sentence to input to the generative AI model:

[0899] User's health data: { "Blood sugar": "110 mg / dL", "Cholesterol": "200 mg / dL", "BMI": "25.0"}. Provide targeted shopping advice.

[0900] Medicine notebook function and product selection

[0901] The server also has a medicine notebook function, automatically managing the medication information entered by the user. The entered medication information is analyzed and important information about medication is notified to the user. At the same time, advice is given on avoiding foods that are incompatible with the medication being taken.

[0902] External tracking data integration and integration

[0903] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app. The device then sends data obtained from the external apps to a server, which then integrates this data with health checkup data. For example, by adding exercise data, the AI ​​can provide detailed advice such as "running for 30 minutes three times a week is recommended."

[0904] Providing advice and notifications

[0905] Finally, the device displays the analysis results and health advice from the server to the user, and also notifies the user of important health risks and product information via push notifications.The notification system uses Firebase Cloud Messaging (FCM), which allows information to be delivered to users quickly and efficiently.

[0906] Specific examples

[0907] For example, a user takes a photo of their health checkup results for October 2023 with their smartphone and uploads it to the app. This data is analyzed by the server, and the value "blood sugar level: 110 mg / dL" is extracted and stored in the database. The generation AI uses this data to predict that the user is at high risk of diabetes and generates personalized advice such as "increase exercise to prevent diabetes." Furthermore, the user can set up a link with an exercise tracking app, and the server will analyze that data to suggest specific amounts of exercise. At the same time, product recommendations that take health risks into consideration are also made, and specific sales notifications are sent, such as "20% off low-sugar snacks."

[0908] In this way, users can effectively utilize their own health checkup data and take appropriate measures. Data from other linked health apps can also be utilized, enabling more comprehensive health management. This system will help users extend their healthy lifespan and facilitate smooth communication with medical professionals.

[0909] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0910] Step 1:

[0911] The user takes a photo of the medical examination result form with the smartphone camera. The captured image is uploaded to the device. The input of this step is the image of the medical examination result, and the output is an image file saved on the device.

[0912] Step 2:

[0913] The device sends the uploaded image to the server. The server then sends the image to an image analysis means, which extracts text data from the image using optical character recognition (OCR) technology. The input for this step is the image of the health check result, and the output is the extracted text data. The specific software used is "Google Cloud Vision API."

[0914] Step 3:

[0915] The server sends the extracted text data to a natural language processing (NLP) module to identify health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.). The input of this step is the text data obtained from OCR, and the output is the identified health checkup items. The specific software used is "SpaCy."

[0916] Step 4:

[0917] The server stores the identified health check items in a database. The input of this step is the data of the health check items, and the output is the information stored in the database. The specific software used is "PostgreSQL."

[0918] Step 5:

[0919] The server inputs the stored health checkup data into the generation AI to predict health risks. The input for this step is the health checkup data stored in the database, and the output is the predicted health risk results. The specific software used is "GPT-4 (OpenAI)."

[0920] Step 6:

[0921] The server generates personalized health advice using generative AI based on the health risk prediction. The input of this step is the health risk prediction result, and the output is personalized health advice. Next, we will show a specific example in which the following prompt sentence is input to the generative AI model:

[0922] User's health data: { "Blood sugar": "110 mg / dL", "Cholesterol": "200 mg / dL", "BMI": "25.0"}. Provide targeted shopping advice.

[0923] Step 7:

[0924] The server then recommends personalized products based on the generated health advice. The input of this step is the health advice provided by the generative AI, and the output is a list of product recommendations.

[0925] Step 8:

[0926] The server manages medication information in conjunction with the medication notebook information and notifies the user of important medication-related information. The input for this step is the medication notebook data, and the output is the notification content.

[0927] Step 9:

[0928] The user sets up a link with an external exercise, sleep, or nutrition tracking app within the app, and the device sends the data acquired from the external app to the server. The input of this step is the external tracking data, and the output is the data integrated by the server.

[0929] Step 10:

[0930] The server aggregates the tracking data and provides more specific advice. The input of this step is the aggregated tracking data, and the output is detailed health advice.

[0931] Step 11:

[0932] Finally, the device displays the analysis results and health advice from the server to the user and notifies them of important information via push notifications. The input for this step is the final health advice and product recommendation information, and the output is the notification content. The specific software used for this is Firebase Cloud Messaging (FCM).

[0933] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0934] This invention is a system that effectively utilizes a user's health checkup data to provide personalized health management. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, a more accurate health management service is provided. This system integrates health risk prediction using AI generated from image analysis of health checkup data, a medicine notebook function, linkage with external tracking data, and the emotion engine function.

[0935] Acquisition and analysis of health checkup data

[0936] Users take a photo of their past health checkup results using their smartphone camera. The system receives the data by uploading the image to the device. The server then sends the image to an image analysis tool and extracts text data from the image using optical character recognition (OCR) technology.

[0937] Identifying and storing text data

[0938] The extracted text data is sent to a natural language processing (NLP) module on the server, which identifies health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.) and stores the identified data in a database.

[0939] Predicting health risks and generating personalized advice

[0940] The server then uses the AI ​​generator to analyze the stored health checkup data and predict the user's health risks. Based on the prediction results, the AI ​​generator generates personalized health advice, such as "increase exercise to prevent diabetes" or "reduce sugar in your diet."

[0941] Use of emotion engine

[0942] The server then uses an emotion engine to recognize the user's emotions. For example, it uses a camera and microphone to analyze the user's facial expressions and tone of voice to identify their current emotional state (e.g., stress, joy, anger, sadness). The recognized emotion data is then incorporated into the health advice provided by the generative AI.

[0943] Medicine notebook function

[0944] The server also has a medicine notebook function, automatically managing the medicine information entered by the user. The entered medicine information is analyzed and important information regarding medication is notified to the user.

[0945] External tracking data integration and integration

[0946] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app. The device then sends data obtained from the external apps to a server, which then integrates this data with health checkup data. For example, by adding exercise data, the AI ​​can provide detailed advice such as "running for 30 minutes three times a week is recommended."

[0947] Providing advice and notifications

[0948] Finally, the device displays the analysis results and health advice from the server to the user, and sends push notifications to inform the user of important health risks and medication information. The emotion engine also recognizes emotions and adjusts the content and manner of notifications. For example, if the user is feeling stressed, health advice will be delivered in a gentler tone.

[0949] Specific examples

[0950] For example, a user takes a photo of their health checkup results with their smartphone and uploads them to the app. This data is analyzed on the server, and a value such as "blood sugar level: 110 mg / dL" is extracted and stored in a database. The generation AI uses this data to predict that the user has a high risk of diabetes and generates personalized advice such as "You should exercise more to prevent diabetes."

[0951] Furthermore, once the user has linked up with an exercise tracking app, the server analyzes the data and suggests specific exercise amounts. If the emotion engine detects the user's stress level, it will provide advice that takes their emotions into consideration, such as "We recommend relaxation exercises to reduce your stress."

[0952] In this way, users can effectively utilize their own health check data and emotional state to take appropriate measures. Furthermore, by utilizing data from other linked health apps and the emotion engine, more comprehensive and responsive health management becomes possible. This system will help users extend their healthy lifespan and facilitate smooth communication with medical professionals.

[0953] The processing flow will be explained below.

[0954] Step 1:

[0955] The user takes a photo of the past health checkup results form using the smartphone camera.

[0956] Step 2:

[0957] The device will upload the captured images to the app.

[0958] Step 3:

[0959] The server receives the uploaded images and transmits them to the image analysis means.

[0960] Step 4:

[0961] The server's image analysis means uses optical character recognition (OCR) technology to extract text data from the image.

[0962] Step 5:

[0963] The server sends the extracted text data to a natural language processing (NLP) module.

[0964] Step 6:

[0965] The server's NLP module analyzes the text data and identifies health check items (e.g., blood sugar level, cholesterol level, BMI, etc.).

[0966] Step 7:

[0967] The server stores the identified data in a database.

[0968] Step 8:

[0969] The server sends the health checkup data stored in the database to the generation AI, which analyzes the data.

[0970] Step 9:

[0971] The server's generated AI predicts the user's health risks.

[0972] Step 10:

[0973] The server's generative AI generates personalized health advice based on predicted health risks.

[0974] Step 11:

[0975] The server formats and prepares the generated health advice for the user.

[0976] Step 12:

[0977] The server analyzes the medication information entered by the user and manages that information.

[0978] Step 13:

[0979] The server initiates a notification mechanism to notify the user of important information about the medication.

[0980] Step 14:

[0981] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app.

[0982] Step 15:

[0983] The terminal transmits the data acquired from the external application to the server.

[0984] Step 16:

[0985] The server integrates the health check data with data from external apps and sends it to the generation AI.

[0986] Step 17:

[0987] The server's generative AI generates detailed personalized advice based on the integrated data.

[0988] Step 18:

[0989] The server uses an emotion engine to recognize the user's emotion.

[0990] Step 19:

[0991] The server's emotion engine analyzes the user's facial expressions and tone of voice to identify their emotional state (e.g., stress, joy, anger, sadness).

[0992] Step 20:

[0993] The server sends the emotional data recognized by the emotion engine to the generative AI, which then adjusts the health advice.

[0994] Step 21:

[0995] The device displays the adjusted analysis results and health advice from the server to the user.

[0996] Step 22:

[0997] The device will send users push notifications about important health risks and medication information.

[0998] (Example)

[0999] For example, a user takes a photo of their health checkup results with their smartphone and uploads them to the app. This data is analyzed on the server, and a value such as "blood sugar level: 110 mg / dL" is extracted and stored in a database. The generation AI uses this data to predict that the user has a high risk of diabetes and generates personalized advice such as "You should exercise more to prevent diabetes."

[1000] Furthermore, once the user has linked up with an exercise tracking app, the server analyzes the data and suggests specific exercise amounts. If the emotion engine detects the user's stress level, it will provide advice that takes their emotions into consideration, such as "We recommend relaxation exercises to reduce your stress."

[1001] In this way, users can effectively utilize their own health check data and emotional state to take appropriate measures. Furthermore, by utilizing data from other linked health apps and the emotion engine, more comprehensive and responsive health management becomes possible. This system will help users extend their healthy lifespan and facilitate smooth communication with medical professionals.

[1002] Example 2

[1003] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1004] It is important for users to predict health risks based on their own health checkup data and receive personalized health advice. However, most systems have complex data acquisition and analysis processes, and it is difficult to provide advice that takes into account the user's emotional state. Therefore, it is a challenge to utilize users' health checkup data and realize highly accurate health management based on their emotional state.

[1005] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring past health checkup data, means for analyzing the acquired health checkup data using image analysis means, means for storing the analyzed data in a database, means for predicting health risks using a generative AI model based on the stored data, means for generating personalized health advice based on the predicted health risks, a terminal for notifying the user of the generated advice and prediction results, emotion analysis means for recognizing the user's emotional state, information acquisition means for acquiring and integrating external tracking data, and means for adjusting the content and method of notification based on the emotion analysis means. This makes it possible to effectively utilize the user's health checkup data, provide specific health risk predictions and personalized advice, and provide responsive health management based on the user's emotional state.

[1006] "Past health checkup data" refers to data that indicates the results of health checkups that the user has taken in the past.

[1007] "Image analysis means" refers to technology used to analyze information within an image, and specifically includes optical character recognition (OCR) technology.

[1008] A "database" is a system for storing and managing acquired and analyzed data.

[1009] A "generative AI model" is an artificial intelligence model that performs analysis and predictions based on a user's health checkup data, and specifically includes a type of generative model.

[1010] "Health risk prediction" refers to the act of analyzing a user's health checkup data and predicting future health risks.

[1011] "Personalized health advice" means individualized health advice provided to an individual user based on their health status and risks.

[1012] A "terminal" is an electronic device such as a smartphone or tablet used by a user.

[1013] "Emotion analysis means" refers to technology for recognizing a user's emotional state, and includes data analysis using a camera or microphone.

[1014] "External tracking data" refers to data about exercise, sleep, nutrition, etc. obtained from external tracking devices or applications.

[1015] "Information acquisition means" refers to the technology or protocol used to acquire necessary data from external devices or applications.

[1016] The "means for adjusting the notification content and notification method" refers to a means for optimally adjusting the content and method of the information to be notified based on the emotional state of the user.

[1017] The present invention is a system for improving a user's health management, which integrates the user's health checkup data and external tracking data, and uses a generative AI model to provide health risk prediction and personalized health advice. Furthermore, it uses emotion analysis means to recognize the user's emotional state and adjust the content of advice and notification method based on that information.

[1018] First, the user takes a photo of the past medical checkup results using the smartphone camera. The system receives the data by uploading the image to the device. The device then uploads the image to cloud storage and notifies the server.

[1019] The server retrieves images from cloud storage and extracts text data from the images using optical character recognition (OCR) technology (e.g., Tesseract OCR). The extracted text data is then analyzed using a natural language processing (NLP) module (e.g., SpaCy) to identify health checkup items. The identified health checkup data (e.g., blood glucose level, cholesterol level, BMI) is stored in a database.

[1020] Based on the stored data, the server uses a generative AI model (e.g., OpenAI GPT-3) to predict the user's health risks. For example, based on the data "blood sugar level 110 mg / dL," it predicts "high risk of diabetes" and generates personalized health advice such as "increase exercise by 30 minutes."

[1021] Furthermore, the server recognizes the user's emotional state using emotion analysis methods (e.g., Microsoft Azure Emotion API). It analyzes the user's facial expressions and tone of voice using the device's camera and microphone to identify their emotional state (e.g., stress, joy, anger, sadness). This emotional data is reflected in the generative AI's advice. For example, if the user is feeling stressed, it will provide health advice in a gentler tone.

[1022] Users can set up integration with external exercise, sleep, and nutrition tracking apps. The device sends data obtained from the external apps to a server, and this data is integrated with health checkup data to provide more detailed advice (e.g., "We recommend running for 30 minutes three times a week").

[1023] It also has a medicine notebook function, and the server automatically manages the medicine information entered by the user. The entered medicine information is analyzed, and important information about taking the medicine (e.g., interactions, precautions, etc.) is notified from the server.

[1024] Finally, the device displays the analysis results and health advice from the server to the user, and notifies the user of important information via push notifications. The content and method of notifications are also adjusted according to the emotions recognized by the emotion analysis means.

[1025] For example, the following prompt sentence is input to the generative AI model:

[1026] Generate specific health risk predictions and personalized advice based on the user's health checkup data. For example, if a blood sugar level is 110 mg / dL, predict a high risk of diabetes and provide specific advice on exercise and dietary improvements. Also, take into account the user's emotional state (e.g., high stress).

[1027] This allows users to effectively utilize their own health checkup data, external tracking data linked to their own health checkup data, and their emotional state to optimally manage their health.

[1028] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1029] Step 1:

[1030] The user takes a photo of a past medical checkup result form using the smartphone camera. The system receives the data by uploading the image to the terminal. The input image data is passed to the terminal. This input image data is then converted into text data through subsequent optical character recognition (OCR) processing.

[1031] Step 2:

[1032] The device uploads the captured images to the cloud storage. A cloud storage URL is generated and notified to the server. The server then receives the cloud storage URL as input for retrieving the images.

[1033] Step 3:

[1034] The server retrieves images from cloud storage and extracts text data from the images using optical character recognition (OCR) technology (e.g., Tesseract OCR). Specifically, the server inputs the retrieved image data into the OCR engine and outputs text data. This text data is used to identify and store health checkup items.

[1035] Step 4:

[1036] The server sends the text data extracted by OCR to a natural language processing (NLP) module (e.g., SpaCy) to identify various health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.). The input is text data, and the output is the identified health checkup items. This identified data is stored in a database.

[1037] Step 5:

[1038] The server stores the data of the specified medical examination items in a database, receiving the specified data as input and obtaining the medical examination data in the format stored in the database as output.

[1039] Step 6:

[1040] The server uses a generative AI model (e.g., OpenAI GPT-3) based on the stored health checkup data to predict the user's health risk. The generative AI model receives the health checkup data as input, and outputs a predicted health risk. For example, a specific prediction such as "Your blood sugar level is 110 mg / dL, so you're at high risk for diabetes" is generated.

[1041] Step 7:

[1042] The server generates personalized health advice using a generative AI model based on the predicted health risks. The predicted health risks are input to the generative AI model, and specific health advice is output. For example, advice such as "Increase your exercise by 30 minutes" is generated.

[1043] Step 8:

[1044] The server recognizes the user's emotional state using emotion analysis means (e.g., Microsoft Azure Emotion API). Data acquired from the camera and microphone is sent as input to the emotion analysis engine, and the user's emotional state is obtained as output. For example, a specific emotional state such as "high stress level" is recognized.

[1045] Step 9:

[1046] The server incorporates the recognized emotional data into the health advice provided by the generative AI and adjusts the content of the advice and notification method. Emotional state and health advice are given as input, and emotion-conscious health advice and notifications are generated as output. For example, if the user is feeling stressed, advice will be provided in a calm tone.

[1047] Step 10:

[1048] Users configure the app to connect to external exercise, sleep, and nutrition tracking apps. The device receives data from the external apps and sends it to a server. The server receives data from the external apps as input and provides integrated health data as output.

[1049] Step 11:

[1050] The server then integrates this data with the health checkup data to generate detailed health advice. The integrated data is given as input to a generative AI model, which outputs detailed health advice. For example, specific advice such as "running for 30 minutes three times a week is recommended" is generated.

[1051] Step 12:

[1052] The server automatically manages medication information entered by the user using the medication notebook function. It analyzes the entered medication information and notifies the user of important medication information, such as drug interactions and precautions.

[1053] Step 13:

[1054] The device displays the analysis results and health advice from the server to the user, and generates push notifications and health advice to be sent to the user as output.

[1055] Step 14:

[1056] The device adjusts the content and method of notifications based on the emotions recognized by the emotion analysis method, ultimately providing appropriate health advice to the user based on their emotions.

[1057] (Application example 2)

[1058] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1059] Health management is becoming increasingly important in modern society, but personalized health support based on individual users' health status and emotions is still lacking. In particular, in-store customer service often fails to provide prompt and accurate health advice that takes into account individual health data and current emotions. Therefore, there is a need for a system that can accurately predict users' health risks, generate personalized health advice based on those risks, and provide health support in-store that takes into account the user's emotions.

[1060] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring past health checkup data, means for analyzing the acquired health checkup data using image analysis means, means for saving the analyzed data in a database, means for predicting health risks using a generation AI based on the saved data, means for generating personalized health advice based on the predicted health risks, means for notifying the user of the generated advice and prediction results, information acquisition means for acquiring and integrating external tracking data, means for recognizing the user's emotions and incorporating the emotion data into the analysis results, and means for providing emotion-based health support via an automated answering device in a physical store. This makes it possible to quickly and accurately provide personalized health advice based on the health condition and emotions of individual users even in physical stores.

[1061] "Past health checkup data" refers to data showing the results of health checkups the user has undergone in the past, and specifically includes numerical information for assessing health status, such as blood sugar levels, cholesterol levels, and BMI.

[1062] The "image analysis means" is a technology for extracting text data from acquired image data, specifically using optical character recognition (OCR) technology.

[1063] "Generative AI" is a type of artificial intelligence model used to generate predictions or advice from specific input data. In this case, it refers to an AI model for predicting health risks.

[1064] "Personalized health advice" means advice, including individualized health guidance and recommendations, generated based on a user's individual health data and predicted health risks.

[1065] "External tracking data" refers to health-related data, such as a user's exercise, sleep, and nutrition, obtained from external devices or applications.

[1066] "Means for recognizing emotions" refers to technology that analyzes a user's facial expressions and tone of voice to identify their emotional state at that time.

[1067] An "automatic answering device" is a device installed in a physical store to support users in managing their health, and is responsible for providing appropriate support and advice based on the user's emotional state.

[1068] "Information acquisition means" refers to a means for acquiring information such as tracking data from the outside and integrating it into the system.

[1069] "Means of notification" refers to the means for informing users of the generated advice and prediction results, and specifically includes push notifications on smartphones.

[1070] This invention relates to a personalized health management system for brick-and-mortar stores. This system analyzes past health checkup data, provides individualized health advice to users, and recognizes users' emotions and incorporates their emotional state into the analysis results to provide health support within the brick-and-mortar stores.

[1071] The server first provides a means for users to obtain past health checkup data. Specifically, the user obtains this data by taking a photo of the health checkup result form using the camera on their smartphone. The obtained image data is uploaded to the server and analyzed using image analysis means. Optical character recognition (OCR) technology is used to extract text data from the image.

[1072] The server then analyzes the extracted text data using a natural language processing (NLP) module to identify health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.), and stores the identified data in a database.

[1073] The server then uses the stored data to predict the user's health risks using generative AI. The generative AI model comprehensively evaluates the user's health status and generates personalized health advice. For example, specific advice such as "You should increase exercise to prevent diabetes" or "You should reduce sugar in your diet" is generated.

[1074] The system also includes a means to recognize the user's emotions. The server uses a camera and microphone to analyze the user's facial expressions and tone of voice to identify their current emotional state (e.g., stress, joy, anger, sadness, etc.). This emotional data is incorporated into the generated health advice, and the content and delivery of notifications and advice are tailored accordingly.

[1075] Additionally, external tracking data such as exercise, sleep, and nutrition data from other tracking apps used by the user can be integrated using information acquisition methods, providing a more complete picture of the user's health and enabling more detailed health advice.

[1076] Based on the accumulated data, information about the user's medication is also managed. This allows information about the medications the user is taking and important notifications to be managed by the server and provided to the user. The generated health advice and prediction results are also provided to medical professionals, who can summarize and display the analysis results and provide appropriate health guidance.

[1077] For example, a user takes a photo of their health checkup results with their smartphone at a store counter and uploads it to the system. This data is analyzed on the server, and the value "Blood glucose level: 110 mg / dL" is extracted and saved. Based on this data, the generation AI predicts that the user is at high risk for diabetes and generates personalized advice such as "You should increase your exercise to prevent diabetes." If the AI ​​recognizes the user's emotional state and determines that stress is high, it provides advice that takes their emotions into consideration, such as "We recommend yoga to reduce stress."

[1078] An example of a prompt is as follows:

[1079] "User's health is at high risk. Dietary changes are required."

[1080] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1081] Step 1:

[1082] Users take photos of their past health checkup results with their smartphone camera and upload the image data to the device. At this stage, the input is image data of the health checkup results, and the output is image data sent to the server. The device converts the captured image into an appropriate format and sends it to the server via a stable communication path.

[1083] Step 2:

[1084] The server converts the received image data into text data using optical character recognition (OCR) technology. The input at this stage is the uploaded health check image data, and the output is text data extracted from the image. The server analyzes the image using an OCR engine such as "Tesseract" and extracts the necessary text information.

[1085] Step 3:

[1086] The server analyzes the extracted text data using a natural language processing (NLP) module to identify specific health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.). The input at this stage is the text data extracted by OCR, and the output is the analyzed health checkup item data. For example, formatted data such as "blood glucose level: 110 mg / dL" is obtained.

[1087] Step 4:

[1088] The server stores the analyzed health check item data in a database. The input at this stage is the data of the identified health check items, and the output is structured data stored in the database. The data will be used for later analysis and prediction by generative AI.

[1089] Step 5:

[1090] Next, the server uses the stored data to predict the user's health risk using a generative AI model. The input at this stage is the health checkup data stored in the database, and the output is the predicted health risk and its details. For example, the assessment may be "high risk of diabetes."

[1091] Step 6:

[1092] The server generates personalized health advice using a generative AI model based on the predicted health risks. The input at this stage is the predicted health risks, and the output is the generated specific health advice. An example of generated advice is "You should exercise more to prevent diabetes."

[1093] Step 7:

[1094] The server analyzes the user's emotional state using emotion recognition means. The user sends data of their facial expressions and tone of voice to the server via a camera or microphone. The input at this stage is data indicating the user's emotional state, and the output is data of the analyzed emotional state. For example, it can recognize that the user is feeling stressed.

[1095] Step 8:

[1096] The server then incorporates the recognized emotional state into the generated health advice. The input at this stage is the health advice and emotional state data, and the output is personalized health advice that takes the emotion into account. An example of generated advice is "We recommend relaxing exercises to reduce stress."

[1097] Step 9:

[1098] The server notifies the user of the generated health advice and prediction results. The device receives the notification from the server and displays it to the user via push notification. The input at this stage is the generated advice and prediction results, and the output is the notification displayed on the device. The notification may include suggested actions for the user or recommendations for device use.

[1099] Step 10:

[1100] The user can then put the advice into practice and provide feedback to the system. At this stage, the input is the user's action data, and the output is the feedback data sent to the system. This allows the system to provide continuous, personalized health management.

[1101] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1102] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1103] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1104] [Fourth embodiment]

[1105] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1106] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1107] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1108] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1109] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1110] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1111] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1112] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1113] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1114] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1115] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1116] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1117] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1118] The present invention is a system that effectively utilizes a user's health checkup data to provide personalized health management. This system has a wide range of functions, including predicting health risks using AI generated from image analysis of health checkup data, a medicine notebook function, and linking with external tracking data. Specific embodiments of the system are described below.

[1119] Acquisition and analysis of health checkup data

[1120] The user takes a photo of the health checkup result form with the smartphone camera. The system receives the data by uploading the image to the terminal. The server sends the received image to an image analysis means and extracts text data from the image using optical character recognition (OCR) technology.

[1121] Identifying and storing text data

[1122] The extracted text data is sent to a natural language processing (NLP) module on the server, which identifies health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.) and stores the identified data in a database.

[1123] Predicting health risks and generating personalized advice

[1124] The server then uses the AI ​​generator to analyze the stored health checkup data and predict the user's health risks. Based on the prediction results, the AI ​​generator generates personalized health advice, such as "increase exercise to prevent diabetes" or "reduce sugar in your diet."

[1125] Medicine notebook function

[1126] The server also has a medicine notebook function, automatically managing the medicine information entered by the user. The entered medicine information is analyzed and important information regarding medication is notified to the user.

[1127] External tracking data integration and integration

[1128] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app. The device then sends data obtained from the external apps to a server, which then integrates this data with health checkup data. For example, by adding exercise data, the AI ​​can provide detailed advice such as "running for 30 minutes three times a week is recommended."

[1129] Providing advice and notifications

[1130] Finally, the device displays the analysis results and health advice from the server to the user, and notifies the user of important health risks and medication information via push notifications.

[1131] Specific examples

[1132] For example, a user takes a photo of their health checkup results for October 2023 with their smartphone and uploads it to the app. This data is analyzed by the server, and the value "blood sugar level: 110 mg / dL" is extracted and stored in the database. The generation AI uses this data to predict that the user has a high risk of diabetes and generates personalized advice such as "increase exercise to prevent diabetes." Furthermore, the user sets up a link with an exercise tracking app, and the server analyzes the data to suggest specific amounts of exercise.

[1133] In this way, users can effectively utilize their own health checkup data and take appropriate measures. Data from other linked health apps can also be utilized, enabling more comprehensive health management. This system will help users extend their healthy lifespan and facilitate smooth communication with medical professionals.

[1134] The processing flow will be explained below.

[1135] Step 1:

[1136] The user takes a photo of the past health checkup results form using the smartphone camera.

[1137] Step 2:

[1138] The device will upload the captured images to the app.

[1139] Step 3:

[1140] The server receives the uploaded images and transmits them to the image analysis means.

[1141] Step 4:

[1142] The server's image analysis means uses optical character recognition (OCR) technology to extract text data from the image.

[1143] Step 5:

[1144] The server sends the extracted text data to a natural language processing (NLP) module.

[1145] Step 6:

[1146] The server's NLP module analyzes the text data and identifies health check items (e.g., blood sugar level, cholesterol level, BMI, etc.).

[1147] Step 7:

[1148] The server stores the identified data in a database.

[1149] Step 8:

[1150] The server sends the health checkup data stored in the database to the generation AI, which analyzes the data.

[1151] Step 9:

[1152] The server's generated AI predicts the user's health risks.

[1153] Step 10:

[1154] The server's generative AI generates personalized health advice based on predicted health risks.

[1155] Step 11:

[1156] The server formats and prepares the generated health advice for the user.

[1157] Step 12:

[1158] The server analyzes the medication information entered by the user and manages that information.

[1159] Step 13:

[1160] The server initiates a notification mechanism to notify the user of important information about the medication.

[1161] Step 14:

[1162] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app.

[1163] Step 15:

[1164] The terminal transmits the data acquired from the external application to the server.

[1165] Step 16:

[1166] The server integrates the health check data with data from external apps and sends it to the generation AI.

[1167] Step 17:

[1168] The server's generative AI generates detailed personalized advice based on the integrated data.

[1169] Step 18:

[1170] The device displays the analysis results and health advice from the server to the user.

[1171] Step 19:

[1172] The device will send users push notifications about important health risks and medication information.

[1173] Example 1

[1174] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1175] Conventional health management systems require users to manually input data, which is time-consuming and laborious, and also carries a high risk of input errors. Furthermore, because they only reference individual health checkup data, it is difficult to comprehensively grasp a user's current health status. Furthermore, it is difficult to integrate and utilize external tracking data. As a result, it is not possible to effectively provide personalized advice to users, and they are unable to provide sufficient support for maintaining their health and preventing disease.

[1176] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1177] In this invention, the server includes a means for uploading images of health checkup data taken by a user using a smartphone, a means for analyzing the uploaded images using an image analysis means and extracting text data using optical character recognition technology, a means for analyzing the extracted text data using a natural language processing module to identify health checkup items and store them in a database, a means for analyzing the stored health checkup data using a generative AI model and predicting the user's health risks, a means for generating personalized health advice based on the predicted health risks, a means for notifying the user of the generated advice and prediction results, an information acquisition means for acquiring and integrating external tracking data, and a means for integrating the acquired external tracking data and health checkup data to generate detailed advice. This allows users to easily import health checkup data and perform comprehensive health management. Furthermore, the automatically generated personalized advice allows users to take appropriate measures to maintain their health and prevent disease.

[1178] "User" means an individual who uses the system to provide health checkup results and external tracking data.

[1179] A "smartphone" is a portable information terminal equipped with a camera and applications that allows users to take and upload health checkup results.

[1180] "Image analysis means" refers to optical character recognition technology for extracting text data from uploaded images of medical examination results.

[1181] "Optical character recognition technology" is a technology that recognizes and extracts character information from images, and is also known as OCR (Optical Character Recognition).

[1182] "Text data" refers to character information extracted using optical character recognition technology.

[1183] The "natural language processing module" is software that analyzes the extracted text data and identifies specific health checkup items.

[1184] "Health check items" refer to the various measurement data and diagnostic results included in the health check results, such as blood sugar levels, cholesterol levels, and BMI.

[1185] A "database" is a system for systematically storing analyzed text data and user information.

[1186] A "generative AI model" is an artificial intelligence model used to analyze stored health checkup data and predict health risks.

[1187] "Health risk" is information indicating the likelihood that the user will have a health problem in the future.

[1188] "Personalized health advice" means specific health measures and advice tailored to the individual user's health condition and lifestyle.

[1189] The "notification means" refers to a means for notifying the user of the generated advice or prediction results, and includes push notifications, etc.

[1190] "External tracking data" refers to health-related data obtained from other tracking applications, such as exercise, sleep, and nutrition.

[1191] "Information acquisition means" refers to the function for acquiring data from external tracking applications and utilizing it within the system.

[1192] "Integration" refers to compiling information obtained from multiple data sources into a single dataset and using it for analysis.

[1193] The system of the present invention allows users to easily input health checkup data, analyze the data, predict health risks, and provide appropriate health advice. It is also possible to integrate external tracking data to provide more detailed advice.

[1194] This system is realized using the following hardware and software.

[1195] Obtaining health checkup data

[1196] The user uses their smartphone to take a photo of the form containing the health checkup results. After taking the photo, the user launches a dedicated application and uploads the image to the application. This application runs on the user's smartphone and has an image upload function.

[1197] Image analysis and text data extraction

[1198] The uploaded image data is sent from the device to the server. The server analyzes the received image and extracts text data using optical character recognition (OCR) technology. Specifically, the server uses the Tesseract OCR engine to analyze the characters in the image. For example, information such as "Blood glucose level: 110 mg / dL" is extracted as text data.

[1199] Identifying and storing text data

[1200] The extracted text data is sent to a natural language processing (NLP) module on the server, which uses the spaCy library to identify health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.) and stores the identified data in a MySQL database on the server.

[1201] Predicting health risks and generating advice

[1202] Based on the stored data, the server uses a generative AI model (specifically, an AI model such as OpenAI's GPT-3) to predict the user's health risks. Based on the prediction, the generative AI generates personalized health advice. For example, an example prompt could be, "Please predict the user's health risks and provide advice based on the following health checkup data: blood sugar level 110 mg / dL, cholesterol level 200 mg / dL, BMI 25."

[1203] Provision of a medicine notebook function

[1204] Users can enter medication information within the app, and the server automatically manages the information. Medication information can also be entered using barcode scanning technology (e.g., the ZBar library). The entered data is analyzed, and important medication information and reminders are sent to the user by the server.

[1205] External tracking data integration and integration

[1206] Within the app, users can set up integration with external exercise, sleep, and nutrition tracking apps (such as Fitbit and MyFitnessPal). The device collects tracking data from these linked applications and sends it to a server. The server then combines this data with health checkup data to generate more detailed health advice. For example, specific advice such as "We recommend running 30 minutes three times a week" may be provided based on the exercise data.

[1207] Providing advice and notifications

[1208] Finally, the server generates analysis results and health advice, which are displayed to the user via the device. Important health risks and medication information are also notified to the user via the device's push notification function (Firebase Cloud Messaging).

[1209] This allows users to comprehensively utilize their own health checkup data and external tracking data to take more specific and appropriate health measures.

[1210] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1211] Step 1:

[1212] The user takes a photo of the medical checkup result form using the smartphone camera. The captured image is the input and triggers the system to start processing. Next, the user launches a dedicated app and uploads the captured image to the app. This operation outputs the image of the medical checkup result and uses it as input for the next step.

[1213] Step 2:

[1214] The terminal sends the images of the health checkup results uploaded by the user to the server. The input is the image data of the health checkup, and by transferring this to the server, the server is ready to perform image analysis. The output is the image data received by the server.

[1215] Step 3:

[1216] The server sends the received image to an image analysis means and extracts text data from the image using optical character recognition (OCR) technology. Specifically, it uses the Tesseract OCR engine to recognize characters in the image and extract them as text data. The input is the image data of the health checkup, and the output is the extracted text data.

[1217] Step 4:

[1218] The server sends the extracted text data to a natural language processing (NLP) module to identify health checkup items. Specifically, it uses the spaCy library to analyze and identify health checkup items such as blood glucose level, cholesterol level, and BMI. The input is the text data extracted by OCR, and the output is the data of the identified health checkup items.

[1219] Step 5:

[1220] The server stores the data of the identified health check items in a database. Here, the input is the text data identified by NLP, and the output is the health check data stored in the database. For example, data such as "Blood glucose level: 110 mg / dL" is stored in a MySQL database.

[1221] Step 6:

[1222] The server uses a generative AI model to analyze the stored health checkup data and predict the user's health risks. The health checkup data stored in the database is used as input, and health risks are predicted by entering prompt statements into the generative AI. The output is the predicted health risks and personalized health advice. For example, advice such as "You should increase exercise to prevent diabetes" is generated.

[1223] Step 7:

[1224] The server notifies the user of predicted health risks and personalized advice. The input is the advice and prediction results obtained from the generative AI, which are sent to the terminal and notified to the user. The output is the predicted health risks and advice displayed on the user's smartphone.

[1225] Step 8:

[1226] The user enters medication information within the app, for example by scanning the medication barcode. The input is the medication information provided by the user, and the output is the medication data sent to the server.

[1227] Step 9:

[1228] The server analyzes the entered medication information and notifies the user of medication reminders and important information. The input is the medication information provided by the user, and the output is the analyzed medication information and notifications of reminders and precautions based on it.

[1229] Step 10:

[1230] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app. The input is the integration information for the external app selected and configured by the user, and the output is the data obtained from the tracking app.

[1231] Step 11:

[1232] The device acquires data from the linked external tracking app and sends it to the server. The input is the data from the tracking app, and the output is the external data sent to the server.

[1233] Step 12:

[1234] The server integrates external data with the original health checkup data to generate personalized, detailed health advice. The input is the external data and the health checkup data stored in the database, and the output is specific advice based on the integrated data. For example, advice such as "We recommend running for 30 minutes three times a week" is provided.

[1235] (Application example 1)

[1236] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1237] Health management systems typically predict health risks based on a user's health checkup data and provide personalized health advice. However, current systems lack the functionality to recommend products based on the user's health condition or to select appropriate products by linking with prescription record information. This makes it difficult for users to easily select products that suit their health condition, particularly when taking specific medications. Another issue is that the generated health advice and prediction results are not adequately communicated, providing insufficient support for users to take appropriate actions.

[1238] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1239] In this invention, the server includes means for acquiring past health checkup data, means for analyzing the acquired health checkup data using image analysis means, means for saving the analyzed data in a database, means for predicting health risks using a generation AI based on the saved data, means for generating personalized health advice based on the predicted health risks, means for notifying the user of the generated advice and prediction results, information acquisition means for acquiring and integrating external tracking data, means for recommending personalized products based on the analyzed health data, and means for selecting products in conjunction with information in the prescription notebook. This allows the user to receive product recommendations tailored to their health condition, enabling them to select appropriate products, and enables the generated health advice and prediction results to be efficiently notified.

[1240] "Past health checkup data" refers to the results data of health checkups that the user has previously undergone.

[1241] "Image analysis means" refers to optical character recognition (OCR) technology and other technologies for extracting text information from images of health checkup data.

[1242] "Database" refers to a storage device for storing analyzed medical examination data and other related information.

[1243] "Generative AI" refers to artificial intelligence algorithms that generate new information based on past data.

[1244] "Health risk" refers to the user's assessment of the likelihood of developing a particular health problem in the future.

[1245] "Health advice" refers to specific suggestions for actions or improvements provided based on the user's health risk predictions.

[1246] "Notification means" refers to the methods and technologies used to notify users of generated health advice and prediction results.

[1247] "Tracking Data" refers to a user's health-related data obtained from external sources, such as exercise, sleep, and nutrition.

[1248] "Product recommendation means" refers to a function that presents appropriate products based on the user's health checkup data and health risk predictions.

[1249] A "medicine notebook" is a tool that allows users to record and manage information about the medications they are taking.

[1250] "Product selection methods" refer to methods and techniques for selecting the most suitable product for a user in conjunction with information in the medicine notebook.

[1251] The present invention is a system that effectively utilizes a user's health checkup data to provide personalized health management and product recommendations. This system has a wide range of functions, including health risk prediction using AI generated from image analysis of health checkup data, product recommendations, a medicine notebook function, and linkage with external tracking data. Specific embodiments of the system are described below.

[1252] Acquisition and analysis of health checkup data

[1253] The user takes a photo of the health check result form using the smartphone camera. The image is uploaded to the device, and the system receives the data. The server then sends the received image to an image analysis tool, which extracts text data from the image using optical character recognition (OCR) technology. The specific software used is "Google Cloud Vision API."

[1254] Identifying and storing text data

[1255] The extracted text data is sent to a natural language processing (NLP) module on the server, which identifies health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.). The identified data is stored in a database. The specific software used is "SpaCy" and "PostgreSQL."

[1256] Predicting health risks and generating personalized advice

[1257] The server then uses the generative AI to analyze the stored health checkup data and predict the user's health risks. Based on the prediction results, the generative AI generates personalized health advice. For example, specific advice such as "You should increase exercise to prevent diabetes" or "You should reduce sugar in your diet" is provided. The specific software used is "GPT-4 (OpenAI)."

[1258] Personalized product recommendations

[1259] The server uses generative AI to make personalized product recommendations based on health risk predictions. For example, it recommends health-conscious product categories such as "low-sugar foods," "snacks without trans fats," and "high-fiber foods." This allows users to easily choose products that best suit their health condition. Below is an example of a prompt sentence to input to the generative AI model:

[1260] User's health data: { "Blood sugar": "110 mg / dL", "Cholesterol": "200 mg / dL", "BMI": "25.0"}. Provide targeted shopping advice.

[1261] Medicine notebook function and product selection

[1262] The server also has a medicine notebook function, automatically managing the medication information entered by the user. The entered medication information is analyzed and important information about medication is notified to the user. At the same time, advice is given on avoiding foods that are incompatible with the medication being taken.

[1263] External tracking data integration and integration

[1264] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app. The device then sends data obtained from the external apps to a server, which then integrates this data with health checkup data. For example, by adding exercise data, the AI ​​can provide detailed advice such as "running for 30 minutes three times a week is recommended."

[1265] Providing advice and notifications

[1266] Finally, the device displays the analysis results and health advice from the server to the user, and also notifies the user of important health risks and product information via push notifications.The notification system uses Firebase Cloud Messaging (FCM), which allows information to be delivered to users quickly and efficiently.

[1267] Specific examples

[1268] For example, a user takes a photo of their health checkup results for October 2023 with their smartphone and uploads it to the app. This data is analyzed by the server, and the value "blood sugar level: 110 mg / dL" is extracted and stored in the database. The generation AI uses this data to predict that the user is at high risk of diabetes and generates personalized advice such as "increase exercise to prevent diabetes." Furthermore, the user can set up a link with an exercise tracking app, and the server will analyze that data to suggest specific amounts of exercise. At the same time, product recommendations that take health risks into consideration are also made, and specific sales notifications are sent, such as "20% off low-sugar snacks."

[1269] In this way, users can effectively utilize their own health checkup data and take appropriate measures. Data from other linked health apps can also be utilized, enabling more comprehensive health management. This system will help users extend their healthy lifespan and facilitate smooth communication with medical professionals.

[1270] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1271] Step 1:

[1272] The user takes a photo of the medical examination result form with the smartphone camera. The captured image is uploaded to the device. The input of this step is the image of the medical examination result, and the output is an image file saved on the device.

[1273] Step 2:

[1274] The device sends the uploaded image to the server. The server then sends the image to an image analysis means, which extracts text data from the image using optical character recognition (OCR) technology. The input for this step is the image of the health check result, and the output is the extracted text data. The specific software used is "Google Cloud Vision API."

[1275] Step 3:

[1276] The server sends the extracted text data to a natural language processing (NLP) module to identify health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.). The input of this step is the text data obtained from OCR, and the output is the identified health checkup items. The specific software used is "SpaCy."

[1277] Step 4:

[1278] The server stores the identified health check items in a database. The input of this step is the data of the health check items, and the output is the information stored in the database. The specific software used is "PostgreSQL."

[1279] Step 5:

[1280] The server inputs the stored health checkup data into the generation AI to predict health risks. The input for this step is the health checkup data stored in the database, and the output is the predicted health risk results. The specific software used is "GPT-4 (OpenAI)."

[1281] Step 6:

[1282] The server generates personalized health advice using generative AI based on the health risk prediction. The input of this step is the health risk prediction result, and the output is personalized health advice. Next, we will show a specific example in which the following prompt sentence is input to the generative AI model:

[1283] User's health data: { "Blood sugar": "110 mg / dL", "Cholesterol": "200 mg / dL", "BMI": "25.0"}. Provide targeted shopping advice.

[1284] Step 7:

[1285] The server then recommends personalized products based on the generated health advice. The input of this step is the health advice provided by the generative AI, and the output is a list of product recommendations.

[1286] Step 8:

[1287] The server manages medication information in conjunction with the medication notebook information and notifies the user of important medication-related information. The input for this step is the medication notebook data, and the output is the notification content.

[1288] Step 9:

[1289] The user sets up a link with an external exercise, sleep, or nutrition tracking app within the app, and the device sends the data acquired from the external app to the server. The input of this step is the external tracking data, and the output is the data integrated by the server.

[1290] Step 10:

[1291] The server aggregates the tracking data and provides more specific advice. The input of this step is the aggregated tracking data, and the output is detailed health advice.

[1292] Step 11:

[1293] Finally, the device displays the analysis results and health advice from the server to the user and notifies them of important information via push notifications. The input for this step is the final health advice and product recommendation information, and the output is the notification content. The specific software used for this is Firebase Cloud Messaging (FCM).

[1294] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1295] This invention is a system that effectively utilizes a user's health checkup data to provide personalized health management. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, a more accurate health management service is provided. This system integrates health risk prediction using AI generated from image analysis of health checkup data, a medicine notebook function, linkage with external tracking data, and the emotion engine function.

[1296] Acquisition and analysis of health checkup data

[1297] Users take a photo of their past health checkup results using their smartphone camera. The system receives the data by uploading the image to the device. The server then sends the image to an image analysis tool and extracts text data from the image using optical character recognition (OCR) technology.

[1298] Identifying and storing text data

[1299] The extracted text data is sent to a natural language processing (NLP) module on the server, which identifies health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.) and stores the identified data in a database.

[1300] Predicting health risks and generating personalized advice

[1301] The server then uses the AI ​​generator to analyze the stored health checkup data and predict the user's health risks. Based on the prediction results, the AI ​​generator generates personalized health advice, such as "increase exercise to prevent diabetes" or "reduce sugar in your diet."

[1302] Use of emotion engine

[1303] The server then uses an emotion engine to recognize the user's emotions. For example, it uses a camera and microphone to analyze the user's facial expressions and tone of voice to identify their current emotional state (e.g., stress, joy, anger, sadness). The recognized emotion data is then incorporated into the health advice provided by the generative AI.

[1304] Medicine notebook function

[1305] The server also has a medicine notebook function, automatically managing the medicine information entered by the user. The entered medicine information is analyzed and important information regarding medication is notified to the user.

[1306] External tracking data integration and integration

[1307] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app. The device then sends data obtained from the external apps to a server, which then integrates this data with health checkup data. For example, by adding exercise data, the AI ​​can provide detailed advice such as "running for 30 minutes three times a week is recommended."

[1308] Providing advice and notifications

[1309] Finally, the device displays the analysis results and health advice from the server to the user, and sends push notifications to inform the user of important health risks and medication information. The emotion engine also recognizes emotions and adjusts the content and manner of notifications. For example, if the user is feeling stressed, health advice will be delivered in a gentler tone.

[1310] Specific examples

[1311] For example, a user takes a photo of their health checkup results with their smartphone and uploads them to the app. This data is analyzed on the server, and a value such as "blood sugar level: 110 mg / dL" is extracted and stored in a database. The generation AI uses this data to predict that the user has a high risk of diabetes and generates personalized advice such as "You should exercise more to prevent diabetes."

[1312] Furthermore, once the user has linked up with an exercise tracking app, the server analyzes the data and suggests specific exercise amounts. If the emotion engine detects the user's stress level, it will provide advice that takes their emotions into consideration, such as "We recommend relaxation exercises to reduce your stress."

[1313] In this way, users can effectively utilize their own health check data and emotional state to take appropriate measures. Furthermore, by utilizing data from other linked health apps and the emotion engine, more comprehensive and responsive health management becomes possible. This system will help users extend their healthy lifespan and facilitate smooth communication with medical professionals.

[1314] The processing flow will be explained below.

[1315] Step 1:

[1316] The user takes a photo of the past health checkup results form using the smartphone camera.

[1317] Step 2:

[1318] The device will upload the captured images to the app.

[1319] Step 3:

[1320] The server receives the uploaded images and transmits them to the image analysis means.

[1321] Step 4:

[1322] The server's image analysis means uses optical character recognition (OCR) technology to extract text data from the image.

[1323] Step 5:

[1324] The server sends the extracted text data to a natural language processing (NLP) module.

[1325] Step 6:

[1326] The server's NLP module analyzes the text data and identifies health check items (e.g., blood sugar level, cholesterol level, BMI, etc.).

[1327] Step 7:

[1328] The server stores the identified data in a database.

[1329] Step 8:

[1330] The server sends the health checkup data stored in the database to the generation AI, which analyzes the data.

[1331] Step 9:

[1332] The server's generated AI predicts the user's health risks.

[1333] Step 10:

[1334] The server's generative AI generates personalized health advice based on predicted health risks.

[1335] Step 11:

[1336] The server formats and prepares the generated health advice for the user.

[1337] Step 12:

[1338] The server analyzes the medication information entered by the user and manages that information.

[1339] Step 13:

[1340] The server initiates a notification mechanism to notify the user of important information about the medication.

[1341] Step 14:

[1342] Users can set up integration with external exercise, sleep, and nutrition tracking apps within the app.

[1343] Step 15:

[1344] The terminal transmits the data acquired from the external application to the server.

[1345] Step 16:

[1346] The server integrates the health check data with data from external apps and sends it to the generation AI.

[1347] Step 17:

[1348] The server's generative AI generates detailed personalized advice based on the integrated data.

[1349] Step 18:

[1350] The server uses an emotion engine to recognize the user's emotion.

[1351] Step 19:

[1352] The server's emotion engine analyzes the user's facial expressions and tone of voice to identify their emotional state (e.g., stress, joy, anger, sadness).

[1353] Step 20:

[1354] The server sends the emotional data recognized by the emotion engine to the generative AI, which then adjusts the health advice.

[1355] Step 21:

[1356] The device displays the adjusted analysis results and health advice from the server to the user.

[1357] Step 22:

[1358] The device will send users push notifications about important health risks and medication information.

[1359] (Example)

[1360] For example, a user takes a photo of their health checkup results with their smartphone and uploads them to the app. This data is analyzed on the server, and a value such as "blood sugar level: 110 mg / dL" is extracted and stored in a database. The generation AI uses this data to predict that the user has a high risk of diabetes and generates personalized advice such as "You should exercise more to prevent diabetes."

[1361] Furthermore, once the user has linked up with an exercise tracking app, the server analyzes the data and suggests specific exercise amounts. If the emotion engine detects the user's stress level, it will provide advice that takes their emotions into consideration, such as "We recommend relaxation exercises to reduce your stress."

[1362] In this way, users can effectively utilize their own health check data and emotional state to take appropriate measures. Furthermore, by utilizing data from other linked health apps and the emotion engine, more comprehensive and responsive health management becomes possible. This system will help users extend their healthy lifespan and facilitate smooth communication with medical professionals.

[1363] Example 2

[1364] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1365] It is important for users to predict health risks based on their own health checkup data and receive personalized health advice. However, most systems have complex data acquisition and analysis processes, and it is difficult to provide advice that takes into account the user's emotional state. Therefore, it is a challenge to utilize users' health checkup data and realize highly accurate health management based on their emotional state.

[1366] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring past health checkup data, means for analyzing the acquired health checkup data using image analysis means, means for storing the analyzed data in a database, means for predicting health risks using a generative AI model based on the stored data, means for generating personalized health advice based on the predicted health risks, a terminal for notifying the user of the generated advice and prediction results, emotion analysis means for recognizing the user's emotional state, information acquisition means for acquiring and integrating external tracking data, and means for adjusting the content and method of notification based on the emotion analysis means. This makes it possible to effectively utilize the user's health checkup data, provide specific health risk predictions and personalized advice, and provide responsive health management based on the user's emotional state.

[1367] "Past health checkup data" refers to data that indicates the results of health checkups that the user has taken in the past.

[1368] "Image analysis means" refers to technology used to analyze information within an image, and specifically includes optical character recognition (OCR) technology.

[1369] A "database" is a system for storing and managing acquired and analyzed data.

[1370] A "generative AI model" is an artificial intelligence model that performs analysis and predictions based on a user's health checkup data, and specifically includes a type of generative model.

[1371] "Health risk prediction" refers to the act of analyzing a user's health checkup data and predicting future health risks.

[1372] "Personalized health advice" means individualized health advice provided to an individual user based on their health status and risks.

[1373] A "terminal" is an electronic device such as a smartphone or tablet used by a user.

[1374] "Emotion analysis means" refers to technology for recognizing a user's emotional state, and includes data analysis using a camera or microphone.

[1375] "External tracking data" refers to data about exercise, sleep, nutrition, etc. obtained from external tracking devices or applications.

[1376] "Information acquisition means" refers to the technology or protocol used to acquire necessary data from external devices or applications.

[1377] The "means for adjusting the notification content and notification method" refers to a means for optimally adjusting the content and method of the information to be notified based on the emotional state of the user.

[1378] The present invention is a system for improving a user's health management, which integrates the user's health checkup data and external tracking data, and uses a generative AI model to provide health risk prediction and personalized health advice. Furthermore, it uses emotion analysis means to recognize the user's emotional state and adjust the content of advice and notification method based on that information.

[1379] First, the user takes a photo of the past medical checkup results using the smartphone camera. The system receives the data by uploading the image to the device. The device then uploads the image to cloud storage and notifies the server.

[1380] The server retrieves images from cloud storage and extracts text data from the images using optical character recognition (OCR) technology (e.g., Tesseract OCR). The extracted text data is then analyzed using a natural language processing (NLP) module (e.g., SpaCy) to identify health checkup items. The identified health checkup data (e.g., blood glucose level, cholesterol level, BMI) is stored in a database.

[1381] Based on the stored data, the server uses a generative AI model (e.g., OpenAI GPT-3) to predict the user's health risks. For example, based on the data "blood sugar level 110 mg / dL," it predicts "high risk of diabetes" and generates personalized health advice such as "increase exercise by 30 minutes."

[1382] Furthermore, the server recognizes the user's emotional state using emotion analysis methods (e.g., Microsoft Azure Emotion API). It analyzes the user's facial expressions and tone of voice using the device's camera and microphone to identify their emotional state (e.g., stress, joy, anger, sadness). This emotional data is reflected in the generative AI's advice. For example, if the user is feeling stressed, it will provide health advice in a gentler tone.

[1383] Users can set up integration with external exercise, sleep, and nutrition tracking apps. The device sends data obtained from the external apps to a server, and this data is integrated with health checkup data to provide more detailed advice (e.g., "We recommend running for 30 minutes three times a week").

[1384] It also has a medicine notebook function, and the server automatically manages the medicine information entered by the user. The entered medicine information is analyzed, and important information about taking the medicine (e.g., interactions, precautions, etc.) is notified from the server.

[1385] Finally, the device displays the analysis results and health advice from the server to the user, and notifies the user of important information via push notifications. The content and method of notifications are also adjusted according to the emotions recognized by the emotion analysis means.

[1386] For example, the following prompt sentence is input to the generative AI model:

[1387] Generate specific health risk predictions and personalized advice based on the user's health checkup data. For example, if a blood sugar level is 110 mg / dL, predict a high risk of diabetes and provide specific advice on exercise and dietary improvements. Also, take into account the user's emotional state (e.g., high stress).

[1388] This allows users to effectively utilize their own health checkup data, external tracking data linked to their own health checkup data, and their emotional state to optimally manage their health.

[1389] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1390] Step 1:

[1391] The user takes a photo of a past medical checkup result form using the smartphone camera. The system receives the data by uploading the image to the terminal. The input image data is passed to the terminal. This input image data is then converted into text data through subsequent optical character recognition (OCR) processing.

[1392] Step 2:

[1393] The device uploads the captured images to the cloud storage. A cloud storage URL is generated and notified to the server. The server then receives the cloud storage URL as input for retrieving the images.

[1394] Step 3:

[1395] The server retrieves images from cloud storage and extracts text data from the images using optical character recognition (OCR) technology (e.g., Tesseract OCR). Specifically, the server inputs the retrieved image data into the OCR engine and outputs text data. This text data is used to identify and store health checkup items.

[1396] Step 4:

[1397] The server sends the text data extracted by OCR to a natural language processing (NLP) module (e.g., SpaCy) to identify various health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.). The input is text data, and the output is the identified health checkup items. This identified data is stored in a database.

[1398] Step 5:

[1399] The server stores the data of the specified medical examination items in a database, receiving the specified data as input and obtaining the medical examination data in the format stored in the database as output.

[1400] Step 6:

[1401] The server uses a generative AI model (e.g., OpenAI GPT-3) based on the stored health checkup data to predict the user's health risk. The generative AI model receives the health checkup data as input, and outputs a predicted health risk. For example, a specific prediction such as "Your blood sugar level is 110 mg / dL, so you're at high risk for diabetes" is generated.

[1402] Step 7:

[1403] The server generates personalized health advice using a generative AI model based on the predicted health risks. The predicted health risks are input to the generative AI model, and specific health advice is output. For example, advice such as "Increase your exercise by 30 minutes" is generated.

[1404] Step 8:

[1405] The server recognizes the user's emotional state using emotion analysis means (e.g., Microsoft Azure Emotion API). Data acquired from the camera and microphone is sent as input to the emotion analysis engine, and the user's emotional state is obtained as output. For example, a specific emotional state such as "high stress level" is recognized.

[1406] Step 9:

[1407] The server incorporates the recognized emotional data into the health advice provided by the generative AI and adjusts the content of the advice and notification method. Emotional state and health advice are given as input, and emotion-conscious health advice and notifications are generated as output. For example, if the user is feeling stressed, advice will be provided in a calm tone.

[1408] Step 10:

[1409] Users configure the app to connect to external exercise, sleep, and nutrition tracking apps. The device receives data from the external apps and sends it to a server. The server receives data from the external apps as input and provides integrated health data as output.

[1410] Step 11:

[1411] The server then integrates this data with the health checkup data to generate detailed health advice. The integrated data is given as input to a generative AI model, which outputs detailed health advice. For example, specific advice such as "running for 30 minutes three times a week is recommended" is generated.

[1412] Step 12:

[1413] The server automatically manages medication information entered by the user using the medication notebook function. It analyzes the entered medication information and notifies the user of important medication information, such as drug interactions and precautions.

[1414] Step 13:

[1415] The device displays the analysis results and health advice from the server to the user, and generates push notifications and health advice to be sent to the user as output.

[1416] Step 14:

[1417] The device adjusts the content and method of notifications based on the emotions recognized by the emotion analysis method, ultimately providing appropriate health advice to the user based on their emotions.

[1418] (Application example 2)

[1419] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1420] Health management is becoming increasingly important in modern society, but personalized health support based on individual users' health status and emotions is still lacking. In particular, in-store customer service often fails to provide prompt and accurate health advice that takes into account individual health data and current emotions. Therefore, there is a need for a system that can accurately predict users' health risks, generate personalized health advice based on those risks, and provide health support in-store that takes into account the user's emotions.

[1421] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring past health checkup data, means for analyzing the acquired health checkup data using image analysis means, means for saving the analyzed data in a database, means for predicting health risks using a generation AI based on the saved data, means for generating personalized health advice based on the predicted health risks, means for notifying the user of the generated advice and prediction results, information acquisition means for acquiring and integrating external tracking data, means for recognizing the user's emotions and incorporating the emotion data into the analysis results, and means for providing emotion-based health support via an automated answering device in a physical store. This makes it possible to quickly and accurately provide personalized health advice based on the health condition and emotions of individual users even in physical stores.

[1422] "Past health checkup data" refers to data showing the results of health checkups the user has undergone in the past, and specifically includes numerical information for assessing health status, such as blood sugar levels, cholesterol levels, and BMI.

[1423] The "image analysis means" is a technology for extracting text data from acquired image data, specifically using optical character recognition (OCR) technology.

[1424] "Generative AI" is a type of artificial intelligence model used to generate predictions or advice from specific input data. In this case, it refers to an AI model for predicting health risks.

[1425] "Personalized health advice" means advice, including individualized health guidance and recommendations, generated based on a user's individual health data and predicted health risks.

[1426] "External tracking data" refers to health-related data, such as a user's exercise, sleep, and nutrition, obtained from external devices or applications.

[1427] "Means for recognizing emotions" refers to technology that analyzes a user's facial expressions and tone of voice to identify their emotional state at that time.

[1428] An "automatic answering device" is a device installed in a physical store to support users in managing their health, and is responsible for providing appropriate support and advice based on the user's emotional state.

[1429] "Information acquisition means" refers to a means for acquiring information such as tracking data from the outside and integrating it into the system.

[1430] "Means of notification" refers to the means for informing users of the generated advice and prediction results, and specifically includes push notifications on smartphones.

[1431] This invention relates to a personalized health management system for brick-and-mortar stores. This system analyzes past health checkup data, provides individualized health advice to users, and recognizes users' emotions and incorporates their emotional state into the analysis results to provide health support within the brick-and-mortar stores.

[1432] The server first provides a means for users to obtain past health checkup data. Specifically, the user obtains this data by taking a photo of the health checkup result form using the camera on their smartphone. The obtained image data is uploaded to the server and analyzed using image analysis means. Optical character recognition (OCR) technology is used to extract text data from the image.

[1433] The server then analyzes the extracted text data using a natural language processing (NLP) module to identify health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.), and stores the identified data in a database.

[1434] The server then uses the stored data to predict the user's health risks using generative AI. The generative AI model comprehensively evaluates the user's health status and generates personalized health advice. For example, specific advice such as "You should increase exercise to prevent diabetes" or "You should reduce sugar in your diet" is generated.

[1435] The system also includes a means to recognize the user's emotions. The server uses a camera and microphone to analyze the user's facial expressions and tone of voice to identify their current emotional state (e.g., stress, joy, anger, sadness, etc.). This emotional data is incorporated into the generated health advice, and the content and delivery of notifications and advice are tailored accordingly.

[1436] Additionally, external tracking data such as exercise, sleep, and nutrition data from other tracking apps used by the user can be integrated using information acquisition methods, providing a more complete picture of the user's health and enabling more detailed health advice.

[1437] Based on the accumulated data, information about the user's medication is also managed. This allows information about the medications the user is taking and important notifications to be managed by the server and provided to the user. The generated health advice and prediction results are also provided to medical professionals, who can summarize and display the analysis results and provide appropriate health guidance.

[1438] For example, a user takes a photo of their health checkup results with their smartphone at a store counter and uploads it to the system. This data is analyzed on the server, and the value "Blood glucose level: 110 mg / dL" is extracted and saved. Based on this data, the generation AI predicts that the user is at high risk for diabetes and generates personalized advice such as "You should increase your exercise to prevent diabetes." If the AI ​​recognizes the user's emotional state and determines that stress is high, it provides advice that takes their emotions into consideration, such as "We recommend yoga to reduce stress."

[1439] An example of a prompt is as follows:

[1440] "User's health is at high risk. Dietary changes are required."

[1441] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1442] Step 1:

[1443] Users take photos of their past health checkup results with their smartphone camera and upload the image data to the device. At this stage, the input is image data of the health checkup results, and the output is image data sent to the server. The device converts the captured image into an appropriate format and sends it to the server via a stable communication path.

[1444] Step 2:

[1445] The server converts the received image data into text data using optical character recognition (OCR) technology. The input at this stage is the uploaded health check image data, and the output is text data extracted from the image. The server analyzes the image using an OCR engine such as "Tesseract" and extracts the necessary text information.

[1446] Step 3:

[1447] The server analyzes the extracted text data using a natural language processing (NLP) module to identify specific health checkup items (e.g., blood glucose level, cholesterol level, BMI, etc.). The input at this stage is the text data extracted by OCR, and the output is the analyzed health checkup item data. For example, formatted data such as "blood glucose level: 110 mg / dL" is obtained.

[1448] Step 4:

[1449] The server stores the analyzed health check item data in a database. The input at this stage is the data of the identified health check items, and the output is structured data stored in the database. The data will be used for later analysis and prediction by generative AI.

[1450] Step 5:

[1451] Next, the server uses the stored data to predict the user's health risk using a generative AI model. The input at this stage is the health checkup data stored in the database, and the output is the predicted health risk and its details. For example, the assessment may be "high risk of diabetes."

[1452] Step 6:

[1453] The server generates personalized health advice using a generative AI model based on the predicted health risks. The input at this stage is the predicted health risks, and the output is the generated specific health advice. An example of generated advice is "You should exercise more to prevent diabetes."

[1454] Step 7:

[1455] The server analyzes the user's emotional state using emotion recognition means. The user sends data of their facial expressions and tone of voice to the server via a camera or microphone. The input at this stage is data indicating the user's emotional state, and the output is data of the analyzed emotional state. For example, it can recognize that the user is feeling stressed.

[1456] Step 8:

[1457] The server then incorporates the recognized emotional state into the generated health advice. The input at this stage is the health advice and emotional state data, and the output is personalized health advice that takes the emotion into account. An example of generated advice is "We recommend relaxing exercises to reduce stress."

[1458] Step 9:

[1459] The server notifies the user of the generated health advice and prediction results. The device receives the notification from the server and displays it to the user via push notification. The input at this stage is the generated advice and prediction results, and the output is the notification displayed on the device. The notification may include suggested actions for the user or recommendations for device use.

[1460] Step 10:

[1461] The user can then put the advice into practice and provide feedback to the system. At this stage, the input is the user's action data, and the output is the feedback data sent to the system. This allows the system to provide continuous, personalized health management.

[1462] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1463] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1464] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1465] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1466] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1467] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1468] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1469] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1470] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1471] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1472] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1473] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1474] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1475] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1476] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1477] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1478] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1479] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1480] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1481] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1482] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1483] The following is further disclosed regarding the above embodiment.

[1484] (Claim 1)

[1485] a means for obtaining past medical examination data;

[1486] A means for analyzing the acquired health checkup data using an image analysis means;

[1487] a means for storing the analyzed data in a database;

[1488] A means to predict health risks using AI generated from stored data;

[1489] means for generating personalized health advice based on the predicted health risks;

[1490] a means for notifying the user of the generated advice or prediction results;

[1491] A system including an information acquisition means for acquiring and integrating external tracking data.

[1492] (Claim 2)

[1493] 10. The system of claim 1, further comprising means for managing and notifying a user of information regarding his or her medication based on the stored data.

[1494] (Claim 3)

[1495] 10. The system of claim 1, further comprising means for summarizing and displaying the analysis results to provide generated health advice and prediction results to a medical professional.

[1496] "Example 1"

[1497] (Claim 1)

[1498] A means for a user to upload an image of health checkup data taken using a smartphone;

[1499] means for analyzing the uploaded image using image analysis means and extracting text data using optical character recognition technology;

[1500] A means for analyzing the extracted text data using a natural language processing module, identifying health checkup items, and storing the identified items in a database;

[1501] A means for analyzing the stored health checkup data using a generative AI model and predicting the user's health risk;

[1502] means for generating personalized health advice based on the predicted health risks;

[1503] means for notifying the user of the generated advice or prediction results;

[1504] an information acquisition means for acquiring and integrating external tracking data;

[1505] The system includes means for integrating acquired external tracking data with health examination data to generate detailed advice.

[1506] (Claim 2)

[1507] The system of claim 1, further comprising means for managing information about the user's medication and notifying them of reminders and precautions based on the stored data.

[1508] (Claim 3)

[1509] 10. The system of claim 1, further comprising means for summarizing and displaying the analysis results to provide generated health advice and prediction results to a medical professional.

[1510] "Application Example 1"

[1511] Extracting application points

[1512] We analyzed the invention and extracted the following distinctive novel features:

[1513] Personalized product recommendations

[1514] Product selection and notification based on health data

[1515] New invention requirements

[1516] Combine the original invention with the following new features:

[1517] Personalized product recommendation method using health checkup data

[1518] Product selection method linked to medicine notebook

[1519] Health advice notification methods

[1520] (Claim 1)

[1521] a means for obtaining past medical examination data;

[1522] A means for analyzing the acquired health checkup data using an image analysis means;

[1523] a means for storing the analyzed data in a database;

[1524] A means to predict health risks using AI generated from stored data;

[1525] means for generating personalized health advice based on the predicted health risks;

[1526] a means for notifying the user of the generated advice or prediction results;

[1527] an information acquisition means for acquiring and integrating external tracking data;

[1528] a means for recommending personalized products based on the analyzed health data; and

[1529] A system that includes a means of selecting products in conjunction with information from a medicine notebook.

[1530] (Claim 2)

[1531] 10. The system of claim 1, further comprising means for managing and notifying a user of information regarding his or her medication based on the stored data.

[1532] (Claim 3)

[1533] 10. The system of claim 1, further comprising means for summarizing and displaying the analysis results to provide generated health advice and prediction results to a medical professional.

[1534] "Example 2: Combining Emotion Engines"

[1535] (Claim 1)

[1536] a means for obtaining past medical examination data;

[1537] A means for analyzing the acquired health checkup data using an image analysis means;

[1538] a means for storing the analyzed data in a database;

[1539] A means of predicting health risks using a generative AI model based on stored data; and

[1540] means for generating personalized health advice based on the predicted health risks;

[1541] a terminal for notifying the user of the generated advice and prediction results;

[1542] emotion analysis means for recognizing the emotional state of a user;

[1543] an information acquisition means for acquiring and integrating external tracking data;

[1544] means for adjusting notification content and notification method based on the sentiment analysis means;

[1545] A system including:

[1546] (Claim 2)

[1547] 10. The system of claim 1, further comprising means for managing and notifying a user of information regarding his or her medication based on the stored data.

[1548] (Claim 3)

[1549] 10. The system of claim 1, further comprising means for summarizing and displaying the analysis results to provide generated health advice and prediction results to a medical professional.

[1550] "Application example 2 when combining emotion engines"

[1551] (Claim 1)

[1552] a means for obtaining past medical examination data;

[1553] A means for analyzing the acquired health checkup data using an image analysis means;

[1554] a means for storing the analyzed data in a database;

[1555] A means to predict health risks using AI generated from stored data;

[1556] means for generating personalized health advice based on the predicted health risks;

[1557] a means for notifying the user of the generated advice or prediction results;

[1558] an information acquisition means for acquiring and integrating external tracking data;

[1559] means for recognizing a user's emotion and incorporating the emotion data into the analysis results;

[1560] A system including a means for providing emotion-based health support through an automated answering device in a physical store.

[1561] (Claim 2)

[1562] 10. The system of claim 1, further comprising means for managing and notifying a user of information regarding his or her medication based on the stored data.

[1563] (Claim 3)

[1564] 10. The system of claim 1, further comprising means for summarizing and displaying the analysis results to provide generated health advice and prediction results to a medical professional. [Explanation of symbols]

[1565] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for obtaining past medical examination data; A means for analyzing the acquired health checkup data using an image analysis means; a means for storing the analyzed data in a database; A means to predict health risks using AI generated from stored data; means for generating personalized health advice based on the predicted health risks; a means for notifying the user of the generated advice or prediction results; A system including an information acquisition means for acquiring and integrating external tracking data.

2. 10. The system of claim 1, further comprising means for managing and notifying a user of information about medications based on the stored data.

3. 10. The system of claim 1, further comprising means for summarizing and displaying the analysis results to provide generated health advice and predictions to a medical professional.

Images