system

Abstract

We provide the system. [Solution] A method for extracting text from images of health checkup results using image recognition technology, A means of analyzing extracted text data to assess health status, A means of suggesting medical institutions and lifestyle improvement measures based on evaluation results, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There are problems that it is difficult to understand the specialized numerical values and terms included in the health diagnosis results, and it is difficult for the examinee to appropriately grasp their own health status and judge the next actions to be taken. In addition, there is also a problem that attribute information and past health diagnosis data cannot be appropriately utilized, and individually customized medical advice is not presented.

Means for Solving the Problems

[0005] This invention provides a system that extracts text from images of health checkup results using image recognition technology. This system analyzes the extracted text data to evaluate the health status and, considering the user's attribute information and past health checkup data, suggests the most suitable medical institution and lifestyle improvement measures based on the evaluation results. Furthermore, it can identify abnormal values ​​by comparing the analysis results with reference values ​​in an internal database. This allows patients to understand their own health status and easily take appropriate medical measures.

[0006] "Image recognition technology" is a technology that extracts information by detecting and analyzing specific patterns or characters from image data.

[0007] A "health checkup result" is a document containing test data and measurements taken at a medical institution to evaluate the state of one's physical health.

[0008] "Text data" refers to information consisting of letters and numbers extracted from an image, and is textual information stored in digital format.

[0009] "Assessing one's health status" means comprehensively judging an individual's health risks and physical condition based on data obtained from health checkup results.

[0010] A "medical institution" is a facility that provides medical services, such as a hospital or clinic.

[0011] "Lifestyle improvement measures" refer to specific advice on reviewing and changing daily habits such as diet, exercise, and sleep in order to improve one's health.

[0012] "Attribute information" refers to basic information associated with an individual, such as the user's gender, age, and place of residence.

[0013] "Determining abnormal values" means comparing acquired numerical data with reference values, identifying values ​​that exceed that range, and determining their potential health risks. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of the data processing device and smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention provides a system that efficiently analyzes health checkup results and provides appropriate action plans by utilizing smart devices that users use on a daily basis. This system can be used simply by the user taking a picture of their health checkup results with their device and uploading it.

[0036] composition

[0037] User behavior

[0038] Users take photos of their health checkup results using the camera on their smart device. They are required to take images that clearly show the examination items. After taking the photos, users upload the images to the system using a dedicated application or web portal.

[0039] Server operation

[0040] The server processes the received image and extracts text data from it using OCR (Optical Character Recognition) technology. Then, it identifies the numerical values ​​for each item in the health checkup from the extracted data and compares them to pre-set reference values ​​to confirm the normal range of the values.

[0041] Analysis and Proposal

[0042] The extracted data is evaluated by an AI analysis engine on the server. This analysis takes into account user attribute information (e.g., gender, age) and past health checkup data to predict future health risks. If the analysis determines that a specific health condition is abnormal, the user will be recommended to a medical institution where further tests are necessary and given suggestions for precautions to take in daily life.

[0043] Specific example

[0044] For example, consider a case where a 40-year-old male user uploads an image of his regular health checkup results. The server recognizes the text in the image and determines, for example, that his AST value is 60 and his γ-GPT value is 85. Since these values ​​are above the normal range, the analysis engine assesses that there may be an abnormality in his liver function. Based on these results, the server recommends that the user undergo a detailed blood test at a nearby gastroenterology clinic and also provides specific lifestyle advice, such as reducing alcohol consumption and doing at least 30 minutes of aerobic exercise daily.

[0045] This system helps users accurately understand their own health status and take appropriate medical action.

[0046] The following describes the processing flow.

[0047] Step 1:

[0048] The user takes a picture of their health checkup results with a smart device. The image must be clear and all test items must be legible.

[0049] Step 2:

[0050] Users access a dedicated application or web portal and upload images of their health check results to the server. Users ensure they select the correct images and that the upload is smooth.

[0051] Step 3:

[0052] The server preprocesses the received image data, performing noise reduction and image adjustments. This improves image quality and makes subsequent text recognition more accurate.

[0053] Step 4:

[0054] The server uses OCR (Optical Character Recognition) technology to extract text data from images. The server specifically aims to clearly recognize the names and numerical values ​​of health checkup items.

[0055] Step 5:

[0056] The server analyzes the extracted text data and identifies specific health checkup values ​​such as AST, γ-GPT, and LAP. These values ​​are then compared to existing reference values ​​to determine if they fall within the normal range.

[0057] Step 6:

[0058] The server considers user attribute information (gender, age, past data, etc.) and uses an AI analysis model to assess their health status. If an abnormality is detected, it estimates its impact and risks.

[0059] Step 7:

[0060] Based on the analysis results, the server generates a specific action plan for the user. This action plan includes suggestions for necessary additional tests, appropriate medical facilities, and lifestyle advice.

[0061] Step 8:

[0062] The terminal presents the user with the action plan received from the server. The terminal displays the information in a visually clear format that the user can understand immediately.

[0063] Step 9:

[0064] Users review the presented action plan and take action to receive additional medical treatment as needed. They also implement lifestyle improvement advice and manage their health.

[0065] (Example 1)

[0066] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0067] Conventional health monitoring systems have had the challenge of making it difficult for users to easily understand their own health status and quickly decide on appropriate medical treatment. In particular, there is a need for a system that can efficiently manage health checkup results and use that information to help users with lifestyle changes and the selection of medical institutions.

[0068] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0069] In this invention, the server includes means for taking a picture of a health checkup document using a camera and inputting the image; means for extracting character data from the image of the health checkup document using optical character recognition technology; means for analyzing the extracted character data and determining the health status by comparing it with evaluation criteria; and means for presenting recommendations from medical institutions and measures for improving lifestyle habits based on the analysis results. This enables the user to accurately understand their own health status and to quickly take necessary medical action or improve their lifestyle habits.

[0070] A "photography device" is a device used to record health examination documents as images.

[0071] Optical Character Recognition (OCR) is a technology that extracts text information from images.

[0072] "Text data" refers to data in string format obtained based on information extracted from an image.

[0073] An "analysis engine" is a program or algorithm used to evaluate a user's health status based on extracted text data.

[0074] "Evaluation criteria" refer to pre-set standard values ​​or conditions used when assessing a person's health status.

[0075] "Medical institution recommendation" refers to the act of suggesting an appropriate medical institution based on the user's health condition.

[0076] "Lifestyle improvement measures" refer to suggestions for changing behaviors and habits that are offered to users in order to improve their health.

[0077] This invention proposes a health management system that users can access on a daily basis using smart devices. The user begins by taking a picture of their health checkup results using the camera on their smart device. This device records the health checkup document as a high-resolution image. The captured image is then uploaded to a server via a dedicated application or web portal.

[0078] The server applies optical character recognition (OCR) technology to the uploaded images to extract text data. This process uses common OCR software such as Tesseract OCR. The server then determines the health status by comparing the extracted text data with evaluation criteria. At this stage, the server efficiently processes the data using the Python Pandas library and employs AI technologies such as TENSORFLOW® as its analysis engine.

[0079] Once the analysis is complete, the server evaluates the results and, taking into account the user's attribute information and past health records, provides recommendations for appropriate medical institutions and lifestyle improvements. For example, for a 40-year-old male user whose AST and γ-GPT levels are above the normal range, the server will recommend additional tests at a nearby medical facility and provide specific suggestions for improving daily habits.

[0080] As a concrete example, the prompt message is as follows: "A 40-year-old male user uploaded an image of his health checkup results. His AST level was detected as 60 and his γ-GPT level as 85. Based on these results, what health risks does he have? Also, what lifestyle changes would you recommend?"

[0081] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0082] Step 1:

[0083] The user takes a picture of their health checkup results using the camera on their smart device. The input is the printed health checkup results, and the output is a high-resolution image file. The user should take the picture so that the examination items are clearly legible and to avoid glare and focus issues.

[0084] Step 2:

[0085] Users upload images they have taken to the server using a dedicated application or web portal. The input is the captured image stored on the user's device, and the output is the image data stored on the server. At this stage, the server receives and stores the image data.

[0086] Step 3:

[0087] The server applies Optical Character Recognition (OCR) technology to uploaded images. The input is image data stored on the server, and the output is text data of health checkup results. The server accurately extracts characters and numbers using software such as Tesseract OCR. During extraction, pre-processing is performed to correct image tilt and remove noise.

[0088] Step 4:

[0089] The server compares the extracted text data with evaluation criteria to determine the user's health status. The input is text data obtained by OCR, and the output is the health status evaluation result. The server uses the Python Pandas library to analyze the data and compare the numerical values ​​of each item with the criteria.

[0090] Step 5:

[0091] The server uses an AI analysis engine to assess health risks, taking into account the user's attribute information and past health records. The input is a numerical value of the health status based on evaluation criteria, and the output is the assessment result of future health risks. The server uses models such as TensorFlow to predict risks and determine specific health conditions.

[0092] Step 6:

[0093] The server compiles the analysis results and presents recommendations to the user. The input is the health risk assessment results from AI analysis, and the output is recommendations for medical institutions and lifestyle improvement measures. The server notifies the user of medical institutions as needed and specific things to be aware of in daily life.

[0094] (Application Example 1)

[0095] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0096] In modern society, personal health management is a crucial issue, but many people find it difficult to accurately understand their own health status and implement appropriate lifestyle improvements based on that understanding. Furthermore, the lack of a system that offers benefits based on health status results in a lack of motivation for health improvement. To address this issue, there is a need for efficient analysis of health checkup information and the provision of motivational support based on that analysis.

[0097] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0098] In this invention, the server includes means for extracting textual information from visual information of test results using image recognition technology, means for analyzing the extracted textual information to evaluate the health status, and means for applying benefits when purchasing health-related products based on the evaluation results. This makes it easier for individuals to understand their own health status and take concrete actions toward improving their health. Furthermore, by applying benefits, it is possible to raise users' health awareness and provide them with an incentive to more actively work toward improving their lifestyle habits.

[0099] "Image recognition technology" is a technology for identifying objects and characteristics from digital images or videos and extracting related information.

[0100] "Visual information" refers to information expressed in the form of images or videos, and is used as visual data.

[0101] "Textual information" refers to data in text format extracted from visual data and used for other information processing purposes.

[0102] "Means for evaluating health status" refers to methods for diagnosing an individual's current health status using extracted textual information and making quantitative or qualitative judgments.

[0103] "Means of applying benefits" refers to the methods used to apply benefits and rewards offered to users when they purchase health-related products, based on the evaluation results of their health checkup.

[0104] A "server" is a remote computer system that stores data and performs computational processing, and is a device that provides information through communication with other devices.

[0105] The embodiment for carrying out this invention is based on the following system configuration. The user first takes a picture of their health checkup results using a smart device they use on a daily basis. This smart device can be a smartphone or a tablet. The user uploads the image to the system through a dedicated application.

[0106] The server uses image recognition technology to extract text information from the received image data. This process utilizes OCR technology such as Google Cloud Vision API. The extracted text information is processed on the server and input into an AI analysis engine to evaluate the user's health status. The AI ​​analysis engine, for example, uses TensorFlow or PyTorch to comprehensively evaluate the user's health status while considering the user's attribute information and past health checkup data.

[0107] The evaluation results are displayed on the user's smart device. This allows users to understand their health status and receive guidance on medical consultations or lifestyle improvements as needed. Furthermore, based on the health check evaluation results, users can receive benefits when purchasing health-related products. These benefits are applied via electronic payment platforms, utilizing services such as Stripe and PayPal.

[0108] For example, a user takes a picture of their health checkup results with the app and uploads it. The server uses OCR to extract the text information, and after evaluation by an analysis engine, it determines that the user has high triglyceride levels. Based on these results, the user is offered supplements to improve their triglyceride levels, and a discount is applied when they purchase the products.

[0109] Examples of prompt statements to input into the generative AI model are as follows:

[0110] "Design an AI model to recommend the most effective health products based on data extracted from health checkup results. This should include the conditions under which benefits and discounts apply."

[0111] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0112] Step 1:

[0113] The user takes a picture of their health checkup results using a smart device. They use a camera app to take a clear picture so that the results are clearly legible. The user then uploads this image to the system using a dedicated application. The input is the image of the health checkup results, and the output is the image data uploaded to the server.

[0114] Step 2:

[0115] The server performs image recognition processing based on the received image data. Specifically, it extracts text data from the image using OCR (Optical Character Recognition) technology. This is done using tools such as the Google Cloud Vision API. The input is image data of health checkup results, and the output is the extracted text information.

[0116] Step 3:

[0117] The server supplies the character information extracted by OCR to the AI ​​analysis engine. This AI analysis engine uses TensorFlow and PyTorch to analyze the character data and evaluate the user's health status. The input is character information, and the output is the health status evaluation result. The analysis also takes into account the user's attribute information and past health checkup data, and abnormal values ​​are also detected.

[0118] Step 4:

[0119] The server processes data based on the health assessment results to recommend the most suitable health-related products to the user. This includes using a generative AI model to suggest products. The input is the health assessment results, and the output is a list of recommended products. The suggested products incorporate elements that support the user's health improvement.

[0120] Step 5:

[0121] The user makes a purchase decision from the presented product list. Payment is made through an electronic payment system, and benefits are applied. This process is implemented using Stripe or PayPal. The input is the product selection result, and the output is the payment completion and benefit application status.

[0122] Step 6:

[0123] The server sends the user a notification confirming the completion of payment and benefit application. This notification includes details of the received benefits and the next steps to take to improve health. The input is the result of payment and benefit application, and the output is the content of the notification sent to the user.

[0124] This process allows users to take effective improvement measures based on their health checkup results, while also raising their health awareness through the benefits they gain.

[0125] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0126] This invention provides a system that enables more personalized health management and behavioral suggestions by considering not only the user's health check results but also their emotional state. In this system, the user takes a picture of their health check results using a smart device, inputs or collects their emotional state along with the image, and provides it to the system.

[0127] composition

[0128] User input and data collection

[0129] Users take pictures of their health checkup results with a smart device and upload the images to a dedicated application. They can also input their current emotional state, or emotional data can be automatically collected using sensors built into the device.

[0130] Server-side processing of image and emotion data

[0131] The server uses OCR technology to convert uploaded images into text and analyzes the extracted data against baseline values. Simultaneously, it uses an emotion engine to analyze the user's emotional data and evaluate the user's psychological state.

[0132] Integrated analysis of health and emotional state

[0133] The server integrates analyzed health and emotional data to comprehensively assess the user's health status. It then considers the impact of emotional state on health behaviors and selects the most effective medical facilities and improvement measures for the user.

[0134] Generation and presentation of action plans

[0135] Based on integrated analysis, the server generates a specific action plan for the user. This action plan includes suggestions for medical institutions appropriate to the user's psychological state and methods for improving lifestyle habits while considering their emotions. The terminal presents this plan to the user, providing information in a visually appealing and easy-to-understand format.

[0136] Specific example

[0137] For example, consider a case where a female user in her 50s self-reports feeling "anxious" along with her health checkup results. The server identifies high total cholesterol levels from the checkup results and assesses a high stress level from emotional data. Based on these results, the server suggests visiting an internist in collaboration with a psychosomatic medicine specialist and recommends practicing yoga or mindfulness to promote emotional stability. In this way, comprehensive health management becomes possible, following not only physical health but also emotional health.

[0138] The following describes the processing flow.

[0139] Step 1:

[0140] The user uses a smart device to take an image of their health checkup results. The image should be taken in high resolution and include all the information in the results.

[0141] Step 2:

[0142] Users log in to a dedicated app or web portal and upload the images they have taken. At the same time, users can input their emotional state, or the device's emotion sensor will automatically collect data.

[0143] Step 3:

[0144] The server preprocesses the received images and extracts text data from them using OCR technology. This text data includes inspection items and numerical information.

[0145] Step 4:

[0146] The server analyzes the text data obtained by OCR to identify health checkup values ​​such as AST, γ-GPT, and LAP, and compares them to reference values ​​to determine if any values ​​are abnormal.

[0147] Step 5:

[0148] The server uses an emotion engine to analyze the user's emotional data. Based on the input emotional state and collected sensor data, it evaluates the user's emotional state.

[0149] Step 6:

[0150] The server integrates health checkup analysis results and emotional data to assess the user's overall health status. It considers the impact of emotions on health and selects appropriate medical facilities.

[0151] Step 7:

[0152] Based on integrated analysis, the server generates an action plan for the user. This includes specific referrals to medical institutions tailored to the user's health and emotional state, as well as lifestyle improvement suggestions that take their emotions into consideration.

[0153] Step 8:

[0154] The terminal displays the action plan received from the server to the user. The plan is displayed on the terminal in a visually clear and easy-to-understand format.

[0155] (Example 2)

[0156] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0157] In modern society, comprehensively managing individual health and emotional states and providing individually optimized action plans is a challenging task. Conventional systems rely solely on health checkup results for evaluation and fail to provide comprehensive health management that considers emotional states, making it difficult to offer effective suggestions for maintaining individual health.

[0158] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0159] In this invention, the server includes means for acquiring text information from images of health checkup results using image processing technology, means for analyzing the acquired text information and evaluating the health and emotional state, and means for generating an optimal action plan based on the evaluation results. This makes it possible to comprehensively evaluate the user's health and emotional state and provide an optimal action plan for the individual.

[0160] "Image processing technology" refers to techniques for extracting or analyzing useful information from digital images, and includes methods and algorithms for manipulating, transforming, and analyzing images.

[0161] "Text information" refers to character data extracted from media such as images, and is information in a format that can be understood and processed by a computer.

[0162] "Health status" refers to a collection of data and indicators related to the user's physical health, including blood pressure, cholesterol levels, and other physiological measurements.

[0163] "Emotional state" is a collection of data and indicators that show the user's current psychological health and emotional tendencies, including stress levels and mood.

[0164] An "action plan" refers to specific behavioral guidelines recommended based on the user's health and emotional assessment results, including suggestions for medical consultations and methods for improving lifestyle habits.

[0165] This invention relates to a system that comprehensively evaluates a user's health check results and emotional state, and provides individually optimized health management. This system consists of multiple elements, each working in conjunction with the others.

[0166] Users use smart devices to capture images of their health checkup results and upload them to a server via a dedicated application. The smart devices are equipped with cameras and image recognition software, which check the quality of the captured images in real time and supply images of appropriate quality to the system.

[0167] The server uses image processing technology to extract text information from images of health checkup results and performs analysis based on this information. The server employs a high-performance OCR (Optical Character Recognition) engine to accurately extract text data from the captured images. The extracted data is compared against health standard values ​​to detect abnormalities. Furthermore, data for recognizing emotions is collected and processed by the analysis engine to evaluate the user's emotional state. This emotional data is acquired through sensors and self-input.

[0168] During the integrated analysis process, the server utilizes a generated AI model to create an optimal action plan for the user based on their health and emotional state. This plan includes, if necessary, recommendations for medical consultations and methods for stabilizing emotions.

[0169] The terminal presents the generated action plan to the user, providing information in a visually easy-to-understand format. The application on the terminal displays information using graphics and infographics, helping the user easily understand and implement the proposed plan.

[0170] As a specific example, if a female user in her 50s takes a picture of her health checkup results and enters "anxiety" as her emotional state, the server analyzes the blood test results and determines that she has high cholesterol levels and a high stress level. Based on these results, the server suggests visiting an appropriate medical institution and recommends practicing yoga or mindfulness to maintain peace of mind.

[0171] An example of a prompt for a generative AI model is: "Use user data to create a comprehensive health plan that takes into account health and emotional states. Include specific suggestions such as selecting a healthcare provider and lifestyle improvements."

[0172] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0173] Step 1:

[0174] Users take images of their health checkup results with a smart device and upload them to a dedicated application. They also input their emotional state or have emotional data automatically collected through the device's sensors. At this stage, the image and emotional data are provided to the system as input. The output becomes the basis for analysis performed by the server.

[0175] Step 2:

[0176] The server receives the uploaded image and extracts text information from it using OCR technology. Specifically, the server activates the OCR engine, recognizes the characters within the image, and outputs the result as digital text. At this stage, the input is image data, and the output is extracted text information.

[0177] Step 3:

[0178] The server performs data analysis to assess health status based on extracted text information. The input is text information obtained by OCR, and the server determines abnormal indicators by comparing it with reference data. In this process, it performs calculations and comparisons of numerical data and outputs analysis results indicating health risks.

[0179] Step 4:

[0180] The server uses an emotion engine to analyze emotional data acquired from the user. Specifically, it processes the input emotional information using a machine learning model to quantify stress levels and emotional tendencies. The input is emotional data, and the output is the analysis result indicating the user's emotional state.

[0181] Step 5:

[0182] The server integrates health and emotional data and utilizes a generative AI model to generate an optimal action plan for the user. The input consists of assessments of health and emotional states. Based on this, the server uses prompts to generate and output an action plan. This action plan includes specific suggestions for medical institutions and methods for improving daily life.

[0183] Step 6:

[0184] The terminal presents the action plan received from the server to the user. Specifically, it displays the information in a visually easy-to-understand format (graphs and charts) on the application, making it easier for the user to recognize the content of the proposed plan and put it into action. The input is the action plan from the server, and the output is the user's understanding and commencement of action.

[0185] (Application Example 2)

[0186] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0187] In modern society, user health management is becoming increasingly important. However, conventional systems evaluate users' health status based solely on physical diagnostic results, lacking appropriate behavioral suggestions tailored to their emotional state and individual lifestyles. As a result, truly personalized health management that users need is not being achieved. To solve this problem, it is necessary to comprehensively evaluate both the user's physical and emotional state and propose optimal actions.

[0188] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0189] In this invention, the server includes means for extracting text from images of health checkup results using image recognition technology, means for analyzing the extracted text data to evaluate the health status, means for processing the user's emotional state data using emotion analysis technology to evaluate the psychological state, means for integrating the health status evaluation results and the psychological state evaluation results to generate personalized action suggestions, and means for analyzing the purchase history based on the action suggestions to recommend products for improving lifestyle habits. This makes it possible to provide comprehensive and personalized action suggestions that take into account the user's physical and emotional health.

[0190] "Image recognition technology" is a technology that analyzes photographs and videos to identify objects and characters contained within them and converts them into a data format that a computer can understand.

[0191] "Analyzing text data" is the process of interpreting the target information based on extracted textual information and evaluating its meaning and trends.

[0192] "Emotion analysis technology" is a technology that automatically evaluates and identifies a user's emotional state based on their input and behavioral data.

[0193] "Assessing psychological state" means determining the user's mental health and stress level at a given time based on their emotional information.

[0194] "Personalized action suggestions" refer to the process of generating optimal improvement measures and recommendations for a specific user, taking into account their health and emotional state.

[0195] "Analyzing purchase history" is the process of analyzing a user's past purchase history to derive behavioral patterns and preferences.

[0196] "Recommending a product" means presenting products that are considered beneficial to the user based on their health status, emotional state, and past purchase history.

[0197] To realize this invention, a system including a smart device and a server is required. Users use a smartphone or other smart device to take images of their health checkup results and upload them to a dedicated application. The device includes image recognition technology and a function to send images to the server.

[0198] The server extracts text data from images using the Google Cloud Vision API or similar image recognition technologies. Next, it analyzes the extracted text data to assess the user's health status. Furthermore, it analyzes emotional state data, either entered by the user or automatically collected by the device, using sentiment analysis technologies (e.g., IBM Watson®'s NLP capabilities). This analysis assesses the user's psychological state.

[0199] The server integrates health and emotional data to generate personalized behavioral suggestions. These suggestions provide users with appropriate health improvement measures and healthcare options. It also analyzes purchase history using data analytics tools such as Amazon Web Services (AWS®) to recommend products for a healthy lifestyle.

[0200] As a concrete example, if a user self-reports feeling "tired," the server might identify that the user has recently purchased a large amount of high-calorie food and suggest caffeinated beverages and stress-relieving products to address both mental and physical needs.

[0201] Examples of prompt statements that can be used are as follows:

[0202] "The user uploaded their health checkup results to the app and entered their current emotional state as 'tired.' The system should suggest products to add to their shopping list to help relieve stress."

[0203] This allows users to receive comprehensive health management based on their physical and emotional state, and to develop concrete action plans.

[0204] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0205] Step 1:

[0206] The user takes an image of their health checkup results using a smart device and uploads it to a dedicated application. The input is the image of the health checkup results taken by the user with their device, and the output is the image data. The terminal sends this image data to the server.

[0207] Step 2:

[0208] The server converts the received image data into text data using the Google Cloud Vision API. The input is image data of health checkup results, and the output is text data obtained using OCR technology. At this stage, characters in the image are recognized and converted into a format that can be processed by a computer.

[0209] Step 3:

[0210] The server analyzes the extracted text data and evaluates the user's health status. The input is text data obtained through OCR processing, and the output is health status evaluation information obtained through analysis. This analysis applies an evaluation algorithm that includes comparison with reference values.

[0211] Step 4:

[0212] The user inputs their emotional state into the application, or emotional data is automatically collected from sensors equipped on the device. The input is data about the user's emotional state, and the output is that emotional data. The device sends this data to the server.

[0213] Step 5:

[0214] The server analyzes emotional data and evaluates the user's psychological state using IBM Watson's NLP capabilities and similar sentiment analysis techniques. The input is emotional data provided by the user, and the output is the emotional evaluation result. In this step, the meaning of the input data is interpreted and the emotional state is quantified.

[0215] Step 6:

[0216] The server integrates health status and emotional assessment results to generate personalized action suggestions. The inputs are health assessment information and emotional assessment information, and the output is the generated action suggestions. This integrated analysis proposes the most suitable health improvement measures for the user.

[0217] Step 7:

[0218] Based on the suggested actions, the server analyzes the user's purchase history using AWS data analysis tools and recommends products to improve lifestyle habits. The input is the user's purchase history data, and the output is a list of recommended products. This process selects and presents healthy products based on past purchase patterns.

[0219] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0220] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0221] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0222] [Second Embodiment]

[0223] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0224] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0225] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0226] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0227] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0228] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0229] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0230] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0231] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0232] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0233] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0234] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0235] This invention provides a system that efficiently analyzes health checkup results and provides appropriate action plans by utilizing smart devices that users use on a daily basis. This system can be used simply by the user taking a picture of their health checkup results with their device and uploading it.

[0236] composition

[0237] User behavior

[0238] Users take photos of their health checkup results using the camera on their smart device. They are required to take images that clearly show the examination items. After taking the photos, users upload the images to the system using a dedicated application or web portal.

[0239] Server operation

[0240] The server processes the received image and extracts text data from it using OCR (Optical Character Recognition) technology. Then, it identifies the numerical values ​​for each item in the health checkup from the extracted data and compares them to pre-set reference values ​​to confirm the normal range of the values.

[0241] Analysis and Proposal

[0242] The extracted data is evaluated by an AI analysis engine on the server. This analysis takes into account user attribute information (e.g., gender, age) and past health checkup data to predict future health risks. If the analysis determines that a specific health condition is abnormal, the user will be recommended to a medical institution where further tests are necessary and will be given suggestions on things to be mindful of in daily life.

[0243] Specific example

[0244] For example, consider a case where a 40-year-old male user uploads an image of his regular health checkup results. The server recognizes the text in the image and determines, for example, that his AST value is 60 and his γ-GPT value is 85. Since these values ​​are above the normal range, the analysis engine assesses that there may be an abnormality in his liver function. Based on these results, the server recommends that the user undergo a detailed blood test at a nearby gastroenterology clinic and also provides specific lifestyle advice, such as reducing alcohol consumption and doing at least 30 minutes of aerobic exercise daily.

[0245] This system helps users accurately understand their own health status and take appropriate medical action.

[0246] The following describes the processing flow.

[0247] Step 1:

[0248] The user takes a picture of their health checkup results with a smart device. The image must be clear and all test items must be legible.

[0249] Step 2:

[0250] Users access a dedicated application or web portal and upload images of their health check results to the server. Users ensure they select the correct images and that the upload is smooth.

[0251] Step 3:

[0252] The server preprocesses the received image data, performing noise reduction and image adjustments. This improves image quality and makes subsequent text recognition more accurate.

[0253] Step 4:

[0254] The server uses OCR (Optical Character Recognition) technology to extract text data from images. The server specifically aims to clearly recognize the names and numerical values ​​of health checkup items.

[0255] Step 5:

[0256] The server analyzes the extracted text data and identifies specific health checkup values ​​such as AST, γ-GPT, and LAP. These values ​​are then compared to existing reference values ​​to determine if they fall within the normal range.

[0257] Step 6:

[0258] The server considers user attribute information (gender, age, past data, etc.) and uses an AI analysis model to assess their health status. If an abnormality is detected, it estimates its impact and risks.

[0259] Step 7:

[0260] Based on the analysis results, the server generates a specific action plan for the user. This action plan includes suggestions for necessary additional tests, appropriate medical facilities, and lifestyle advice.

[0261] Step 8:

[0262] The terminal presents the user with the action plan received from the server. The terminal displays the information in a visually clear format that the user can understand immediately.

[0263] Step 9:

[0264] Users review the presented action plan and take action to receive additional medical treatment as needed. They also implement lifestyle improvement advice and manage their health.

[0265] (Example 1)

[0266] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0267] Conventional health monitoring systems have had the challenge of making it difficult for users to easily understand their own health status and quickly decide on appropriate medical treatment. In particular, there is a need for a system that can efficiently manage health checkup results and use that information to help users with lifestyle changes and the selection of medical institutions.

[0268] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0269] In this invention, the server includes means for taking a picture of a health checkup document using a camera and inputting the image; means for extracting character data from the image of the health checkup document using optical character recognition technology; means for analyzing the extracted character data and determining the health status by comparing it with evaluation criteria; and means for presenting recommendations from medical institutions and measures for improving lifestyle habits based on the analysis results. This enables the user to accurately understand their own health status and to quickly take necessary medical action or improve their lifestyle habits.

[0270] A "photography device" is a device used to record health examination documents as images.

[0271] Optical Character Recognition (OCR) is a technology that extracts text information from images.

[0272] "Text data" refers to data in string format obtained based on information extracted from an image.

[0273] An "analysis engine" is a program or algorithm used to evaluate a user's health status based on extracted text data.

[0274] "Evaluation criteria" refer to pre-set standard values ​​or conditions used when assessing a person's health status.

[0275] "Medical institution recommendation" refers to the act of suggesting an appropriate medical institution based on the user's health condition.

[0276] "Lifestyle improvement measures" refer to suggestions for changing behaviors and habits that are offered to users in order to improve their health.

[0277] This invention proposes a health management system that users can access on a daily basis using smart devices. The user begins by taking a picture of their health checkup results using the camera on their smart device. This device records the health checkup document as a high-resolution image. The captured image is then uploaded to a server via a dedicated application or web portal.

[0278] The server applies optical character recognition (OCR) technology to uploaded images to extract text data. This process uses common OCR software such as Tesseract OCR. The server then determines the health status by comparing the extracted text data with evaluation criteria. At this stage, the server efficiently processes the data using the Python Pandas library and employs AI technologies such as TensorFlow for its analysis engine.

[0279] Once the analysis is complete, the server evaluates the results and, taking into account the user's attribute information and past health records, provides recommendations for appropriate medical institutions and lifestyle improvements. For example, for a 40-year-old male user whose AST and γ-GPT levels are above the normal range, the server will recommend additional tests at a nearby medical facility and provide specific suggestions for improving daily habits.

[0280] As a concrete example, the prompt message is as follows: "A 40-year-old male user uploaded an image of his health checkup results. His AST level was detected as 60 and his γ-GPT level as 85. Based on these results, what health risks does he have? Also, what lifestyle changes would you recommend?"

[0281] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0282] Step 1:

[0283] The user takes a photo of the health check results using the camera of the smart device. The input is the health check results on paper, and the output is a high-resolution image file. At this time, the user takes the photo so that the inspection items can be clearly read and pays attention to avoid light reflection and out-of-focus.

[0284] Step 2:

[0285] The user uploads the taken photo to the server using a dedicated application or web portal. The input is the taken photo saved on the user device, and the output is the image data stored on the server. At this stage, the server receives and stores the image data.

[0286] Step 3:

[0287] The server applies optical character recognition (OCR) technology to the uploaded image. The input is the image data stored on the server, and the output is the text data of the health check results. The server accurately extracts characters and numerical values using software such as Tesseract OCR. During extraction, preprocessing is performed to correct the tilt of the image and remove noise.

[0288] Step 4:

[0289] The server compares the extracted text data with the evaluation criteria to determine the user's health status. The input is the text data obtained by OCR, and the output is the evaluation result of the health status. The server performs data analysis using the Pandas library in Python and compares the numerical values of each item with the criteria.

[0290] Step 5:

[0291] The server uses an AI analysis engine to assess health risks, taking into account the user's attribute information and past health records. The input is a numerical value of the health status based on evaluation criteria, and the output is the assessment result of future health risks. The server uses models such as TensorFlow to predict risks and determine specific health conditions.

[0292] Step 6:

[0293] The server compiles the analysis results and presents recommendations to the user. The input is the health risk assessment results from AI analysis, and the output is recommendations for medical institutions and lifestyle improvement measures. The server notifies the user of medical institutions as needed and specific things to be aware of in daily life.

[0294] (Application Example 1)

[0295] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0296] In modern society, personal health management is a crucial issue, but many people find it difficult to accurately understand their own health status and implement appropriate lifestyle improvements based on that understanding. Furthermore, the lack of a system that offers benefits based on health status results in a lack of motivation for health improvement. To address this issue, there is a need for efficient analysis of health checkup information and the provision of motivational support based on that analysis.

[0297] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0298] In this invention, the server includes means for extracting textual information from visual information of test results using image recognition technology, means for analyzing the extracted textual information to evaluate the health status, and means for applying benefits when purchasing health-related products based on the evaluation results. This makes it easier for individuals to understand their own health status and take concrete actions toward improving their health. Furthermore, by applying benefits, it is possible to raise users' health awareness and provide them with an incentive to more actively work toward improving their lifestyle habits.

[0299] "Image recognition technology" is a technology for identifying objects and characteristics from digital images or videos and extracting related information.

[0300] "Visual information" refers to information expressed in the form of images or videos, and is used as visual data.

[0301] "Textual information" refers to data in text format extracted from visual data and used for other information processing purposes.

[0302] "Means for evaluating health status" refers to methods for diagnosing an individual's current health status using extracted textual information and making quantitative or qualitative judgments.

[0303] "Means of applying benefits" refers to the methods used to apply benefits and rewards offered to users when they purchase health-related products, based on the evaluation results of their health checkup.

[0304] A "server" is a remote computer system that stores data and performs computational processing, and is a device that provides information through communication with other devices.

[0305] The embodiments for implementing this invention are based on the following system configurations. The user first takes a photo of the health check result using a smart device that they use daily. As this smart device, a smartphone or a tablet is used. The user uploads the image to the system through a dedicated application.

[0306] The server uses image recognition technology to extract character information from the received image data. For this process, OCR technology such as Google Cloud Vision API is used, for example. The extracted character information is processed on the server and input into an AI analysis engine to evaluate the health status. As the AI analysis engine, for example, TensorFlow or PyTorch is used to comprehensively evaluate the health status while considering the user's attribute information and past health check data.

[0307] The result of the evaluation is presented to the user's smart device. As a result, the user can grasp their own health status and receive medical institution visit guidance or obtain improvement measures for their lifestyle habits as needed. Also, based on the evaluation result of the health check, the user can receive benefits when purchasing health-related products. The benefits are applied through an electronic payment platform, and services such as Stripe or PayPal are used.

[0308] For example, a certain user takes a photo of the health check result with the app and uploads it. The server extracts character information using OCR and, as a result of evaluation by the analysis engine, it is determined that the triglyceride level is high. Based on this result, supplements for improving triglycerides are proposed to the user, and a discount benefit is applied when purchasing the product.

[0309] Examples of the prompt sentences input into the generative AI model are as follows.

[0310] "Please design an AI model for recommending the most effective health products based on the data extracted from the health check result. Also include the conditions for applying benefits and discounts."

[0311] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0312] Step 1:

[0313] The user takes a picture of their health checkup results using a smart device. They use a camera app to take a clear picture so that the results are clearly legible. The user then uploads this image to the system using a dedicated application. The input is the image of the health checkup results, and the output is the image data uploaded to the server.

[0314] Step 2:

[0315] The server performs image recognition processing based on the received image data. Specifically, it extracts text data from the image using OCR (Optical Character Recognition) technology. This is done using tools such as the Google Cloud Vision API. The input is image data of health checkup results, and the output is the extracted text information.

[0316] Step 3:

[0317] The server supplies the character information extracted by OCR to the AI ​​analysis engine. This AI analysis engine uses TensorFlow and PyTorch to analyze the character data and evaluate the user's health status. The input is character information, and the output is the health status evaluation result. The analysis also takes into account the user's attribute information and past health checkup data, and abnormal values ​​are also detected.

[0318] Step 4:

[0319] The server processes data based on the health assessment results to recommend the most suitable health-related products to the user. This includes using a generative AI model to suggest products. The input is the health assessment results, and the output is a list of recommended products. The suggested products incorporate elements that support the user's health improvement.

[0320] Step 5:

[0321] The user makes a purchase decision from the presented product list. Payment is made through an electronic payment system, and benefits are applied. This process is implemented using Stripe or PayPal. The input is the product selection result, and the output is the payment completion and benefit application status.

[0322] Step 6:

[0323] The server sends the user a notification confirming the completion of payment and benefit application. This notification includes details of the received benefits and the next steps to take to improve health. The input is the result of payment and benefit application, and the output is the content of the notification sent to the user.

[0324] This process allows users to take effective improvement measures based on their health checkup results, while also raising their health awareness through the benefits they gain.

[0325] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0326] This invention provides a system that enables more personalized health management and behavioral suggestions by considering not only the user's health check results but also their emotional state. In this system, the user takes a picture of their health check results using a smart device, inputs or collects their emotional state along with the image, and provides it to the system.

[0327] composition

[0328] User input and data collection

[0329] Users take pictures of their health checkup results with a smart device and upload the images to a dedicated application. They can also input their current emotional state, or emotional data can be automatically collected using sensors built into the device.

[0330] Server-side processing of image and emotion data

[0331] The server uses OCR technology to convert uploaded images into text and analyzes the extracted data against baseline values. Simultaneously, it uses an emotion engine to analyze the user's emotional data and evaluate the user's psychological state.

[0332] Integrated analysis of health and emotional state

[0333] The server integrates analyzed health and emotional data to comprehensively assess the user's health status. It then considers the impact of emotional state on health behaviors and selects the most effective medical facilities and improvement measures for the user.

[0334] Generation and presentation of action plans

[0335] Based on integrated analysis, the server generates a specific action plan for the user. This action plan includes suggestions for medical institutions appropriate to the user's psychological state and methods for improving lifestyle habits while considering their emotions. The terminal presents this plan to the user, providing information in a visually appealing and easy-to-understand format.

[0336] Specific example

[0337] For example, consider a case where a female user in her 50s self-reports feeling "anxious" along with her health checkup results. The server identifies high total cholesterol levels from the checkup results and assesses a high stress level from emotional data. Based on these results, the server suggests visiting an internist in collaboration with a psychosomatic medicine specialist and recommends practicing yoga or mindfulness to promote emotional stability. In this way, comprehensive health management becomes possible, following not only physical health but also emotional health.

[0338] The following describes the processing flow.

[0339] Step 1:

[0340] The user uses a smart device to take an image of their health checkup results. The image should be taken in high resolution and include all the information in the results.

[0341] Step 2:

[0342] Users log in to a dedicated app or web portal and upload the images they have taken. At the same time, users can input their emotional state, or the device's emotion sensor will automatically collect data.

[0343] Step 3:

[0344] The server preprocesses the received images and extracts text data from them using OCR technology. This text data includes inspection items and numerical information.

[0345] Step 4:

[0346] The server analyzes the text data obtained by OCR to identify health checkup values ​​such as AST, γ-GPT, and LAP, and compares them to reference values ​​to determine if any values ​​are abnormal.

[0347] Step 5:

[0348] The server uses an emotion engine to analyze the user's emotional data. Based on the input emotional state and collected sensor data, it evaluates the user's emotional state.

[0349] Step 6:

[0350] The server integrates health checkup analysis results and emotional data to assess the user's overall health status. It considers the impact of emotions on health and selects appropriate medical facilities.

[0351] Step 7:

[0352] Based on integrated analysis, the server generates an action plan for the user. This includes specific referrals to medical institutions tailored to the user's health and emotional state, as well as lifestyle improvement suggestions that take their emotions into consideration.

[0353] Step 8:

[0354] The terminal displays the action plan received from the server to the user. The plan is displayed on the terminal in a visually clear and easy-to-understand format.

[0355] (Example 2)

[0356] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0357] In modern society, comprehensively managing individual health and emotional states and providing individually optimized action plans is a challenging task. Conventional systems rely solely on health checkup results for evaluation and fail to provide comprehensive health management that considers emotional states, making it difficult to offer effective suggestions for maintaining individual health.

[0358] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0359] In this invention, the server includes means for acquiring text information from images of health checkup results using image processing technology, means for analyzing the acquired text information and evaluating the health and emotional state, and means for generating an optimal action plan based on the evaluation results. This makes it possible to comprehensively evaluate the user's health and emotional state and provide an optimal action plan for the individual.

[0360] "Image processing technology" refers to techniques for extracting or analyzing useful information from digital images, and includes methods and algorithms for manipulating, transforming, and analyzing images.

[0361] "Text information" refers to character data extracted from media such as images, and is information in a format that can be understood and processed by a computer.

[0362] "Health status" refers to a collection of data and indicators related to the user's physical health, including blood pressure, cholesterol levels, and other physiological measurements.

[0363] "Emotional state" is a collection of data and indicators that show the user's current psychological health and emotional tendencies, including stress levels and mood.

[0364] An "action plan" refers to specific behavioral guidelines recommended based on the user's health and emotional assessment results, including suggestions for medical consultations and methods for improving lifestyle habits.

[0365] This invention relates to a system that comprehensively evaluates a user's health check results and emotional state, and provides individually optimized health management. This system consists of multiple elements, each working in conjunction with the others.

[0366] Users use smart devices to capture images of their health checkup results and upload them to a server via a dedicated application. The smart devices are equipped with cameras and image recognition software, which check the quality of the captured images in real time and supply images of appropriate quality to the system.

[0367] The server uses image processing technology to extract text information from images of health checkup results and performs analysis based on this information. The server employs a high-performance OCR (Optical Character Recognition) engine to accurately extract text data from the captured images. The extracted data is compared against health standard values ​​to detect abnormalities. Furthermore, data for recognizing emotions is collected and processed by the analysis engine to evaluate the user's emotional state. This emotional data is acquired through sensors and self-input.

[0368] During the integrated analysis process, the server utilizes a generated AI model to create an optimal action plan for the user based on their health and emotional state. This plan includes, if necessary, recommendations for medical consultations and methods for stabilizing emotions.

[0369] The terminal presents the generated action plan to the user, providing information in a visually easy-to-understand format. The application on the terminal displays information using graphics and infographics, helping the user easily understand and implement the proposed plan.

[0370] As a specific example, if a female user in her 50s takes a picture of her health checkup results and enters "anxiety" as her emotional state, the server analyzes the blood test results and determines that she has high cholesterol levels and a high stress level. Based on these results, the server suggests visiting an appropriate medical institution and recommends practicing yoga or mindfulness to maintain peace of mind.

[0371] An example of a prompt for a generative AI model is: "Use user data to create a comprehensive health plan that takes into account health and emotional states. Include specific suggestions such as selecting a healthcare provider and lifestyle improvements."

[0372] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0373] Step 1:

[0374] Users take images of their health checkup results with a smart device and upload them to a dedicated application. They also input their emotional state or have emotional data automatically collected through the device's sensors. At this stage, the image and emotional data are provided to the system as input. The output becomes the basis for analysis performed by the server.

[0375] Step 2:

[0376] The server receives the uploaded image and extracts text information from it using OCR technology. Specifically, the server activates the OCR engine, recognizes the characters within the image, and outputs the result as digital text. At this stage, the input is image data, and the output is extracted text information.

[0377] Step 3:

[0378] The server performs data analysis to assess health status based on extracted text information. The input is text information obtained by OCR, and the server determines abnormal indicators by comparing it with reference data. In this process, it performs calculations and comparisons of numerical data and outputs analysis results indicating health risks.

[0379] Step 4:

[0380] The server uses an emotion engine to analyze emotional data acquired from the user. Specifically, it processes the input emotional information using a machine learning model to quantify stress levels and emotional tendencies. The input is emotional data, and the output is the analysis result indicating the user's emotional state.

[0381] Step 5:

[0382] The server integrates health and emotional data and utilizes a generative AI model to generate an optimal action plan for the user. The input consists of assessments of health and emotional states. Based on this, the server uses prompts to generate and output an action plan. This action plan includes specific suggestions for medical institutions and methods for improving daily life.

[0383] Step 6:

[0384] The terminal presents the action plan received from the server to the user. Specifically, it displays the information in a visually easy-to-understand format (graphs and charts) on the application, making it easier for the user to recognize the content of the proposed plan and put it into action. The input is the action plan from the server, and the output is the user's understanding and commencement of action.

[0385] (Application Example 2)

[0386] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0387] In modern society, user health management is becoming increasingly important. However, conventional systems evaluate users' health status based solely on physical diagnostic results, lacking appropriate behavioral suggestions tailored to their emotional state and individual lifestyles. As a result, truly personalized health management that users need is not being achieved. To solve this problem, it is necessary to comprehensively evaluate both the user's physical and emotional state and propose optimal actions.

[0388] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0389] In this invention, the server includes means for extracting text from images of health checkup results using image recognition technology, means for analyzing the extracted text data to evaluate the health status, means for processing the user's emotional state data using emotion analysis technology to evaluate the psychological state, means for integrating the health status evaluation results and the psychological state evaluation results to generate personalized action suggestions, and means for analyzing the purchase history based on the action suggestions to recommend products for improving lifestyle habits. This makes it possible to provide comprehensive and personalized action suggestions that take into account the user's physical and emotional health.

[0390] "Image recognition technology" is a technology that analyzes photographs and videos to identify objects and characters contained within them and converts them into a data format that a computer can understand.

[0391] "Analyzing text data" is the process of interpreting the target information based on extracted textual information and evaluating its meaning and trends.

[0392] "Emotion analysis technology" is a technology that automatically evaluates and identifies a user's emotional state based on their input and behavioral data.

[0393] "Assessing psychological state" means determining the user's mental health and stress level at a given time based on their emotional information.

[0394] "Personalized action suggestions" refer to the process of generating optimal improvement measures and recommendations for a specific user, taking into account their health and emotional state.

[0395] "Analyzing purchase history" is the process of analyzing a user's past purchase history to derive behavioral patterns and preferences.

[0396] "Recommending a product" means presenting products that are considered beneficial to the user based on their health status, emotional state, and past purchase history.

[0397] To realize this invention, a system including a smart device and a server is required. Users use a smartphone or other smart device to take images of their health checkup results and upload them to a dedicated application. The device includes image recognition technology and a function to send images to the server.

[0398] The server extracts text data from images using the Google Cloud Vision API or similar image recognition technologies. Next, it analyzes the extracted text data to assess the user's health status. Furthermore, it analyzes emotional state data, either entered by the user or automatically collected by the device, using sentiment analysis technologies (e.g., IBM Watson's NLP capabilities). This analysis assesses the user's psychological state.

[0399] The server integrates health and emotional data to generate personalized behavioral suggestions. These suggestions provide users with appropriate health improvement measures and healthcare options. It also analyzes purchase history using data analytics tools such as Amazon Web Services (AWS) to recommend products for a healthy lifestyle.

[0400] As a concrete example, if a user self-reports feeling "tired," the server might identify that the user has recently purchased a large amount of high-calorie food and suggest caffeinated beverages and stress-relieving products to address both mental and physical needs.

[0401] Examples of prompt statements that can be used are as follows:

[0402] "The user uploaded their health checkup results to the app and entered their current emotional state as 'tired.' The system should suggest products to add to their shopping list to help relieve stress."

[0403] This allows users to receive comprehensive health management based on their physical and emotional state, and to develop concrete action plans.

[0404] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0405] Step 1:

[0406] The user takes an image of their health checkup results using a smart device and uploads it to a dedicated application. The input is the image of the health checkup results taken by the user with their device, and the output is the image data. The terminal sends this image data to the server.

[0407] Step 2:

[0408] The server converts the received image data into text data using the Google Cloud Vision API. The input is image data of health checkup results, and the output is text data obtained using OCR technology. At this stage, characters in the image are recognized and converted into a format that can be processed by a computer.

[0409] Step 3:

[0410] The server analyzes the extracted text data and evaluates the user's health status. The input is text data obtained through OCR processing, and the output is health status evaluation information obtained through analysis. This analysis applies an evaluation algorithm that includes comparison with reference values.

[0411] Step 4:

[0412] The user inputs their emotional state into the application, or emotional data is automatically collected from sensors equipped on the device. The input is data about the user's emotional state, and the output is that emotional data. The device sends this data to the server.

[0413] Step 5:

[0414] The server analyzes emotional data and evaluates the user's psychological state using IBM Watson's NLP capabilities and similar sentiment analysis techniques. The input is emotional data provided by the user, and the output is the emotional evaluation result. In this step, the meaning of the input data is interpreted and the emotional state is quantified.

[0415] Step 6:

[0416] The server integrates health status and emotional assessment results to generate personalized action suggestions. The inputs are health assessment information and emotional assessment information, and the output is the generated action suggestions. This integrated analysis proposes the most suitable health improvement measures for the user.

[0417] Step 7:

[0418] Based on the suggested actions, the server analyzes the user's purchase history using AWS data analysis tools and recommends products to improve lifestyle habits. The input is the user's purchase history data, and the output is a list of recommended products. This process selects and presents healthy products based on past purchase patterns.

[0419] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0420] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0421] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0422] [Third Embodiment]

[0423] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0424] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0425] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0426] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0427] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0428] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0429] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0430] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0431] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0432] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0433] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0434] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0435] This invention provides a system that efficiently analyzes health checkup results and provides appropriate action plans by utilizing smart devices that users use on a daily basis. This system can be used simply by the user taking a picture of their health checkup results with their device and uploading it.

[0436] composition

[0437] User behavior

[0438] Users take photos of their health checkup results using the camera on their smart device. They are required to take images that clearly show the examination items. After taking the photos, users upload the images to the system using a dedicated application or web portal.

[0439] Server operation

[0440] The server processes the received image and extracts text data from it using OCR (Optical Character Recognition) technology. Then, it identifies the numerical values ​​for each item in the health checkup from the extracted data and compares them to pre-set reference values ​​to confirm the normal range of the values.

[0441] Analysis and Proposal

[0442] The extracted data is evaluated by an AI analysis engine on the server. This analysis takes into account user attribute information (e.g., gender, age) and past health checkup data to predict future health risks. If the analysis determines that a specific health condition is abnormal, the user will be recommended to a medical institution where further tests are necessary and will be given suggestions on things to be mindful of in daily life.

[0443] Specific example

[0444] For example, consider a case where a 40-year-old male user uploads an image of his regular health checkup results. The server recognizes the text in the image and determines, for example, that his AST value is 60 and his γ-GPT value is 85. Since these values ​​are above the normal range, the analysis engine assesses that there may be an abnormality in his liver function. Based on these results, the server recommends that the user undergo a detailed blood test at a nearby gastroenterology clinic and also provides specific lifestyle advice, such as reducing alcohol consumption and doing at least 30 minutes of aerobic exercise daily.

[0445] This system helps users accurately understand their own health status and take appropriate medical action.

[0446] The following describes the processing flow.

[0447] Step 1:

[0448] The user takes a picture of their health checkup results with a smart device. The image must be clear and all test items must be legible.

[0449] Step 2:

[0450] Users access a dedicated application or web portal and upload images of their health check results to the server. Users ensure they select the correct images and that the upload is smooth.

[0451] Step 3:

[0452] The server preprocesses the received image data, performing noise reduction and image adjustments. This improves image quality and makes subsequent text recognition more accurate.

[0453] Step 4:

[0454] The server uses OCR (Optical Character Recognition) technology to extract text data from images. The server specifically aims to clearly recognize the names and numerical values ​​of health checkup items.

[0455] Step 5:

[0456] The server analyzes the extracted text data and identifies specific health checkup values ​​such as AST, γ-GPT, and LAP. These values ​​are then compared to existing reference values ​​to determine if they fall within the normal range.

[0457] Step 6:

[0458] The server considers user attribute information (gender, age, past data, etc.) and uses an AI analysis model to assess their health status. If an abnormality is detected, it estimates its impact and risks.

[0459] Step 7:

[0460] Based on the analysis results, the server generates a specific action plan for the user. This action plan includes suggestions for necessary additional tests, appropriate medical facilities, and lifestyle advice.

[0461] Step 8:

[0462] The terminal presents the user with the action plan received from the server. The terminal displays the information in a visually clear format that the user can understand immediately.

[0463] Step 9:

[0464] Users review the presented action plan and take action to receive additional medical treatment as needed. They also implement lifestyle improvement advice and manage their health.

[0465] (Example 1)

[0466] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0467] Conventional health monitoring systems have had the challenge of making it difficult for users to easily understand their own health status and quickly decide on appropriate medical treatment. In particular, there is a need for a system that can efficiently manage health checkup results and use that information to help users with lifestyle changes and the selection of medical institutions.

[0468] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0469] In this invention, the server includes means for taking a picture of a health checkup document using a camera and inputting the image; means for extracting character data from the image of the health checkup document using optical character recognition technology; means for analyzing the extracted character data and determining the health status by comparing it with evaluation criteria; and means for presenting recommendations from medical institutions and measures for improving lifestyle habits based on the analysis results. This enables the user to accurately understand their own health status and to quickly take necessary medical action or improve their lifestyle habits.

[0470] A "photography device" is a device used to record health examination documents as images.

[0471] Optical Character Recognition (OCR) is a technology that extracts text information from images.

[0472] "Text data" refers to data in string format obtained based on information extracted from an image.

[0473] An "analysis engine" is a program or algorithm used to evaluate a user's health status based on extracted text data.

[0474] "Evaluation criteria" refer to pre-set standard values ​​or conditions used when assessing a person's health status.

[0475] "Medical institution recommendation" refers to the act of suggesting an appropriate medical institution based on the user's health condition.

[0476] "Lifestyle improvement measures" refer to suggestions for changing behaviors and habits that are offered to users in order to improve their health.

[0477] This invention proposes a health management system that users can access on a daily basis using smart devices. The user begins by taking a picture of their health checkup results using the camera on their smart device. This device records the health checkup document as a high-resolution image. The captured image is then uploaded to a server via a dedicated application or web portal.

[0478] The server applies optical character recognition (OCR) technology to uploaded images to extract text data. This process uses common OCR software such as Tesseract OCR. The server then determines the health status by comparing the extracted text data with evaluation criteria. At this stage, the server efficiently processes the data using the Python Pandas library and employs AI technologies such as TensorFlow for its analysis engine.

[0479] Once the analysis is complete, the server evaluates the results and, taking into account the user's attribute information and past health records, provides recommendations for appropriate medical institutions and lifestyle improvements. For example, for a 40-year-old male user whose AST and γ-GPT levels are above the normal range, the server will recommend additional tests at a nearby medical facility and provide specific suggestions for improving daily habits.

[0480] As a concrete example, the prompt message is as follows: "A 40-year-old male user uploaded an image of his health checkup results. His AST level was detected as 60 and his γ-GPT level as 85. Based on these results, what health risks does he have? Also, what lifestyle changes would you recommend?"

[0481] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0482] Step 1:

[0483] The user takes a picture of their health checkup results using the camera on their smart device. The input is the printed health checkup results, and the output is a high-resolution image file. The user is required to take the picture so that the examination items are clearly legible and to avoid glare and focus issues.

[0484] Step 2:

[0485] Users upload images they have taken to the server using a dedicated application or web portal. The input is the captured image stored on the user's device, and the output is the image data stored on the server. At this stage, the server receives and stores the image data.

[0486] Step 3:

[0487] The server applies Optical Character Recognition (OCR) technology to uploaded images. The input is image data stored on the server, and the output is text data of health checkup results. The server accurately extracts characters and numbers using software such as Tesseract OCR. During extraction, pre-processing is performed to correct image tilt and remove noise.

[0488] Step 4:

[0489] The server compares the extracted text data with evaluation criteria to determine the user's health status. The input is text data obtained by OCR, and the output is the health status evaluation result. The server uses the Python Pandas library to analyze the data and compare the numerical values ​​of each item with the criteria.

[0490] Step 5:

[0491] The server uses an AI analysis engine to assess health risks, taking into account the user's attribute information and past health records. The input is a numerical value of the health status based on evaluation criteria, and the output is the assessment result of future health risks. The server uses models such as TensorFlow to predict risks and determine specific health conditions.

[0492] Step 6:

[0493] The server compiles the analysis results and presents recommendations to the user. The input is the health risk assessment results from AI analysis, and the output is recommendations for medical institutions and lifestyle improvement measures. The server notifies the user of medical institutions as needed and specific things to be aware of in daily life.

[0494] (Application Example 1)

[0495] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0496] In modern society, personal health management is a crucial issue, but many people find it difficult to accurately understand their own health status and implement appropriate lifestyle improvements based on that understanding. Furthermore, the lack of a system that offers benefits based on health status results in a lack of motivation for health improvement. To address this issue, there is a need for efficient analysis of health checkup information and the provision of motivational support based on that analysis.

[0497] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0498] In this invention, the server includes means for extracting textual information from visual information of test results using image recognition technology, means for analyzing the extracted textual information to evaluate the health status, and means for applying benefits when purchasing health-related products based on the evaluation results. This makes it easier for individuals to understand their own health status and take concrete actions toward improving their health. Furthermore, by applying benefits, it is possible to raise users' health awareness and provide them with an incentive to more actively work toward improving their lifestyle habits.

[0499] "Image recognition technology" is a technology for identifying objects and characteristics from digital images or videos and extracting related information.

[0500] "Visual information" refers to information expressed in the form of images or videos, and is used as visual data.

[0501] "Textual information" refers to data in text format extracted from visual data and used for other information processing purposes.

[0502] "Means for evaluating health status" refers to methods for diagnosing an individual's current health status using extracted textual information and making quantitative or qualitative judgments.

[0503] "Means of applying benefits" refers to the methods used to apply benefits and rewards offered to users when they purchase health-related products, based on the evaluation results of their health checkup.

[0504] A "server" is a remote computer system that stores data and performs computational processing, and is a device that provides information through communication with other devices.

[0505] The embodiment for carrying out this invention is based on the following system configuration. The user first takes a picture of their health checkup results using a smart device they use on a daily basis. This smart device can be a smartphone or a tablet. The user uploads the image to the system through a dedicated application.

[0506] The server uses image recognition technology to extract text information from the received image data. This process utilizes OCR technology such as the Google Cloud Vision API. The extracted text information is processed on the server and input into an AI analysis engine to evaluate the user's health status. The AI ​​analysis engine, for example, uses TensorFlow or PyTorch, and comprehensively evaluates the user's health status while considering the user's attribute information and past health checkup data.

[0507] The evaluation results are displayed on the user's smart device. This allows users to understand their health status and receive guidance on medical consultations or lifestyle improvements as needed. Furthermore, based on the health check evaluation results, users can receive benefits when purchasing health-related products. These benefits are applied via electronic payment platforms, utilizing services such as Stripe and PayPal.

[0508] For example, a user takes a picture of their health checkup results with the app and uploads it. The server uses OCR to extract the text information, and after evaluation by an analysis engine, it determines that the user has high triglyceride levels. Based on these results, the user is offered supplements to improve their triglyceride levels, and a discount is applied when they purchase the products.

[0509] Examples of prompt statements to input into the generative AI model are as follows:

[0510] "Design an AI model to recommend the most effective health products based on data extracted from health checkup results. This should include the conditions under which benefits and discounts apply."

[0511] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0512] Step 1:

[0513] The user takes a picture of their health checkup results using a smart device. They use a camera app to take a clear picture so that the results are clearly legible. The user then uploads this image to the system using a dedicated application. The input is the image of the health checkup results, and the output is the image data uploaded to the server.

[0514] Step 2:

[0515] The server performs image recognition processing based on the received image data. Specifically, it extracts text data from the image using OCR (Optical Character Recognition) technology. This is done using tools such as the Google Cloud Vision API. The input is image data of health checkup results, and the output is the extracted text information.

[0516] Step 3:

[0517] The server supplies the character information extracted by OCR to the AI ​​analysis engine. This AI analysis engine uses TensorFlow and PyTorch to analyze the character data and evaluate the user's health status. The input is character information, and the output is the health status evaluation result. The analysis also takes into account the user's attribute information and past health checkup data, and abnormal values ​​are also detected.

[0518] Step 4:

[0519] The server processes data based on the health assessment results to recommend the most suitable health-related products to the user. This includes using a generative AI model to suggest products. The input is the health assessment results, and the output is a list of recommended products. The suggested products incorporate elements that support the user's health improvement.

[0520] Step 5:

[0521] The user makes a purchase decision from the presented product list. Payment is made through an electronic payment system, and benefits are applied. This process is implemented using Stripe or PayPal. The input is the product selection result, and the output is the payment completion and benefit application status.

[0522] Step 6:

[0523] The server sends the user a notification confirming the completion of payment and benefit application. This notification includes details of the received benefits and the next steps to take to improve health. The input is the result of payment and benefit application, and the output is the content of the notification sent to the user.

[0524] This process allows users to take effective improvement measures based on their health checkup results, while also raising their health awareness through the benefits they gain.

[0525] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0526] This invention provides a system that enables more personalized health management and behavioral suggestions by considering not only the user's health check results but also their emotional state. In this system, the user takes a picture of their health check results using a smart device, inputs or collects their emotional state along with the image, and provides it to the system.

[0527] composition

[0528] User input and data collection

[0529] Users take pictures of their health checkup results with a smart device and upload the images to a dedicated application. They can also input their current emotional state, or emotional data can be automatically collected using sensors built into the device.

[0530] Server-side processing of image and emotion data

[0531] The server uses OCR technology to convert uploaded images into text and analyzes the extracted data against baseline values. Simultaneously, it uses an emotion engine to analyze the user's emotional data and evaluate the user's psychological state.

[0532] Integrated analysis of health and emotional state

[0533] The server integrates analyzed health and emotional data to comprehensively assess the user's health status. It then considers the impact of emotional state on health behaviors and selects the most effective medical facilities and improvement measures for the user.

[0534] Generation and presentation of action plans

[0535] Based on integrated analysis, the server generates a specific action plan for the user. This action plan includes suggestions for medical institutions appropriate to the user's psychological state and methods for improving lifestyle habits while considering their emotions. The terminal presents this plan to the user, providing information in a visually appealing and easy-to-understand format.

[0536] Specific example

[0537] For example, consider a case where a female user in her 50s self-reports feeling "anxious" along with her health checkup results. The server identifies high total cholesterol levels from the checkup results and assesses a high stress level from emotional data. Based on these results, the server suggests visiting an internist in collaboration with a psychosomatic medicine specialist and recommends practicing yoga or mindfulness to promote emotional stability. In this way, comprehensive health management becomes possible, following not only physical health but also emotional health.

[0538] The following describes the processing flow.

[0539] Step 1:

[0540] The user uses a smart device to take an image of their health checkup results. The image should be taken in high resolution and include all the information in the results.

[0541] Step 2:

[0542] Users log in to a dedicated app or web portal and upload the images they have taken. At the same time, users can input their emotional state, or the device's emotion sensor will automatically collect data.

[0543] Step 3:

[0544] The server preprocesses the received images and extracts text data from them using OCR technology. This text data includes inspection items and numerical information.

[0545] Step 4:

[0546] The server analyzes the text data obtained by OCR to identify health checkup values ​​such as AST, γ-GPT, and LAP, and compares them to reference values ​​to determine if any values ​​are abnormal.

[0547] Step 5:

[0548] The server uses an emotion engine to analyze the user's emotional data. Based on the input emotional state and collected sensor data, it evaluates the user's emotional state.

[0549] Step 6:

[0550] The server integrates health checkup analysis results and emotional data to assess the user's overall health status. It considers the impact of emotions on health and selects appropriate medical facilities.

[0551] Step 7:

[0552] Based on integrated analysis, the server generates an action plan for the user. This includes specific referrals to medical institutions tailored to the user's health and emotional state, as well as lifestyle improvement suggestions that take their emotions into consideration.

[0553] Step 8:

[0554] The terminal displays the action plan received from the server to the user. The plan is displayed on the terminal in a visually clear and easy-to-understand format.

[0555] (Example 2)

[0556] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0557] In modern society, comprehensively managing individual health and emotional states and providing individually optimized action plans is a challenging task. Conventional systems rely solely on health checkup results for evaluation and fail to provide comprehensive health management that considers emotional states, making it difficult to offer effective suggestions for maintaining individual health.

[0558] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0559] In this invention, the server includes means for acquiring text information from images of health checkup results using image processing technology, means for analyzing the acquired text information and evaluating the health and emotional state, and means for generating an optimal action plan based on the evaluation results. This makes it possible to comprehensively evaluate the user's health and emotional state and provide an optimal action plan for the individual.

[0560] "Image processing technology" refers to techniques for extracting or analyzing useful information from digital images, and includes methods and algorithms for manipulating, transforming, and analyzing images.

[0561] "Text information" refers to character data extracted from media such as images, and is information in a format that can be understood and processed by a computer.

[0562] "Health status" refers to a collection of data and indicators related to the user's physical health, including blood pressure, cholesterol levels, and other physiological measurements.

[0563] "Emotional state" is a collection of data and indicators that show the user's current psychological health and emotional tendencies, including stress levels and mood.

[0564] An "action plan" refers to specific behavioral guidelines recommended based on the user's health and emotional assessment results, including suggestions for medical consultations and methods for improving lifestyle habits.

[0565] This invention relates to a system that comprehensively evaluates a user's health check results and emotional state, and provides individually optimized health management. This system consists of multiple elements, each working in conjunction with the others.

[0566] Users use smart devices to capture images of their health checkup results and upload them to a server via a dedicated application. The smart devices are equipped with cameras and image recognition software, which check the quality of the captured images in real time and supply images of appropriate quality to the system.

[0567] The server uses image processing technology to extract text information from images of health checkup results and performs analysis based on this information. The server employs a high-performance OCR (Optical Character Recognition) engine to accurately extract text data from the captured images. The extracted data is compared against health standard values ​​to detect abnormalities. Furthermore, data for recognizing emotions is collected and processed by the analysis engine to evaluate the user's emotional state. This emotional data is acquired through sensors and self-input.

[0568] During the integrated analysis process, the server utilizes a generated AI model to create an optimal action plan for the user based on their health and emotional state. This plan includes, if necessary, recommendations for medical consultations and methods for stabilizing emotions.

[0569] The terminal presents the generated action plan to the user, providing information in a visually easy-to-understand format. The application on the terminal displays information using graphics and infographics, helping the user easily understand and implement the proposed plan.

[0570] As a specific example, if a female user in her 50s takes a picture of her health checkup results and enters "anxiety" as her emotional state, the server analyzes the blood test results and determines that she has high cholesterol levels and a high stress level. Based on these results, the server suggests visiting an appropriate medical institution and recommends practicing yoga or mindfulness to maintain peace of mind.

[0571] An example of a prompt for a generative AI model is: "Use user data to create a comprehensive health plan that takes into account health and emotional states. Include specific suggestions such as selecting a healthcare provider and lifestyle improvements."

[0572] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0573] Step 1:

[0574] Users take images of their health checkup results with a smart device and upload them to a dedicated application. They also input their emotional state or have emotional data automatically collected through the device's sensors. At this stage, the image and emotional data are provided to the system as input. The output becomes the basis for analysis performed by the server.

[0575] Step 2:

[0576] The server receives the uploaded image and extracts text information from it using OCR technology. Specifically, the server activates the OCR engine, recognizes the characters within the image, and outputs the result as digital text. At this stage, the input is image data, and the output is extracted text information.

[0577] Step 3:

[0578] The server performs data analysis to assess health status based on extracted text information. The input is text information obtained by OCR, and the server determines abnormal indicators by comparing it with reference data. In this process, it performs calculations and comparisons of numerical data and outputs analysis results indicating health risks.

[0579] Step 4:

[0580] The server uses an emotion engine to analyze emotional data acquired from the user. Specifically, it processes the input emotional information using a machine learning model to quantify stress levels and emotional tendencies. The input is emotional data, and the output is the analysis result indicating the user's emotional state.

[0581] Step 5:

[0582] The server integrates health and emotional data and utilizes a generative AI model to generate an optimal action plan for the user. The input consists of assessments of health and emotional states. Based on this, the server uses prompts to generate and output an action plan. This action plan includes specific suggestions for medical institutions and methods for improving daily life.

[0583] Step 6:

[0584] The terminal presents the action plan received from the server to the user. Specifically, it displays the information in a visually easy-to-understand format (graphs and charts) on the application, making it easier for the user to recognize the content of the proposed plan and put it into action. The input is the action plan from the server, and the output is the user's understanding and commencement of action.

[0585] (Application Example 2)

[0586] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0587] In modern society, user health management is becoming increasingly important. However, conventional systems evaluate users' health status based solely on physical diagnostic results, lacking appropriate behavioral suggestions tailored to their emotional state and individual lifestyles. As a result, truly personalized health management that users need is not being achieved. To solve this problem, it is necessary to comprehensively evaluate both the user's physical and emotional state and propose optimal actions.

[0588] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0589] In this invention, the server includes means for extracting text from images of health checkup results using image recognition technology, means for analyzing the extracted text data to evaluate the health status, means for processing the user's emotional state data using emotion analysis technology to evaluate the psychological state, means for integrating the health status evaluation results and the psychological state evaluation results to generate personalized action suggestions, and means for analyzing the purchase history based on the action suggestions to recommend products for improving lifestyle habits. This makes it possible to provide comprehensive and personalized action suggestions that take into account the user's physical and emotional health.

[0590] "Image recognition technology" is a technology that analyzes photographs and videos to identify objects and characters contained within them and converts them into a data format that a computer can understand.

[0591] "Analyzing text data" is the process of interpreting the target information based on extracted textual information and evaluating its meaning and trends.

[0592] "Emotion analysis technology" is a technology that automatically evaluates and identifies a user's emotional state based on their input and behavioral data.

[0593] "Assessing psychological state" means determining the user's mental health and stress level at a given time based on their emotional information.

[0594] "Personalized action suggestions" refer to the process of generating optimal improvement measures and recommendations for a specific user, taking into account their health and emotional state.

[0595] "Analyzing purchase history" is the process of analyzing a user's past purchase history to derive behavioral patterns and preferences.

[0596] "Recommending a product" means presenting products that are considered beneficial to the user based on their health status, emotional state, and past purchase history.

[0597] To realize this invention, a system including a smart device and a server is required. Users use a smartphone or other smart device to take images of their health checkup results and upload them to a dedicated application. The device includes image recognition technology and a function to send images to the server.

[0598] The server extracts text data from images using the Google Cloud Vision API or similar image recognition technologies. Next, it analyzes the extracted text data to assess the user's health status. Furthermore, it analyzes emotional state data, either entered by the user or automatically collected by the device, using sentiment analysis technologies (e.g., IBM Watson's NLP capabilities). This analysis assesses the user's psychological state.

[0599] The server integrates health and emotional data to generate personalized behavioral suggestions. These suggestions provide users with appropriate health improvement measures and healthcare options. It also analyzes purchase history using data analytics tools such as Amazon Web Services (AWS) to recommend products for a healthy lifestyle.

[0600] As a concrete example, if a user self-reports feeling "tired," the server might identify that the user has recently purchased a large amount of high-calorie food and suggest caffeinated beverages and stress-relieving products to address both mental and physical needs.

[0601] Examples of prompt statements that can be used are as follows:

[0602] "The user uploaded their health checkup results to the app and entered their current emotional state as 'tired.' The system should suggest products to add to their shopping list to help relieve stress."

[0603] This allows users to receive comprehensive health management based on their physical and emotional state, and to develop concrete action plans.

[0604] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0605] Step 1:

[0606] The user takes an image of their health checkup results using a smart device and uploads it to a dedicated application. The input is the image of the health checkup results taken by the user with their device, and the output is the image data. The terminal sends this image data to the server.

[0607] Step 2:

[0608] The server converts the received image data into text data using the Google Cloud Vision API. The input is image data of health checkup results, and the output is text data obtained using OCR technology. At this stage, characters in the image are recognized and converted into a format that can be processed by a computer.

[0609] Step 3:

[0610] The server analyzes the extracted text data and evaluates the user's health status. The input is text data obtained through OCR processing, and the output is health status evaluation information obtained through analysis. This analysis applies an evaluation algorithm that includes comparison with reference values.

[0611] Step 4:

[0612] The user inputs their emotional state into the application, or emotional data is automatically collected from sensors equipped on the device. The input is data about the user's emotional state, and the output is that emotional data. The device sends this data to the server.

[0613] Step 5:

[0614] The server analyzes emotional data and evaluates the user's psychological state using IBM Watson's NLP capabilities and similar sentiment analysis techniques. The input is emotional data provided by the user, and the output is the emotional evaluation result. In this step, the meaning of the input data is interpreted and the emotional state is quantified.

[0615] Step 6:

[0616] The server integrates health status and emotional assessment results to generate personalized action suggestions. The inputs are health assessment information and emotional assessment information, and the output is the generated action suggestions. This integrated analysis proposes the most suitable health improvement measures for the user.

[0617] Step 7:

[0618] Based on the suggested actions, the server analyzes the user's purchase history using AWS data analysis tools and recommends products to improve lifestyle habits. The input is the user's purchase history data, and the output is a list of recommended products. This process selects and presents healthy products based on past purchase patterns.

[0619] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0620] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0621] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0622] [Fourth Embodiment]

[0623] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0624] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0625] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0626] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0627] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0628] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0629] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0630] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0631] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0632] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0633] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0634] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0635] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0636] This invention provides a system that efficiently analyzes health checkup results and provides appropriate action plans by utilizing smart devices that users use on a daily basis. This system can be used simply by the user taking a picture of their health checkup results with their device and uploading it.

[0637] composition

[0638] User behavior

[0639] Users take photos of their health checkup results using the camera on their smart device. They are required to take images that clearly show the examination items. After taking the photos, users upload the images to the system using a dedicated application or web portal.

[0640] Server operation

[0641] The server processes the received image and extracts text data from it using OCR (Optical Character Recognition) technology. Then, it identifies the numerical values ​​for each item in the health checkup from the extracted data and compares them to pre-set reference values ​​to confirm the normal range of the values.

[0642] Analysis and Proposal

[0643] The extracted data is evaluated by an AI analysis engine on the server. This analysis takes into account user attribute information (e.g., gender, age) and past health checkup data to predict future health risks. If the analysis determines that a specific health condition is abnormal, the user will be recommended to a medical institution where further tests are necessary and will be given suggestions on things to be mindful of in daily life.

[0644] Specific example

[0645] For example, consider a case where a 40-year-old male user uploads an image of his regular health checkup results. The server recognizes the text in the image and determines, for example, that his AST value is 60 and his γ-GPT value is 85. Since these values ​​are above the normal range, the analysis engine assesses that there may be an abnormality in his liver function. Based on these results, the server recommends that the user undergo a detailed blood test at a nearby gastroenterology clinic and also provides specific lifestyle advice, such as reducing alcohol consumption and doing at least 30 minutes of aerobic exercise daily.

[0646] This system helps users accurately understand their own health status and take appropriate medical action.

[0647] The following describes the processing flow.

[0648] Step 1:

[0649] The user takes a picture of their health checkup results with a smart device. The image must be clear and all test items must be legible.

[0650] Step 2:

[0651] Users access a dedicated application or web portal and upload images of their health check results to the server. Users ensure they select the correct images and that the upload is smooth.

[0652] Step 3:

[0653] The server preprocesses the received image data, performing noise reduction and image adjustments. This improves image quality and makes subsequent text recognition more accurate.

[0654] Step 4:

[0655] The server uses OCR (Optical Character Recognition) technology to extract text data from images. The server specifically aims to clearly recognize the names and numerical values ​​of health checkup items.

[0656] Step 5:

[0657] The server analyzes the extracted text data and identifies specific health checkup values ​​such as AST, γ-GPT, and LAP. These values ​​are then compared to existing reference values ​​to determine if they fall within the normal range.

[0658] Step 6:

[0659] The server considers user attribute information (gender, age, past data, etc.) and uses an AI analysis model to assess their health status. If an abnormality is detected, it estimates its impact and risks.

[0660] Step 7:

[0661] Based on the analysis results, the server generates a specific action plan for the user. This action plan includes suggestions for necessary additional tests, appropriate medical facilities, and lifestyle advice.

[0662] Step 8:

[0663] The terminal presents the user with the action plan received from the server. The terminal displays the information in a visually clear format that the user can understand immediately.

[0664] Step 9:

[0665] Users review the presented action plan and take action to receive additional medical treatment as needed. They also implement lifestyle improvement advice and manage their health.

[0666] (Example 1)

[0667] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0668] Conventional health monitoring systems have had the challenge of making it difficult for users to easily understand their own health status and quickly decide on appropriate medical treatment. In particular, there is a need for a system that can efficiently manage health checkup results and use that information to help users with lifestyle changes and the selection of medical institutions.

[0669] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0670] In this invention, the server includes means for taking a picture of a health checkup document using a camera and inputting the image; means for extracting character data from the image of the health checkup document using optical character recognition technology; means for analyzing the extracted character data and determining the health status by comparing it with evaluation criteria; and means for presenting recommendations from medical institutions and measures for improving lifestyle habits based on the analysis results. This enables the user to accurately understand their own health status and to quickly take necessary medical action or improve their lifestyle habits.

[0671] A "photography device" is a device used to record health examination documents as images.

[0672] Optical Character Recognition (OCR) is a technology that extracts text information from images.

[0673] "Text data" refers to data in string format obtained based on information extracted from an image.

[0674] An "analysis engine" is a program or algorithm used to evaluate a user's health status based on extracted text data.

[0675] "Evaluation criteria" refer to pre-set standard values ​​or conditions used when assessing a person's health status.

[0676] "Medical institution recommendation" refers to the act of suggesting an appropriate medical institution based on the user's health condition.

[0677] "Lifestyle improvement measures" refer to suggestions for changing behaviors and habits that are offered to users in order to improve their health.

[0678] This invention proposes a health management system that users can access on a daily basis using smart devices. The user begins by taking a picture of their health checkup results using the camera on their smart device. This device records the health checkup document as a high-resolution image. The captured image is then uploaded to a server via a dedicated application or web portal.

[0679] The server applies optical character recognition (OCR) technology to uploaded images to extract text data. This process uses common OCR software such as Tesseract OCR. The server then determines the health status by comparing the extracted text data with evaluation criteria. At this stage, the server efficiently processes the data using the Python Pandas library and employs AI technologies such as TensorFlow for its analysis engine.

[0680] Once the analysis is complete, the server evaluates the results and, taking into account the user's attribute information and past health records, provides recommendations for appropriate medical institutions and lifestyle improvements. For example, for a 40-year-old male user whose AST and γ-GPT levels are above the normal range, the server will recommend additional tests at a nearby medical facility and provide specific suggestions for improving daily habits.

[0681] As a concrete example, the prompt message is as follows: "A 40-year-old male user uploaded an image of his health checkup results. His AST level was detected as 60 and his γ-GPT level as 85. Based on these results, what health risks does he have? Also, what lifestyle changes would you recommend?"

[0682] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0683] Step 1:

[0684] The user takes a picture of their health checkup results using the camera on their smart device. The input is the printed health checkup results, and the output is a high-resolution image file. The user is required to take the picture so that the examination items are clearly legible and to avoid glare and focus issues.

[0685] Step 2:

[0686] Users upload images they have taken to the server using a dedicated application or web portal. The input is the captured image stored on the user's device, and the output is the image data stored on the server. At this stage, the server receives and stores the image data.

[0687] Step 3:

[0688] The server applies Optical Character Recognition (OCR) technology to uploaded images. The input is image data stored on the server, and the output is text data of health checkup results. The server accurately extracts characters and numbers using software such as Tesseract OCR. During extraction, pre-processing is performed to correct image tilt and remove noise.

[0689] Step 4:

[0690] The server compares the extracted text data with evaluation criteria to determine the user's health status. The input is text data obtained by OCR, and the output is the health status evaluation result. The server uses the Python Pandas library to analyze the data and compare the numerical values ​​of each item with the criteria.

[0691] Step 5:

[0692] The server uses an AI analysis engine to assess health risks, taking into account the user's attribute information and past health records. The input is a numerical value of the health status based on evaluation criteria, and the output is the assessment result of future health risks. The server uses models such as TensorFlow to predict risks and determine specific health conditions.

[0693] Step 6:

[0694] The server compiles the analysis results and presents recommendations to the user. The input is the health risk assessment results from AI analysis, and the output is recommendations for medical institutions and lifestyle improvement measures. The server notifies the user of medical institutions as needed and specific things to be aware of in daily life.

[0695] (Application Example 1)

[0696] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0697] In modern society, personal health management is a crucial issue, but many people find it difficult to accurately understand their own health status and implement appropriate lifestyle improvements based on that understanding. Furthermore, the lack of a system that offers benefits based on health status results in a lack of motivation for health improvement. To address this issue, there is a need for efficient analysis of health checkup information and the provision of motivational support based on that analysis.

[0698] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0699] In this invention, the server includes means for extracting textual information from visual information of test results using image recognition technology, means for analyzing the extracted textual information to evaluate the health status, and means for applying benefits when purchasing health-related products based on the evaluation results. This makes it easier for individuals to understand their own health status and take concrete actions toward improving their health. Furthermore, by applying benefits, it is possible to raise users' health awareness and provide them with an incentive to more actively work toward improving their lifestyle habits.

[0700] "Image recognition technology" is a technology for identifying objects and characteristics from digital images or videos and extracting related information.

[0701] "Visual information" refers to information expressed in the form of images or videos, and is used as visual data.

[0702] "Textual information" refers to data in text format extracted from visual data and used for other information processing purposes.

[0703] "Means for evaluating health status" refers to methods for diagnosing an individual's current health status using extracted textual information and making quantitative or qualitative judgments.

[0704] "Means of applying benefits" refers to the methods used to apply benefits and rewards offered to users when they purchase health-related products, based on the evaluation results of their health checkup.

[0705] A "server" is a remote computer system that stores data and performs computational processing, and is a device that provides information through communication with other devices.

[0706] The embodiment for carrying out this invention is based on the following system configuration. The user first takes a picture of their health checkup results using a smart device they use on a daily basis. This smart device can be a smartphone or a tablet. The user uploads the image to the system through a dedicated application.

[0707] The server uses image recognition technology to extract text information from the received image data. This process utilizes OCR technology such as the Google Cloud Vision API. The extracted text information is processed on the server and input into an AI analysis engine to evaluate the user's health status. The AI ​​analysis engine, for example, uses TensorFlow or PyTorch, and comprehensively evaluates the user's health status while considering the user's attribute information and past health checkup data.

[0708] The evaluation results are displayed on the user's smart device. This allows users to understand their health status and receive guidance on medical consultations or lifestyle improvements as needed. Furthermore, based on the health check evaluation results, users can receive benefits when purchasing health-related products. These benefits are applied via electronic payment platforms, utilizing services such as Stripe and PayPal.

[0709] For example, a user takes a picture of their health checkup results with the app and uploads it. The server uses OCR to extract the text information, and after evaluation by an analysis engine, it determines that the user has high triglyceride levels. Based on these results, the user is offered supplements to improve their triglyceride levels, and a discount is applied when they purchase the products.

[0710] Examples of prompt statements to input into the generative AI model are as follows:

[0711] "Design an AI model to recommend the most effective health products based on data extracted from health checkup results. This should include the conditions under which benefits and discounts apply."

[0712] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0713] Step 1:

[0714] The user takes a picture of their health checkup results using a smart device. They use a camera app to take a clear picture so that the results are clearly legible. The user then uploads this image to the system using a dedicated application. The input is the image of the health checkup results, and the output is the image data uploaded to the server.

[0715] Step 2:

[0716] The server performs image recognition processing based on the received image data. Specifically, it extracts text data from the image using OCR (Optical Character Recognition) technology. This is done using tools such as the Google Cloud Vision API. The input is image data of health checkup results, and the output is the extracted text information.

[0717] Step 3:

[0718] The server supplies the character information extracted by OCR to the AI ​​analysis engine. This AI analysis engine uses TensorFlow and PyTorch to analyze the character data and evaluate the user's health status. The input is character information, and the output is the health status evaluation result. The analysis also takes into account the user's attribute information and past health checkup data, and abnormal values ​​are also detected.

[0719] Step 4:

[0720] The server processes data based on the health assessment results to recommend the most suitable health-related products to the user. This includes using a generative AI model to suggest products. The input is the health assessment results, and the output is a list of recommended products. The suggested products incorporate elements that support the user's health improvement.

[0721] Step 5:

[0722] The user makes a purchase decision from the presented product list. Payment is made through an electronic payment system, and benefits are applied. This process is implemented using Stripe or PayPal. The input is the product selection result, and the output is the payment completion and benefit application status.

[0723] Step 6:

[0724] The server sends the user a notification confirming the completion of payment and benefit application. This notification includes details of the received benefits and the next steps to take to improve health. The input is the result of payment and benefit application, and the output is the content of the notification sent to the user.

[0725] This process allows users to take effective improvement measures based on their health checkup results, while also raising their health awareness through the benefits they gain.

[0726] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0727] This invention provides a system that enables more personalized health management and behavioral suggestions by considering not only the user's health check results but also their emotional state. In this system, the user takes a picture of their health check results using a smart device, inputs or collects their emotional state along with the image, and provides it to the system.

[0728] composition

[0729] User input and data collection

[0730] Users take pictures of their health checkup results with a smart device and upload the images to a dedicated application. They can also input their current emotional state, or emotional data can be automatically collected using sensors built into the device.

[0731] Server-side processing of image and emotion data

[0732] The server uses OCR technology to convert uploaded images into text and analyzes the extracted data against baseline values. Simultaneously, it uses an emotion engine to analyze the user's emotional data and evaluate the user's psychological state.

[0733] Integrated analysis of health and emotional state

[0734] The server integrates analyzed health and emotional data to comprehensively assess the user's health status. It then considers the impact of emotional state on health behaviors and selects the most effective medical facilities and improvement measures for the user.

[0735] Generation and presentation of action plans

[0736] Based on integrated analysis, the server generates a specific action plan for the user. This action plan includes suggestions for medical institutions appropriate to the user's psychological state and methods for improving lifestyle habits while considering their emotions. The terminal presents this plan to the user, providing information in a visually appealing and easy-to-understand format.

[0737] Specific example

[0738] For example, consider a case where a female user in her 50s self-reports feeling "anxious" along with her health checkup results. The server identifies high total cholesterol levels from the checkup results and assesses a high stress level from emotional data. Based on these results, the server suggests visiting an internist in collaboration with a psychosomatic medicine specialist and recommends practicing yoga or mindfulness to promote emotional stability. In this way, comprehensive health management becomes possible, following not only physical health but also emotional health.

[0739] The following describes the processing flow.

[0740] Step 1:

[0741] The user uses a smart device to take an image of their health checkup results. The image should be taken in high resolution and include all the information in the results.

[0742] Step 2:

[0743] Users log in to a dedicated app or web portal and upload the images they have taken. At the same time, users can input their emotional state, or the device's emotion sensor will automatically collect data.

[0744] Step 3:

[0745] The server preprocesses the received images and extracts text data from them using OCR technology. This text data includes inspection items and numerical information.

[0746] Step 4:

[0747] The server analyzes the text data obtained by OCR to identify health checkup values ​​such as AST, γ-GPT, and LAP, and compares them to reference values ​​to determine if any values ​​are abnormal.

[0748] Step 5:

[0749] The server uses an emotion engine to analyze the user's emotional data. Based on the input emotional state and collected sensor data, it evaluates the user's emotional state.

[0750] Step 6:

[0751] The server integrates health checkup analysis results and emotional data to assess the user's overall health status. It considers the impact of emotions on health and selects appropriate medical facilities.

[0752] Step 7:

[0753] Based on integrated analysis, the server generates an action plan for the user. This includes specific referrals to medical institutions tailored to the user's health and emotional state, as well as lifestyle improvement suggestions that take their emotions into consideration.

[0754] Step 8:

[0755] The terminal displays the action plan received from the server to the user. The plan is displayed on the terminal in a visually clear and easy-to-understand format.

[0756] (Example 2)

[0757] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0758] In modern society, comprehensively managing individual health and emotional states and providing individually optimized action plans is a challenging task. Conventional systems rely solely on health checkup results for evaluation and fail to provide comprehensive health management that considers emotional states, making it difficult to offer effective suggestions for maintaining individual health.

[0759] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0760] In this invention, the server includes means for acquiring text information from images of health checkup results using image processing technology, means for analyzing the acquired text information and evaluating the health and emotional state, and means for generating an optimal action plan based on the evaluation results. This makes it possible to comprehensively evaluate the user's health and emotional state and provide an optimal action plan for the individual.

[0761] "Image processing technology" refers to techniques for extracting or analyzing useful information from digital images, and includes methods and algorithms for manipulating, transforming, and analyzing images.

[0762] "Text information" refers to character data extracted from media such as images, and is information in a format that can be understood and processed by a computer.

[0763] "Health status" refers to a collection of data and indicators related to the user's physical health, including blood pressure, cholesterol levels, and other physiological measurements.

[0764] "Emotional state" is a collection of data and indicators that show the user's current psychological health and emotional tendencies, including stress levels and mood.

[0765] An "action plan" refers to specific behavioral guidelines recommended based on the user's health and emotional assessment results, including suggestions for medical consultations and methods for improving lifestyle habits.

[0766] This invention relates to a system that comprehensively evaluates a user's health check results and emotional state, and provides individually optimized health management. This system consists of multiple elements, each working in conjunction with the others.

[0767] Users use smart devices to capture images of their health checkup results and upload them to a server via a dedicated application. The smart devices are equipped with cameras and image recognition software, which check the quality of the captured images in real time and supply images of appropriate quality to the system.

[0768] The server uses image processing technology to extract text information from images of health checkup results and performs analysis based on this information. The server employs a high-performance OCR (Optical Character Recognition) engine to accurately extract text data from the captured images. The extracted data is compared against health standard values ​​to detect abnormalities. Furthermore, data for recognizing emotions is collected and processed by the analysis engine to evaluate the user's emotional state. This emotional data is acquired through sensors and self-input.

[0769] During the integrated analysis process, the server utilizes a generated AI model to create an optimal action plan for the user based on their health and emotional state. This plan includes, if necessary, recommendations for medical consultations and methods for stabilizing emotions.

[0770] The terminal presents the generated action plan to the user, providing information in a visually easy-to-understand format. The application on the terminal displays information using graphics and infographics, helping the user easily understand and implement the proposed plan.

[0771] As a specific example, if a female user in her 50s takes a picture of her health checkup results and enters "anxiety" as her emotional state, the server analyzes the blood test results and determines that she has high cholesterol levels and a high stress level. Based on these results, the server suggests visiting an appropriate medical institution and recommends practicing yoga or mindfulness to maintain peace of mind.

[0772] An example of a prompt for a generative AI model is: "Use user data to create a comprehensive health plan that takes into account health and emotional states. Include specific suggestions such as selecting a healthcare provider and lifestyle improvements."

[0773] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0774] Step 1:

[0775] Users take images of their health checkup results with a smart device and upload them to a dedicated application. They also input their emotional state or have emotional data automatically collected through the device's sensors. At this stage, the image and emotional data are provided to the system as input. The output becomes the basis for analysis performed by the server.

[0776] Step 2:

[0777] The server receives the uploaded image and extracts text information from it using OCR technology. Specifically, the server activates the OCR engine, recognizes the characters within the image, and outputs the result as digital text. At this stage, the input is image data, and the output is extracted text information.

[0778] Step 3:

[0779] The server performs data analysis to assess health status based on extracted text information. The input is text information obtained by OCR, and the server determines abnormal indicators by comparing it with reference data. In this process, it performs calculations and comparisons of numerical data and outputs analysis results indicating health risks.

[0780] Step 4:

[0781] The server uses an emotion engine to analyze emotional data acquired from the user. Specifically, it processes the input emotional information using a machine learning model to quantify stress levels and emotional tendencies. The input is emotional data, and the output is the analysis result indicating the user's emotional state.

[0782] Step 5:

[0783] The server integrates health and emotional data and utilizes a generative AI model to generate an optimal action plan for the user. The input consists of assessments of health and emotional states. Based on this, the server uses prompts to generate and output an action plan. This action plan includes specific suggestions for medical institutions and methods for improving daily life.

[0784] Step 6:

[0785] The terminal presents the action plan received from the server to the user. Specifically, it displays the information in a visually easy-to-understand format (graphs and charts) on the application, making it easier for the user to recognize the content of the proposed plan and put it into action. The input is the action plan from the server, and the output is the user's understanding and commencement of action.

[0786] (Application Example 2)

[0787] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0788] In modern society, user health management is becoming increasingly important. However, conventional systems evaluate users' health status based solely on physical diagnostic results, lacking appropriate behavioral suggestions tailored to their emotional state and individual lifestyles. As a result, truly personalized health management that users need is not being achieved. To solve this problem, it is necessary to comprehensively evaluate both the user's physical and emotional state and propose optimal actions.

[0789] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0790] In this invention, the server includes means for extracting text from images of health checkup results using image recognition technology, means for analyzing the extracted text data to evaluate the health status, means for processing the user's emotional state data using emotion analysis technology to evaluate the psychological state, means for integrating the health status evaluation results and the psychological state evaluation results to generate personalized action suggestions, and means for analyzing the purchase history based on the action suggestions to recommend products for improving lifestyle habits. This makes it possible to provide comprehensive and personalized action suggestions that take into account the user's physical and emotional health.

[0791] "Image recognition technology" is a technology that analyzes photographs and videos to identify objects and characters contained within them and converts them into a data format that a computer can understand.

[0792] "Analyzing text data" is the process of interpreting the target information based on extracted textual information and evaluating its meaning and trends.

[0793] "Emotion analysis technology" is a technology that automatically evaluates and identifies a user's emotional state based on their input and behavioral data.

[0794] "Assessing psychological state" means determining the user's mental health and stress level at a given time based on their emotional information.

[0795] "Personalized action suggestions" refer to the process of generating optimal improvement measures and recommendations for a specific user, taking into account their health and emotional state.

[0796] "Analyzing purchase history" is the process of analyzing a user's past purchase history to derive behavioral patterns and preferences.

[0797] "Recommending a product" means presenting products that are considered beneficial to the user based on their health status, emotional state, and past purchase history.

[0798] To realize this invention, a system including a smart device and a server is required. Users use a smartphone or other smart device to take images of their health checkup results and upload them to a dedicated application. The device includes image recognition technology and a function to send images to the server.

[0799] The server extracts text data from images using the Google Cloud Vision API or similar image recognition technologies. Next, it analyzes the extracted text data to assess the user's health status. Furthermore, it analyzes emotional state data, either entered by the user or automatically collected by the device, using sentiment analysis technologies (e.g., IBM Watson's NLP capabilities). This analysis assesses the user's psychological state.

[0800] The server integrates health and emotional data to generate personalized behavioral suggestions. These suggestions provide users with appropriate health improvement measures and healthcare options. It also analyzes purchase history using data analytics tools such as Amazon Web Services (AWS) to recommend products for a healthy lifestyle.

[0801] As a concrete example, if a user self-reports feeling "tired," the server might identify that the user has recently purchased a large amount of high-calorie food and suggest caffeinated beverages and stress-relieving products to address both mental and physical needs.

[0802] Examples of prompt statements that can be used are as follows:

[0803] "The user uploaded their health checkup results to the app and entered their current emotional state as 'tired.' The system should suggest products to add to their shopping list to help relieve stress."

[0804] This allows users to receive comprehensive health management based on their physical and emotional state, and to develop concrete action plans.

[0805] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0806] Step 1:

[0807] The user takes an image of their health checkup results using a smart device and uploads it to a dedicated application. The input is the image of the health checkup results taken by the user with their device, and the output is the image data. The terminal sends this image data to the server.

[0808] Step 2:

[0809] The server converts the received image data into text data using the Google Cloud Vision API. The input is image data of health checkup results, and the output is text data obtained using OCR technology. At this stage, characters in the image are recognized and converted into a format that can be processed by a computer.

[0810] Step 3:

[0811] The server analyzes the extracted text data and evaluates the user's health status. The input is text data obtained through OCR processing, and the output is health status evaluation information obtained through analysis. This analysis applies an evaluation algorithm that includes comparison with reference values.

[0812] Step 4:

[0813] The user inputs their emotional state into the application, or emotional data is automatically collected from sensors equipped on the device. The input is data about the user's emotional state, and the output is that emotional data. The device sends this data to the server.

[0814] Step 5:

[0815] The server analyzes emotional data and evaluates the user's psychological state using IBM Watson's NLP capabilities and similar sentiment analysis techniques. The input is emotional data provided by the user, and the output is the emotional evaluation result. In this step, the meaning of the input data is interpreted and the emotional state is quantified.

[0816] Step 6:

[0817] The server integrates health status and emotional assessment results to generate personalized action suggestions. The inputs are health assessment information and emotional assessment information, and the output is the generated action suggestions. This integrated analysis proposes the most suitable health improvement measures for the user.

[0818] Step 7:

[0819] Based on the suggested actions, the server analyzes the user's purchase history using AWS data analysis tools and recommends products to improve lifestyle habits. The input is the user's purchase history data, and the output is a list of recommended products. This process selects and presents healthy products based on past purchase patterns.

[0820] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0821] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0822] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0823] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0824] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0825] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0826] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0827] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0828] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0829] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0830] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0831] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0832] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0833] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0834] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0835] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0836] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0837] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0838] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0839] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0840] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0841] The following is further disclosed regarding the embodiments described above.

[0842] (Claim 1)

[0843] A method for extracting text from images of health checkup results using image recognition technology,

[0844] A means of analyzing extracted text data to assess health status,

[0845] A means of suggesting medical institutions and lifestyle improvement measures based on evaluation results,

[0846] A system that includes this.

[0847] (Claim 2)

[0848] The system according to claim 1, which evaluates the user's health status taking into account the user's attribute information and past health check data.

[0849] (Claim 3)

[0850] The system according to claim 1, which determines abnormal values ​​by comparing extracted text data with reference values ​​in an internal database.

[0851] "Example 1"

[0852] (Claim 1)

[0853] A means for taking pictures of health examination documents using a camera and inputting the images,

[0854] A method for extracting character data from images of health examination documents using optical character recognition technology,

[0855] A means of analyzing extracted text data and comparing it with evaluation criteria to determine health status,

[0856] Based on the analysis results, a means of providing recommendations from medical institutions and suggestions for improving lifestyle habits,

[0857] A system that includes this.

[0858] (Claim 2)

[0859] The system according to claim 1, which evaluates health risks using an analysis engine, taking into account the user's attribute information and past health records.

[0860] (Claim 3)

[0861] The system according to claim 1, which detects anomalies by comparing character data obtained from a user with registered information of comparison criteria.

[0862] "Application Example 1"

[0863] (Claim 1)

[0864] A means of extracting textual information from visual information of inspection results using image recognition technology,

[0865] A means of analyzing extracted textual information to evaluate health status,

[0866] A means of suggesting medical institutions and lifestyle improvement measures based on evaluation results,

[0867] A method of applying benefits when purchasing health-related products based on evaluation results,

[0868] A system that includes this.

[0869] (Claim 2)

[0870] The system according to claim 1, which evaluates the user's health status taking into account the user's attribute information and past examination data.

[0871] (Claim 3)

[0872] The system according to claim 1, which determines abnormal values ​​by comparing extracted character information with reference values ​​in an internal database.

[0873] "Example 2 of combining an emotion engine"

[0874] (Claim 1)

[0875] A means of obtaining text information from images of health checkup results using image processing technology,

[0876] A means for analyzing acquired text information and evaluating health and emotional states,

[0877] A means for generating an optimal action plan based on evaluation results,

[0878] Means for presenting the generated action plan,

[0879] A system that includes this.

[0880] (Claim 2)

[0881] The system according to claim 1, which comprehensively evaluates the user's health status by taking into account the user's physiological and emotional information.

[0882] (Claim 3)

[0883] The system according to claim 1, which determines an anomaly indicator by comparing acquired text information with reference data.

[0884] "Application example 2 when combining with an emotional engine"

[0885] (Claim 1)

[0886] A method for extracting text from images of health checkup results using image recognition technology,

[0887] A means of analyzing extracted text data to assess health status,

[0888] A means of processing user emotional state data using emotion analysis technology to evaluate psychological state,

[0889] A means for integrating the results of health status assessments and psychological status assessments to generate personalized behavioral suggestions,

[0890] A method for analyzing purchase history based on behavioral suggestions and recommending products for improving lifestyle habits,

[0891] A system that includes this.

[0892] (Claim 2)

[0893] The system according to claim 1, which evaluates the user's health status taking into account the user's attribute information and past health check data.

[0894] (Claim 3)

[0895] The system according to claim 1, which determines outliers by comparing extracted text data with reference values ​​in an internal database and provides products based on an integrated evaluation. [Explanation of symbols]

[0896] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A method for extracting text from images of health checkup results using image recognition technology, A means of analyzing extracted text data to assess health status, A means of suggesting medical institutions and lifestyle improvement measures based on evaluation results, A system that includes this.

2. The system according to claim 1, which evaluates the user's health status taking into account the user's attribute information and past health check data.

3. The system according to claim 1, which determines abnormal values ​​by comparing extracted text data with reference values ​​in an internal database.

Images