system

Abstract

Provide a system. A means for acquiring an image of a medical examination result; means for extracting text data from the captured image; means for analyzing the extracted text data to detect outliers; A means for generating an appropriate action plan for the user based on the analysis results; means for notifying the user of the generated action plan; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Interpreting conventional health checkup results requires specialized knowledge and is extremely difficult for the general public to understand. Furthermore, when abnormal values ​​are viewed in a complex manner, it is unclear what to do next or which medical department to visit, preventing specific actions from being taken. Furthermore, finding the appropriate medical institution for each region is also a challenge. The present invention aims to solve these problems and provide assistance for easily understanding health checkup results and taking specific actions. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by the following means. First, it provides a means for acquiring an image of the health checkup results. Next, it uses a means for extracting text data from the acquired image. It further includes a means for analyzing the extracted text data and detecting abnormal values. It is equipped with a means for generating an appropriate action plan for the user based on the analysis results and includes a means for notifying the user of the generated action plan. In addition, it includes a means for taking user attribute information into account in the analysis and a means for providing the user with information on the most suitable medical department and medical institution based on the generated action plan, thereby supporting the user in the series of processes for understanding the health checkup results and taking specific actions.

[0006] "Health checkup results" are documents or data summarizing the test results of an individual who underwent a health checkup, and include the numerical values ​​of various test items such as blood tests, urine tests, and physical measurements.

[0007] The "means for acquiring images" refers to a means for capturing images of the medical examination results as digital data using a photographing device such as a camera.

[0008] The "means for extracting text data" refers to a means for reading text information from the acquired image using OCR (Optical Character Recognition) technology and converting it into digital character data.

[0009] The "means for detecting abnormal values" is a means for analyzing the extracted text data, determining whether the numerical values ​​of each test item are within the normal range, and identifying abnormal values.

[0010] The "means for generating an action plan" is a means for creating a plan to recommend specific actions to be taken by the user (e.g., visiting a specific medical department or changing lifestyle habits) based on the analysis results.

[0011] "Means for notifying the user" refers to means for sending the generated action plan to the user's device and notifying the user by means of push notification, display, or the like.

[0012] "Attribute information" refers to personal information about a user, such as age, gender, and place of residence, and is a factor that is taken into consideration when analyzing health checkup results.

[0013] "Means for providing information on medical departments and medical institutions" refers to means for providing information on the appropriate medical department that the user should visit and related medical institutions near the user's place of residence based on the detected abnormal values. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The system of the present invention acquires and analyzes information from images of health checkup results, and presents a specific action plan to the user based on the results. This system is composed of three elements: the user, a terminal, and a server.

[0036] System Overview

[0037] Photographing and uploading health checkup results

[0038] User: Takes a photo of the health check result sheet using the camera on a device such as a smartphone and uploads the image to the application.

[0039] Device: The captured image is temporarily stored on the device and prepared for transmission to the server.

[0040] Receiving and analyzing image data

[0041] Server: Receives image data sent from the device and checks the image quality. Next, it uses OCR (Optical Character Recognition) technology to extract text data from the image. This text data includes test item names such as "AST," "γ-GTP," and "LAP," as well as their numerical values.

[0042] Text data analysis

[0043] Server: Based on the extracted text data, the server analyzes whether the values ​​for each test item are within the normal range or abnormal. This analysis takes into account the user's attribute information (age, gender, place of residence, etc.).

[0044] Action plan generation and notification

[0045] Server: Based on the analysis results, it generates an action plan for the user. For example, if the AST level is high, it recommends a consultation with an internal medicine doctor and provides information on appropriate medical institutions in the area.

[0046] Server: Sends the generated action plan and analysis results to the user's device.

[0047] Device: Notifies the user of the received action plan and displays it in the application.

[0048] Specific scenarios

[0049] For example, let's say a male user in his 30s has undergone a health checkup in Shibuya Ward, Tokyo, and has received the results. When this user takes a photo of the health checkup result sheet with his smartphone camera and uploads it to the app, the system operates as follows:

[0050] User: Open the smartphone app and use the camera to take a photo of the health check result sheet and upload it.

[0051] Device: The captured image is temporarily saved and sent to the server.

[0052] Server: Receives image data and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L" using OCR.

[0053] Server: The extracted data is passed to an AI engine, which takes into account the user's attributes such as age and gender to detect outliers.

[0054] Server: Detects when AST levels are above the normal range and generates an action plan recommending a consultation with an internal medicine doctor. At the same time, it generates a list of reputable internal medicine clinics in Shibuya Ward.

[0055] Server: Sends the action plan and clinic information to the user's device.

[0056] On the device: The app will notify the user and display the results. The user can then check the recommended clinic information within the app and make an appointment if necessary.

[0057] In this way, the system of the present invention supports the user in understanding the health checkup results and intuitively determining the specific actions to be taken next.

[0058] The processing flow will be explained below.

[0059] Step 1:

[0060] The user launches the smartphone app and takes a photo of the health check result sheet with the camera.

[0061] Step 2:

[0062] The user checks the image they have taken and presses the upload button.

[0063] Step 3:

[0064] The terminal temporarily stores the captured image and prepares the data to be sent to the server.

[0065] Step 4:

[0066] The terminal transmits the image data to the server.

[0067] Step 5:

[0068] Check the quality of the image data received by the server and, if necessary, pre-process the image.

[0069] Step 6:

[0070] The server uses OCR technology to extract text data from the image.

[0071] Step 7:

[0072] The server stores the extracted text data in a database.

[0073] Step 8:

[0074] The server passes the text data to the AI ​​engine and begins analysis.

[0075] Step 9:

[0076] Based on the results analyzed by the AI ​​engine, the server determines whether the values ​​of each test item (e.g., AST, γ-GTP, LAP, etc.) are within the normal range or abnormal.

[0077] Step 10:

[0078] Based on the analysis results, the server performs a detailed analysis of the anomalies, taking into account the user's attribute information such as age, gender, and place of residence.

[0079] Step 11:

[0080] The server uses the analysis results to generate an appropriate action plan for the user.

[0081] Step 12:

[0082] Based on the action plan generated by the server, the user decides what action to take next (e.g., visiting a specific medical department or changing lifestyle habits).

[0083] Step 13:

[0084] The server lists recommended medical institutions and medical departments based on the user's location.

[0085] Step 14:

[0086] The server sends the action plan and related information to the user's terminal.

[0087] Step 15:

[0088] The device notifies the user of the action plan and information on recommended medical institutions it has received.

[0089] Step 16:

[0090] The user can check the notification contents within the app and make an appointment with a medical institution if necessary.

[0091] Example 1

[0092] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0093] Currently, many users receive their health checkup results, but find it difficult to understand the contents and take appropriate action. Furthermore, reviewing and interpreting the results takes time, making it difficult to quickly respond to specific abnormal values. Furthermore, results are often judged based on uniform standards without taking into account the user's attribute information, making it difficult to provide the optimal response for each individual user.

[0094] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0095] In this invention, the server includes a means for a user to acquire an image of the health checkup results, a means for extracting text data from the acquired image, a means for analyzing the extracted text data to detect abnormal values, a means for generating an action plan based on the analysis results taking into account the user's attribute information, and a means for notifying the generated action plan and the analysis results from the server to the user's terminal. This allows the user to quickly and accurately understand the health checkup results and take appropriate action. Furthermore, by taking the user's attribute information into account during the analysis, more personalized advice can be provided, enabling responses tailored to each individual's health condition.

[0096] A "user" is a person who uses this system to upload health checkup results and receive analysis results and action plans.

[0097] "Image of medical examination results" refers to image data of sheets or documents containing the results of a medical examination that the user underwent, taken with a smartphone or digital camera.

[0098] "Text data" refers to text information extracted from an image using OCR technology, and includes test item names and their numerical values.

[0099] An "abnormal value" refers to a value outside the normal range in each test item of a health checkup, and is a value that requires the user to take measures such as visiting a medical institution or improving their lifestyle.

[0100] "User attribute information" is data including personal information such as the user's age, gender, and place of residence, and is an element that is taken into consideration when analyzing the health checkup results.

[0101] An "action plan" is specific advice and instructions provided to the user based on the analysis results, including necessary medical visits and lifestyle improvements.

[0102] A "terminal" is a device used by a user to take images of health checkup results and upload them to the system, and refers to electronic devices such as smartphones and tablets.

[0103] The "server" is a computer system that receives image data of health checkup results sent from a user's terminal, analyzes the data, and sends the results and action plans to the user's terminal.

[0104] "Means to confirm quality" refers to a technology that evaluates whether the images obtained from health checkup results are of a quality that can be analyzed, and is a process that automatically checks resolution, brightness, distortion, etc.

[0105] The "means for requesting re-imaging" is a mechanism that notifies the user to re-imaging the medical examination results if the image quality is determined to be insufficient.

[0106] "Information on medical departments and medical institutions" refers to detailed information on medical specialties recommended for the user based on their health condition and the corresponding medical facilities in the area.

[0107] The system of the present invention supports users in obtaining useful information from health checkup results and taking specific actions based on the results. This system is broadly composed of three elements: the user, the terminal, and the server.

[0108] Photographing and uploading health checkup results

[0109] 1. User: The user takes a photo of the health checkup result sheet using a camera on a device such as a smartphone. The image is then uploaded via a dedicated app. It is recommended that the user checks in advance to ensure that the photo has been taken properly.

[0110] Temporarily save image data and prepare for transmission

[0111] 2. Device: The captured image is temporarily stored on the device. During this process, the image's resolution and clarity are checked and it is prepared for transmission to the server. The connection status is also checked and the optimal transmission method is selected.

[0112] Receiving and analyzing image data

[0113] 3. Server: Receives image data sent from the device. Checks the quality of the received image and requests the user to retake the image if it is not suitable. Once the image quality is confirmed, extracts text data from the image using OCR (Optical Character Recognition) technology. Specifically, OCR tools such as Google® Cloud Vision API and Tesseract can be used.

[0114] 4. Server: Analyzes the extracted text data and determines whether the values ​​for each test item are within the normal range or abnormal. This analysis takes into account the user's attribute information (age, gender, place of residence, etc.). The analysis is carried out using AI analysis engines such as TENSORFLOW (registered trademark) and PyTorch, which are written in the Python language.

[0115] Action plan generation and notification

[0116] 5. Server: Generates a specific action plan for the user based on the analysis results. For example, if the AST value is high, it recommends a consultation with an internal medicine doctor and provides information on local medical institutions.

[0117] 6. Server: Sends the generated action plan and analysis results to the user's device.

[0118] 7. Device: The received action plan is notified to the user and displayed within the application. The user can then review the provided specialist information and recommended actions, and make an appointment or follow up if necessary.

[0119] Specific examples

[0120] For example, let's say a male user in his 30s has undergone a health checkup in Shibuya Ward, Tokyo, and has received the results. When this user takes a photo of the health checkup result sheet with his smartphone camera and uploads it to the app, the system operates as follows:

[0121] User: Opens the smartphone app and uses the camera to take a photo of the health check result sheet and upload it.

[0122] Device: The captured image is temporarily saved and sent to the server.

[0123] Server: Receives image data and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L" using OCR.

[0124] Server: The extracted data is passed to an AI engine, which takes into account the user's attributes such as age and gender to detect outliers.

[0125] Server: Detects when AST levels are above the normal range and generates an action plan recommending a consultation with an internal medicine doctor. At the same time, it generates a list of reputable internal medicine clinics in Shibuya Ward.

[0126] Server: Sends the action plan and clinic information to the user's device.

[0127] On the device: The app will notify the user and display the results. The user can then check the recommended clinic information within the app and make an appointment if necessary.

[0128] Prompt Sentence Examples

[0129] "When a male user in his 30s uploads his health checkup results, if the images are unclear, send him a notification urging him to take a new photo. If his AST value is high, recommend that he see an internal medicine doctor and suggest a reputable clinic in Shibuya Ward."

[0130] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0131] Program processing flow

[0132] Step 1:

[0133] User: Launches the smartphone app and takes a photo of the health check result sheet with the camera.

[0134] Input: Medical examination result sheet of the subject.

[0135] Output: Image file of the health check result sheet.

[0136] Specific operation: The user uses the camera function of the smartphone to take a picture of the health check result sheet, fitting it within the frame. After taking the picture, the user can check the preview of the image.

[0137] Step 2:

[0138] User: Check the captured image and press the upload button within the app.

[0139] Input: Image file of health check result sheet.

[0140] Output: The upload request.

[0141] Specific operation: The user checks whether the image they have taken is accurate, and if there are no problems, presses the "Upload" button in the app.

[0142] Step 3:

[0143] Terminal: Temporarily stores image data and prepares it for transmission to the server.

[0144] Input: Image file of health check result sheet.

[0145] Output: Server ready to send status.

[0146] Specific operation: The device saves the image file to a temporary folder, checks the status of Wi-Fi and mobile data communication, and selects the optimal communication method.

[0147] Step 4:

[0148] Terminal: Sends image data to the server.

[0149] Input: Image file of health check result sheet.

[0150] Output: Image sent to server.

[0151] Specific operation: The device sends the image file to the server's upload API endpoint. When the sending is complete, a success message is received.

[0152] Step 5:

[0153] Server: Receives image data and stores it in a database.

[0154] Input: Image data.

[0155] Output: Image data stored in a database.

[0156] Specific operation: The server receives the uploaded image and saves it in the database from the temporary storage folder.

[0157] Step 6:

[0158] Server: Checks the quality of the image and requests a retake if it is not satisfactory.

[0159] Input: Stored image data.

[0160] Output: Image quality check result, notification of request to retake if inadequate.

[0161] Specific operation: The server automatically checks the image resolution, brightness, distortion, etc., and if they are inappropriate, it notifies the user's device to request a re-take of the image.

[0162] Step 7:

[0163] Server: Extracts text data from images using OCR technology.

[0164] Input: Quality checked image data.

[0165] Output: The extracted text data.

[0166] Specific operation: The server uses the Google Cloud Vision API and Tesseract to extract text data such as "AST," "γ-GTP," and "LAP" from the image.

[0167] Step 8:

[0168] Server: Analyzes the text data and detects outliers taking into account user attribute information.

[0169] Input: Text data, user attribute information.

[0170] Output: The analyzed outlier data.

[0171] Specific operation: The server passes the extracted text data to an AI analysis engine (TensorFlow or PyTorch), which analyzes the data taking into account the user's attribute information such as age, gender, and place of residence.

[0172] Step 9:

[0173] Server: Generates an action plan based on the analysis results.

[0174] Input: Analyzed outlier data, user attribute information.

[0175] Output: Action plan.

[0176] Specific actions: The server generates a specific action plan for the user based on the abnormal values. For example, if the AST value is high, it recommends a consultation with an internal medicine doctor and also provides information on local medical institutions.

[0177] Step 10:

[0178] Server: Sends the generated action plan and analysis results to the user's device.

[0179] Input: Action plan, analysis results.

[0180] Output: Sent to user's device.

[0181] Specific operation: Calls an API that formats the action plan and analysis results and sends them to the user's device.

[0182] Step 11:

[0183] On the device: The user is notified of the received action plan and it is displayed within the app.

[0184] Input: Action plan and analysis results sent from the server.

[0185] Output: Action plans and analysis results displayed to the user.

[0186] What it does: The device analyzes the data it receives and generates a notification, which displays the analysis results and action plan to the user within the application.

[0187] Step 12:

[0188] User: Review the action plan and take the necessary steps.

[0189] Input: Action plan and analysis results displayed within the app.

[0190] Output: Action taken, necessary procedures (e.g., medical appointment).

[0191] Specific operation: The user checks the medical institution information provided in the app and, if necessary, makes an appointment or makes a medical appointment.

[0192] (Application example 1)

[0193] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0194] In modern society, the need to be health-conscious is increasing, but many people find it difficult to properly understand their health checkup results and take appropriate action. It is also difficult to recommend appropriate products and ensure the convenience of purchasing them. For this reason, there is a need for a system that allows users to create specific action plans based on their health checkup results, select appropriate products according to those plans, and easily purchase them in physical stores.

[0195] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0196] In this invention, the server includes means for acquiring an image of the medical checkup result, means for extracting text data from the acquired image, means for analyzing the extracted text data to detect abnormal values, means for generating an appropriate action plan for the user based on the analysis results, means for notifying the user of the generated action plan, and means for suggesting optimal products to the user based on the analysis results of the medical checkup result. This enables the user to easily understand the medical checkup result and take appropriate action, and provides an environment in which products can be suggested and purchased seamlessly according to the action plan.

[0197] "Image of health checkup results" is an image file containing health data obtained when a user undergoes a health checkup.

[0198] "Text data" is text information extracted from an image using OCR technology.

[0199] An "abnormal value" refers to a value in a test item in a health checkup result that exceeds the normal range.

[0200] An "action plan" is a plan of specific actions that the user should take based on the analysis of the health check results.

[0201] "Notification" is a means of communicating information to inform the user of the generated action plan.

[0202] "User attribute information" refers to information that indicates individual characteristics of a user, such as age, gender, and place of residence.

[0203] A "medical department" is a medical department at a medical institution that specializes in treating specific diseases or parts of the body.

[0204] "Medical institutions" are facilities that provide medical services, such as hospitals and clinics.

[0205] "Products" are items such as food, supplements, and daily necessities that are recommended to users based on the analysis of their health checkup results.

[0206] A "physical store" is a physical sales location where users can actually visit and purchase products.

[0207] "Navigation" is a means of guidance to guide a user to a destination.

[0208] An embodiment of the present invention will be described below: The system is composed of three elements: a user, a terminal, and a server.

[0209] Photographing and uploading health checkup results

[0210] Users take a photo of their health checkup result sheet using a device such as a smartphone and upload the image to the application.

[0211] The terminal temporarily stores the captured image and prepares it for transmission to the server.

[0212] Receiving and analyzing image data

[0213] The server receives the image data sent from the device and checks the image quality. It then uses OCR (Optical Character Recognition) technology to extract text data from the image. This text data includes test item names such as "AST," "γ-GTP," and "LAP," as well as their numerical values.

[0214] Text data analysis

[0215] The server analyzes the extracted text data to determine whether the values ​​for each test item are within the normal range or are abnormal, taking into account the user's attribute information (age, gender, place of residence, etc.).

[0216] Action plan generation and notification

[0217] Based on the analysis results, the server generates an action plan for the user. For example, if the AST level is high, it may recommend a consultation with an internal medicine doctor and provide information on appropriate medical institutions in the area.

[0218] The generated action plan is then notified to the user and displayed within the application, where the user can check the recommended medical institutions and product information and make purchases or reservations as necessary.

[0219] Product suggestions and navigation

[0220] The server then recommends optimal products to users based on the analysis of their health checkup results. For example, foods and supplements that are useful for improving health. Furthermore, the server provides a navigation function to make it easy to purchase these products in physical stores. Users can use the application to find and purchase the recommended products at nearby physical stores.

[0221] Technology used

[0222] In the present invention, the following hardware and software are used.

[0223] Hardware:

[0224] Smartphone (with camera function)

[0225] Servers (with high-performance processors and storage)

[0226] Physical store sales system

[0227] software:

[0228] pytesseract: OCR engine (extracts text from images)

[0229] Pillow: Image Processing Library (PIL)

[0230] requests: Communication with external API (store information acquisition)

[0231] JSON: Data Format

[0232] Specific examples

[0233] For example, if a male user in his 30s receives a medical checkup result showing an AST level of 45 U / L and a γ-GTP level of 75 U / L, he takes a photo of the medical checkup result sheet with his smartphone and uploads it to the app. The following process will occur:

[0234] 1. The device temporarily saves the image and sends it to the server.

[0235] 2. The server receives the image data and uses OCR to convert it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L."

[0236] 3. The server passes the extracted data to an AI engine, which takes into account the user's attribute information such as age and gender to detect outliers.

[0237] 4. The server detects that the AST level is above the normal range and suggests products to help improve health, along with an action plan that recommends consulting an internal medicine doctor.

[0238] 5. Users can check recommended products and medical institutions within the app and make reservations or purchases as needed.

[0239] Prompt Sentence Examples

[0240] Below is an example prompt to input to the generative AI model.

[0241] We provide images of health checkup results. Extract important health data from these images and recommend the best products for users based on the analysis results.

[0242] User information: Male, 30 years old, living in Tokyo

[0243] Health data: AST: 45 U / L, γ-GTP: 75 U / L

[0244] Analysis and recommendations: Recommendations for medical consultation, suggestions for places to buy low-fat yogurt, etc.

[0245] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0246] Step 1:

[0247] Users take a photo of their health checkup result sheet with their smartphone and upload the image to the application. At this time, users open the application and use the camera function to capture an image of the health checkup result sheet.

[0248] Input: Image of health check result sheet

[0249] Output: Temporarily saved image file

[0250] Specific operation: The user launches the camera app, takes a photo of the health check result sheet, and uploads the image to the application. The application temporarily saves the image file on the device.

[0251] Step 2:

[0252] The device temporarily stores the captured image and prepares it for transmission to the server. At this time, an application on the device converts the image file into the data format required for transmission to the server.

[0253] Input: Temporarily saved image file

[0254] Output: Image data formatted for sending to the server

[0255] Specific operation: The application on the terminal converts the image file into an appropriate format (e.g., JPEG or PNG) and sends it to the server over the network.

[0256] Step 3:

[0257] The server receives the image data sent from the terminal. At this time, the server checks the quality of the image and requests retransmission if it is unclear.

[0258] Input: Image data sent from the device

[0259] Output: Quality-checked image data

[0260] Specific operation: The server receives the image, evaluates the quality using an image processing algorithm, and if the quality is not satisfactory, asks the user to take the photo again.

[0261] Step 4:

[0262] The server extracts text data from the image using OCR technology (e.g., pytesseract), where the server inputs the image into the OCR engine and obtains the text information.

[0263] Input: Quality-checked image data

[0264] Output: Extracted text data

[0265] Specific operation: The server analyzes the image using pytesseract and extracts text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L."

[0266] Step 5:

[0267] The server analyzes the extracted text data to detect outliers, using an AI engine and user attribute information.

[0268] Input: Extracted text data, user attribute information

[0269] Output: Outlier detection results

[0270] Specific operation: The server inputs the extracted numerical data into an AI engine, taking into account attribute information such as age, gender, and place of residence, and compares it with normal values. If an abnormal value is detected, detailed information about it is generated.

[0271] Step 6:

[0272] The server generates an action plan based on the analysis results, and the AI ​​engine assembles data to suggest appropriate actions and products to the user.

[0273] Input: Outlier detection results

[0274] Output: Generated action plan

[0275] Specific operation: Based on the results of the abnormal value detection, the server generates an action plan such as "We recommend that you visit an internal medicine doctor" or "We recommend that you consume low-fat yogurt."

[0276] Step 7:

[0277] The server notifies the user of the generated action plan, which can be done via push notification or in-app display.

[0278] Input: Generated action plan

[0279] Output: User informed action plan

[0280] Specific operation: The server sends the generated action plan to the terminal, and the terminal displays the received notification to the user.

[0281] Step 8:

[0282] Users can check recommended products and medical institutions within the application and are guided to purchase products at physical stores. At this time, users can refer to the in-app map function and store information.

[0283] Input: Notified action plan, recommended product information

[0284] Output: Purchased items, medical appointments

[0285] Specific operation: The user checks the product information and medical institution information displayed within the app and navigates to the nearest store to purchase the product.

[0286] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0287] The system of the present invention acquires and analyzes information from images of health checkup results and presents a specific action plan to the user based on the results. By incorporating an emotion engine that recognizes the user's emotions, the system provides a more appropriate action plan that also takes into account the user's psychological state. The system is comprised of three elements: the user, the terminal, and the server.

[0288] System Overview

[0289] Photographing and uploading health checkup results

[0290] User: Takes a photo of the health check result sheet using the camera on a device such as a smartphone and uploads the image to the application.

[0291] Device: The captured image is temporarily stored on the device and prepared for transmission to the server.

[0292] Receiving and analyzing image data

[0293] Server: Receives image data sent from the device and checks the image quality. Next, it uses OCR (Optical Character Recognition) technology to extract text data from the image. This text data includes test item names such as "AST," "γ-GTP," and "LAP," as well as their numerical values.

[0294] Text data analysis

[0295] Server: Based on the extracted text data, the server analyzes whether the values ​​for each test item are within the normal range or abnormal. This analysis takes into account the user's attribute information (age, gender, place of residence, etc.).

[0296] Emotion recognition by emotion engine

[0297] Device or server: Analyze the user's current emotions using the emotion engine. For example, if the results of a health checkup cause stress or anxiety to the user, the emotion engine can recognize this.

[0298] Action plan generation and notification

[0299] Server: Based on the analysis results, taking into account the user's attribute information and the analysis results of the emotion engine, the server generates an action plan for the user. For example, if the AST value is high, it will recommend a consultation with an internal medicine doctor and provide information on appropriate medical institutions in the area. If the user is feeling anxious, it will include explanations and responses to reassure them.

[0300] Server: Sends the generated action plan and analysis results to the user's device.

[0301] On the device, the received action plan is notified to the user and displayed in the application, with the language and expressions adjusted to match the user's emotions.

[0302] Specific scenarios

[0303] For example, let's say a male user in his 30s has undergone a health checkup in Shibuya Ward, Tokyo, and has received the results. When this user takes a photo of the health checkup result sheet with his smartphone camera and uploads it to the app, the system operates as follows:

[0304] Photographing and analyzing health checkup results

[0305] User: Open the smartphone app and use the camera to take a photo of the health check result sheet and upload it.

[0306] Device: The captured image is temporarily saved and sent to the server.

[0307] Server: Receives image data and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L" using OCR.

[0308] Server: The extracted data is passed to an AI engine, which takes into account the user's attributes such as age and gender to detect outliers.

[0309] Emotion recognition and action plan generation

[0310] Device or server: The emotion engine recognizes the user's heart rate and facial expressions to determine whether the user is feeling anxious or stressed about the results.

[0311] Server: Detects when AST levels are above the normal range and generates an action plan that takes the user's anxiety into account. For example, it may recommend a consultation with an internal medicine doctor while providing reassuring explanations. It also generates a list of reputable internal medicine clinics in Shibuya Ward.

[0312] Communicating results and encouraging action

[0313] Server: Sends the action plan and clinic information to the user's device.

[0314] On the device: The app will notify the user of the results, and the explanation will be given in a way that is sensitive to the user's emotions, allowing the user to take the next step with peace of mind.

[0315] In this way, the system of the present invention takes into consideration the psychological state felt by the user through the health checkup results, and supports the user in intuitively deciding what specific action to take next.

[0316] The processing flow will be explained below.

[0317] Step 1:

[0318] The user launches the smartphone app and takes a photo of the health check result sheet with the camera.

[0319] Step 2:

[0320] The user checks the image they have taken and presses the upload button.

[0321] Step 3:

[0322] The terminal temporarily stores the captured image and prepares the data to be sent to the server.

[0323] Step 4:

[0324] The terminal transmits the image data to the server.

[0325] Step 5:

[0326] Check the quality of the image data received by the server and, if necessary, pre-process the image.

[0327] Step 6:

[0328] The server uses OCR technology to extract text data from the image.

[0329] Step 7:

[0330] The server stores the extracted text data in a database.

[0331] Step 8:

[0332] The server passes the text data to the AI ​​engine and begins analysis.

[0333] Step 9:

[0334] Based on the results analyzed by the AI ​​engine, the server determines whether the values ​​of each test item (e.g., AST, γ-GTP, LAP, etc.) are within the normal range or abnormal.

[0335] Step 10:

[0336] Based on the analysis results, the server performs a detailed analysis of the anomalies, taking into account the user's attribute information such as age, gender, and place of residence.

[0337] Step 11:

[0338] The emotion engine uses data acquired by the server from the user's terminal or other devices to analyze the user's emotions, for example, detecting anxiety or stress by analyzing voice tone and facial expressions.

[0339] Step 12:

[0340] The server adjusts the action plan based on the results of the emotion engine, adding more reassuring explanations and support if the user feels anxious.

[0341] Step 13:

[0342] The server generates an action plan for the user based on the analysis results and emotion recognition results.

[0343] Step 14:

[0344] The server lists information on recommended medical institutions and medical departments based on the user's location.

[0345] Step 15:

[0346] The server sends the action plan and related information to the user's terminal.

[0347] Step 16:

[0348] The device notifies the user of the action plan and information on recommended medical institutions it has received.

[0349] Step 17:

[0350] The user can check the notification contents within the app and make an appointment with a medical institution if necessary.

[0351] Step 18:

[0352] The device uses an emotion engine to obtain user feedback and reflect it in future analysis and action plan generation.

[0353] Example 2

[0354] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0355] Conventional health checkup result analysis systems required users to understand the results and decide for themselves what action to take next, which often led to anxiety and stress in users. Furthermore, when abnormal values ​​were detected, the systems lacked specific instructions for appropriate action or the ability to provide psychological support. This made it difficult for users to move on to the next action quickly and with peace of mind.

[0356] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring an image of the medical examination result, a means for extracting text data from the acquired image, a means for analyzing the extracted text data and detecting abnormal values, a means for analyzing the emotional state of the user, a means for generating an appropriate action plan for the user based on the analysis results, and a means for notifying the user of the generated action plan. This allows an appropriate action plan to be proposed taking into consideration the psychological state of the user, allowing the user to proceed with the next action with peace of mind.

[0357] "Image of medical examination results" refers to a digital image of the sheet containing the medical examination results received by the user.

[0358] "Text data" refers to the text information extracted from images of health checkup results, including numerical values ​​and names of test items.

[0359] An "abnormal value" is a value in which the numerical value of the analyzed text data deviates from a preset normal range.

[0360] An "action plan" is a specific course of action that is suggested to the user based on the analysis results and the user's attribute information.

[0361] "Emotional state" refers to the psychological state the user is currently feeling, including emotions such as stress and anxiety.

[0362] The "analysis results" are the comprehensive results of the analysis performed based on the text data of the health checkup results.

[0363] "User attribute information" refers to information related to an individual, such as the user's age, gender, and place of residence.

[0364] "Information on medical departments and medical institutions" is detailed data including information on medical facilities and medical departments recommended to users.

[0365] "Means for acquiring images" refers to a method of acquiring the health check result sheet as a digital image using a camera such as a smartphone.

[0366] "Means for extracting text data" refers to the process of using OCR technology to read character information from an image and extract it as text data.

[0367] The "means for detecting abnormal values ​​through analysis" is a method for analyzing extracted text data and detecting values ​​outside the standard range.

[0368] The "means for notifying the user" is a method for sending the generated action plan to the user's terminal and notifying the user.

[0369] The system of the present invention analyzes images obtained from health checkup results and presents a specific action plan to the user. This system operates with three elements: the user, the terminal, and the server. It also has a function to analyze the user's emotional state, which can reduce the psychological burden.

[0370] Hardware and Software Used

[0371] Device: Smartphone (e.g. iPhone (registered trademark) or ANDROID (registered trademark) smartphone)

[0372] Server: Cloud server (e.g., AWS (registered trademark), GCP)

[0373] OCR software: Tesseract OCR

[0374] Emotion engine: Emotion analysis software (e.g., Affectiva, Emotion API)

[0375] Explanation of program processing

[0376] 1. User: The user takes a photo of the health check result sheet with their smartphone camera and uploads the image to the application by tapping the "Upload Diagnosis Results" button in the app.

[0377] 2. Device: The device temporarily stores the captured image in its internal storage and checks the image's file format, resolution, brightness, etc. If the image quality is insufficient, a message is displayed prompting the user to take a new photo. If the image quality is confirmed to be sufficient, it is sent to the server.

[0378] 3. Server: The server receives the image data sent from the device and extracts text data from the image using OCR software (e.g., Tesseract OCR). In this example, items such as "AST: 45 U / L" and "γ-GTP: 75 U / L" are extracted.

[0379] 4. Server: Analyzes the extracted text data and determines whether the values ​​for each test item are within the normal range or abnormal. This analysis uses an AI engine and takes into account the user's attributes such as age, gender, and place of residence.

[0380] 5. Device or server: Using the emotion engine, the system analyzes the user's heart rate, facial expression data, etc. For example, if the user is feeling anxious or stressed about the results of a health check, the system can recognize this.

[0381] 6. Server: Based on the analysis results and those of the emotion engine, the server generates an action plan appropriate for the user. For example, if the AST value is high, it will recommend a consultation with an internal medicine doctor and provide a list of appropriate medical institutions in the area. Furthermore, if the user is feeling anxious, it will provide reassuring explanations.

[0382] 7. Server: Sends the generated action plan and analysis results to the user's device.

[0383] 8. Device: The device notifies the user of the received action plan and displays details within the application, using appropriate language that reflects the user's emotional state to help the user feel at ease and take the next step.

[0384] Specific examples

[0385] For example, consider the case where a male user in his 30s has undergone a health checkup in Shibuya Ward, Tokyo, and receives the results. When the user takes a photo of the health checkup result sheet with their smartphone camera and uploads it to the app, the system operates as follows:

[0386] User: Open the smartphone app and use the camera to take a photo of the health check result sheet and upload it.

[0387] Device: The captured image is temporarily saved and sent to the server.

[0388] Server: Receives image data and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L" using OCR.

[0389] Server: The extracted data is passed to an AI engine, which takes into account the user's attributes such as age and gender to detect outliers.

[0390] Device or server: The emotion engine recognizes the user's heart rate and facial expressions to determine whether the user is feeling anxious or stressed about the results.

[0391] Server: Detects when AST levels are above the normal range and generates an action plan that takes the user's anxiety into account. For example, it may recommend a consultation with an internal medicine doctor while providing reassuring explanations. It also generates a list of reputable internal medicine clinics in Shibuya Ward.

[0392] Server: Sends the action plan and clinic information to the user's device.

[0393] On the device: The app will notify the user of the results, and the explanation will be given in a way that is sensitive to the user's emotions, allowing the user to take the next step with peace of mind.

[0394] Example prompts for generative AI models

[0395] "Would you like to analyze your health checkup results based on your age and AST values?"

[0396] "Generate an action plan appropriate for the user based on the health check results."

[0397] "Understand your users' current feelings and reflect them in your action plan."

[0398] These prompts can be used to input the generative AI model, allowing the system to provide a more accurate and tailored service to the user.

[0399] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0400] Step 1: Photograph and upload your health check results

[0401] User: Takes a photo of the health check result sheet with the smartphone camera and uploads the image to the application.

[0402] Specific operation: Launch the app, tap the "Upload diagnosis results" button to activate the camera function, take a picture of the sheet, and tap the "Send" button to upload the image to the app.

[0403] Input: Health check result sheet

[0404] Output: Captured image file

[0405] Step 2: Temporarily save the image and check its quality

[0406] On the device: The device temporarily stores the captured image in its internal storage. It checks the image's file format, resolution, brightness, etc., and if the quality is insufficient, displays a message to the user urging them to take the photo again.

[0407] Specific behavior: The device will save the image in JPEG format and ensure that the resolution is at least 300 dpi. If the quality is insufficient, it will display the message "Retake photo required."

[0408] Input: Captured image

[0409] Output: Save image files to internal storage, quality check results

[0410] Step 3: Sending images

[0411] Terminal: Prepares the image whose quality has been confirmed to be sent to the server, and sends the image data to the server.

[0412] Specific operation: The device sends the saved image file to the server via the API.

[0413] Input: Quality-checked image

[0414] Output: Sending image data to the server

[0415] Step 4: Receiving image data and OCR processing

[0416] Server: The server receives the image data sent from the device and uses OCR software to extract text data from the image.

[0417] Specific operation: The server passes the image file to an OCR engine (e.g., Tesseract OCR) and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L."

[0418] Input: Received image data

[0419] Output: Extracted text data

[0420] Step 5: Analyzing the text data

[0421] Server: Analyzes the extracted text data and determines whether the values ​​for each test item are within the normal range or abnormal. The analysis also takes into account the user's attribute information such as age, gender, and place of residence.

[0422] Specific operation: The server retrieves the normal value range from the database and determines, for example, that an AST value of 40 U / L or higher is abnormal. It also takes into account the user's age and gender information when making a diagnosis.

[0423] Input: Extracted text data, user attribute information

[0424] Output: Analysis result (normal or abnormal)

[0425] Step 6: Analyze emotional state

[0426] Device or server: The emotion engine is used to analyze the user's heart rate, facial expression data, etc., to determine the anxiety or stress the user feels in response to the health check results.

[0427] How it works: The device uses the device's camera and sensors to capture the user's facial expressions and heart rate, and sends the data to the emotion engine. The server analyzes this data and determines whether the user is feeling anxious.

[0428] Input: facial expression data, heart rate, etc.

[0429] Output: Emotion analysis results

[0430] Step 7: Generate an action plan

[0431] Server: Generates an optimal action plan for the user based on the analysis results and emotion recognition results. For example, it recommends a visit to an internal medicine doctor if the AST value is high or if the user feels anxious.

[0432] Specific operation: Based on the analysis results, the server takes into account user attribute information and emotion recognition results and generates a specific action plan such as "recommend visiting an internal medicine doctor" along with information on local medical institutions.

[0433] Input: Analysis results, sentiment analysis results, user attribute information

[0434] Output: Action plan

[0435] Step 8: Communicate your action plan

[0436] Server: Sends the generated action plan and analysis results to the user's device.

[0437] Specific operation: Generates an action plan in JSON format and sends it to the user's device.

[0438] Input: Action plan, analysis results

[0439] Output: Notification data sent to the user's terminal

[0440] Step 9: View your action plan

[0441] Device: The device notifies the user of the received action plan and displays the details within the application.

[0442] What it does: The app will notify the user via push notification that an action plan has been received and will display it in the app along with a detailed explanation. For example, it may say, "We recommend you see a doctor," along with a list of clinics.

[0443] Input: Notification data

[0444] Output: Display and notification to the user

[0445] The above is the specific processing flow of the system program. By showing in detail the data input and output at each processing step, and the data processing and calculations that are performed based on that, the operation of the entire system can be clearly understood.

[0446] (Application example 2)

[0447] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0448] In modern society, there is a growing demand for systems that suggest appropriate action plans based on health checkup results. However, current systems only consider the user's health status and are unable to provide optimal action plans that take into account psychological state and stress. Furthermore, there is a lack of effective means to provide personalized healthcare advertisements based on health status and emotions. This can lead to delays in users taking appropriate actions, further increasing health-related anxiety and stress.

[0449] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring an image of the health checkup results, means for extracting text data from the acquired image, means for analyzing the extracted text data and detecting abnormal values, means for generating an appropriate action plan for the user based on the analysis results, means for notifying the user of the generated action plan, means for recognizing the user's emotions, and means for generating and displaying advertisements based on the user's emotions. This makes it possible to provide an optimal action plan and healthcare advertisements by comprehensively considering the user's health condition and emotions.

[0450] "Health checkup results" are data summarizing the numerical values ​​and evaluations of various tests obtained from the health checkups taken by the user.

[0451] "Means for acquiring images" refers to the technology and devices that collect paper or digital forms of health examination results as digital images using a camera, scanner, etc.

[0452] "Means for extracting text data" refers to techniques and devices that identify textual information from captured images and convert it into digital text.

[0453] The "means for detecting abnormal values" refers to technology and devices that analyze and identify abnormal values ​​and patterns related to the user's health status based on the extracted text data.

[0454] "Means for generating an appropriate action plan" refers to technology and devices that suggest specific actions that users should take based on analyzed health data.

[0455] "Means for notifying" refers to the technology and devices used to communicate the generated action plan or advertisement to the user.

[0456] "Means for recognizing emotions" refers to technology and devices that analyze a user's psychological state from facial expressions, tone of voice, heart rate, etc., and identify emotions.

[0457] "Means for generating and displaying advertisements" refers to technology and devices that generate and display advertisements for appropriate products and services based on the user's health condition and emotions.

[0458] The system for implementing the present invention mainly uses the following hardware and software.

[0459] Hardware

[0460] 1. Smartphone camera: Used to capture images of health checkup results.

[0461] 2. Server: Used to analyze data and generate results.

[0462] software

[0463] 1. OCR library (pytesseract): Used to extract text data from images of medical examination results.

[0464] 2. Image processing library (OpenCV): Used to read and preprocess images.

[0465] 3. HTTP Request Library (Requests): Used to send and receive data.

[0466] 4. Emotion Recognition Engine: Used to analyze the user's emotions.

[0467] System Operation

[0468] User: Takes a photo of the health check result sheet with the smartphone camera and uploads the image to the application, which obtains the image data.

[0469] On the device: The captured image is converted into text data using an OCR library, and image preprocessing is also performed using an image processing library. The converted text data is temporarily stored and sent to the server for further processing.

[0470] Server: Analyzes the received text data and detects outliers. The analysis takes into account the user's attribute information (age, gender, place of residence, etc.). In addition, an emotion recognition engine is used to analyze the user's emotions. For example, it determines whether the user is feeling stressed or anxious.

[0471] Server: Generates an appropriate action plan for the user based on the analysis results. For example, if the liver function value (AST) is high, it recommends a consultation with an internal medicine doctor and provides information on medical institutions in the area. It also generates advertisements based on the user's emotions. If it determines that the user's emotions are unstable, it displays advertisements for products and services that will give them a sense of security.

[0472] Device: The device notifies the user of action plans and advertisements sent from the server. This information is provided in a way that reflects the user's emotions, allowing the user to feel at ease and take the next step.

[0473] Specific examples

[0474] For example, suppose a male user in his 30s takes a photo of his health checkup results sheet and uploads it. Text data is extracted using OCR, and analysis is performed based on that data. The analysis results indicate that his AST level is 45 U / L, which is above the normal range, and based on the user's heart rate and facial expression analysis, it is recognized that he is feeling anxious. As a result, the system generates an action plan recommending a consultation with an internal medicine doctor and provides information about local medical institutions. It also displays advertisements for reassuring herbal teas.

[0475] Prompt Sentence Examples

[0476] "User's health report shows high AST (45 U / L) and they feel anxious."

[0477] In this way, the present invention can provide optimal action plans and advertisements by comprehensively considering the user's health and psychological state.

[0478] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0479] Step 1:

[0480] User: Take a photo of the health check result sheet

[0481] Input: Health check result sheet

[0482] Specific operation: The user takes a photo of the health check result sheet using the smartphone camera. The image is then uploaded to the application.

[0483] Output: Images of the medical examination results

[0484] Step 2:

[0485] Terminal: Extracting text data using OCR

[0486] Input: Images of medical examination results

[0487] Specific operation: The device uses an OCR library (pytesseract) to extract character data from the image, and uses OpenCV to perform image preprocessing (noise removal, grayscale conversion, etc.).

[0488] Output: Extracted text data

[0489] Step 3:

[0490] Terminal: Temporarily saves text data and prepares it for sending to the server

[0491] Input: Extracted text data

[0492] Specific operation: The device temporarily stores the text data extracted by OCR and forms a data structure for sending to the server.

[0493] Output: Text data in a format that can be sent to the server

[0494] Step 4:

[0495] Server: Receiving and analyzing text data

[0496] Input: Text data sent

[0497] Specific operation: The server analyzes the received text data and detects abnormalities based on the user's attribute information (age, gender, place of residence, etc.). It evaluates the health check results using a database and AI model.

[0498] Output: Outlier analysis results

[0499] Step 5:

[0500] Server: Emotion Recognition

[0501] Input: Anomaly analysis results and user information

[0502] What it does: The server uses an emotion recognition engine to analyze the user's emotions from their heart rate and facial expressions, and if stress or anxiety is detected, it records it.

[0503] Output: User's emotional state

[0504] Step 6:

[0505] Server: Generate action plan

[0506] Input: Outlier analysis results and user emotional state

[0507] Specific operation: The server generates an appropriate action plan for the user based on the analysis results and emotional state. For example, if a high AST level is detected, it will recommend a consultation with an internal medicine doctor and provide information on local medical institutions.

[0508] Output: Action plan

[0509] Step 7:

[0510] Server: Generates and displays ads

[0511] Input: Action plan and user's emotional state

[0512] Specific operation: The server generates advertisements for products and services that provide a sense of security based on the user's emotional state, and sends the advertisements along with an action plan to the device.

[0513] Output: Action plan and advertisement

[0514] Step 8:

[0515] Device: Action plan and advertising notifications

[0516] Input: Action Plan and Advertisement

[0517] Specific operation: The device notifies the user of the action plan and advertisement sent from the server, providing information in a way that takes into account the user's emotional state, allowing them to take the next action with peace of mind.

[0518] Output: Final notification and display to the user

[0519] Through these steps, the system can comprehensively evaluate the user's health check results and emotional state, and provide optimal action plans and advertisements.

[0520] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0521] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0522] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0523] [Second embodiment]

[0524] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0525] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0526] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0527] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0528] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0529] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0530] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0531] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0532] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0533] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0534] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0535] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0536] The system of the present invention acquires and analyzes information from images of health checkup results, and presents a specific action plan to the user based on the results. This system is composed of three elements: the user, a terminal, and a server.

[0537] System Overview

[0538] Photographing and uploading health checkup results

[0539] User: Takes a photo of the health check result sheet using the camera on a device such as a smartphone and uploads the image to the application.

[0540] Device: The captured image is temporarily stored on the device and prepared for transmission to the server.

[0541] Receiving and analyzing image data

[0542] Server: Receives image data sent from the device and checks the image quality. Next, it uses OCR (Optical Character Recognition) technology to extract text data from the image. This text data includes test item names such as "AST," "γ-GTP," and "LAP," as well as their numerical values.

[0543] Text data analysis

[0544] Server: Based on the extracted text data, the server analyzes whether the values ​​for each test item are within the normal range or abnormal. This analysis takes into account the user's attribute information (age, gender, place of residence, etc.).

[0545] Action plan generation and notification

[0546] Server: Based on the analysis results, it generates an action plan for the user. For example, if the AST level is high, it recommends a consultation with an internal medicine doctor and provides information on appropriate medical institutions in the area.

[0547] Server: Sends the generated action plan and analysis results to the user's device.

[0548] Device: Notifies the user of the received action plan and displays it in the application.

[0549] Specific scenarios

[0550] For example, let's say a male user in his 30s has undergone a health checkup in Shibuya Ward, Tokyo, and has received the results. When this user takes a photo of the health checkup result sheet with his smartphone camera and uploads it to the app, the system operates as follows:

[0551] User: Open the smartphone app and use the camera to take a photo of the health check result sheet and upload it.

[0552] Device: The captured image is temporarily saved and sent to the server.

[0553] Server: Receives image data and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L" using OCR.

[0554] Server: The extracted data is passed to an AI engine, which takes into account the user's attributes such as age and gender to detect outliers.

[0555] Server: Detects when AST levels are above the normal range and generates an action plan recommending a consultation with an internal medicine doctor. At the same time, it generates a list of reputable internal medicine clinics in Shibuya Ward.

[0556] Server: Sends the action plan and clinic information to the user's device.

[0557] On the device: The app will notify the user and display the results. The user can then check the recommended clinic information within the app and make an appointment if necessary.

[0558] In this way, the system of the present invention supports the user in understanding the health checkup results and intuitively determining the specific actions to be taken next.

[0559] The processing flow will be explained below.

[0560] Step 1:

[0561] The user launches the smartphone app and takes a photo of the health check result sheet with the camera.

[0562] Step 2:

[0563] The user checks the image they have taken and presses the upload button.

[0564] Step 3:

[0565] The terminal temporarily stores the captured image and prepares the data to be sent to the server.

[0566] Step 4:

[0567] The terminal transmits the image data to the server.

[0568] Step 5:

[0569] Check the quality of the image data received by the server and, if necessary, pre-process the image.

[0570] Step 6:

[0571] The server uses OCR technology to extract text data from the image.

[0572] Step 7:

[0573] The server stores the extracted text data in a database.

[0574] Step 8:

[0575] The server passes the text data to the AI ​​engine and begins analysis.

[0576] Step 9:

[0577] Based on the results analyzed by the AI ​​engine, the server determines whether the values ​​of each test item (e.g., AST, γ-GTP, LAP, etc.) are within the normal range or abnormal.

[0578] Step 10:

[0579] Based on the analysis results, the server performs a detailed analysis of the anomalies, taking into account the user's attribute information such as age, gender, and place of residence.

[0580] Step 11:

[0581] The server uses the analysis results to generate an appropriate action plan for the user.

[0582] Step 12:

[0583] Based on the action plan generated by the server, the user decides what action to take next (e.g., visiting a specific medical department or changing lifestyle habits).

[0584] Step 13:

[0585] The server lists recommended medical institutions and medical departments based on the user's location.

[0586] Step 14:

[0587] The server sends the action plan and related information to the user's terminal.

[0588] Step 15:

[0589] The device notifies the user of the action plan and information on recommended medical institutions it has received.

[0590] Step 16:

[0591] The user can check the notification contents within the app and make an appointment with a medical institution if necessary.

[0592] Example 1

[0593] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0594] Currently, many users receive their health checkup results, but find it difficult to understand the contents and take appropriate action. Furthermore, reviewing and interpreting the results takes time, making it difficult to quickly respond to specific abnormal values. Furthermore, results are often judged based on uniform standards without taking into account the user's attribute information, making it difficult to provide the optimal response for each individual user.

[0595] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0596] In this invention, the server includes a means for a user to acquire an image of the health checkup results, a means for extracting text data from the acquired image, a means for analyzing the extracted text data to detect abnormal values, a means for generating an action plan based on the analysis results taking into account the user's attribute information, and a means for notifying the generated action plan and the analysis results from the server to the user's terminal. This allows the user to quickly and accurately understand the health checkup results and take appropriate action. Furthermore, by taking the user's attribute information into account during the analysis, more personalized advice can be provided, enabling responses tailored to each individual's health condition.

[0597] A "user" is a person who uses this system to upload health checkup results and receive analysis results and action plans.

[0598] "Image of medical examination results" refers to image data of sheets or documents containing the results of a medical examination that the user underwent, taken with a smartphone or digital camera.

[0599] "Text data" refers to text information extracted from an image using OCR technology, and includes test item names and their numerical values.

[0600] An "abnormal value" refers to a value outside the normal range in each test item of a health checkup, and is a value that requires the user to take measures such as visiting a medical institution or improving their lifestyle.

[0601] "User attribute information" is data including personal information such as the user's age, gender, and place of residence, and is an element that is taken into consideration when analyzing the health checkup results.

[0602] An "action plan" is specific advice and instructions provided to the user based on the analysis results, including necessary medical visits and lifestyle improvements.

[0603] A "terminal" is a device used by a user to take images of health checkup results and upload them to the system, and refers to electronic devices such as smartphones and tablets.

[0604] The "server" is a computer system that receives image data of health checkup results sent from a user's terminal, analyzes the data, and sends the results and action plans to the user's terminal.

[0605] "Means to confirm quality" refers to a technology that evaluates whether the images obtained from health checkup results are of a quality that can be analyzed, and is a process that automatically checks resolution, brightness, distortion, etc.

[0606] The "means for requesting re-imaging" is a mechanism that notifies the user to re-imaging the medical examination results if the image quality is determined to be insufficient.

[0607] "Information on medical departments and medical institutions" refers to detailed information on medical specialties recommended for the user based on their health condition and the corresponding medical facilities in the area.

[0608] The system of the present invention supports users in obtaining useful information from health checkup results and taking specific actions based on the results. This system is broadly composed of three elements: the user, the terminal, and the server.

[0609] Photographing and uploading health checkup results

[0610] 1. User: The user takes a photo of the health checkup result sheet using a camera on a device such as a smartphone. The image is then uploaded via a dedicated app. It is recommended that the user checks in advance to ensure that the photo has been taken properly.

[0611] Temporarily save image data and prepare for transmission

[0612] 2. Device: The captured image is temporarily stored on the device. During this process, the image's resolution and clarity are checked and it is prepared for transmission to the server. The connection status is also checked and the optimal transmission method is selected.

[0613] Receiving and analyzing image data

[0614] 3. Server: Receives image data sent from the device. Checks the quality of the received image and requests the user to retake the image if it is not satisfactory. Once the image quality is confirmed, extracts text data from the image using OCR (Optical Character Recognition) technology. Specifically, OCR tools such as Google Cloud Vision API and Tesseract can be used.

[0615] 4. Server: Analyzes the extracted text data and determines whether the values ​​for each test item are within the normal range or abnormal. This analysis takes into account the user's attribute information (age, gender, place of residence, etc.). The analysis is carried out using AI analysis engines such as TensorFlow and PyTorch, which are written in the Python language.

[0616] Action plan generation and notification

[0617] 5. Server: Generates a specific action plan for the user based on the analysis results. For example, if the AST value is high, it recommends a consultation with an internal medicine doctor and provides information on local medical institutions.

[0618] 6. Server: Sends the generated action plan and analysis results to the user's device.

[0619] 7. Device: The received action plan is notified to the user and displayed within the application. The user can then review the provided specialist information and recommended actions, and make an appointment or follow up if necessary.

[0620] Specific examples

[0621] For example, let's say a male user in his 30s has undergone a health checkup in Shibuya Ward, Tokyo, and has received the results. When this user takes a photo of the health checkup result sheet with his smartphone camera and uploads it to the app, the system operates as follows:

[0622] User: Opens the smartphone app and uses the camera to take a photo of the health check result sheet and upload it.

[0623] Device: The captured image is temporarily saved and sent to the server.

[0624] Server: Receives image data and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L" using OCR.

[0625] Server: The extracted data is passed to an AI engine, which takes into account the user's attributes such as age and gender to detect outliers.

[0626] Server: Detects when AST levels are above the normal range and generates an action plan recommending a consultation with an internal medicine doctor. At the same time, it generates a list of reputable internal medicine clinics in Shibuya Ward.

[0627] Server: Sends the action plan and clinic information to the user's device.

[0628] On the device: The app will notify the user and display the results. The user can then check the recommended clinic information within the app and make an appointment if necessary.

[0629] Prompt Sentence Examples

[0630] "When a male user in his 30s uploads his health checkup results, if the images are unclear, send him a notification urging him to take a new photo. If his AST value is high, recommend that he see an internal medicine doctor and suggest a reputable clinic in Shibuya Ward."

[0631] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0632] Program processing flow

[0633] Step 1:

[0634] User: Launches the smartphone app and takes a photo of the health check result sheet with the camera.

[0635] Input: Medical examination result sheet of the subject.

[0636] Output: Image file of the health check result sheet.

[0637] Specific operation: The user uses the camera function of the smartphone to take a picture of the health check result sheet, fitting it within the frame. After taking the picture, the user can check the preview of the image.

[0638] Step 2:

[0639] User: Check the captured image and press the upload button within the app.

[0640] Input: Image file of health check result sheet.

[0641] Output: The upload request.

[0642] Specific operation: The user checks whether the image they have taken is accurate, and if there are no problems, presses the "Upload" button in the app.

[0643] Step 3:

[0644] Terminal: Temporarily stores image data and prepares it for transmission to the server.

[0645] Input: Image file of health check result sheet.

[0646] Output: Server ready to send status.

[0647] Specific operation: The device saves the image file to a temporary folder, checks the status of Wi-Fi and mobile data communication, and selects the optimal communication method.

[0648] Step 4:

[0649] Terminal: Sends image data to the server.

[0650] Input: Image file of health check result sheet.

[0651] Output: Image sent to server.

[0652] Specific operation: The device sends the image file to the server's upload API endpoint. When the sending is complete, a success message is received.

[0653] Step 5:

[0654] Server: Receives image data and stores it in a database.

[0655] Input: Image data.

[0656] Output: Image data stored in a database.

[0657] Specific operation: The server receives the uploaded image and saves it in the database from the temporary storage folder.

[0658] Step 6:

[0659] Server: Checks the quality of the image and requests a retake if it is not satisfactory.

[0660] Input: Stored image data.

[0661] Output: Image quality check result, notification of request to retake if inadequate.

[0662] Specific operation: The server automatically checks the image resolution, brightness, distortion, etc., and if they are inappropriate, it notifies the user's device to request a re-take of the image.

[0663] Step 7:

[0664] Server: Extracts text data from images using OCR technology.

[0665] Input: Quality checked image data.

[0666] Output: The extracted text data.

[0667] Specific operation: The server uses the Google Cloud Vision API and Tesseract to extract text data such as "AST," "γ-GTP," and "LAP" from the image.

[0668] Step 8:

[0669] Server: Analyzes the text data and detects outliers taking into account user attribute information.

[0670] Input: Text data, user attribute information.

[0671] Output: The analyzed outlier data.

[0672] Specific operation: The server passes the extracted text data to an AI analysis engine (TensorFlow or PyTorch), which analyzes the data taking into account the user's attribute information such as age, gender, and place of residence.

[0673] Step 9:

[0674] Server: Generates an action plan based on the analysis results.

[0675] Input: Analyzed outlier data, user attribute information.

[0676] Output: Action plan.

[0677] Specific actions: The server generates a specific action plan for the user based on the abnormal values. For example, if the AST value is high, it recommends a consultation with an internal medicine doctor and also provides information on local medical institutions.

[0678] Step 10:

[0679] Server: Sends the generated action plan and analysis results to the user's device.

[0680] Input: Action plan, analysis results.

[0681] Output: Sent to user's device.

[0682] Specific operation: Calls an API that formats the action plan and analysis results and sends them to the user's device.

[0683] Step 11:

[0684] On the device: The user is notified of the received action plan and it is displayed within the app.

[0685] Input: Action plan and analysis results sent from the server.

[0686] Output: Action plans and analysis results displayed to the user.

[0687] What it does: The device analyzes the data it receives and generates a notification, which displays the analysis results and action plan to the user within the application.

[0688] Step 12:

[0689] User: Review the action plan and take the necessary steps.

[0690] Input: Action plan and analysis results displayed within the app.

[0691] Output: Action taken, necessary procedures (e.g., medical appointment).

[0692] Specific operation: The user checks the medical institution information provided in the app and, if necessary, makes an appointment or makes a medical appointment.

[0693] (Application example 1)

[0694] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0695] In modern society, the need to be health-conscious is increasing, but many people find it difficult to properly understand their health checkup results and take appropriate action. It is also difficult to recommend appropriate products and ensure the convenience of purchasing them. For this reason, there is a need for a system that allows users to create specific action plans based on their health checkup results, select appropriate products according to those plans, and easily purchase them in physical stores.

[0696] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0697] In this invention, the server includes means for acquiring an image of the medical checkup result, means for extracting text data from the acquired image, means for analyzing the extracted text data to detect abnormal values, means for generating an appropriate action plan for the user based on the analysis results, means for notifying the user of the generated action plan, and means for suggesting optimal products to the user based on the analysis results of the medical checkup result. This enables the user to easily understand the medical checkup result and take appropriate action, and provides an environment in which products can be suggested and purchased seamlessly according to the action plan.

[0698] "Image of health checkup results" is an image file containing health data obtained when a user undergoes a health checkup.

[0699] "Text data" is text information extracted from an image using OCR technology.

[0700] An "abnormal value" refers to a value in a test item in a health checkup result that exceeds the normal range.

[0701] An "action plan" is a plan of specific actions that the user should take based on the analysis of the health check results.

[0702] "Notification" is a means of communicating information to inform the user of the generated action plan.

[0703] "User attribute information" refers to information that indicates individual characteristics of a user, such as age, gender, and place of residence.

[0704] A "medical department" is a medical department at a medical institution that specializes in treating specific diseases or parts of the body.

[0705] "Medical institutions" are facilities that provide medical services, such as hospitals and clinics.

[0706] "Products" are items such as food, supplements, and daily necessities that are recommended to users based on the analysis of their health checkup results.

[0707] A "physical store" is a physical sales location where users can actually visit and purchase products.

[0708] "Navigation" is a means of guidance to guide a user to a destination.

[0709] An embodiment of the present invention will be described below: The system is composed of three elements: a user, a terminal, and a server.

[0710] Photographing and uploading health checkup results

[0711] Users take a photo of their health checkup result sheet using a device such as a smartphone and upload the image to the application.

[0712] The terminal temporarily stores the captured image and prepares it for transmission to the server.

[0713] Receiving and analyzing image data

[0714] The server receives the image data sent from the device and checks the image quality. It then uses OCR (Optical Character Recognition) technology to extract text data from the image. This text data includes test item names such as "AST," "γ-GTP," and "LAP," as well as their numerical values.

[0715] Text data analysis

[0716] The server analyzes the extracted text data to determine whether the values ​​for each test item are within the normal range or are abnormal, taking into account the user's attribute information (age, gender, place of residence, etc.).

[0717] Action plan generation and notification

[0718] Based on the analysis results, the server generates an action plan for the user. For example, if the AST level is high, it may recommend a consultation with an internal medicine doctor and provide information on appropriate medical institutions in the area.

[0719] The generated action plan is then notified to the user and displayed within the application, where the user can check the recommended medical institutions and product information and make purchases or reservations as necessary.

[0720] Product suggestions and navigation

[0721] The server then recommends optimal products to users based on the analysis of their health checkup results. For example, foods and supplements that are useful for improving health. Furthermore, the server provides a navigation function to make it easy to purchase these products in physical stores. Users can use the application to find and purchase the recommended products at nearby physical stores.

[0722] Technology used

[0723] In the present invention, the following hardware and software are used.

[0724] Hardware:

[0725] Smartphone (with camera function)

[0726] Servers (with high-performance processors and storage)

[0727] Physical store sales system

[0728] software:

[0729] pytesseract: OCR engine (extracts text from images)

[0730] Pillow: Image Processing Library (PIL)

[0731] requests: Communication with external API (store information acquisition)

[0732] JSON: Data Format

[0733] Specific examples

[0734] For example, if a male user in his 30s receives a medical checkup result showing an AST level of 45 U / L and a γ-GTP level of 75 U / L, he takes a photo of the medical checkup result sheet with his smartphone and uploads it to the app. The following process will occur:

[0735] 1. The device temporarily saves the image and sends it to the server.

[0736] 2. The server receives the image data and uses OCR to convert it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L."

[0737] 3. The server passes the extracted data to an AI engine, which takes into account the user's attribute information such as age and gender to detect outliers.

[0738] 4. The server detects that the AST level is above the normal range and suggests products to help improve health, along with an action plan that recommends consulting an internal medicine doctor.

[0739] 5. Users can check recommended products and medical institutions within the app and make reservations or purchases as needed.

[0740] Prompt Sentence Examples

[0741] Below is an example prompt to input to the generative AI model.

[0742] We provide images of health checkup results. Extract important health data from these images and recommend the best products for users based on the analysis results.

[0743] User information: Male, 30 years old, living in Tokyo

[0744] Health data: AST: 45 U / L, γ-GTP: 75 U / L

[0745] Analysis and recommendations: Recommendations for medical consultation, suggestions for places to buy low-fat yogurt, etc.

[0746] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0747] Step 1:

[0748] Users take a photo of their health checkup result sheet with their smartphone and upload the image to the application. At this time, users open the application and use the camera function to capture an image of the health checkup result sheet.

[0749] Input: Image of health check result sheet

[0750] Output: Temporarily saved image file

[0751] Specific operation: The user launches the camera app, takes a photo of the health check result sheet, and uploads the image to the application. The application temporarily saves the image file on the device.

[0752] Step 2:

[0753] The device temporarily stores the captured image and prepares it for transmission to the server. At this time, an application on the device converts the image file into the data format required for transmission to the server.

[0754] Input: Temporarily saved image file

[0755] Output: Image data formatted for sending to the server

[0756] Specific operation: The application on the terminal converts the image file into an appropriate format (e.g., JPEG or PNG) and sends it to the server over the network.

[0757] Step 3:

[0758] The server receives the image data sent from the terminal. At this time, the server checks the quality of the image and requests retransmission if it is unclear.

[0759] Input: Image data sent from the device

[0760] Output: Quality-checked image data

[0761] Specific operation: The server receives the image, evaluates the quality using an image processing algorithm, and if the quality is not satisfactory, asks the user to take the photo again.

[0762] Step 4:

[0763] The server extracts text data from the image using OCR technology (e.g., pytesseract), where the server inputs the image into the OCR engine and obtains the text information.

[0764] Input: Quality-checked image data

[0765] Output: Extracted text data

[0766] Specific operation: The server analyzes the image using pytesseract and extracts text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L."

[0767] Step 5:

[0768] The server analyzes the extracted text data to detect outliers, using an AI engine and user attribute information.

[0769] Input: Extracted text data, user attribute information

[0770] Output: Outlier detection results

[0771] Specific operation: The server inputs the extracted numerical data into an AI engine, taking into account attribute information such as age, gender, and place of residence, and compares it with normal values. If an abnormal value is detected, detailed information about it is generated.

[0772] Step 6:

[0773] The server generates an action plan based on the analysis results, and the AI ​​engine assembles data to suggest appropriate actions and products to the user.

[0774] Input: Outlier detection results

[0775] Output: Generated action plan

[0776] Specific operation: Based on the results of the abnormal value detection, the server generates an action plan such as "We recommend that you visit an internal medicine doctor" or "We recommend that you consume low-fat yogurt."

[0777] Step 7:

[0778] The server notifies the user of the generated action plan, which can be done via push notification or in-app display.

[0779] Input: Generated action plan

[0780] Output: User informed action plan

[0781] Specific operation: The server sends the generated action plan to the terminal, and the terminal displays the received notification to the user.

[0782] Step 8:

[0783] Users can check recommended products and medical institutions within the application and are guided to purchase products at physical stores. At this time, users can refer to the in-app map function and store information.

[0784] Input: Notified action plan, recommended product information

[0785] Output: Purchased items, medical appointments

[0786] Specific operation: The user checks the product information and medical institution information displayed within the app and navigates to the nearest store to purchase the product.

[0787] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0788] The system of the present invention acquires and analyzes information from images of health checkup results and presents a specific action plan to the user based on the results. By incorporating an emotion engine that recognizes the user's emotions, the system provides a more appropriate action plan that also takes into account the user's psychological state. The system is comprised of three elements: the user, the terminal, and the server.

[0789] System Overview

[0790] Photographing and uploading health checkup results

[0791] User: Takes a photo of the health check result sheet using the camera on a device such as a smartphone and uploads the image to the application.

[0792] Device: The captured image is temporarily stored on the device and prepared for transmission to the server.

[0793] Receiving and analyzing image data

[0794] Server: Receives image data sent from the device and checks the image quality. Next, it uses OCR (Optical Character Recognition) technology to extract text data from the image. This text data includes test item names such as "AST," "γ-GTP," and "LAP," as well as their numerical values.

[0795] Text data analysis

[0796] Server: Based on the extracted text data, the server analyzes whether the values ​​for each test item are within the normal range or abnormal. This analysis takes into account the user's attribute information (age, gender, place of residence, etc.).

[0797] Emotion recognition by emotion engine

[0798] Device or server: Analyze the user's current emotions using the emotion engine. For example, if the results of a health checkup cause stress or anxiety to the user, the emotion engine can recognize this.

[0799] Action plan generation and notification

[0800] Server: Based on the analysis results, taking into account the user's attribute information and the analysis results of the emotion engine, the server generates an action plan for the user. For example, if the AST value is high, it will recommend a consultation with an internal medicine doctor and provide information on appropriate medical institutions in the area. If the user is feeling anxious, it will include explanations and responses to reassure them.

[0801] Server: Sends the generated action plan and analysis results to the user's device.

[0802] On the device, the received action plan is notified to the user and displayed in the application, with the language and expressions adjusted to match the user's emotions.

[0803] Specific scenarios

[0804] For example, let's say a male user in his 30s has undergone a health checkup in Shibuya Ward, Tokyo, and has received the results. When this user takes a photo of the health checkup result sheet with his smartphone camera and uploads it to the app, the system operates as follows:

[0805] Photographing and analyzing health checkup results

[0806] User: Open the smartphone app and use the camera to take a photo of the health check result sheet and upload it.

[0807] Device: The captured image is temporarily saved and sent to the server.

[0808] Server: Receives image data and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L" using OCR.

[0809] Server: The extracted data is passed to an AI engine, which takes into account the user's attributes such as age and gender to detect outliers.

[0810] Emotion recognition and action plan generation

[0811] Device or server: The emotion engine recognizes the user's heart rate and facial expressions to determine whether the user is feeling anxious or stressed about the results.

[0812] Server: Detects when AST levels are above the normal range and generates an action plan that takes the user's anxiety into account. For example, it may recommend a consultation with an internal medicine doctor while providing reassuring explanations. It also generates a list of reputable internal medicine clinics in Shibuya Ward.

[0813] Communicating results and encouraging action

[0814] Server: Sends the action plan and clinic information to the user's device.

[0815] On the device: The app will notify the user of the results, and the explanation will be given in a way that is sensitive to the user's emotions, allowing the user to take the next step with peace of mind.

[0816] In this way, the system of the present invention takes into consideration the psychological state felt by the user through the health checkup results, and supports the user in intuitively deciding what specific action to take next.

[0817] The processing flow will be explained below.

[0818] Step 1:

[0819] The user launches the smartphone app and takes a photo of the health check result sheet with the camera.

[0820] Step 2:

[0821] The user checks the image they have taken and presses the upload button.

[0822] Step 3:

[0823] The terminal temporarily stores the captured image and prepares the data to be sent to the server.

[0824] Step 4:

[0825] The terminal transmits the image data to the server.

[0826] Step 5:

[0827] Check the quality of the image data received by the server and, if necessary, pre-process the image.

[0828] Step 6:

[0829] The server uses OCR technology to extract text data from the image.

[0830] Step 7:

[0831] The server stores the extracted text data in a database.

[0832] Step 8:

[0833] The server passes the text data to the AI ​​engine and begins analysis.

[0834] Step 9:

[0835] Based on the results analyzed by the AI ​​engine, the server determines whether the values ​​of each test item (e.g., AST, γ-GTP, LAP, etc.) are within the normal range or abnormal.

[0836] Step 10:

[0837] Based on the analysis results, the server performs a detailed analysis of the anomalies, taking into account the user's attribute information such as age, gender, and place of residence.

[0838] Step 11:

[0839] The emotion engine uses data acquired by the server from the user's terminal or other devices to analyze the user's emotions, for example, detecting anxiety or stress by analyzing voice tone and facial expressions.

[0840] Step 12:

[0841] The server adjusts the action plan based on the results of the emotion engine, adding more reassuring explanations and support if the user feels anxious.

[0842] Step 13:

[0843] The server generates an action plan for the user based on the analysis results and emotion recognition results.

[0844] Step 14:

[0845] The server lists information on recommended medical institutions and medical departments based on the user's location.

[0846] Step 15:

[0847] The server sends the action plan and related information to the user's terminal.

[0848] Step 16:

[0849] The device notifies the user of the action plan and information on recommended medical institutions it has received.

[0850] Step 17:

[0851] The user can check the notification contents within the app and make an appointment with a medical institution if necessary.

[0852] Step 18:

[0853] The device uses an emotion engine to obtain user feedback and reflect it in future analysis and action plan generation.

[0854] Example 2

[0855] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0856] Conventional health checkup result analysis systems required users to understand the results and decide for themselves what action to take next, which often led to anxiety and stress in users. Furthermore, when abnormal values ​​were detected, the systems lacked specific instructions for appropriate action or the ability to provide psychological support. This made it difficult for users to move on to the next action quickly and with peace of mind.

[0857] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring an image of the medical examination result, a means for extracting text data from the acquired image, a means for analyzing the extracted text data and detecting abnormal values, a means for analyzing the emotional state of the user, a means for generating an appropriate action plan for the user based on the analysis results, and a means for notifying the user of the generated action plan. This allows an appropriate action plan to be proposed taking into consideration the psychological state of the user, allowing the user to proceed with the next action with peace of mind.

[0858] "Image of medical examination results" refers to a digital image of the sheet containing the medical examination results received by the user.

[0859] "Text data" refers to the text information extracted from images of health checkup results, including numerical values ​​and names of test items.

[0860] An "abnormal value" is a value in which the numerical value of the analyzed text data deviates from a preset normal range.

[0861] An "action plan" is a specific course of action that is suggested to the user based on the analysis results and the user's attribute information.

[0862] "Emotional state" refers to the psychological state the user is currently feeling, including emotions such as stress and anxiety.

[0863] The "analysis results" are the comprehensive results of the analysis performed based on the text data of the health checkup results.

[0864] "User attribute information" refers to information related to an individual, such as the user's age, gender, and place of residence.

[0865] "Information on medical departments and medical institutions" is detailed data including information on medical facilities and medical departments recommended to users.

[0866] "Means for acquiring images" refers to a method of acquiring the health check result sheet as a digital image using a camera such as a smartphone.

[0867] "Means for extracting text data" refers to the process of using OCR technology to read character information from an image and extract it as text data.

[0868] The "means for detecting abnormal values ​​through analysis" is a method for analyzing extracted text data and detecting values ​​outside the standard range.

[0869] The "means for notifying the user" is a method for sending the generated action plan to the user's terminal and notifying the user.

[0870] The system of the present invention analyzes images obtained from health checkup results and presents a specific action plan to the user. This system operates with three elements: the user, the terminal, and the server. It also has a function to analyze the user's emotional state, which can reduce the psychological burden.

[0871] Hardware and Software Used

[0872] Device: Smartphone (e.g. iPhone or Android smartphone)

[0873] Server: Cloud server (e.g. AWS, GCP)

[0874] OCR software: Tesseract OCR

[0875] Emotion engine: Emotion analysis software (e.g., Affectiva, Emotion API)

[0876] Explanation of program processing

[0877] 1. User: The user takes a photo of the health check result sheet with their smartphone camera and uploads the image to the application by tapping the "Upload Diagnosis Results" button in the app.

[0878] 2. Device: The device temporarily stores the captured image in its internal storage and checks the image's file format, resolution, brightness, etc. If the image quality is insufficient, a message is displayed prompting the user to take a new photo. If the image quality is confirmed to be sufficient, it is sent to the server.

[0879] 3. Server: The server receives the image data sent from the device and extracts text data from the image using OCR software (e.g., Tesseract OCR). In this example, items such as "AST: 45 U / L" and "γ-GTP: 75 U / L" are extracted.

[0880] 4. Server: Analyzes the extracted text data and determines whether the values ​​for each test item are within the normal range or abnormal. This analysis uses an AI engine and takes into account the user's attributes such as age, gender, and place of residence.

[0881] 5. Device or server: Using the emotion engine, the system analyzes the user's heart rate, facial expression data, etc. For example, if the user is feeling anxious or stressed about the results of a health check, the system can recognize this.

[0882] 6. Server: Based on the analysis results and those of the emotion engine, the server generates an action plan appropriate for the user. For example, if the AST value is high, it will recommend a consultation with an internal medicine doctor and provide a list of appropriate medical institutions in the area. Furthermore, if the user is feeling anxious, it will provide reassuring explanations.

[0883] 7. Server: Sends the generated action plan and analysis results to the user's device.

[0884] 8. Device: The device notifies the user of the received action plan and displays details within the application, using appropriate language that reflects the user's emotional state to help the user feel at ease and take the next step.

[0885] Specific examples

[0886] For example, consider the case where a male user in his 30s has undergone a health checkup in Shibuya Ward, Tokyo, and receives the results. When the user takes a photo of the health checkup result sheet with their smartphone camera and uploads it to the app, the system operates as follows:

[0887] User: Open the smartphone app and use the camera to take a photo of the health check result sheet and upload it.

[0888] Device: The captured image is temporarily saved and sent to the server.

[0889] Server: Receives image data and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L" using OCR.

[0890] Server: The extracted data is passed to an AI engine, which takes into account the user's attributes such as age and gender to detect outliers.

[0891] Device or server: The emotion engine recognizes the user's heart rate and facial expressions to determine whether the user is feeling anxious or stressed about the results.

[0892] Server: Detects when AST levels are above the normal range and generates an action plan that takes the user's anxiety into account. For example, it may recommend a consultation with an internal medicine doctor while providing reassuring explanations. It also generates a list of reputable internal medicine clinics in Shibuya Ward.

[0893] Server: Sends the action plan and clinic information to the user's device.

[0894] On the device: The app will notify the user of the results, and the explanation will be given in a way that is sensitive to the user's emotions, allowing the user to take the next step with peace of mind.

[0895] Example prompts for generative AI models

[0896] "Would you like to analyze your health checkup results based on your age and AST values?"

[0897] "Generate an action plan appropriate for the user based on the health check results."

[0898] "Understand your users' current feelings and reflect them in your action plan."

[0899] These prompts can be used to input the generative AI model, allowing the system to provide a more accurate and tailored service to the user.

[0900] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0901] Step 1: Photograph and upload your health check results

[0902] User: Takes a photo of the health check result sheet with the smartphone camera and uploads the image to the application.

[0903] Specific operation: Launch the app, tap the "Upload diagnosis results" button to activate the camera function, take a picture of the sheet, and tap the "Send" button to upload the image to the app.

[0904] Input: Health check result sheet

[0905] Output: Captured image file

[0906] Step 2: Temporarily save the image and check its quality

[0907] On the device: The device temporarily stores the captured image in its internal storage. It checks the image's file format, resolution, brightness, etc., and if the quality is insufficient, displays a message to the user urging them to take the photo again.

[0908] Specific behavior: The device will save the image in JPEG format and ensure that the resolution is at least 300 dpi. If the quality is insufficient, it will display the message "Retake photo required."

[0909] Input: Captured image

[0910] Output: Save image files to internal storage, quality check results

[0911] Step 3: Sending images

[0912] Terminal: Prepares the image whose quality has been confirmed to be sent to the server, and sends the image data to the server.

[0913] Specific operation: The device sends the saved image file to the server via the API.

[0914] Input: Quality-checked image

[0915] Output: Sending image data to the server

[0916] Step 4: Receiving image data and OCR processing

[0917] Server: The server receives the image data sent from the device and uses OCR software to extract text data from the image.

[0918] Specific operation: The server passes the image file to an OCR engine (e.g., Tesseract OCR) and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L."

[0919] Input: Received image data

[0920] Output: Extracted text data

[0921] Step 5: Analyzing the text data

[0922] Server: Analyzes the extracted text data and determines whether the values ​​for each test item are within the normal range or abnormal. The analysis also takes into account the user's attribute information such as age, gender, and place of residence.

[0923] Specific operation: The server retrieves the normal value range from the database and determines, for example, that an AST value of 40 U / L or higher is abnormal. It also takes into account the user's age and gender information when making a diagnosis.

[0924] Input: Extracted text data, user attribute information

[0925] Output: Analysis result (normal or abnormal)

[0926] Step 6: Analyze emotional state

[0927] Device or server: The emotion engine is used to analyze the user's heart rate, facial expression data, etc., to determine the anxiety or stress the user feels in response to the health check results.

[0928] How it works: The device uses the device's camera and sensors to capture the user's facial expressions and heart rate, and sends the data to the emotion engine. The server analyzes this data and determines whether the user is feeling anxious.

[0929] Input: facial expression data, heart rate, etc.

[0930] Output: Emotion analysis results

[0931] Step 7: Generate an action plan

[0932] Server: Generates an optimal action plan for the user based on the analysis results and emotion recognition results. For example, it recommends a visit to an internal medicine doctor if the AST value is high or if the user feels anxious.

[0933] Specific operation: Based on the analysis results, the server takes into account user attribute information and emotion recognition results and generates a specific action plan such as "recommend visiting an internal medicine doctor" along with information on local medical institutions.

[0934] Input: Analysis results, sentiment analysis results, user attribute information

[0935] Output: Action plan

[0936] Step 8: Communicate your action plan

[0937] Server: Sends the generated action plan and analysis results to the user's device.

[0938] Specific operation: Generates an action plan in JSON format and sends it to the user's device.

[0939] Input: Action plan, analysis results

[0940] Output: Notification data sent to the user's terminal

[0941] Step 9: View your action plan

[0942] Device: The device notifies the user of the received action plan and displays the details within the application.

[0943] What it does: The app will notify the user via push notification that an action plan has been received and will display it in the app along with a detailed explanation. For example, it may say, "We recommend you see a doctor," along with a list of clinics.

[0944] Input: Notification data

[0945] Output: Display and notification to the user

[0946] The above is the specific processing flow of the system program. By showing in detail the data input and output at each processing step, and the data processing and calculations that are performed based on that, the operation of the entire system can be clearly understood.

[0947] (Application example 2)

[0948] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0949] In modern society, there is a growing demand for systems that suggest appropriate action plans based on health checkup results. However, current systems only consider the user's health status and are unable to provide optimal action plans that take into account psychological state and stress. Furthermore, there is a lack of effective means to provide personalized healthcare advertisements based on health status and emotions. This can lead to delays in users taking appropriate actions, further increasing health-related anxiety and stress.

[0950] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring an image of the health checkup results, means for extracting text data from the acquired image, means for analyzing the extracted text data and detecting abnormal values, means for generating an appropriate action plan for the user based on the analysis results, means for notifying the user of the generated action plan, means for recognizing the user's emotions, and means for generating and displaying advertisements based on the user's emotions. This makes it possible to provide an optimal action plan and healthcare advertisements by comprehensively considering the user's health condition and emotions.

[0951] "Health checkup results" are data summarizing the numerical values ​​and evaluations of various tests obtained from the health checkups taken by the user.

[0952] "Means for acquiring images" refers to the technology and devices that collect paper or digital forms of health examination results as digital images using a camera, scanner, etc.

[0953] "Means for extracting text data" refers to techniques and devices that identify textual information from captured images and convert it into digital text.

[0954] The "means for detecting abnormal values" refers to technology and devices that analyze and identify abnormal values ​​and patterns related to the user's health status based on the extracted text data.

[0955] "Means for generating an appropriate action plan" refers to technology and devices that suggest specific actions that users should take based on analyzed health data.

[0956] "Means for notifying" refers to the technology and devices used to communicate the generated action plan or advertisement to the user.

[0957] "Means for recognizing emotions" refers to technology and devices that analyze a user's psychological state from facial expressions, tone of voice, heart rate, etc., and identify emotions.

[0958] "Means for generating and displaying advertisements" refers to technology and devices that generate and display advertisements for appropriate products and services based on the user's health condition and emotions.

[0959] The system for implementing the present invention mainly uses the following hardware and software.

[0960] Hardware

[0961] 1. Smartphone camera: Used to capture images of health checkup results.

[0962] 2. Server: Used to analyze data and generate results.

[0963] software

[0964] 1. OCR library (pytesseract): Used to extract text data from images of medical examination results.

[0965] 2. Image processing library (OpenCV): Used to read and preprocess images.

[0966] 3. HTTP Request Library (Requests): Used to send and receive data.

[0967] 4. Emotion Recognition Engine: Used to analyze the user's emotions.

[0968] System Operation

[0969] User: Takes a photo of the health check result sheet with the smartphone camera and uploads the image to the application, which obtains the image data.

[0970] On the device: The captured image is converted into text data using an OCR library, and image preprocessing is also performed using an image processing library. The converted text data is temporarily stored and sent to the server for further processing.

[0971] Server: Analyzes the received text data and detects outliers. The analysis takes into account the user's attribute information (age, gender, place of residence, etc.). In addition, an emotion recognition engine is used to analyze the user's emotions. For example, it determines whether the user is feeling stressed or anxious.

[0972] Server: Generates an appropriate action plan for the user based on the analysis results. For example, if the liver function value (AST) is high, it recommends a consultation with an internal medicine doctor and provides information on medical institutions in the area. It also generates advertisements based on the user's emotions. If it determines that the user's emotions are unstable, it displays advertisements for products and services that will give them a sense of security.

[0973] Device: The device notifies the user of action plans and advertisements sent from the server. This information is provided in a way that reflects the user's emotions, allowing the user to feel at ease and take the next step.

[0974] Specific examples

[0975] For example, suppose a male user in his 30s takes a photo of his health checkup results sheet and uploads it. Text data is extracted using OCR, and analysis is performed based on that data. The analysis results indicate that his AST level is 45 U / L, which is above the normal range, and based on the user's heart rate and facial expression analysis, it is recognized that he is feeling anxious. As a result, the system generates an action plan recommending a consultation with an internal medicine doctor and provides information about local medical institutions. It also displays advertisements for reassuring herbal teas.

[0976] Prompt Sentence Examples

[0977] "User's health report shows high AST (45 U / L) and they feel anxious."

[0978] In this way, the present invention can provide optimal action plans and advertisements by comprehensively considering the user's health and psychological state.

[0979] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0980] Step 1:

[0981] User: Take a photo of the health check result sheet

[0982] Input: Health check result sheet

[0983] Specific operation: The user takes a photo of the health check result sheet using the smartphone camera. The image is then uploaded to the application.

[0984] Output: Images of the medical examination results

[0985] Step 2:

[0986] Terminal: Extracting text data using OCR

[0987] Input: Images of medical examination results

[0988] Specific operation: The device uses an OCR library (pytesseract) to extract character data from the image, and uses OpenCV to perform image preprocessing (noise removal, grayscale conversion, etc.).

[0989] Output: Extracted text data

[0990] Step 3:

[0991] Terminal: Temporarily saves text data and prepares it for sending to the server

[0992] Input: Extracted text data

[0993] Specific operation: The device temporarily stores the text data extracted by OCR and forms a data structure for sending to the server.

[0994] Output: Text data in a format that can be sent to the server

[0995] Step 4:

[0996] Server: Receiving and analyzing text data

[0997] Input: Text data sent

[0998] Specific operation: The server analyzes the received text data and detects abnormalities based on the user's attribute information (age, gender, place of residence, etc.). It evaluates the health check results using a database and AI model.

[0999] Output: Outlier analysis results

[1000] Step 5:

[1001] Server: Emotion Recognition

[1002] Input: Anomaly analysis results and user information

[1003] What it does: The server uses an emotion recognition engine to analyze the user's emotions from their heart rate and facial expressions, and if stress or anxiety is detected, it records it.

[1004] Output: User's emotional state

[1005] Step 6:

[1006] Server: Generate action plan

[1007] Input: Outlier analysis results and user emotional state

[1008] Specific operation: The server generates an appropriate action plan for the user based on the analysis results and emotional state. For example, if a high AST level is detected, it will recommend a consultation with an internal medicine doctor and provide information on local medical institutions.

[1009] Output: Action plan

[1010] Step 7:

[1011] Server: Generates and displays ads

[1012] Input: Action plan and user's emotional state

[1013] Specific operation: The server generates advertisements for products and services that provide a sense of security based on the user's emotional state, and sends the advertisements along with an action plan to the device.

[1014] Output: Action plan and advertisement

[1015] Step 8:

[1016] Device: Action plan and advertising notifications

[1017] Input: Action Plan and Advertisement

[1018] Specific operation: The device notifies the user of the action plan and advertisement sent from the server, providing information in a way that takes into account the user's emotional state, allowing them to take the next action with peace of mind.

[1019] Output: Final notification and display to the user

[1020] Through these steps, the system can comprehensively evaluate the user's health check results and emotional state, and provide optimal action plans and advertisements.

[1021] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1022] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1023] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1024] [Third embodiment]

[1025] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1026] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1028] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1029] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1030] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1032] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1033] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1035] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1036] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1037] The system of the present invention acquires and analyzes information from images of health checkup results, and presents a specific action plan to the user based on the results. This system is composed of three elements: the user, a terminal, and a server.

[1038] System Overview

[1039] Photographing and uploading health checkup results

[1040] User: Takes a photo of the health check result sheet using the camera on a device such as a smartphone and uploads the image to the application.

[1041] Device: The captured image is temporarily stored on the device and prepared for transmission to the server.

[1042] Receiving and analyzing image data

[1043] Server: Receives image data sent from the device and checks the image quality. Next, it uses OCR (Optical Character Recognition) technology to extract text data from the image. This text data includes test item names such as "AST," "γ-GTP," and "LAP," as well as their numerical values.

[1044] Text data analysis

[1045] Server: Based on the extracted text data, the server analyzes whether the values ​​for each test item are within the normal range or abnormal. This analysis takes into account the user's attribute information (age, gender, place of residence, etc.).

[1046] Action plan generation and notification

[1047] Server: Based on the analysis results, it generates an action plan for the user. For example, if the AST level is high, it recommends a consultation with an internal medicine doctor and provides information on appropriate medical institutions in the area.

[1048] Server: Sends the generated action plan and analysis results to the user's device.

[1049] Device: Notifies the user of the received action plan and displays it in the application.

[1050] Specific scenarios

[1051] For example, let's say a male user in his 30s has undergone a health checkup in Shibuya Ward, Tokyo, and has received the results. When this user takes a photo of the health checkup result sheet with his smartphone camera and uploads it to the app, the system operates as follows:

[1052] User: Open the smartphone app and use the camera to take a photo of the health check result sheet and upload it.

[1053] Device: The captured image is temporarily saved and sent to the server.

[1054] Server: Receives image data and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L" using OCR.

[1055] Server: The extracted data is passed to an AI engine, which takes into account the user's attributes such as age and gender to detect outliers.

[1056] Server: Detects when AST levels are above the normal range and generates an action plan recommending a consultation with an internal medicine doctor. At the same time, it generates a list of reputable internal medicine clinics in Shibuya Ward.

[1057] Server: Sends the action plan and clinic information to the user's device.

[1058] On the device: The app will notify the user and display the results. The user can then check the recommended clinic information within the app and make an appointment if necessary.

[1059] In this way, the system of the present invention supports the user in understanding the health checkup results and intuitively determining the specific actions to be taken next.

[1060] The processing flow will be explained below.

[1061] Step 1:

[1062] The user launches the smartphone app and takes a photo of the health check result sheet with the camera.

[1063] Step 2:

[1064] The user checks the image they have taken and presses the upload button.

[1065] Step 3:

[1066] The terminal temporarily stores the captured image and prepares the data to be sent to the server.

[1067] Step 4:

[1068] The terminal transmits the image data to the server.

[1069] Step 5:

[1070] Check the quality of the image data received by the server and, if necessary, pre-process the image.

[1071] Step 6:

[1072] The server uses OCR technology to extract text data from the image.

[1073] Step 7:

[1074] The server stores the extracted text data in a database.

[1075] Step 8:

[1076] The server passes the text data to the AI ​​engine and begins analysis.

[1077] Step 9:

[1078] Based on the results analyzed by the AI ​​engine, the server determines whether the values ​​of each test item (e.g., AST, γ-GTP, LAP, etc.) are within the normal range or abnormal.

[1079] Step 10:

[1080] Based on the analysis results, the server performs a detailed analysis of the anomalies, taking into account the user's attribute information such as age, gender, and place of residence.

[1081] Step 11:

[1082] The server uses the analysis results to generate an appropriate action plan for the user.

[1083] Step 12:

[1084] Based on the action plan generated by the server, the user decides what action to take next (e.g., visiting a specific medical department or changing lifestyle habits).

[1085] Step 13:

[1086] The server lists recommended medical institutions and medical departments based on the user's location.

[1087] Step 14:

[1088] The server sends the action plan and related information to the user's terminal.

[1089] Step 15:

[1090] The device notifies the user of the action plan and information on recommended medical institutions it has received.

[1091] Step 16:

[1092] The user can check the notification contents within the app and make an appointment with a medical institution if necessary.

[1093] Example 1

[1094] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1095] Currently, many users receive their health checkup results, but find it difficult to understand the contents and take appropriate action. Furthermore, reviewing and interpreting the results takes time, making it difficult to quickly respond to specific abnormal values. Furthermore, results are often judged based on uniform standards without taking into account the user's attribute information, making it difficult to provide the optimal response for each individual user.

[1096] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1097] In this invention, the server includes a means for a user to acquire an image of the health checkup results, a means for extracting text data from the acquired image, a means for analyzing the extracted text data to detect abnormal values, a means for generating an action plan based on the analysis results taking into account the user's attribute information, and a means for notifying the generated action plan and the analysis results from the server to the user's terminal. This allows the user to quickly and accurately understand the health checkup results and take appropriate action. Furthermore, by taking the user's attribute information into account during the analysis, more personalized advice can be provided, enabling responses tailored to each individual's health condition.

[1098] A "user" is a person who uses this system to upload health checkup results and receive analysis results and action plans.

[1099] "Image of medical examination results" refers to image data of sheets or documents containing the results of a medical examination that the user underwent, taken with a smartphone or digital camera.

[1100] "Text data" refers to text information extracted from an image using OCR technology, and includes test item names and their numerical values.

[1101] An "abnormal value" refers to a value outside the normal range in each test item of a health checkup, and is a value that requires the user to take measures such as visiting a medical institution or improving their lifestyle.

[1102] "User attribute information" is data including personal information such as the user's age, gender, and place of residence, and is an element that is taken into consideration when analyzing the health checkup results.

[1103] An "action plan" is specific advice and instructions provided to the user based on the analysis results, including necessary medical visits and lifestyle improvements.

[1104] A "terminal" is a device used by a user to take images of health checkup results and upload them to the system, and refers to electronic devices such as smartphones and tablets.

[1105] The "server" is a computer system that receives image data of health checkup results sent from a user's terminal, analyzes the data, and sends the results and action plans to the user's terminal.

[1106] "Means to confirm quality" refers to a technology that evaluates whether the images obtained from health checkup results are of a quality that can be analyzed, and is a process that automatically checks resolution, brightness, distortion, etc.

[1107] The "means for requesting re-imaging" is a mechanism that notifies the user to re-imaging the medical examination results if the image quality is determined to be insufficient.

[1108] "Information on medical departments and medical institutions" refers to detailed information on medical specialties recommended for the user based on their health condition and the corresponding medical facilities in the area.

[1109] The system of the present invention supports users in obtaining useful information from health checkup results and taking specific actions based on the results. This system is broadly composed of three elements: the user, the terminal, and the server.

[1110] Photographing and uploading health checkup results

[1111] 1. User: The user takes a photo of the health checkup result sheet using a camera on a device such as a smartphone. The image is then uploaded via a dedicated app. It is recommended that the user checks in advance to ensure that the photo has been taken properly.

[1112] Temporarily save image data and prepare for transmission

[1113] 2. Device: The captured image is temporarily stored on the device. During this process, the image's resolution and clarity are checked and it is prepared for transmission to the server. The connection status is also checked and the optimal transmission method is selected.

[1114] Receiving and analyzing image data

[1115] 3. Server: Receives image data sent from the device. Checks the quality of the received image and requests the user to retake the image if it is not satisfactory. Once the image quality is confirmed, extracts text data from the image using OCR (Optical Character Recognition) technology. Specifically, OCR tools such as Google Cloud Vision API and Tesseract can be used.

[1116] 4. Server: Analyzes the extracted text data and determines whether the values ​​for each test item are within the normal range or abnormal. This analysis takes into account the user's attribute information (age, gender, place of residence, etc.). The analysis is carried out using AI analysis engines such as TensorFlow and PyTorch, which are written in the Python language.

[1117] Action plan generation and notification

[1118] 5. Server: Generates a specific action plan for the user based on the analysis results. For example, if the AST value is high, it recommends a consultation with an internal medicine doctor and provides information on local medical institutions.

[1119] 6. Server: Sends the generated action plan and analysis results to the user's device.

[1120] 7. Device: The received action plan is notified to the user and displayed within the application. The user can then review the provided specialist information and recommended actions, and make an appointment or follow up if necessary.

[1121] Specific examples

[1122] For example, let's say a male user in his 30s has undergone a health checkup in Shibuya Ward, Tokyo, and has received the results. When this user takes a photo of the health checkup result sheet with his smartphone camera and uploads it to the app, the system operates as follows:

[1123] User: Opens the smartphone app and uses the camera to take a photo of the health check result sheet and upload it.

[1124] Device: The captured image is temporarily saved and sent to the server.

[1125] Server: Receives image data and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L" using OCR.

[1126] Server: The extracted data is passed to an AI engine, which takes into account the user's attributes such as age and gender to detect outliers.

[1127] Server: Detects when AST levels are above the normal range and generates an action plan recommending a consultation with an internal medicine doctor. At the same time, it generates a list of reputable internal medicine clinics in Shibuya Ward.

[1128] Server: Sends the action plan and clinic information to the user's device.

[1129] On the device: The app will notify the user and display the results. The user can then check the recommended clinic information within the app and make an appointment if necessary.

[1130] Prompt Sentence Examples

[1131] "When a male user in his 30s uploads his health checkup results, if the images are unclear, send him a notification urging him to take a new photo. If his AST value is high, recommend that he see an internal medicine doctor and suggest a reputable clinic in Shibuya Ward."

[1132] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1133] Program processing flow

[1134] Step 1:

[1135] User: Launches the smartphone app and takes a photo of the health check result sheet with the camera.

[1136] Input: Medical examination result sheet of the subject.

[1137] Output: Image file of the health check result sheet.

[1138] Specific operation: The user uses the camera function of the smartphone to take a picture of the health check result sheet, fitting it within the frame. After taking the picture, the user can check the preview of the image.

[1139] Step 2:

[1140] User: Check the captured image and press the upload button within the app.

[1141] Input: Image file of health check result sheet.

[1142] Output: The upload request.

[1143] Specific operation: The user checks whether the image they have taken is accurate, and if there are no problems, presses the "Upload" button in the app.

[1144] Step 3:

[1145] Terminal: Temporarily stores image data and prepares it for transmission to the server.

[1146] Input: Image file of health check result sheet.

[1147] Output: Server ready to send status.

[1148] Specific operation: The device saves the image file to a temporary folder, checks the status of Wi-Fi and mobile data communication, and selects the optimal communication method.

[1149] Step 4:

[1150] Terminal: Sends image data to the server.

[1151] Input: Image file of health check result sheet.

[1152] Output: Image sent to server.

[1153] Specific operation: The device sends the image file to the server's upload API endpoint. When the sending is complete, a success message is received.

[1154] Step 5:

[1155] Server: Receives image data and stores it in a database.

[1156] Input: Image data.

[1157] Output: Image data stored in a database.

[1158] Specific operation: The server receives the uploaded image and saves it in the database from the temporary storage folder.

[1159] Step 6:

[1160] Server: Checks the quality of the image and requests a retake if it is not satisfactory.

[1161] Input: Stored image data.

[1162] Output: Image quality check result, notification of request to retake if inadequate.

[1163] Specific operation: The server automatically checks the image resolution, brightness, distortion, etc., and if they are inappropriate, it notifies the user's device to request a re-take of the image.

[1164] Step 7:

[1165] Server: Extracts text data from images using OCR technology.

[1166] Input: Quality checked image data.

[1167] Output: The extracted text data.

[1168] Specific operation: The server uses the Google Cloud Vision API and Tesseract to extract text data such as "AST," "γ-GTP," and "LAP" from the image.

[1169] Step 8:

[1170] Server: Analyzes the text data and detects outliers taking into account user attribute information.

[1171] Input: Text data, user attribute information.

[1172] Output: The analyzed outlier data.

[1173] Specific operation: The server passes the extracted text data to an AI analysis engine (TensorFlow or PyTorch), which analyzes the data taking into account the user's attribute information such as age, gender, and place of residence.

[1174] Step 9:

[1175] Server: Generates an action plan based on the analysis results.

[1176] Input: Analyzed outlier data, user attribute information.

[1177] Output: Action plan.

[1178] Specific actions: The server generates a specific action plan for the user based on the abnormal values. For example, if the AST value is high, it recommends a consultation with an internal medicine doctor and also provides information on local medical institutions.

[1179] Step 10:

[1180] Server: Sends the generated action plan and analysis results to the user's device.

[1181] Input: Action plan, analysis results.

[1182] Output: Sent to user's device.

[1183] Specific operation: Calls an API that formats the action plan and analysis results and sends them to the user's device.

[1184] Step 11:

[1185] On the device: The user is notified of the received action plan and it is displayed within the app.

[1186] Input: Action plan and analysis results sent from the server.

[1187] Output: Action plans and analysis results displayed to the user.

[1188] What it does: The device analyzes the data it receives and generates a notification, which displays the analysis results and action plan to the user within the application.

[1189] Step 12:

[1190] User: Review the action plan and take the necessary steps.

[1191] Input: Action plan and analysis results displayed within the app.

[1192] Output: Action taken, necessary procedures (e.g., medical appointment).

[1193] Specific operation: The user checks the medical institution information provided in the app and, if necessary, makes an appointment or makes a medical appointment.

[1194] (Application example 1)

[1195] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1196] In modern society, the need to be health-conscious is increasing, but many people find it difficult to properly understand their health checkup results and take appropriate action. It is also difficult to recommend appropriate products and ensure the convenience of purchasing them. For this reason, there is a need for a system that allows users to create specific action plans based on their health checkup results, select appropriate products according to those plans, and easily purchase them in physical stores.

[1197] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1198] In this invention, the server includes means for acquiring an image of the medical checkup result, means for extracting text data from the acquired image, means for analyzing the extracted text data to detect abnormal values, means for generating an appropriate action plan for the user based on the analysis results, means for notifying the user of the generated action plan, and means for suggesting optimal products to the user based on the analysis results of the medical checkup result. This enables the user to easily understand the medical checkup result and take appropriate action, and provides an environment in which products can be suggested and purchased seamlessly according to the action plan.

[1199] "Image of health checkup results" is an image file containing health data obtained when a user undergoes a health checkup.

[1200] "Text data" is text information extracted from an image using OCR technology.

[1201] An "abnormal value" refers to a value in a test item in a health checkup result that exceeds the normal range.

[1202] An "action plan" is a plan of specific actions that the user should take based on the analysis of the health check results.

[1203] "Notification" is a means of communicating information to inform the user of the generated action plan.

[1204] "User attribute information" refers to information that indicates individual characteristics of a user, such as age, gender, and place of residence.

[1205] A "medical department" is a medical department at a medical institution that specializes in treating specific diseases or parts of the body.

[1206] "Medical institutions" are facilities that provide medical services, such as hospitals and clinics.

[1207] "Products" are items such as food, supplements, and daily necessities that are recommended to users based on the analysis of their health checkup results.

[1208] A "physical store" is a physical sales location where users can actually visit and purchase products.

[1209] "Navigation" is a means of guidance to guide a user to a destination.

[1210] An embodiment of the present invention will be described below: The system is composed of three elements: a user, a terminal, and a server.

[1211] Photographing and uploading health checkup results

[1212] Users take a photo of their health checkup result sheet using a device such as a smartphone and upload the image to the application.

[1213] The terminal temporarily stores the captured image and prepares it for transmission to the server.

[1214] Receiving and analyzing image data

[1215] The server receives the image data sent from the device and checks the image quality. It then uses OCR (Optical Character Recognition) technology to extract text data from the image. This text data includes test item names such as "AST," "γ-GTP," and "LAP," as well as their numerical values.

[1216] Text data analysis

[1217] The server analyzes the extracted text data to determine whether the values ​​for each test item are within the normal range or are abnormal, taking into account the user's attribute information (age, gender, place of residence, etc.).

[1218] Action plan generation and notification

[1219] Based on the analysis results, the server generates an action plan for the user. For example, if the AST level is high, it may recommend a consultation with an internal medicine doctor and provide information on appropriate medical institutions in the area.

[1220] The generated action plan is then notified to the user and displayed within the application, where the user can check the recommended medical institutions and product information and make purchases or reservations as necessary.

[1221] Product suggestions and navigation

[1222] The server then recommends optimal products to users based on the analysis of their health checkup results. For example, foods and supplements that are useful for improving health. Furthermore, the server provides a navigation function to make it easy to purchase these products in physical stores. Users can use the application to find and purchase the recommended products at nearby physical stores.

[1223] Technology used

[1224] In the present invention, the following hardware and software are used.

[1225] Hardware:

[1226] Smartphone (with camera function)

[1227] Servers (with high-performance processors and storage)

[1228] Physical store sales system

[1229] software:

[1230] pytesseract: OCR engine (extracts text from images)

[1231] Pillow: Image Processing Library (PIL)

[1232] requests: Communication with external API (store information acquisition)

[1233] JSON: Data Format

[1234] Specific examples

[1235] For example, if a male user in his 30s receives a medical checkup result showing an AST level of 45 U / L and a γ-GTP level of 75 U / L, he takes a photo of the medical checkup result sheet with his smartphone and uploads it to the app. The following process will occur:

[1236] 1. The device temporarily saves the image and sends it to the server.

[1237] 2. The server receives the image data and uses OCR to convert it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L."

[1238] 3. The server passes the extracted data to an AI engine, which takes into account the user's attribute information such as age and gender to detect outliers.

[1239] 4. The server detects that the AST level is above the normal range and suggests products to help improve health, along with an action plan that recommends consulting an internal medicine doctor.

[1240] 5. Users can check recommended products and medical institutions within the app and make reservations or purchases as needed.

[1241] Prompt Sentence Examples

[1242] Below is an example prompt to input to the generative AI model.

[1243] We provide images of health checkup results. Extract important health data from these images and recommend the best products for users based on the analysis results.

[1244] User information: Male, 30 years old, living in Tokyo

[1245] Health data: AST: 45 U / L, γ-GTP: 75 U / L

[1246] Analysis and recommendations: Recommendations for medical consultation, suggestions for places to buy low-fat yogurt, etc.

[1247] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1248] Step 1:

[1249] Users take a photo of their health checkup result sheet with their smartphone and upload the image to the application. At this time, users open the application and use the camera function to capture an image of the health checkup result sheet.

[1250] Input: Image of health check result sheet

[1251] Output: Temporarily saved image file

[1252] Specific operation: The user launches the camera app, takes a photo of the health check result sheet, and uploads the image to the application. The application temporarily saves the image file on the device.

[1253] Step 2:

[1254] The device temporarily stores the captured image and prepares it for transmission to the server. At this time, an application on the device converts the image file into the data format required for transmission to the server.

[1255] Input: Temporarily saved image file

[1256] Output: Image data formatted for sending to the server

[1257] Specific operation: The application on the terminal converts the image file into an appropriate format (e.g., JPEG or PNG) and sends it to the server over the network.

[1258] Step 3:

[1259] The server receives the image data sent from the terminal. At this time, the server checks the quality of the image and requests retransmission if it is unclear.

[1260] Input: Image data sent from the device

[1261] Output: Quality-checked image data

[1262] Specific operation: The server receives the image, evaluates the quality using an image processing algorithm, and if the quality is not satisfactory, asks the user to take the photo again.

[1263] Step 4:

[1264] The server extracts text data from the image using OCR technology (e.g., pytesseract), where the server inputs the image into the OCR engine and obtains the text information.

[1265] Input: Quality-checked image data

[1266] Output: Extracted text data

[1267] Specific operation: The server analyzes the image using pytesseract and extracts text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L."

[1268] Step 5:

[1269] The server analyzes the extracted text data to detect outliers, using an AI engine and user attribute information.

[1270] Input: Extracted text data, user attribute information

[1271] Output: Outlier detection results

[1272] Specific operation: The server inputs the extracted numerical data into an AI engine, taking into account attribute information such as age, gender, and place of residence, and compares it with normal values. If an abnormal value is detected, detailed information about it is generated.

[1273] Step 6:

[1274] The server generates an action plan based on the analysis results, and the AI ​​engine assembles data to suggest appropriate actions and products to the user.

[1275] Input: Outlier detection results

[1276] Output: Generated action plan

[1277] Specific operation: Based on the results of the abnormal value detection, the server generates an action plan such as "We recommend that you visit an internal medicine doctor" or "We recommend that you consume low-fat yogurt."

[1278] Step 7:

[1279] The server notifies the user of the generated action plan, which can be done via push notification or in-app display.

[1280] Input: Generated action plan

[1281] Output: User informed action plan

[1282] Specific operation: The server sends the generated action plan to the terminal, and the terminal displays the received notification to the user.

[1283] Step 8:

[1284] Users can check recommended products and medical institutions within the application and are guided to purchase products at physical stores. At this time, users can refer to the in-app map function and store information.

[1285] Input: Notified action plan, recommended product information

[1286] Output: Purchased items, medical appointments

[1287] Specific operation: The user checks the product information and medical institution information displayed within the app and navigates to the nearest store to purchase the product.

[1288] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1289] The system of the present invention acquires and analyzes information from images of health checkup results and presents a specific action plan to the user based on the results. By incorporating an emotion engine that recognizes the user's emotions, the system provides a more appropriate action plan that also takes into account the user's psychological state. The system is comprised of three elements: the user, the terminal, and the server.

[1290] System Overview

[1291] Photographing and uploading health checkup results

[1292] User: Takes a photo of the health check result sheet using the camera on a device such as a smartphone and uploads the image to the application.

[1293] Device: The captured image is temporarily stored on the device and prepared for transmission to the server.

[1294] Receiving and analyzing image data

[1295] Server: Receives image data sent from the device and checks the image quality. Next, it uses OCR (Optical Character Recognition) technology to extract text data from the image. This text data includes test item names such as "AST," "γ-GTP," and "LAP," as well as their numerical values.

[1296] Text data analysis

[1297] Server: Based on the extracted text data, the server analyzes whether the values ​​for each test item are within the normal range or abnormal. This analysis takes into account the user's attribute information (age, gender, place of residence, etc.).

[1298] Emotion recognition by emotion engine

[1299] Device or server: Analyze the user's current emotions using the emotion engine. For example, if the results of a health checkup cause stress or anxiety to the user, the emotion engine can recognize this.

[1300] Action plan generation and notification

[1301] Server: Based on the analysis results, taking into account the user's attribute information and the analysis results of the emotion engine, the server generates an action plan for the user. For example, if the AST value is high, it will recommend a consultation with an internal medicine doctor and provide information on appropriate medical institutions in the area. If the user is feeling anxious, it will include explanations and responses to reassure them.

[1302] Server: Sends the generated action plan and analysis results to the user's device.

[1303] On the device, the received action plan is notified to the user and displayed in the application, with the language and expressions adjusted to match the user's emotions.

[1304] Specific scenarios

[1305] For example, let's say a male user in his 30s has undergone a health checkup in Shibuya Ward, Tokyo, and has received the results. When this user takes a photo of the health checkup result sheet with his smartphone camera and uploads it to the app, the system operates as follows:

[1306] Photographing and analyzing health checkup results

[1307] User: Open the smartphone app and use the camera to take a photo of the health check result sheet and upload it.

[1308] Device: The captured image is temporarily saved and sent to the server.

[1309] Server: Receives image data and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L" using OCR.

[1310] Server: The extracted data is passed to an AI engine, which takes into account the user's attributes such as age and gender to detect outliers.

[1311] Emotion recognition and action plan generation

[1312] Device or server: The emotion engine recognizes the user's heart rate and facial expressions to determine whether the user is feeling anxious or stressed about the results.

[1313] Server: Detects when AST levels are above the normal range and generates an action plan that takes the user's anxiety into account. For example, it may recommend a consultation with an internal medicine doctor while providing reassuring explanations. It also generates a list of reputable internal medicine clinics in Shibuya Ward.

[1314] Communicating results and encouraging action

[1315] Server: Sends the action plan and clinic information to the user's device.

[1316] On the device: The app will notify the user of the results, and the explanation will be given in a way that is sensitive to the user's emotions, allowing the user to take the next step with peace of mind.

[1317] In this way, the system of the present invention takes into consideration the psychological state felt by the user through the health checkup results, and supports the user in intuitively deciding what specific action to take next.

[1318] The processing flow will be explained below.

[1319] Step 1:

[1320] The user launches the smartphone app and takes a photo of the health check result sheet with the camera.

[1321] Step 2:

[1322] The user checks the image they have taken and presses the upload button.

[1323] Step 3:

[1324] The terminal temporarily stores the captured image and prepares the data to be sent to the server.

[1325] Step 4:

[1326] The terminal transmits the image data to the server.

[1327] Step 5:

[1328] Check the quality of the image data received by the server and, if necessary, pre-process the image.

[1329] Step 6:

[1330] The server uses OCR technology to extract text data from the image.

[1331] Step 7:

[1332] The server stores the extracted text data in a database.

[1333] Step 8:

[1334] The server passes the text data to the AI ​​engine and begins analysis.

[1335] Step 9:

[1336] Based on the results analyzed by the AI ​​engine, the server determines whether the values ​​of each test item (e.g., AST, γ-GTP, LAP, etc.) are within the normal range or abnormal.

[1337] Step 10:

[1338] Based on the analysis results, the server performs a detailed analysis of the anomalies, taking into account the user's attribute information such as age, gender, and place of residence.

[1339] Step 11:

[1340] The emotion engine uses data acquired by the server from the user's terminal or other devices to analyze the user's emotions, for example, detecting anxiety or stress by analyzing voice tone and facial expressions.

[1341] Step 12:

[1342] The server adjusts the action plan based on the results of the emotion engine, adding more reassuring explanations and support if the user feels anxious.

[1343] Step 13:

[1344] The server generates an action plan for the user based on the analysis results and emotion recognition results.

[1345] Step 14:

[1346] The server lists information on recommended medical institutions and medical departments based on the user's location.

[1347] Step 15:

[1348] The server sends the action plan and related information to the user's terminal.

[1349] Step 16:

[1350] The device notifies the user of the action plan and information on recommended medical institutions it has received.

[1351] Step 17:

[1352] The user can check the notification contents within the app and make an appointment with a medical institution if necessary.

[1353] Step 18:

[1354] The device uses an emotion engine to obtain user feedback and reflect it in future analysis and action plan generation.

[1355] Example 2

[1356] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1357] Conventional health checkup result analysis systems required users to understand the results and decide for themselves what action to take next, which often led to anxiety and stress in users. Furthermore, when abnormal values ​​were detected, the systems lacked specific instructions for appropriate action or the ability to provide psychological support. This made it difficult for users to move on to the next action quickly and with peace of mind.

[1358] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring an image of the medical examination result, a means for extracting text data from the acquired image, a means for analyzing the extracted text data and detecting abnormal values, a means for analyzing the emotional state of the user, a means for generating an appropriate action plan for the user based on the analysis results, and a means for notifying the user of the generated action plan. This allows an appropriate action plan to be proposed taking into consideration the psychological state of the user, allowing the user to proceed with the next action with peace of mind.

[1359] "Image of medical examination results" refers to a digital image of the sheet containing the medical examination results received by the user.

[1360] "Text data" refers to the text information extracted from images of health checkup results, including numerical values ​​and names of test items.

[1361] An "abnormal value" is a value in which the numerical value of the analyzed text data deviates from a preset normal range.

[1362] An "action plan" is a specific course of action that is suggested to the user based on the analysis results and the user's attribute information.

[1363] "Emotional state" refers to the psychological state the user is currently feeling, including emotions such as stress and anxiety.

[1364] The "analysis results" are the comprehensive results of the analysis performed based on the text data of the health checkup results.

[1365] "User attribute information" refers to information related to an individual, such as the user's age, gender, and place of residence.

[1366] "Information on medical departments and medical institutions" is detailed data including information on medical facilities and medical departments recommended to users.

[1367] "Means for acquiring images" refers to a method of acquiring the health check result sheet as a digital image using a camera such as a smartphone.

[1368] "Means for extracting text data" refers to the process of using OCR technology to read character information from an image and extract it as text data.

[1369] The "means for detecting abnormal values ​​through analysis" is a method for analyzing extracted text data and detecting values ​​outside the standard range.

[1370] The "means for notifying the user" is a method for sending the generated action plan to the user's terminal and notifying the user.

[1371] The system of the present invention analyzes images obtained from health checkup results and presents a specific action plan to the user. This system operates with three elements: the user, the terminal, and the server. It also has a function to analyze the user's emotional state, which can reduce the psychological burden.

[1372] Hardware and Software Used

[1373] Device: Smartphone (e.g. iPhone or Android smartphone)

[1374] Server: Cloud server (e.g. AWS, GCP)

[1375] OCR software: Tesseract OCR

[1376] Emotion engine: Emotion analysis software (e.g., Affectiva, Emotion API)

[1377] Explanation of program processing

[1378] 1. User: The user takes a photo of the health check result sheet with their smartphone camera and uploads the image to the application by tapping the "Upload Diagnosis Results" button in the app.

[1379] 2. Device: The device temporarily stores the captured image in its internal storage and checks the image's file format, resolution, brightness, etc. If the image quality is insufficient, a message is displayed prompting the user to take a new photo. If the image quality is confirmed to be sufficient, it is sent to the server.

[1380] 3. Server: The server receives the image data sent from the device and extracts text data from the image using OCR software (e.g., Tesseract OCR). In this example, items such as "AST: 45 U / L" and "γ-GTP: 75 U / L" are extracted.

[1381] 4. Server: Analyzes the extracted text data and determines whether the values ​​for each test item are within the normal range or abnormal. This analysis uses an AI engine and takes into account the user's attributes such as age, gender, and place of residence.

[1382] 5. Device or server: Using the emotion engine, the system analyzes the user's heart rate, facial expression data, etc. For example, if the user is feeling anxious or stressed about the results of a health check, the system can recognize this.

[1383] 6. Server: Based on the analysis results and those of the emotion engine, the server generates an action plan appropriate for the user. For example, if the AST value is high, it will recommend a consultation with an internal medicine doctor and provide a list of appropriate medical institutions in the area. Furthermore, if the user is feeling anxious, it will provide reassuring explanations.

[1384] 7. Server: Sends the generated action plan and analysis results to the user's device.

[1385] 8. Device: The device notifies the user of the received action plan and displays details within the application, using appropriate language that reflects the user's emotional state to help the user feel at ease and take the next step.

[1386] Specific examples

[1387] For example, consider the case where a male user in his 30s has undergone a health checkup in Shibuya Ward, Tokyo, and receives the results. When the user takes a photo of the health checkup result sheet with their smartphone camera and uploads it to the app, the system operates as follows:

[1388] User: Open the smartphone app and use the camera to take a photo of the health check result sheet and upload it.

[1389] Device: The captured image is temporarily saved and sent to the server.

[1390] Server: Receives image data and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L" using OCR.

[1391] Server: The extracted data is passed to an AI engine, which takes into account the user's attributes such as age and gender to detect outliers.

[1392] Device or server: The emotion engine recognizes the user's heart rate and facial expressions to determine whether the user is feeling anxious or stressed about the results.

[1393] Server: Detects when AST levels are above the normal range and generates an action plan that takes the user's anxiety into account. For example, it may recommend a consultation with an internal medicine doctor while providing reassuring explanations. It also generates a list of reputable internal medicine clinics in Shibuya Ward.

[1394] Server: Sends the action plan and clinic information to the user's device.

[1395] On the device: The app will notify the user of the results, and the explanation will be given in a way that is sensitive to the user's emotions, allowing the user to take the next step with peace of mind.

[1396] Example prompts for generative AI models

[1397] "Would you like to analyze your health checkup results based on your age and AST values?"

[1398] "Generate an action plan appropriate for the user based on the health check results."

[1399] "Understand your users' current feelings and reflect them in your action plan."

[1400] These prompts can be used to input the generative AI model, allowing the system to provide a more accurate and tailored service to the user.

[1401] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1402] Step 1: Photograph and upload your health check results

[1403] User: Takes a photo of the health check result sheet with the smartphone camera and uploads the image to the application.

[1404] Specific operation: Launch the app, tap the "Upload diagnosis results" button to activate the camera function, take a picture of the sheet, and tap the "Send" button to upload the image to the app.

[1405] Input: Health check result sheet

[1406] Output: Captured image file

[1407] Step 2: Temporarily save the image and check its quality

[1408] On the device: The device temporarily stores the captured image in its internal storage. It checks the image's file format, resolution, brightness, etc., and if the quality is insufficient, displays a message to the user urging them to take the photo again.

[1409] Specific behavior: The device will save the image in JPEG format and ensure that the resolution is at least 300 dpi. If the quality is insufficient, it will display the message "Retake photo required."

[1410] Input: Captured image

[1411] Output: Save image files to internal storage, quality check results

[1412] Step 3: Sending images

[1413] Terminal: Prepares the image whose quality has been confirmed to be sent to the server, and sends the image data to the server.

[1414] Specific operation: The device sends the saved image file to the server via the API.

[1415] Input: Quality-checked image

[1416] Output: Sending image data to the server

[1417] Step 4: Receiving image data and OCR processing

[1418] Server: The server receives the image data sent from the device and uses OCR software to extract text data from the image.

[1419] Specific operation: The server passes the image file to an OCR engine (e.g., Tesseract OCR) and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L."

[1420] Input: Received image data

[1421] Output: Extracted text data

[1422] Step 5: Analyzing the text data

[1423] Server: Analyzes the extracted text data and determines whether the values ​​for each test item are within the normal range or abnormal. The analysis also takes into account the user's attribute information such as age, gender, and place of residence.

[1424] Specific operation: The server retrieves the normal value range from the database and determines, for example, that an AST value of 40 U / L or higher is abnormal. It also takes into account the user's age and gender information when making a diagnosis.

[1425] Input: Extracted text data, user attribute information

[1426] Output: Analysis result (normal or abnormal)

[1427] Step 6: Analyze emotional state

[1428] Device or server: The emotion engine is used to analyze the user's heart rate, facial expression data, etc., to determine the anxiety or stress the user feels in response to the health check results.

[1429] How it works: The device uses the device's camera and sensors to capture the user's facial expressions and heart rate, and sends the data to the emotion engine. The server analyzes this data and determines whether the user is feeling anxious.

[1430] Input: facial expression data, heart rate, etc.

[1431] Output: Emotion analysis results

[1432] Step 7: Generate an action plan

[1433] Server: Generates an optimal action plan for the user based on the analysis results and emotion recognition results. For example, it recommends a visit to an internal medicine doctor if the AST value is high or if the user feels anxious.

[1434] Specific operation: Based on the analysis results, the server takes into account user attribute information and emotion recognition results and generates a specific action plan such as "recommend visiting an internal medicine doctor" along with information on local medical institutions.

[1435] Input: Analysis results, sentiment analysis results, user attribute information

[1436] Output: Action plan

[1437] Step 8: Communicate your action plan

[1438] Server: Sends the generated action plan and analysis results to the user's device.

[1439] Specific operation: Generates an action plan in JSON format and sends it to the user's device.

[1440] Input: Action plan, analysis results

[1441] Output: Notification data sent to the user's terminal

[1442] Step 9: View your action plan

[1443] Device: The device notifies the user of the received action plan and displays the details within the application.

[1444] What it does: The app will notify the user via push notification that an action plan has been received and will display it in the app along with a detailed explanation. For example, it may say, "We recommend you see a doctor," along with a list of clinics.

[1445] Input: Notification data

[1446] Output: Display and notification to the user

[1447] The above is the specific processing flow of the system program. By showing in detail the data input and output at each processing step, and the data processing and calculations that are performed based on that, the operation of the entire system can be clearly understood.

[1448] (Application example 2)

[1449] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1450] In modern society, there is a growing demand for systems that suggest appropriate action plans based on health checkup results. However, current systems only consider the user's health status and are unable to provide optimal action plans that take into account psychological state and stress. Furthermore, there is a lack of effective means to provide personalized healthcare advertisements based on health status and emotions. This can lead to delays in users taking appropriate actions, further increasing health-related anxiety and stress.

[1451] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring an image of the health checkup results, means for extracting text data from the acquired image, means for analyzing the extracted text data and detecting abnormal values, means for generating an appropriate action plan for the user based on the analysis results, means for notifying the user of the generated action plan, means for recognizing the user's emotions, and means for generating and displaying advertisements based on the user's emotions. This makes it possible to provide an optimal action plan and healthcare advertisements by comprehensively considering the user's health condition and emotions.

[1452] "Health checkup results" are data summarizing the numerical values ​​and evaluations of various tests obtained from the health checkups taken by the user.

[1453] "Means for acquiring images" refers to the technology and devices that collect paper or digital forms of health examination results as digital images using a camera, scanner, etc.

[1454] "Means for extracting text data" refers to techniques and devices that identify textual information from captured images and convert it into digital text.

[1455] The "means for detecting abnormal values" refers to technology and devices that analyze and identify abnormal values ​​and patterns related to the user's health status based on the extracted text data.

[1456] "Means for generating an appropriate action plan" refers to technology and devices that suggest specific actions that users should take based on analyzed health data.

[1457] "Means for notifying" refers to the technology and devices used to communicate the generated action plan or advertisement to the user.

[1458] "Means for recognizing emotions" refers to technology and devices that analyze a user's psychological state from facial expressions, tone of voice, heart rate, etc., and identify emotions.

[1459] "Means for generating and displaying advertisements" refers to technology and devices that generate and display advertisements for appropriate products and services based on the user's health condition and emotions.

[1460] The system for implementing the present invention mainly uses the following hardware and software.

[1461] Hardware

[1462] 1. Smartphone camera: Used to capture images of health checkup results.

[1463] 2. Server: Used to analyze data and generate results.

[1464] software

[1465] 1. OCR library (pytesseract): Used to extract text data from images of medical examination results.

[1466] 2. Image processing library (OpenCV): Used to read and preprocess images.

[1467] 3. HTTP Request Library (Requests): Used to send and receive data.

[1468] 4. Emotion Recognition Engine: Used to analyze the user's emotions.

[1469] System Operation

[1470] User: Takes a photo of the health check result sheet with the smartphone camera and uploads the image to the application, which obtains the image data.

[1471] On the device: The captured image is converted into text data using an OCR library, and image preprocessing is also performed using an image processing library. The converted text data is temporarily stored and sent to the server for further processing.

[1472] Server: Analyzes the received text data and detects outliers. The analysis takes into account the user's attribute information (age, gender, place of residence, etc.). In addition, an emotion recognition engine is used to analyze the user's emotions. For example, it determines whether the user is feeling stressed or anxious.

[1473] Server: Generates an appropriate action plan for the user based on the analysis results. For example, if the liver function value (AST) is high, it recommends a consultation with an internal medicine doctor and provides information on medical institutions in the area. It also generates advertisements based on the user's emotions. If it determines that the user's emotions are unstable, it displays advertisements for products and services that will give them a sense of security.

[1474] Device: The device notifies the user of action plans and advertisements sent from the server. This information is provided in a way that reflects the user's emotions, allowing the user to feel at ease and take the next step.

[1475] Specific examples

[1476] For example, suppose a male user in his 30s takes a photo of his health checkup results sheet and uploads it. Text data is extracted using OCR, and analysis is performed based on that data. The analysis results indicate that his AST level is 45 U / L, which is above the normal range, and based on the user's heart rate and facial expression analysis, it is recognized that he is feeling anxious. As a result, the system generates an action plan recommending a consultation with an internal medicine doctor and provides information about local medical institutions. It also displays advertisements for reassuring herbal teas.

[1477] Prompt Sentence Examples

[1478] "User's health report shows high AST (45 U / L) and they feel anxious."

[1479] In this way, the present invention can provide optimal action plans and advertisements by comprehensively considering the user's health and psychological state.

[1480] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1481] Step 1:

[1482] User: Take a photo of the health check result sheet

[1483] Input: Health check result sheet

[1484] Specific operation: The user takes a photo of the health check result sheet using the smartphone camera. The image is then uploaded to the application.

[1485] Output: Images of the medical examination results

[1486] Step 2:

[1487] Terminal: Extracting text data using OCR

[1488] Input: Images of medical examination results

[1489] Specific operation: The device uses an OCR library (pytesseract) to extract character data from the image, and uses OpenCV to perform image preprocessing (noise removal, grayscale conversion, etc.).

[1490] Output: Extracted text data

[1491] Step 3:

[1492] Terminal: Temporarily saves text data and prepares it for sending to the server

[1493] Input: Extracted text data

[1494] Specific operation: The device temporarily stores the text data extracted by OCR and forms a data structure for sending to the server.

[1495] Output: Text data in a format that can be sent to the server

[1496] Step 4:

[1497] Server: Receiving and analyzing text data

[1498] Input: Text data sent

[1499] Specific operation: The server analyzes the received text data and detects abnormalities based on the user's attribute information (age, gender, place of residence, etc.). It evaluates the health check results using a database and AI model.

[1500] Output: Outlier analysis results

[1501] Step 5:

[1502] Server: Emotion Recognition

[1503] Input: Anomaly analysis results and user information

[1504] What it does: The server uses an emotion recognition engine to analyze the user's emotions from their heart rate and facial expressions, and if stress or anxiety is detected, it records it.

[1505] Output: User's emotional state

[1506] Step 6:

[1507] Server: Generate action plan

[1508] Input: Outlier analysis results and user emotional state

[1509] Specific operation: The server generates an appropriate action plan for the user based on the analysis results and emotional state. For example, if a high AST level is detected, it will recommend a consultation with an internal medicine doctor and provide information on local medical institutions.

[1510] Output: Action plan

[1511] Step 7:

[1512] Server: Generates and displays ads

[1513] Input: Action plan and user's emotional state

[1514] Specific operation: The server generates advertisements for products and services that provide a sense of security based on the user's emotional state, and sends the advertisements along with an action plan to the device.

[1515] Output: Action plan and advertisement

[1516] Step 8:

[1517] Device: Action plan and advertising notifications

[1518] Input: Action Plan and Advertisement

[1519] Specific operation: The device notifies the user of the action plan and advertisement sent from the server, providing information in a way that takes into account the user's emotional state, allowing them to take the next action with peace of mind.

[1520] Output: Final notification and display to the user

[1521] Through these steps, the system can comprehensively evaluate the user's health check results and emotional state, and provide optimal action plans and advertisements.

[1522] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1523] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1524] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1525] [Fourth embodiment]

[1526] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1527] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1528] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1529] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1530] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1531] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1532] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1533] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1534] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1535] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1536] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1537] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1538] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1539] The system of the present invention acquires and analyzes information from images of health checkup results, and presents a specific action plan to the user based on the results. This system is composed of three elements: the user, a terminal, and a server.

[1540] System Overview

[1541] Photographing and uploading health checkup results

[1542] User: Takes a photo of the health check result sheet using the camera on a device such as a smartphone and uploads the image to the application.

[1543] Device: The captured image is temporarily stored on the device and prepared for transmission to the server.

[1544] Receiving and analyzing image data

[1545] Server: Receives image data sent from the device and checks the image quality. Next, it uses OCR (Optical Character Recognition) technology to extract text data from the image. This text data includes test item names such as "AST," "γ-GTP," and "LAP," as well as their numerical values.

[1546] Text data analysis

[1547] Server: Based on the extracted text data, the server analyzes whether the values ​​for each test item are within the normal range or abnormal. This analysis takes into account the user's attribute information (age, gender, place of residence, etc.).

[1548] Action plan generation and notification

[1549] Server: Based on the analysis results, it generates an action plan for the user. For example, if the AST level is high, it recommends a consultation with an internal medicine doctor and provides information on appropriate medical institutions in the area.

[1550] Server: Sends the generated action plan and analysis results to the user's device.

[1551] Device: Notifies the user of the received action plan and displays it in the application.

[1552] Specific scenarios

[1553] For example, let's say a male user in his 30s has undergone a health checkup in Shibuya Ward, Tokyo, and has received the results. When this user takes a photo of the health checkup result sheet with his smartphone camera and uploads it to the app, the system operates as follows:

[1554] User: Open the smartphone app and use the camera to take a photo of the health check result sheet and upload it.

[1555] Device: The captured image is temporarily saved and sent to the server.

[1556] Server: Receives image data and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L" using OCR.

[1557] Server: The extracted data is passed to an AI engine, which takes into account the user's attributes such as age and gender to detect outliers.

[1558] Server: Detects when AST levels are above the normal range and generates an action plan recommending a consultation with an internal medicine doctor. At the same time, it generates a list of reputable internal medicine clinics in Shibuya Ward.

[1559] Server: Sends the action plan and clinic information to the user's device.

[1560] On the device: The app will notify the user and display the results. The user can then check the recommended clinic information within the app and make an appointment if necessary.

[1561] In this way, the system of the present invention supports the user in understanding the health checkup results and intuitively determining the specific actions to be taken next.

[1562] The processing flow will be explained below.

[1563] Step 1:

[1564] The user launches the smartphone app and takes a photo of the health check result sheet with the camera.

[1565] Step 2:

[1566] The user checks the image they have taken and presses the upload button.

[1567] Step 3:

[1568] The terminal temporarily stores the captured image and prepares the data to be sent to the server.

[1569] Step 4:

[1570] The terminal transmits the image data to the server.

[1571] Step 5:

[1572] Check the quality of the image data received by the server and, if necessary, pre-process the image.

[1573] Step 6:

[1574] The server uses OCR technology to extract text data from the image.

[1575] Step 7:

[1576] The server stores the extracted text data in a database.

[1577] Step 8:

[1578] The server passes the text data to the AI ​​engine and begins analysis.

[1579] Step 9:

[1580] Based on the results analyzed by the AI ​​engine, the server determines whether the values ​​of each test item (e.g., AST, γ-GTP, LAP, etc.) are within the normal range or abnormal.

[1581] Step 10:

[1582] Based on the analysis results, the server performs a detailed analysis of the anomalies, taking into account the user's attribute information such as age, gender, and place of residence.

[1583] Step 11:

[1584] The server uses the analysis results to generate an appropriate action plan for the user.

[1585] Step 12:

[1586] Based on the action plan generated by the server, the user decides what action to take next (e.g., visiting a specific medical department or changing lifestyle habits).

[1587] Step 13:

[1588] The server lists recommended medical institutions and medical departments based on the user's location.

[1589] Step 14:

[1590] The server sends the action plan and related information to the user's terminal.

[1591] Step 15:

[1592] The device notifies the user of the action plan and information on recommended medical institutions it has received.

[1593] Step 16:

[1594] The user can check the notification contents within the app and make an appointment with a medical institution if necessary.

[1595] Example 1

[1596] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1597] Currently, many users receive their health checkup results, but find it difficult to understand the contents and take appropriate action. Furthermore, reviewing and interpreting the results takes time, making it difficult to quickly respond to specific abnormal values. Furthermore, results are often judged based on uniform standards without taking into account the user's attribute information, making it difficult to provide the optimal response for each individual user.

[1598] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1599] In this invention, the server includes a means for a user to acquire an image of the health checkup results, a means for extracting text data from the acquired image, a means for analyzing the extracted text data to detect abnormal values, a means for generating an action plan based on the analysis results taking into account the user's attribute information, and a means for notifying the generated action plan and the analysis results from the server to the user's terminal. This allows the user to quickly and accurately understand the health checkup results and take appropriate action. Furthermore, by taking the user's attribute information into account during the analysis, more personalized advice can be provided, enabling responses tailored to each individual's health condition.

[1600] A "user" is a person who uses this system to upload health checkup results and receive analysis results and action plans.

[1601] "Image of medical examination results" refers to image data of sheets or documents containing the results of a medical examination that the user underwent, taken with a smartphone or digital camera.

[1602] "Text data" refers to text information extracted from an image using OCR technology, and includes test item names and their numerical values.

[1603] An "abnormal value" refers to a value outside the normal range in each test item of a health checkup, and is a value that requires the user to take measures such as visiting a medical institution or improving their lifestyle.

[1604] "User attribute information" is data including personal information such as the user's age, gender, and place of residence, and is an element that is taken into consideration when analyzing the health checkup results.

[1605] An "action plan" is specific advice and instructions provided to the user based on the analysis results, including necessary medical visits and lifestyle improvements.

[1606] A "terminal" is a device used by a user to take images of health checkup results and upload them to the system, and refers to electronic devices such as smartphones and tablets.

[1607] The "server" is a computer system that receives image data of health checkup results sent from a user's terminal, analyzes the data, and sends the results and action plans to the user's terminal.

[1608] "Means to confirm quality" refers to a technology that evaluates whether the images obtained from health checkup results are of a quality that can be analyzed, and is a process that automatically checks resolution, brightness, distortion, etc.

[1609] The "means for requesting re-imaging" is a mechanism that notifies the user to re-imaging the medical examination results if the image quality is determined to be insufficient.

[1610] "Information on medical departments and medical institutions" refers to detailed information on medical specialties recommended for the user based on their health condition and the corresponding medical facilities in the area.

[1611] The system of the present invention supports users in obtaining useful information from health checkup results and taking specific actions based on the results. This system is broadly composed of three elements: the user, the terminal, and the server.

[1612] Photographing and uploading health checkup results

[1613] 1. User: The user takes a photo of the health checkup result sheet using a camera on a device such as a smartphone. The image is then uploaded via a dedicated app. It is recommended that the user checks in advance to ensure that the photo has been taken properly.

[1614] Temporarily save image data and prepare for transmission

[1615] 2. Device: The captured image is temporarily stored on the device. During this process, the image's resolution and clarity are checked and it is prepared for transmission to the server. The connection status is also checked and the optimal transmission method is selected.

[1616] Receiving and analyzing image data

[1617] 3. Server: Receives image data sent from the device. Checks the quality of the received image and requests the user to retake the image if it is not satisfactory. Once the image quality is confirmed, extracts text data from the image using OCR (Optical Character Recognition) technology. Specifically, OCR tools such as Google Cloud Vision API and Tesseract can be used.

[1618] 4. Server: Analyzes the extracted text data and determines whether the values ​​for each test item are within the normal range or abnormal. This analysis takes into account the user's attribute information (age, gender, place of residence, etc.). The analysis is carried out using AI analysis engines such as TensorFlow and PyTorch, which are written in the Python language.

[1619] Action plan generation and notification

[1620] 5. Server: Generates a specific action plan for the user based on the analysis results. For example, if the AST value is high, it recommends a consultation with an internal medicine doctor and provides information on local medical institutions.

[1621] 6. Server: Sends the generated action plan and analysis results to the user's device.

[1622] 7. Device: The received action plan is notified to the user and displayed within the application. The user can then review the provided specialist information and recommended actions, and make an appointment or follow up if necessary.

[1623] Specific examples

[1624] For example, let's say a male user in his 30s has undergone a health checkup in Shibuya Ward, Tokyo, and has received the results. When this user takes a photo of the health checkup result sheet with his smartphone camera and uploads it to the app, the system operates as follows:

[1625] User: Opens the smartphone app and uses the camera to take a photo of the health check result sheet and upload it.

[1626] Device: The captured image is temporarily saved and sent to the server.

[1627] Server: Receives image data and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L" using OCR.

[1628] Server: The extracted data is passed to an AI engine, which takes into account the user's attributes such as age and gender to detect outliers.

[1629] Server: Detects when AST levels are above the normal range and generates an action plan recommending a consultation with an internal medicine doctor. At the same time, it generates a list of reputable internal medicine clinics in Shibuya Ward.

[1630] Server: Sends the action plan and clinic information to the user's device.

[1631] On the device: The app will notify the user and display the results. The user can then check the recommended clinic information within the app and make an appointment if necessary.

[1632] Prompt Sentence Examples

[1633] "When a male user in his 30s uploads his health checkup results, if the images are unclear, send him a notification urging him to take a new photo. If his AST value is high, recommend that he see an internal medicine doctor and suggest a reputable clinic in Shibuya Ward."

[1634] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1635] Program processing flow

[1636] Step 1:

[1637] User: Launches the smartphone app and takes a photo of the health check result sheet with the camera.

[1638] Input: Medical examination result sheet of the subject.

[1639] Output: Image file of the health check result sheet.

[1640] Specific operation: The user uses the camera function of the smartphone to take a picture of the health check result sheet, fitting it within the frame. After taking the picture, the user can check the preview of the image.

[1641] Step 2:

[1642] User: Check the captured image and press the upload button within the app.

[1643] Input: Image file of health check result sheet.

[1644] Output: The upload request.

[1645] Specific operation: The user checks whether the image they have taken is accurate, and if there are no problems, presses the "Upload" button in the app.

[1646] Step 3:

[1647] Terminal: Temporarily stores image data and prepares it for transmission to the server.

[1648] Input: Image file of health check result sheet.

[1649] Output: Server ready to send status.

[1650] Specific operation: The device saves the image file to a temporary folder, checks the status of Wi-Fi and mobile data communication, and selects the optimal communication method.

[1651] Step 4:

[1652] Terminal: Sends image data to the server.

[1653] Input: Image file of health check result sheet.

[1654] Output: Image sent to server.

[1655] Specific operation: The device sends the image file to the server's upload API endpoint. When the sending is complete, a success message is received.

[1656] Step 5:

[1657] Server: Receives image data and stores it in a database.

[1658] Input: Image data.

[1659] Output: Image data stored in a database.

[1660] Specific operation: The server receives the uploaded image and saves it in the database from the temporary storage folder.

[1661] Step 6:

[1662] Server: Checks the quality of the image and requests a retake if it is not satisfactory.

[1663] Input: Stored image data.

[1664] Output: Image quality check result, notification of request to retake if inadequate.

[1665] Specific operation: The server automatically checks the image resolution, brightness, distortion, etc., and if they are inappropriate, it notifies the user's device to request a re-take of the image.

[1666] Step 7:

[1667] Server: Extracts text data from images using OCR technology.

[1668] Input: Quality checked image data.

[1669] Output: The extracted text data.

[1670] Specific operation: The server uses the Google Cloud Vision API and Tesseract to extract text data such as "AST," "γ-GTP," and "LAP" from the image.

[1671] Step 8:

[1672] Server: Analyzes the text data and detects outliers taking into account user attribute information.

[1673] Input: Text data, user attribute information.

[1674] Output: The analyzed outlier data.

[1675] Specific operation: The server passes the extracted text data to an AI analysis engine (TensorFlow or PyTorch), which analyzes the data taking into account the user's attribute information such as age, gender, and place of residence.

[1676] Step 9:

[1677] Server: Generates an action plan based on the analysis results.

[1678] Input: Analyzed outlier data, user attribute information.

[1679] Output: Action plan.

[1680] Specific actions: The server generates a specific action plan for the user based on the abnormal values. For example, if the AST value is high, it recommends a consultation with an internal medicine doctor and also provides information on local medical institutions.

[1681] Step 10:

[1682] Server: Sends the generated action plan and analysis results to the user's device.

[1683] Input: Action plan, analysis results.

[1684] Output: Sent to user's device.

[1685] Specific operation: Calls an API that formats the action plan and analysis results and sends them to the user's device.

[1686] Step 11:

[1687] On the device: The user is notified of the received action plan and it is displayed within the app.

[1688] Input: Action plan and analysis results sent from the server.

[1689] Output: Action plans and analysis results displayed to the user.

[1690] What it does: The device analyzes the data it receives and generates a notification, which displays the analysis results and action plan to the user within the application.

[1691] Step 12:

[1692] User: Review the action plan and take the necessary steps.

[1693] Input: Action plan and analysis results displayed within the app.

[1694] Output: Action taken, necessary procedures (e.g., medical appointment).

[1695] Specific operation: The user checks the medical institution information provided in the app and, if necessary, makes an appointment or makes a medical appointment.

[1696] (Application example 1)

[1697] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1698] In modern society, the need to be health-conscious is increasing, but many people find it difficult to properly understand their health checkup results and take appropriate action. It is also difficult to recommend appropriate products and ensure the convenience of purchasing them. For this reason, there is a need for a system that allows users to create specific action plans based on their health checkup results, select appropriate products according to those plans, and easily purchase them in physical stores.

[1699] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1700] In this invention, the server includes means for acquiring an image of the medical checkup result, means for extracting text data from the acquired image, means for analyzing the extracted text data to detect abnormal values, means for generating an appropriate action plan for the user based on the analysis results, means for notifying the user of the generated action plan, and means for suggesting optimal products to the user based on the analysis results of the medical checkup result. This enables the user to easily understand the medical checkup result and take appropriate action, and provides an environment in which products can be suggested and purchased seamlessly according to the action plan.

[1701] "Image of health checkup results" is an image file containing health data obtained when a user undergoes a health checkup.

[1702] "Text data" is text information extracted from an image using OCR technology.

[1703] An "abnormal value" refers to a value in a test item in a health checkup result that exceeds the normal range.

[1704] An "action plan" is a plan of specific actions that the user should take based on the analysis of the health check results.

[1705] "Notification" is a means of communicating information to inform the user of the generated action plan.

[1706] "User attribute information" refers to information that indicates individual characteristics of a user, such as age, gender, and place of residence.

[1707] A "medical department" is a medical department at a medical institution that specializes in treating specific diseases or parts of the body.

[1708] "Medical institutions" are facilities that provide medical services, such as hospitals and clinics.

[1709] "Products" are items such as food, supplements, and daily necessities that are recommended to users based on the analysis of their health checkup results.

[1710] A "physical store" is a physical sales location where users can actually visit and purchase products.

[1711] "Navigation" is a means of guidance to guide a user to a destination.

[1712] An embodiment of the present invention will be described below: The system is composed of three elements: a user, a terminal, and a server.

[1713] Photographing and uploading health checkup results

[1714] Users take a photo of their health checkup result sheet using a device such as a smartphone and upload the image to the application.

[1715] The terminal temporarily stores the captured image and prepares it for transmission to the server.

[1716] Receiving and analyzing image data

[1717] The server receives the image data sent from the device and checks the image quality. It then uses OCR (Optical Character Recognition) technology to extract text data from the image. This text data includes test item names such as "AST," "γ-GTP," and "LAP," as well as their numerical values.

[1718] Text data analysis

[1719] The server analyzes the extracted text data to determine whether the values ​​for each test item are within the normal range or are abnormal, taking into account the user's attribute information (age, gender, place of residence, etc.).

[1720] Action plan generation and notification

[1721] Based on the analysis results, the server generates an action plan for the user. For example, if the AST level is high, it may recommend a consultation with an internal medicine doctor and provide information on appropriate medical institutions in the area.

[1722] The generated action plan is then notified to the user and displayed within the application, where the user can check the recommended medical institutions and product information and make purchases or reservations as necessary.

[1723] Product suggestions and navigation

[1724] The server then recommends optimal products to users based on the analysis of their health checkup results. For example, foods and supplements that are useful for improving health. Furthermore, the server provides a navigation function to make it easy to purchase these products in physical stores. Users can use the application to find and purchase the recommended products at nearby physical stores.

[1725] Technology used

[1726] In the present invention, the following hardware and software are used.

[1727] Hardware:

[1728] Smartphone (with camera function)

[1729] Servers (with high-performance processors and storage)

[1730] Physical store sales system

[1731] software:

[1732] pytesseract: OCR engine (extracts text from images)

[1733] Pillow: Image Processing Library (PIL)

[1734] requests: Communication with external API (store information acquisition)

[1735] JSON: Data Format

[1736] Specific examples

[1737] For example, if a male user in his 30s receives a medical checkup result showing an AST level of 45 U / L and a γ-GTP level of 75 U / L, he takes a photo of the medical checkup result sheet with his smartphone and uploads it to the app. The following process will occur:

[1738] 1. The device temporarily saves the image and sends it to the server.

[1739] 2. The server receives the image data and uses OCR to convert it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L."

[1740] 3. The server passes the extracted data to an AI engine, which takes into account the user's attribute information such as age and gender to detect outliers.

[1741] 4. The server detects that the AST level is above the normal range and suggests products to help improve health, along with an action plan that recommends consulting an internal medicine doctor.

[1742] 5. Users can check recommended products and medical institutions within the app and make reservations or purchases as needed.

[1743] Prompt Sentence Examples

[1744] Below is an example prompt to input to the generative AI model.

[1745] We provide images of health checkup results. Extract important health data from these images and recommend the best products for users based on the analysis results.

[1746] User information: Male, 30 years old, living in Tokyo

[1747] Health data: AST: 45 U / L, γ-GTP: 75 U / L

[1748] Analysis and recommendations: Recommendations for medical consultation, suggestions for places to buy low-fat yogurt, etc.

[1749] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1750] Step 1:

[1751] Users take a photo of their health checkup result sheet with their smartphone and upload the image to the application. At this time, users open the application and use the camera function to capture an image of the health checkup result sheet.

[1752] Input: Image of health check result sheet

[1753] Output: Temporarily saved image file

[1754] Specific operation: The user launches the camera app, takes a photo of the health check result sheet, and uploads the image to the application. The application temporarily saves the image file on the device.

[1755] Step 2:

[1756] The device temporarily stores the captured image and prepares it for transmission to the server. At this time, an application on the device converts the image file into the data format required for transmission to the server.

[1757] Input: Temporarily saved image file

[1758] Output: Image data formatted for sending to the server

[1759] Specific operation: The application on the terminal converts the image file into an appropriate format (e.g., JPEG or PNG) and sends it to the server over the network.

[1760] Step 3:

[1761] The server receives the image data sent from the terminal. At this time, the server checks the quality of the image and requests retransmission if it is unclear.

[1762] Input: Image data sent from the device

[1763] Output: Quality-checked image data

[1764] Specific operation: The server receives the image, evaluates the quality using an image processing algorithm, and if the quality is not satisfactory, asks the user to take the photo again.

[1765] Step 4:

[1766] The server extracts text data from the image using OCR technology (e.g., pytesseract), where the server inputs the image into the OCR engine and obtains the text information.

[1767] Input: Quality-checked image data

[1768] Output: Extracted text data

[1769] Specific operation: The server analyzes the image using pytesseract and extracts text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L."

[1770] Step 5:

[1771] The server analyzes the extracted text data to detect outliers, using an AI engine and user attribute information.

[1772] Input: Extracted text data, user attribute information

[1773] Output: Outlier detection results

[1774] Specific operation: The server inputs the extracted numerical data into an AI engine, taking into account attribute information such as age, gender, and place of residence, and compares it with normal values. If an abnormal value is detected, detailed information about it is generated.

[1775] Step 6:

[1776] The server generates an action plan based on the analysis results, and the AI ​​engine assembles data to suggest appropriate actions and products to the user.

[1777] Input: Outlier detection results

[1778] Output: Generated action plan

[1779] Specific operation: Based on the results of the abnormal value detection, the server generates an action plan such as "We recommend that you visit an internal medicine doctor" or "We recommend that you consume low-fat yogurt."

[1780] Step 7:

[1781] The server notifies the user of the generated action plan, which can be done via push notification or in-app display.

[1782] Input: Generated action plan

[1783] Output: User informed action plan

[1784] Specific operation: The server sends the generated action plan to the terminal, and the terminal displays the received notification to the user.

[1785] Step 8:

[1786] Users can check recommended products and medical institutions within the application and are guided to purchase products at physical stores. At this time, users can refer to the in-app map function and store information.

[1787] Input: Notified action plan, recommended product information

[1788] Output: Purchased items, medical appointments

[1789] Specific operation: The user checks the product information and medical institution information displayed within the app and navigates to the nearest store to purchase the product.

[1790] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1791] The system of the present invention acquires and analyzes information from images of health checkup results and presents a specific action plan to the user based on the results. By incorporating an emotion engine that recognizes the user's emotions, the system provides a more appropriate action plan that also takes into account the user's psychological state. The system is comprised of three elements: the user, the terminal, and the server.

[1792] System Overview

[1793] Photographing and uploading health checkup results

[1794] User: Takes a photo of the health check result sheet using the camera on a device such as a smartphone and uploads the image to the application.

[1795] Device: The captured image is temporarily stored on the device and prepared for transmission to the server.

[1796] Receiving and analyzing image data

[1797] Server: Receives image data sent from the device and checks the image quality. Next, it uses OCR (Optical Character Recognition) technology to extract text data from the image. This text data includes test item names such as "AST," "γ-GTP," and "LAP," as well as their numerical values.

[1798] Text data analysis

[1799] Server: Based on the extracted text data, the server analyzes whether the values ​​for each test item are within the normal range or abnormal. This analysis takes into account the user's attribute information (age, gender, place of residence, etc.).

[1800] Emotion recognition by emotion engine

[1801] Device or server: Analyze the user's current emotions using the emotion engine. For example, if the results of a health checkup cause stress or anxiety to the user, the emotion engine can recognize this.

[1802] Action plan generation and notification

[1803] Server: Based on the analysis results, taking into account the user's attribute information and the analysis results of the emotion engine, the server generates an action plan for the user. For example, if the AST value is high, it will recommend a consultation with an internal medicine doctor and provide information on appropriate medical institutions in the area. If the user is feeling anxious, it will include explanations and responses to reassure them.

[1804] Server: Sends the generated action plan and analysis results to the user's device.

[1805] On the device, the received action plan is notified to the user and displayed in the application, with the language and expressions adjusted to match the user's emotions.

[1806] Specific scenarios

[1807] For example, let's say a male user in his 30s has undergone a health checkup in Shibuya Ward, Tokyo, and has received the results. When this user takes a photo of the health checkup result sheet with his smartphone camera and uploads it to the app, the system operates as follows:

[1808] Photographing and analyzing health checkup results

[1809] User: Open the smartphone app and use the camera to take a photo of the health check result sheet and upload it.

[1810] Device: The captured image is temporarily saved and sent to the server.

[1811] Server: Receives image data and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L" using OCR.

[1812] Server: The extracted data is passed to an AI engine, which takes into account the user's attributes such as age and gender to detect outliers.

[1813] Emotion recognition and action plan generation

[1814] Device or server: The emotion engine recognizes the user's heart rate and facial expressions to determine whether the user is feeling anxious or stressed about the results.

[1815] Server: Detects when AST levels are above the normal range and generates an action plan that takes the user's anxiety into account. For example, it may recommend a consultation with an internal medicine doctor while providing reassuring explanations. It also generates a list of reputable internal medicine clinics in Shibuya Ward.

[1816] Communicating results and encouraging action

[1817] Server: Sends the action plan and clinic information to the user's device.

[1818] On the device: The app will notify the user of the results, and the explanation will be given in a way that is sensitive to the user's emotions, allowing the user to take the next step with peace of mind.

[1819] In this way, the system of the present invention takes into consideration the psychological state felt by the user through the health checkup results, and supports the user in intuitively deciding what specific action to take next.

[1820] The processing flow will be explained below.

[1821] Step 1:

[1822] The user launches the smartphone app and takes a photo of the health check result sheet with the camera.

[1823] Step 2:

[1824] The user checks the image they have taken and presses the upload button.

[1825] Step 3:

[1826] The terminal temporarily stores the captured image and prepares the data to be sent to the server.

[1827] Step 4:

[1828] The terminal transmits the image data to the server.

[1829] Step 5:

[1830] Check the quality of the image data received by the server and, if necessary, pre-process the image.

[1831] Step 6:

[1832] The server uses OCR technology to extract text data from the image.

[1833] Step 7:

[1834] The server stores the extracted text data in a database.

[1835] Step 8:

[1836] The server passes the text data to the AI ​​engine and begins analysis.

[1837] Step 9:

[1838] Based on the results analyzed by the AI ​​engine, the server determines whether the values ​​of each test item (e.g., AST, γ-GTP, LAP, etc.) are within the normal range or abnormal.

[1839] Step 10:

[1840] Based on the analysis results, the server performs a detailed analysis of the anomalies, taking into account the user's attribute information such as age, gender, and place of residence.

[1841] Step 11:

[1842] The emotion engine uses data acquired by the server from the user's terminal or other devices to analyze the user's emotions, for example, detecting anxiety or stress by analyzing voice tone and facial expressions.

[1843] Step 12:

[1844] The server adjusts the action plan based on the results of the emotion engine, adding more reassuring explanations and support if the user feels anxious.

[1845] Step 13:

[1846] The server generates an action plan for the user based on the analysis results and emotion recognition results.

[1847] Step 14:

[1848] The server lists information on recommended medical institutions and medical departments based on the user's location.

[1849] Step 15:

[1850] The server sends the action plan and related information to the user's terminal.

[1851] Step 16:

[1852] The device notifies the user of the action plan and information on recommended medical institutions it has received.

[1853] Step 17:

[1854] The user can check the notification contents within the app and make an appointment with a medical institution if necessary.

[1855] Step 18:

[1856] The device uses an emotion engine to obtain user feedback and reflect it in future analysis and action plan generation.

[1857] Example 2

[1858] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1859] Conventional health checkup result analysis systems required users to understand the results and decide for themselves what action to take next, which often led to anxiety and stress in users. Furthermore, when abnormal values ​​were detected, the systems lacked specific instructions for appropriate action or the ability to provide psychological support. This made it difficult for users to move on to the next action quickly and with peace of mind.

[1860] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring an image of the medical examination result, a means for extracting text data from the acquired image, a means for analyzing the extracted text data and detecting abnormal values, a means for analyzing the emotional state of the user, a means for generating an appropriate action plan for the user based on the analysis results, and a means for notifying the user of the generated action plan. This allows an appropriate action plan to be proposed taking into consideration the psychological state of the user, allowing the user to proceed with the next action with peace of mind.

[1861] "Image of medical examination results" refers to a digital image of the sheet containing the medical examination results received by the user.

[1862] "Text data" refers to the text information extracted from images of health checkup results, including numerical values ​​and names of test items.

[1863] An "abnormal value" is a value in which the numerical value of the analyzed text data deviates from a preset normal range.

[1864] An "action plan" is a specific course of action that is suggested to the user based on the analysis results and the user's attribute information.

[1865] "Emotional state" refers to the psychological state the user is currently feeling, including emotions such as stress and anxiety.

[1866] The "analysis results" are the comprehensive results of the analysis performed based on the text data of the health checkup results.

[1867] "User attribute information" refers to information related to an individual, such as the user's age, gender, and place of residence.

[1868] "Information on medical departments and medical institutions" is detailed data including information on medical facilities and medical departments recommended to users.

[1869] "Means for acquiring images" refers to a method of acquiring the health check result sheet as a digital image using a camera such as a smartphone.

[1870] "Means for extracting text data" refers to the process of using OCR technology to read character information from an image and extract it as text data.

[1871] The "means for detecting abnormal values ​​through analysis" is a method for analyzing extracted text data and detecting values ​​outside the standard range.

[1872] The "means for notifying the user" is a method for sending the generated action plan to the user's terminal and notifying the user.

[1873] The system of the present invention analyzes images obtained from health checkup results and presents a specific action plan to the user. This system operates with three elements: the user, the terminal, and the server. It also has a function to analyze the user's emotional state, which can reduce the psychological burden.

[1874] Hardware and Software Used

[1875] Device: Smartphone (e.g. iPhone or Android smartphone)

[1876] Server: Cloud server (e.g. AWS, GCP)

[1877] OCR software: Tesseract OCR

[1878] Emotion engine: Emotion analysis software (e.g., Affectiva, Emotion API)

[1879] Explanation of program processing

[1880] 1. User: The user takes a photo of the health check result sheet with their smartphone camera and uploads the image to the application by tapping the "Upload Diagnosis Results" button in the app.

[1881] 2. Device: The device temporarily stores the captured image in its internal storage and checks the image's file format, resolution, brightness, etc. If the image quality is insufficient, a message is displayed prompting the user to take a new photo. If the image quality is confirmed to be sufficient, it is sent to the server.

[1882] 3. Server: The server receives the image data sent from the device and extracts text data from the image using OCR software (e.g., Tesseract OCR). In this example, items such as "AST: 45 U / L" and "γ-GTP: 75 U / L" are extracted.

[1883] 4. Server: Analyzes the extracted text data and determines whether the values ​​for each test item are within the normal range or abnormal. This analysis uses an AI engine and takes into account the user's attributes such as age, gender, and place of residence.

[1884] 5. Device or server: Using the emotion engine, the system analyzes the user's heart rate, facial expression data, etc. For example, if the user is feeling anxious or stressed about the results of a health check, the system can recognize this.

[1885] 6. Server: Based on the analysis results and those of the emotion engine, the server generates an action plan appropriate for the user. For example, if the AST value is high, it will recommend a consultation with an internal medicine doctor and provide a list of appropriate medical institutions in the area. Furthermore, if the user is feeling anxious, it will provide reassuring explanations.

[1886] 7. Server: Sends the generated action plan and analysis results to the user's device.

[1887] 8. Device: The device notifies the user of the received action plan and displays details within the application, using appropriate language that reflects the user's emotional state to help the user feel at ease and take the next step.

[1888] Specific examples

[1889] For example, consider the case where a male user in his 30s has undergone a health checkup in Shibuya Ward, Tokyo, and receives the results. When the user takes a photo of the health checkup result sheet with their smartphone camera and uploads it to the app, the system operates as follows:

[1890] User: Open the smartphone app and use the camera to take a photo of the health check result sheet and upload it.

[1891] Device: The captured image is temporarily saved and sent to the server.

[1892] Server: Receives image data and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L" using OCR.

[1893] Server: The extracted data is passed to an AI engine, which takes into account the user's attributes such as age and gender to detect outliers.

[1894] Device or server: The emotion engine recognizes the user's heart rate and facial expressions to determine whether the user is feeling anxious or stressed about the results.

[1895] Server: Detects when AST levels are above the normal range and generates an action plan that takes the user's anxiety into account. For example, it may recommend a consultation with an internal medicine doctor while providing reassuring explanations. It also generates a list of reputable internal medicine clinics in Shibuya Ward.

[1896] Server: Sends the action plan and clinic information to the user's device.

[1897] On the device: The app will notify the user of the results, and the explanation will be given in a way that is sensitive to the user's emotions, allowing the user to take the next step with peace of mind.

[1898] Example prompts for generative AI models

[1899] "Would you like to analyze your health checkup results based on your age and AST values?"

[1900] "Generate an action plan appropriate for the user based on the health check results."

[1901] "Understand your users' current feelings and reflect them in your action plan."

[1902] These prompts can be used to input the generative AI model, allowing the system to provide a more accurate and tailored service to the user.

[1903] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1904] Step 1: Photograph and upload your health check results

[1905] User: Takes a photo of the health check result sheet with the smartphone camera and uploads the image to the application.

[1906] Specific operation: Launch the app, tap the "Upload diagnosis results" button to activate the camera function, take a picture of the sheet, and tap the "Send" button to upload the image to the app.

[1907] Input: Health check result sheet

[1908] Output: Captured image file

[1909] Step 2: Temporarily save the image and check its quality

[1910] On the device: The device temporarily stores the captured image in its internal storage. It checks the image's file format, resolution, brightness, etc., and if the quality is insufficient, displays a message to the user urging them to take the photo again.

[1911] Specific behavior: The device will save the image in JPEG format and ensure that the resolution is at least 300 dpi. If the quality is insufficient, it will display the message "Retake photo required."

[1912] Input: Captured image

[1913] Output: Save image files to internal storage, quality check results

[1914] Step 3: Sending images

[1915] Terminal: Prepares the image whose quality has been confirmed to be sent to the server, and sends the image data to the server.

[1916] Specific operation: The device sends the saved image file to the server via the API.

[1917] Input: Quality-checked image

[1918] Output: Sending image data to the server

[1919] Step 4: Receiving image data and OCR processing

[1920] Server: The server receives the image data sent from the device and uses OCR software to extract text data from the image.

[1921] Specific operation: The server passes the image file to an OCR engine (e.g., Tesseract OCR) and converts it into text data such as "AST: 45 U / L" and "γ-GTP: 75 U / L."

[1922] Input: Received image data

[1923] Output: Extracted text data

[1924] Step 5: Analyzing the text data

[1925] Server: Analyzes the extracted text data and determines whether the values ​​for each test item are within the normal range or abnormal. The analysis also takes into account the user's attribute information such as age, gender, and place of residence.

[1926] Specific operation: The server retrieves the normal value range from the database and determines, for example, that an AST value of 40 U / L or higher is abnormal. It also takes into account the user's age and gender information when making a diagnosis.

[1927] Input: Extracted text data, user attribute information

[1928] Output: Analysis result (normal or abnormal)

[1929] Step 6: Analyze emotional state

[1930] Device or server: The emotion engine is used to analyze the user's heart rate, facial expression data, etc., to determine the anxiety or stress the user feels in response to the health check results.

[1931] How it works: The device uses the device's camera and sensors to capture the user's facial expressions and heart rate, and sends the data to the emotion engine. The server analyzes this data and determines whether the user is feeling anxious.

[1932] Input: facial expression data, heart rate, etc.

[1933] Output: Emotion analysis results

[1934] Step 7: Generate an action plan

[1935] Server: Generates an optimal action plan for the user based on the analysis results and emotion recognition results. For example, it recommends a visit to an internal medicine doctor if the AST value is high or if the user feels anxious.

[1936] Specific operation: Based on the analysis results, the server takes into account user attribute information and emotion recognition results and generates a specific action plan such as "recommend visiting an internal medicine doctor" along with information on local medical institutions.

[1937] Input: Analysis results, sentiment analysis results, user attribute information

[1938] Output: Action plan

[1939] Step 8: Communicate your action plan

[1940] Server: Sends the generated action plan and analysis results to the user's device.

[1941] Specific operation: Generates an action plan in JSON format and sends it to the user's device.

[1942] Input: Action plan, analysis results

[1943] Output: Notification data sent to the user's terminal

[1944] Step 9: View your action plan

[1945] Device: The device notifies the user of the received action plan and displays the details within the application.

[1946] What it does: The app will notify the user via push notification that an action plan has been received and will display it in the app along with a detailed explanation. For example, it may say, "We recommend you see a doctor," along with a list of clinics.

[1947] Input: Notification data

[1948] Output: Display and notification to the user

[1949] The above is the specific processing flow of the system program. By showing in detail the data input and output at each processing step, and the data processing and calculations that are performed based on that, the operation of the entire system can be clearly understood.

[1950] (Application example 2)

[1951] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1952] In modern society, there is a growing demand for systems that suggest appropriate action plans based on health checkup results. However, current systems only consider the user's health status and are unable to provide optimal action plans that take into account psychological state and stress. Furthermore, there is a lack of effective means to provide personalized healthcare advertisements based on health status and emotions. This can lead to delays in users taking appropriate actions, further increasing health-related anxiety and stress.

[1953] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring an image of the health checkup results, means for extracting text data from the acquired image, means for analyzing the extracted text data and detecting abnormal values, means for generating an appropriate action plan for the user based on the analysis results, means for notifying the user of the generated action plan, means for recognizing the user's emotions, and means for generating and displaying advertisements based on the user's emotions. This makes it possible to provide an optimal action plan and healthcare advertisements by comprehensively considering the user's health condition and emotions.

[1954] "Health checkup results" are data summarizing the numerical values ​​and evaluations of various tests obtained from the health checkups taken by the user.

[1955] "Means for acquiring images" refers to the technology and devices that collect paper or digital forms of health examination results as digital images using a camera, scanner, etc.

[1956] "Means for extracting text data" refers to techniques and devices that identify textual information from captured images and convert it into digital text.

[1957] The "means for detecting abnormal values" refers to technology and devices that analyze and identify abnormal values ​​and patterns related to the user's health status based on the extracted text data.

[1958] "Means for generating an appropriate action plan" refers to technology and devices that suggest specific actions that users should take based on analyzed health data.

[1959] "Means for notifying" refers to the technology and devices used to communicate the generated action plan or advertisement to the user.

[1960] "Means for recognizing emotions" refers to technology and devices that analyze a user's psychological state from facial expressions, tone of voice, heart rate, etc., and identify emotions.

[1961] "Means for generating and displaying advertisements" refers to technology and devices that generate and display advertisements for appropriate products and services based on the user's health condition and emotions.

[1962] The system for implementing the present invention mainly uses the following hardware and software.

[1963] Hardware

[1964] 1. Smartphone camera: Used to capture images of health checkup results.

[1965] 2. Server: Used to analyze data and generate results.

[1966] software

[1967] 1. OCR library (pytesseract): Used to extract text data from images of medical examination results.

[1968] 2. Image processing library (OpenCV): Used to read and preprocess images.

[1969] 3. HTTP Request Library (Requests): Used to send and receive data.

[1970] 4. Emotion Recognition Engine: Used to analyze the user's emotions.

[1971] System Operation

[1972] User: Takes a photo of the health check result sheet with the smartphone camera and uploads the image to the application, which obtains the image data.

[1973] On the device: The captured image is converted into text data using an OCR library, and image preprocessing is also performed using an image processing library. The converted text data is temporarily stored and sent to the server for further processing.

[1974] Server: Analyzes the received text data and detects outliers. The analysis takes into account the user's attribute information (age, gender, place of residence, etc.). In addition, an emotion recognition engine is used to analyze the user's emotions. For example, it determines whether the user is feeling stressed or anxious.

[1975] Server: Generates an appropriate action plan for the user based on the analysis results. For example, if the liver function value (AST) is high, it recommends a consultation with an internal medicine doctor and provides information on medical institutions in the area. It also generates advertisements based on the user's emotions. If it determines that the user's emotions are unstable, it displays advertisements for products and services that will give them a sense of security.

[1976] Device: The device notifies the user of action plans and advertisements sent from the server. This information is provided in a way that reflects the user's emotions, allowing the user to feel at ease and take the next step.

[1977] Specific examples

[1978] For example, suppose a male user in his 30s takes a photo of his health checkup results sheet and uploads it. Text data is extracted using OCR, and analysis is performed based on that data. The analysis results indicate that his AST level is 45 U / L, which is above the normal range, and based on the user's heart rate and facial expression analysis, it is recognized that he is feeling anxious. As a result, the system generates an action plan recommending a consultation with an internal medicine doctor and provides information about local medical institutions. It also displays advertisements for reassuring herbal teas.

[1979] Prompt Sentence Examples

[1980] "User's health report shows high AST (45 U / L) and they feel anxious."

[1981] In this way, the present invention can provide optimal action plans and advertisements by comprehensively considering the user's health and psychological state.

[1982] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1983] Step 1:

[1984] User: Take a photo of the health check result sheet

[1985] Input: Health check result sheet

[1986] Specific operation: The user takes a photo of the health check result sheet using the smartphone camera. The image is then uploaded to the application.

[1987] Output: Images of the medical examination results

[1988] Step 2:

[1989] Terminal: Extracting text data using OCR

[1990] Input: Images of medical examination results

[1991] Specific operation: The device uses an OCR library (pytesseract) to extract character data from the image, and uses OpenCV to perform image preprocessing (noise removal, grayscale conversion, etc.).

[1992] Output: Extracted text data

[1993] Step 3:

[1994] Terminal: Temporarily saves text data and prepares it for sending to the server

[1995] Input: Extracted text data

[1996] Specific operation: The device temporarily stores the text data extracted by OCR and forms a data structure for sending to the server.

[1997] Output: Text data in a format that can be sent to the server

[1998] Step 4:

[1999] Server: Receiving and analyzing text data

[2000] Input: Text data sent

[2001] Specific operation: The server analyzes the received text data and detects abnormalities based on the user's attribute information (age, gender, place of residence, etc.). It evaluates the health check results using a database and AI model.

[2002] Output: Outlier analysis results

[2003] Step 5:

[2004] Server: Emotion Recognition

[2005] Input: Anomaly analysis results and user information

[2006] What it does: The server uses an emotion recognition engine to analyze the user's emotions from their heart rate and facial expressions, and if stress or anxiety is detected, it records it.

[2007] Output: User's emotional state

[2008] Step 6:

[2009] Server: Generate action plan

[2010] Input: Outlier analysis results and user emotional state

[2011] Specific operation: The server generates an appropriate action plan for the user based on the analysis results and emotional state. For example, if a high AST level is detected, it will recommend a consultation with an internal medicine doctor and provide information on local medical institutions.

[2012] Output: Action plan

[2013] Step 7:

[2014] Server: Generates and displays ads

[2015] Input: Action plan and user's emotional state

[2016] Specific operation: The server generates advertisements for products and services that provide a sense of security based on the user's emotional state, and sends the advertisements along with an action plan to the device.

[2017] Output: Action plan and advertisement

[2018] Step 8:

[2019] Device: Action plan and advertising notifications

[2020] Input: Action Plan and Advertisement

[2021] Specific operation: The device notifies the user of the action plan and advertisement sent from the server, providing information in a way that takes into account the user's emotional state, allowing them to take the next action with peace of mind.

[2022] Output: Final notification and display to the user

[2023] Through these steps, the system can comprehensively evaluate the user's health check results and emotional state, and provide optimal action plans and advertisements.

[2024] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2025] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2026] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2027] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2028] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2029] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2030] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2031] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2032] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2033] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2034] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2035] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2036] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2037] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2038] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2039] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2040] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2041] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2042] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2043] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2044] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2045] The following is further disclosed regarding the above embodiment.

[2046] (Claim 1)

[2047] A means for acquiring an image of the medical examination result;

[2048] means for extracting text data from the captured image;

[2049] means for analyzing the extracted text data to detect outliers;

[2050] A means for generating an appropriate action plan for the user based on the analysis results;

[2051] means for notifying the user of the generated action plan;

[2052] A system including:

[2053] (Claim 2)

[2054] 10. The system according to claim 1, further comprising means for taking into account user attribute information in the analysis.

[2055] (Claim 3)

[2056] 2. The system according to claim 1, further comprising means for providing the user with information on the most suitable medical department and medical institution based on the generated action plan.

[2057] "Example 1"

[2058] (Claim 1)

[2059] A means for a user to acquire an image of the medical examination result;

[2060] means for extracting text data from the captured image;

[2061] means for analyzing the extracted text data to detect outliers;

[2062] A means for generating an action plan based on the analysis results in consideration of the attribute information of the user;

[2063] a means for notifying the generated action plan and analysis results from the server to the user's terminal;

[2064] A system including:

[2065] (Claim 2)

[2066] 10. The system of claim 1, wherein the server includes means for checking the quality of the image and requesting a retake.

[2067] (Claim 3)

[2068] 10. The system according to claim 1, further comprising means for providing the user with information on the most suitable medical department and medical institution based on the generated action plan.

[2069] "Application Example 1"

[2070] (Claim 1)

[2071] A means for acquiring an image of the medical examination result;

[2072] means for extracting text data from the captured image;

[2073] means for analyzing the extracted text data to detect outliers;

[2074] A means for generating an appropriate action plan for the user based on the analysis results;

[2075] means for notifying the user of the generated action plan;

[2076] A means to suggest optimal products to users based on the analysis of health checkup results,

[2077] A system including:

[2078] (Claim 2)

[2079] 10. The system according to claim 1, further comprising means for taking into account user attribute information in the analysis.

[2080] (Claim 3)

[2081] 2. The system according to claim 1, further comprising means for providing the user with information on the most suitable medical department and medical institution based on the generated action plan.

[2082] (Claim 4)

[2083] The system according to claim 1, further comprising a means for suggesting optimal products to the user based on the generated action plan and navigating the user to purchase the products in a physical store.

[2084] "Example 2: Combining Emotion Engines"

[2085] (Claim 1)

[2086] A means for acquiring an image of the medical examination result;

[2087] means for extracting text data from the captured image;

[2088] means for analyzing the extracted text data to detect outliers;

[2089] A means for generating an appropriate action plan for the user based on the analysis results;

[2090] a system including means for analyzing an emotional state of a user;

[2091] means for notifying the user of the generated action plan;

[2092] A system including:

[2093] (Claim 2)

[2094] 10. The system according to claim 1, further comprising means for taking into account user attribute information in the analysis.

[2095] (Claim 3)

[2096] 2. The system according to claim 1, further comprising means for providing the user with information on the most suitable medical department and medical institution based on the generated action plan.

[2097] "Application example 2 when combining emotion engines"

[2098] (Claim 1)

[2099] A means for acquiring an image of the medical examination result;

[2100] means for extracting text data from the captured image;

[2101] means for analyzing the extracted text data to detect outliers;

[2102] A means for generating an appropriate action plan for the user based on the analysis results;

[2103] means for notifying the user of the generated action plan;

[2104] a means for recognizing a user's emotions;

[2105] means for generating and displaying advertisements based on user sentiment;

[2106] A system including:

[2107] (Claim 2)

[2108] 10. The system according to claim 1, further comprising means for taking into account user attribute information in the analysis.

[2109] (Claim 3)

[2110] 2. The system according to claim 1, further comprising means for providing the user with information on the most suitable medical department and medical institution based on the generated action plan. [Explanation of symbols]

[2111] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for acquiring an image of the medical examination result; means for extracting text data from the captured image; means for analyzing the extracted text data to detect outliers; A means for generating an appropriate action plan for the user based on the analysis results; means for notifying the user of the generated action plan; A system including:

2. The system according to claim 1 , further comprising means for taking into account user attribute information in the analysis.

3. The system according to claim 1, further comprising means for providing the user with information on the most suitable medical department and medical institution based on the generated action plan.

Images