system

Abstract

A system is provided.SOLUTION: A system comprising: means for generating personalized health advice based on information input by a user; means for extracting text information from a captured food-label image using OCR; means for matching the extracted text information with a food-ingredient database and analyzing the ingredient information using a generated AI to assess a risk associated with a health condition of the user; means for notifying the user of the assessed risk information as an alert; and means for analyzing the user's speech input by speech recognition and using the generated AI to provide an appropriate answer and advice.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Many people today are health-conscious, but making the right food choices requires specialized knowledge, and their busy daily lives leave them with little time or space to acquire such knowledge. As a result, they may ingest foods or ingredients that have a negative impact on their health. It is also difficult to properly identify ingredients that should be avoided depending on high blood pressure, diabetes, or other health conditions, and to select ingredients accordingly. Given this background, there is a need for a system that allows for effective and easy food selection based on specific health conditions and goals. [Means for solving the problem]

[0005] This system solves problems related to food selection. Specifically, it provides a means for generating personalized health advice based on health information entered by the user and a means for photographing food labels and extracting text information using OCR (optical character recognition). It also provides a means for comparing the extracted text information with a food ingredient database, analyzing the ingredient information using a generation AI, and assessing the risks associated with the user's health. It also provides a means for notifying the user of the assessed risk information as an alert, and a means for analyzing the user's voice input using voice recognition and providing appropriate answers and advice using a generation AI. Furthermore, the system stores the food selection data and advice received by the user in a database and uses this data to improve the accuracy of future personalization, thereby supporting continuous health management for each user. This allows users to easily choose healthy foods, which can ultimately help maintain and improve their health.

[0006] "User" refers to an individual who uses this system to make food choices and manage health information.

[0007] "Health Advice" refers to advice about food choices and lifestyle habits provided based on the user's health status and goals.

[0008] "OCR (Optical Character Recognition)" refers to the technology that reads characters contained in an image and converts them into text data.

[0009] A "food composition database" refers to a system that stores composition information for various foods and provides related information through matching and searching.

[0010] "Generative AI" refers to technology that uses artificial intelligence to analyze food ingredients and health risks and generate appropriate advice.

[0011] "Ingredient information" refers to detailed data on the various nutrients and additives contained in food.

[0012] "Risk assessment" refers to the process of analyzing the effects of specific food ingredients on the user's health and determining their risk.

[0013] An "alert" refers to a notification to warn a user about a health risk.

[0014] "Speech recognition" refers to the technology that analyzes the user's voice and converts it into text data.

[0015] "Database" refers to a system that systematically stores information such as user-selected food data and received advice, and uses it for future searches and analysis.

[0016] "Personalization accuracy" refers to the degree to which advice and support for individual users is adapted to the user's specific needs and circumstances. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention provides a system that allows users to easily select healthy foods, and includes a program that operates by combining various technologies to achieve this. Specific embodiments of this system are described below.

[0039] User registration and initial settings

[0040] 1. The user installs the application and, when launching it for the first time, enters basic information such as name, age, gender, medical history, and dietary restrictions on the account creation screen.

[0041] 2. The terminal sends the information entered by the user to the server.

[0042] 3. The server generates a user profile based on the received information and tailors health advice to the user. This profile is stored in a database for future analysis and advice provision.

[0043] Food Identification

[0044] 1. The user opens the application and takes a photo of the label of a food item they are considering purchasing.

[0045] 2. The device passes the captured image to an OCR (Optical Character Recognition) module to extract text information from the image.

[0046] Example: If a food label contains information such as "phosphates," "sodium," and "sugars," these will be read as text.

[0047] 3. The terminal sends the extracted text data to the server.

[0048] Ingredient analysis and risk assessment

[0049] 1. The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[0050] 2. The server uses generative AI to analyze the ingredients and assess the risks associated with the user's health.

[0051] Example: If the analysis reveals that a user with high blood pressure selects a high-salt food, a risk assessment will be made, such as "This food contains a high amount of salt and may have a negative impact on high blood pressure."

[0052] 3. The server generates the results of the risk assessment as alert information and sends it to the terminal.

[0053] Viewing Alerts and Information

[0054] 1. The terminal displays the alert information and detailed data received from the server on the application screen.

[0055] Example: The application screen displays the results of the risk assessment using text and icons, visually communicating the risks to the user.

[0056] 2. The user reviews the information displayed and asks additional questions if necessary.

[0057] Voice recognition and additional information provision

[0058] 1. If the user wants more information, they can ask a question out loud. For example, "What should I eat instead of this food?"

[0059] 2. The device uses voice recognition to convert the user's question into text and send it to the server.

[0060] 3. The server uses generative AI to generate the best answer based on the question.

[0061] Example: Create a "list of low-salt foods" based on a food database and health guidelines and send it to your device.

[0062] 4. The device will play the received answer aloud and also display it as text, allowing the user to receive information both visually and audibly.

[0063] Data accumulation and feedback

[0064] 1. The server stores the user's food selection data and received advice in a database.

[0065] 2. The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments.

[0066] In this way, the present invention realizes a system that allows users to easily select healthy foods. By providing users with checked ingredient information and risk alerts, the system helps users make healthy food choices. This system allows users to make healthy food choices without spending time or effort.

[0067] The processing flow will be explained below.

[0068] Food selection system processing procedure

[0069] Step 1:

[0070] Users install the app and enter basic information such as their name, age, gender, medical history, and dietary restrictions when they first launch it.

[0071] Step 2:

[0072] The terminal transmits the input information to the server.

[0073] Step 3:

[0074] The server generates a user profile based on the received information and sets basic health advice tailored to the user. This profile is used for future analysis and advice provision.

[0075] Step 4:

[0076] The user launches the app and takes a photo of the label of the food item they plan to purchase.

[0077] Step 5:

[0078] The device passes the captured image to an OCR (optical character recognition) module, which extracts text information from the image.

[0079] Step 6:

[0080] The terminal transmits the extracted text data to the server.

[0081] Step 7:

[0082] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[0083] Step 8:

[0084] The server uses generative AI to analyze the ingredients and assess the risks associated with the user's health.

[0085] Step 9:

[0086] The server generates the evaluated risk information as an alert and transmits it to the terminal for notification to the user.

[0087] Step 10:

[0088] The device displays the alert information and detailed data received from the server on the app screen, using text and icons to visually communicate risk to the user.

[0089] Step 11:

[0090] The user reviews the displayed information and verbally asks additional questions if necessary.

[0091] Step 12:

[0092] The terminal uses a voice recognition function to convert the user's voice into text and send it to the server.

[0093] Step 13:

[0094] The server uses generative AI to generate the best answer based on the question and sends that answer to the device.

[0095] Step 14:

[0096] The device will play back the received answer aloud and also display it as text, allowing the user to receive information both visually and audibly.

[0097] Step 15:

[0098] The server stores the food data selected by the user and the advice received in a database, which is used to improve the accuracy of future advice and risk assessments.

[0099] This series of processes allows users to easily choose healthy foods and maintain and improve their own health.

[0100] Example 1

[0101] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0102] Choosing healthy foods is an important issue in modern society. However, many users have difficulty making appropriate choices because they do not fully understand the ingredients and health risks of foods. It is also difficult to obtain personalized advice tailored to their own health status. As a result, inappropriate food choices can have a negative impact on health. Furthermore, there is a lack of systems that can quickly respond to users' questions or requests for additional information. To address these issues, a system is needed that analyzes users' health information and food ingredients and provides individually customized advice and risk assessments.

[0103] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0104] In this invention, the server includes a means for generating personalized health advice based on information entered by the user, a means for extracting text information from a photographed food label image using OCR (optical character recognition), and a means for comparing the extracted text information with a food ingredient database, analyzing the ingredient information using a generation AI, and assessing risks associated with the user's health condition. This enables accurate advice and risk assessment that are individually customized based on the user's health condition.

[0105] "User" refers to an individual who uses the system to select healthy foods.

[0106] "Terminal" refers to a smartphone, tablet, or other electronic device used by a user to run applications and input, display, and communicate information.

[0107] "Server" refers to a remote computer system that processes data received from users and devices and provides advice and risk assessments.

[0108] "Health Advice" refers to specific recommendations to promote healthy living based on information entered by the user.

[0109] "Food label image" refers to an image of the label attached to the package of food that the user is considering purchasing, taken with a camera.

[0110] "OCR (Optical Character Recognition)" refers to a technology that analyzes the text information contained in an image and extracts it as machine-readable character data.

[0111] A "food composition database" refers to a collection of data that collects and stores information on the composition of various foods.

[0112] "Generative AI" refers to artificial intelligence that uses the technology of generative AI models (e.g., GPT-3) to generate appropriate answers and advice to human questions.

[0113] "Risk assessment" refers to analyzing the impact of ingredients contained in food on the user's health and determining the level of risk.

[0114] "Alert" refers to a warning message intended to notify the user of health risks or other important information.

[0115] "Speech recognition" refers to the technology that converts a user's voice input into text data.

[0116] A "profile" refers to a data set that records a user's basic information and health status.

[0117] "Historical Data" refers to a record of the foods a user selects and the advice they receive.

[0118] "Personalization accuracy" refers to the accuracy and suitability of the advice and information provided to an individual user.

[0119] The present invention provides a system that allows a user to easily select healthy foods. An embodiment of this system will now be described in detail.

[0120] First, the user installs the application and enters basic information such as name, age, gender, medical history, and dietary restrictions on the account creation screen when first starting the application. This basic information is used to create a user profile and is sent from the device to the server. The server generates a user profile based on the received information and stores it in a database. This allows for future analysis and provision of health advice tailored to the user.

[0121] Next, when the user takes a photo of the label of the food they are considering purchasing, an OCR (optical character recognition) module is used to extract text information from the image, using the Tesseract OCR library as an example. The extracted text data is sent from the device to a server and compared with a food ingredient database.

[0122] The server uses the received text data to compare it with a food ingredient database to obtain detailed information about each ingredient. It then analyzes the ingredients using a generative AI model (e.g., GPT-3) to assess the risk associated with the user's health. The assessed risk information is generated as an alert and sent to the device. The device displays the received alert information on the application screen and communicates the risk assessment results to the user visually and audibly.

[0123] If the user wants more detailed information, they can ask by voice. The device uses voice recognition to convert the user's question into text and send it to the server. The server then uses generative AI to generate the best answer based on the question, providing information such as a "list of low-salt foods" based on a food database and health guidelines. The device then plays back the received answer by voice and also displays it as text, allowing the user to receive information both visually and audibly.

[0124] In addition, all user-selected food data and advice received are stored in a database. The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments. Data mining algorithms can be used to extract patterns from the accumulated data and train models to improve the accuracy of advice.

[0125] (Example)

[0126] For example, if a user uses the application to scan a "potato chips" label, the OCR module extracts ingredients such as "sodium," "fat," and "sugar." The server analyzes these ingredients, and if the user is overweight and has a history of high blood pressure, an alert message will be displayed stating, "This food contains high amounts of sodium and fat, which may have adverse health effects." Furthermore, if the user asks "What are some alternative foods?" by voice, the server will list "low-sodium and low-calorie foods," and the device will provide that information via voice and text.

[0127] (Example of a prompt)

[0128] "Analyze the ingredients in this food and assess the risk."

[0129] "What healthy options can I use to replace this food?"

[0130] "Create a recommended food list for users with high blood pressure."

[0131] In this way, users can make healthy food choices without spending time or effort.

[0132] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0133] Step 1: Enter and submit basic user information

[0134] The user installs and launches the application, and enters basic information such as name, age, gender, medical history, and dietary restrictions on the account creation screen. The device then formats this information into a database and sends it to the server.

[0135] Input: Basic information entered by the user (name, age, gender, medical history, dietary restrictions)

[0136] Data processing: Convert input information into JSON format

[0137] Output: Send basic user information data in JSON format to the server

[0138] What it does: Parses data entered into form fields into JSON format and sends it via a REST API.

[0139] Step 2: Generate and save a profile on the server

[0140] The server generates a user profile based on the received user information and stores it in a database, and prepares to provide personalized health advice based on the profile.

[0141] Input: Received user basic information data

[0142] Data processing: Insert user information into the database

[0143] Output: User profile stored in the database

[0144] What it does: Runs a script that inserts user information into a MySQL database and generates profile data.

[0145] Step 3: Photograph food labels and extract text using OCR

[0146] The user launches the application and takes a photo of the label of a food item they are considering purchasing. The device inputs the image into the OCR module and extracts text information from the image.

[0147] Input: A photograph of a food label image

[0148] Data processing: Convert images to text using OCR processing

[0149] Output: Extracted text data

[0150] What it does: Uses the Tesseract OCR library to extract text from image data and save it as a string.

[0151] Step 4: Sending text data to the server

[0152] The terminal transmits the extracted text data to the server.

[0153] Input: Extracted text data

[0154] Data processing: Convert text data into JSON format

[0155] Output: Send JSON format text data to the server

[0156] Specific operation: The extracted text data is converted into JSON format and sent to the server via REST API.

[0157] Step 5: Obtaining ingredient information and risk assessment on the server

[0158] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient, then uses a generative AI model to analyze the ingredients and assess the risk associated with the user's health.

[0159] Input: Received text data

[0160] Data processing: Query the database using ingredient names as keys, and perform risk assessment using a generative AI model

[0161] Output: Risk assessment results

[0162] Specific operation: Query ingredient information based on text data, provide prompts to the generative AI model, and perform risk assessment.

[0163] Step 6: Generate and send alert information

[0164] The server generates alert information based on the results of the risk assessment and sends it to the terminal.

[0165] Input: Risk assessment results

[0166] Data processing: Converting risk assessment results into alert information

[0167] Output: Generated alert information

[0168] Specific operation: Based on the risk assessment, the alert information is parsed into JSON format and sent to the terminal.

[0169] Step 7: View alert information

[0170] The terminal displays the alert information and detailed data received from the server on the application screen.

[0171] Input: Received alert information

[0172] Data processing: Binding alert information to UI components

[0173] Output: Alert information displayed on the application screen

[0174] Specific behavior: Parses alert information and binds the data to UI components (e.g. TextView, ImageView).

[0175] Step 8: Request additional information via voice recognition

[0176] If the user wants additional information, they can ask a question by voice, and the device will use its voice recognition function to convert the question into text and send it to the server.

[0177] Input: User's voice question

[0178] Data processing: Converting voice data into text

[0179] Output: Textual question data

[0180] Specific operation: Activates the voice input function and converts the voice data into text using the Google Speech-to-Text API.

[0181] Step 9: Generate additional information on the server

[0182] The server uses generative AI to generate the best answer based on the question and sends it to the device.

[0183] Input: Textual question data

[0184] Data processing: Question data is used as a prompt and provided to the generative AI model to generate an answer

[0185] Output: Generated response data

[0186] Specific operation: The question content is provided as a prompt to the AI ​​model, an answer is generated, the answer data is converted to JSON format, and sent to the device.

[0187] Step 10: View and play answers

[0188] The terminal will play back the received answer aloud and also display it as text.

[0189] Input: Generated response data

[0190] Data processing: Converting response data into voice and text

[0191] Output: Spoken and textual answers

[0192] Specific operation: Text data is converted to audio using the Google Text-to-Speech API, played back to the user, and displayed in a text view.

[0193] Step 11: Data collection and feedback

[0194] The server stores the user's food selections and advice received in a database, accumulating feedback, and then analyzes the data to improve the accuracy of future advice and risk assessments.

[0195] Input: User food choice data, received advice

[0196] Data processing: Inserting data into databases, generating datasets for analysis

[0197] Output: Accumulated historical data, analysis results

[0198] Specific operation: User activity history is inserted into a database, and data mining algorithms are used to extract and analyze patterns from the accumulated data.

[0199] (Application example 1)

[0200] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0201] Conventional health management systems require users to spend a lot of time and effort selecting the right foods, making it difficult to provide appropriate health advice in real time. Furthermore, they are inadequate in terms of protecting user privacy and ensuring data security, making it difficult for users to manage their health data with peace of mind.

[0202] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0203] In this invention, the server includes means for generating personalized health advice based on information entered by the user, means for extracting text information from a photographed food label image using OCR (optical character recognition), means for comparing the extracted text information with a food ingredient database and analyzing the ingredient information using a generation AI to evaluate risks associated with the user's health status, means for visually and audibly notifying the user of the evaluated risk information in real time, and means for analyzing the user's voice input using voice recognition and providing appropriate answers and advice using a generation AI, thereby enabling users to make healthy food choices in real time while also ensuring privacy protection and data security.

[0204] A "user" is someone who uses the system to select healthy foods.

[0205] "Health advice" is a suggestion for appropriate food choices and lifestyle improvements based on the user's health condition.

[0206] "Food label images" are photographs or images containing ingredient and nutritional information found on food packaging.

[0207] "OCR (Optical Character Recognition)" is a technology that extracts character information from image data.

[0208] "Text information" refers to character data such as food ingredients and nutritional information extracted by OCR.

[0209] A "food composition database" is a collection of data that records the composition and nutritional information of various foods.

[0210] "Generative AI" is a system that uses machine learning and artificial intelligence technologies to analyze data, assess risks, and generate advice.

[0211] "Risk assessment" refers to the analysis and determination of the potential health risks posed by specific foods based on the user's health status and input information.

[0212] An "alert" is information that alerts or warns the user.

[0213] "Speech recognition" is a technology that analyzes a user's voice and converts it into text information.

[0214] A "smart device" is an electronic device with advanced functions, including internet connectivity and various sensors.

[0215] "Personalization accuracy" refers to the degree to which the advice and information provided is relevant to the characteristics and preferences of each individual user.

[0216] "Real-time" is a time concept that refers to data processing and information provision occurring almost instantaneously.

[0217] "Visual information" refers to information such as images and text that is presented through a display or screen.

[0218] "Audio information" refers to audio messages delivered through a speaker or speech synthesis system.

[0219] "Privacy protection" is a security measure to protect users' personal information and health data from third parties.

[0220] "Data security" means that data collected, stored and processed is protected from unauthorized access and destruction.

[0221] To realize the present invention, it is necessary to design a system using the following hardware and software.

[0222] First, the system uses a smart device (e.g., smart glasses or a smartphone) that includes a camera, display, microphone, and audio output device. A cloud-based computing service (e.g., AWS) is used as the server. For analysis and database management, an OCR module (e.g., Tesseract), generative AI (e.g., OpenAI GPT-3), and a database (e.g., MySQL) are used. For security software, Symantec Endpoint Protection is used.

[0223] 1. User registration and data management

[0224] The device presents a form for users to register their health data when they first launch the application. Users enter basic information such as name, age, gender, medical history, and dietary restrictions. This information is stored in a database and encrypted by endpoint protection software.

[0225] 2. Food Label Identification

[0226] Users take a photo of a food label using their smart device's camera. The image is sent to an OCR module to extract text information. The extracted text data is then sent to a cloud server where the generative AI analyzes the ingredient information.

[0227] 3. Component analysis and risk assessment

[0228] The cloud server analyzes the received text information by comparing it with a food composition database. The generating AI then compares it with the user's health profile and assesses health risks. For example, if a user with high blood pressure chooses a high-salt food, a risk assessment will be made, such as "This food contains a high amount of salt and may have a negative impact on high blood pressure," and the user will be notified in real time.

[0229] 4. Real-time alerts

[0230] The assessed risk information is not only visually displayed on the smart device display, but also audibly notified via the audio output device, allowing users to receive information both visually and audibly.

[0231] 5. User voice questions and feedback

[0232] If the user wants more detailed information, they can ask a question by voice. For example, they might ask, "What should I eat instead of this food?" The smart device uses its voice recognition function to convert the user's question into text and send it to a cloud server. Based on the question, the generative AI generates an answer from a food database and health guidelines, and provides feedback to the user in real time via text and voice.

[0233] Specific examples

[0234] A user selects a "low-sugar yogurt" at the supermarket and scans the label with the camera on the smart glasses. The OCR reads the ingredient information for the "low-sugar yogurt" and sends it to a cloud server. The generative AI analyzes it and displays advice on the smart glasses, such as "These ingredients are glycemic, but slightly higher in fat. Consider low-fat options."

[0235] Example prompts for generative AI models

[0236] User Profile:

[0237] Name: Tanaka Taro

[0238] Age: 45

[0239] Gender: Male

[0240] Medical history: high blood pressure

[0241] Dietary restrictions: Low salt

[0242] Input food: Low-sugar yogurt

[0243] Ingredients: Carbohydrates, fat, calcium

[0244] Questions asked:

[0245] "Is this food suitable for me?"

[0246] advice:

[0247] "These ingredients are blood sugar friendly, but they are a little higher in fat. Consider lower-fat options."

[0248] In this way, the present invention realizes a system that allows users to make healthy food choices in real time while ensuring privacy and data security.

[0249] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0250] Step 1: User registration and data management

[0251] When a user launches the application for the first time, they enter basic information such as their name, age, gender, medical history, dietary restrictions, etc. The device then sends this information to a database, where it is stored as encrypted data using endpoint protection software.

[0252] Input: User's basic information (name, age, gender, medical history, dietary restrictions)

[0253] Output: Encrypted user profile data

[0254] Specific operation: The information entered into the form on the terminal is sent to the server and stored in the destination database.

[0255] Step 2: Food Label Identification

[0256] The user takes a photo of a food label using the camera on their smart device, and the device passes the image to the OCR module to extract text information, which is then sent to the server.

[0257] Input: Food label image

[0258] Output: Extracted text information

[0259] Specific operation: The camera takes a photo of the food label, extracts text information using the OCR module (Tesseract), and sends it to the server.

[0260] Step 3: Ingredient analysis and risk assessment

[0261] The server compares the received text information with a food ingredient database to obtain detailed ingredient information, and uses generative AI to assess the risk based on the user's health status.

[0262] Input: Extracted text information, user profile

[0263] Output: Risk assessment results and advice on countermeasures

[0264] Specific operation: Using a generative AI model (GPT-3), ingredient information is analyzed and compared with the user's health profile to perform risk assessment.

[0265] Step 4: Real-time alerts

[0266] The terminal notifies the user of the risk assessment results received from the server visually and audibly, showing risk information on the display and explaining risk details via the audio output device.

[0267] Input: Risk assessment results

[0268] Output: Visual and audio risk information

[0269] Specific operation: The risk assessment results are displayed on the smart device display and notified to the user via voice output.

[0270] Step 5: User voice questions and feedback

[0271] The user asks a follow-up question by voice. The device uses voice recognition to convert the user's question into text and send it to the server. The server then uses generative AI to generate an answer and provides real-time feedback to the user in text and voice.

[0272] Input: User's voice question

[0273] Output: Answers and advice from generative AI

[0274] Specific operation: The question is converted into text using the voice recognition function, and the generative AI model (GPT-3) generates an appropriate answer based on the question. The generated answer is then fed back to the user via the device's display and voice output.

[0275] Step 6: Accumulating data and improving personalization accuracy

[0276] The server stores the food selections and advice received by the user in a database, and the accumulated data will be used to improve the accuracy of personalization in the future.

[0277] Input: Selected food data, received advice

[0278] Output: Updated user profile data

[0279] Specific operation: The server stores the selected food data and advice in a database and uses it as analytical data to improve the accuracy of future advice.

[0280] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0281] The present invention provides a system that allows users to easily select healthy foods and provides personalized advice taking into account the user's emotional state. Specific embodiments of this system are described below.

[0282] User registration and initial settings

[0283] 1. The user installs the application and enters basic information such as name, age, gender, medical history, and dietary restrictions when launching it for the first time.

[0284] 2. The terminal sends the entered information to the server.

[0285] 3. The server generates a user profile based on the received information and creates personalized health advice for the user. This profile is used for future analysis and advice provision.

[0286] Food Identification

[0287] 1. The user opens the application and takes a photo of the label of the food item they plan to purchase.

[0288] 2. The device passes the captured image to an OCR (Optical Character Recognition) module to extract text information from the image.

[0289] 3. The terminal sends the extracted text data to the server.

[0290] Ingredient analysis and risk assessment

[0291] 1. The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[0292] 2. The server uses generative AI to analyze the ingredients and assess the risks associated with the user's health.

[0293] Example: If the analysis reveals that a user with high blood pressure selects a high-salt food, a risk assessment will be made, such as "This food contains a high amount of salt and may have a negative impact on high blood pressure."

[0294] 3. The server generates the assessed risk information as an alert and sends it to the terminal to notify the user.

[0295] Viewing Alerts and Information

[0296] 1. The device displays the alert information and detailed data received from the server on the app screen.

[0297] Example: The app screen displays the risk assessment results using text and icons, visually communicating the risks to the user.

[0298] 2. The user reviews the displayed information and asks additional questions verbally if necessary.

[0299] Voice recognition and additional information provision

[0300] 1. If the user needs more information, they ask a question by voice. For example, "What should I eat instead of this food?"

[0301] 2. The device uses its voice recognition function to convert the user's voice into text and send it to the server.

[0302] 3. The server uses generative AI to generate the optimal answer based on the question and sends it to the device.

[0303] 4. The device will play back the received answer aloud and also display it as text, providing the user with both visual and auditory information.

[0304] Utilizing the Emotion Engine

[0305] 1. The emotion engine recognizes emotions from the user's voice or text input.

[0306] Example: Based on the user's tone of voice and text content, the emotion engine determines that the user is "stressed."

[0307] 2. The server customizes the advice it provides depending on the perceived emotional state.

[0308] Example: A user who is feeling stressed is offered advice on foods that have a relaxing effect and lifestyle habits to reduce stress.

[0309] 3. The server automatically selects a risk alert wording that is appropriate for the recognized emotion.

[0310] Example: Presenting risk alerts in a gentler tone to emotionally unstable users.

[0311] Data accumulation and feedback

[0312] 1. The server stores the user's food selection data, received advice, and recognized emotional state in a database.

[0313] 2. The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments.

[0314] In this way, the present invention provides a system that allows users to select appropriate foods taking into account their health and emotional state, thereby providing more personalized health support and helping users lead a healthier life with ease.

[0315] The processing flow will be explained below.

[0316] Processing procedure of a food selection system that combines an emotion engine

[0317] Step 1:

[0318] The user installs the application and enters basic information such as name, age, gender, medical history, and dietary restrictions when launching it for the first time.

[0319] Step 2:

[0320] The terminal transmits the input information to the server.

[0321] Step 3:

[0322] The server generates a user profile based on the received information, creates basic health advice tailored to the user, and stores it in a database.

[0323] Step 4:

[0324] The user launches the application and takes a photo of the label of the food item they plan to purchase.

[0325] Step 5:

[0326] The device passes the captured image to an OCR (optical character recognition) module, which extracts text information from the image.

[0327] How it works: The OCR module analyzes the image and extracts text such as ingredient information, calorie information, and additives.

[0328] Step 6:

[0329] The terminal transmits the extracted text data to the server.

[0330] Step 7:

[0331] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[0332] Step 8:

[0333] The server uses generative AI to analyze the ingredients and assess the risks associated with the user's health.

[0334] Specific operation: The health risk assessment module analyzes ingredient information and makes assessments such as "risk of high blood pressure due to high salt content."

[0335] Step 9:

[0336] The server generates the evaluated risk information as an alert and transmits it to the terminal for notification to the user.

[0337] Step 10:

[0338] The device displays the alert information and detailed data received from the server on the app screen.

[0339] Example: The app screen might say something like, "This food contains high salt content and may have a negative impact on high blood pressure."

[0340] Step 11:

[0341] The user reviews the displayed information and verbally asks additional questions if necessary.

[0342] Step 12:

[0343] The terminal uses a voice recognition function to convert the user's voice into text and send it to the server.

[0344] Specific operation: The speech recognition module analyzes the speech and converts it into text.

[0345] Step 13:

[0346] The server uses generative AI to generate the best answer based on the question and sends that answer to the device.

[0347] Example: Generate specific suggestions such as "List of low-sodium foods."

[0348] Step 14:

[0349] The device will play the received answer aloud and also display it as text.

[0350] Specific operation: Passes the text to the text-to-speech (TTS) module and reads it aloud.

[0351] Utilizing the Emotion Engine

[0352] Step 15:

[0353] The emotion engine recognizes emotions from the user's voice and text input.

[0354] Example: Recognizing emotions such as "anxiety" and "stress" from the user's voice.

[0355] Step 16:

[0356] The server customizes the advice it provides depending on the perceived emotional state.

[0357] Example: For users who are under stress, we suggest foods that have a relaxing effect and lifestyle habits that will help relieve stress.

[0358] Step 17:

[0359] The server automatically selects a risk alert presentation that is appropriate for the recognized emotion.

[0360] Specific behavior: For users who are emotionally anxious, a gentle tone of alert is displayed.

[0361] Data accumulation and feedback

[0362] Step 18:

[0363] The server stores the user's food selections, received advice, and recognized emotional state in a database.

[0364] Step 19:

[0365] The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments.

[0366] Example: A data analytics module analyzes accumulated data to improve the advice-giving algorithm.

[0367] Through this process, users can select the best foods based on their health and emotional state, providing more personalized health support and helping them live a healthier life.

[0368] Example 2

[0369] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0370] In recent years, with the increasing interest in health, there is a demand for personalized food selection support that takes into account individual health and emotional states. However, when users select healthy foods, it is not easy to accurately understand ingredient information and receive appropriate advice based on their current physical condition and emotional state. Furthermore, the advice provided is uniform and does not address individual emotional states, which may reduce user satisfaction.

[0371] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0372] In this invention, the server includes means for generating personalized health advice based on information entered by the user, means for extracting text information from a photographed food label image using optical character recognition, means for comparing the extracted text information with a food ingredient database and analyzing the ingredient information using a generation AI to evaluate risks associated with the user's health condition, means for notifying the user of the evaluated risk information as an alert, means for analyzing the user's voice input using voice recognition and providing appropriate answers and advice using a generation AI, means for analyzing the user's emotional state and customizing the advice to be provided depending on the emotional state, and means for selecting a method of expressing the risk alert appropriate to the emotional state. This enables the user to receive personalized advice that takes into account not only their health condition but also their emotional state.

[0373] "User" refers to an individual who uses the system and is the target of assistance in managing health conditions and food choices.

[0374] "Personalized health advice" refers to health guidance and recommendations tailored to a particular user based on user input.

[0375] Optical character recognition is a technology that extracts text information from images and photographs, and is used to incorporate food label information into the system.

[0376] The "Food Composition Database" is a database that collects information on the composition of various foods and provides basic data for composition analysis and risk assessment.

[0377] "Generative AI" refers to artificial intelligence technology that performs natural language processing and information generation based on large amounts of data, and is used to generate health advice and answers for users.

[0378] "Risk alert" refers to warning information that notifies users of health risks associated with food choices.

[0379] "Speech recognition" is a technology that converts a user's voice input into text data and is used to process spoken queries and commands.

[0380] "Emotional state" refers to the user's psychological and emotional state, and represents emotions such as stress or joy that can be inferred from voice and text input.

[0381] "Personalization" refers to providing optimized information and services that take into account the characteristics of each individual user.

[0382] "Food label" refers to the text and images containing ingredient and nutritional information that appear on food packaging.

[0383] MODE FOR CARRYING OUT THE INVENTION

[0384] The present invention provides a system that allows users to easily select healthy foods and provides personalized advice taking into account the user's emotional state. Specific embodiments of the system are described below.

[0385] User registration and initial settings

[0386] First, the user installs and launches the application on their device. When launching the application for the first time, the user enters basic information such as their name, age, gender, medical history, and dietary restrictions. The device then sends this information to the server. The server then creates a user profile based on the received information and creates initial health advice tailored to the user. For example, a user who needs to limit their fat intake may be provided with advice recommending low-fat foods.

[0387] Food Identification

[0388] The user uses the application to take a photo of the label of the food they plan to purchase with their camera. The device then passes the image to an optical character recognition (OCR) module, which extracts text information from the image. Specifically, the device uses an OCR service such as Google Cloud Vision API. The extracted text information is then sent from the device to a server.

[0389] Ingredient analysis and risk assessment

[0390] The server compares the received text information with a food ingredient database (e.g., USDA Nutrient Database) to obtain detailed information about each ingredient. Next, it uses generative AI (e.g., OpenAI's GPT-4) to analyze the ingredient information and evaluate the risk associated with the user's health. An example of a specific prompt is, "How does this ingredient affect high blood pressure?" The evaluated risk information is generated as an alert and sent to the device.

[0391] Viewing Alerts and Information

[0392] The device displays the alert information and detailed data received from the server on the app screen. For example, the app screen displays the risk assessment results using text and icons, visually communicating the risk to the user. The user can check the displayed information and ask additional questions by voice if necessary.

[0393] Voice recognition and additional information provision

[0394] If the user needs additional information, they can ask a question by voice. For example, they might ask, "What should I eat instead of this food?" The device uses a voice recognition function (e.g., Google Speech-to-Text) to convert the user's voice into text and send it to the server. The server then uses a generative AI to generate the optimal answer based on the question and sends it to the device. The device then plays the received answer aloud and also displays it as text.

[0395] Utilizing the Emotion Engine

[0396] An emotion engine (e.g., Microsoft Azure Emotion API) recognizes the user's emotional state from their voice or text input. Specifically, it determines whether the user is "feeling stressed" based on the user's tone of voice and the content of the text. The server then customizes the advice it provides based on the recognized emotional state. For example, a user who is feeling stressed could be offered advice about foods and lifestyle habits that have a relaxing effect. At the same time, the server selects a risk alert presentation style appropriate to the user's emotional state and sends it to the device.

[0397] Data accumulation and feedback

[0398] The server stores the user's food selections, advice received, and perceived emotional state in a database. This accumulated data is used to improve the accuracy of future advice and risk assessments. Specifically, the data is analyzed using data analytics tools (e.g., Python's Pandas and Scikit-learn), and the algorithm is improved based on the analysis results.

[0399] In this way, the present invention provides a system that allows users to select appropriate foods taking into account their health and emotional state, thereby providing more personalized health support and helping users lead a healthier life with ease.

[0400] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0401] Step 1:

[0402] Users install and launch the application on their device, then enter basic information such as their name, age, gender, medical history, and dietary restrictions.

[0403] Input: User's basic information (name, age, gender, medical history, dietary restrictions, etc.)

[0404] Specific action: The user enters information into an input form within the app and presses the "Submit" button.

[0405] Output: Request data with basic information entered

[0406] Step 2:

[0407] The terminal transmits the input basic information to the server.

[0408] Input: Request data (basic user information)

[0409] Specific operation: The terminal generates an HTTP request and sends data to the server.

[0410] Output: Basic information sent to the server

[0411] Step 3:

[0412] The server generates a user profile based on the received basic information, which also includes initial health advice for the user.

[0413] Input: User basic information

[0414] Specific operations: The server stores the information in a database and runs the profile generation algorithm.

[0415] Output: Generated user profile and initial advice

[0416] Step 4:

[0417] The user uses the application to take a photo of the label of the food item they plan to purchase with their camera.

[0418] Input: Food label image

[0419] Specific operation: The user activates the camera function and takes a picture of the food label.

[0420] Output: Photographed food label image

[0421] Step 5:

[0422] The device passes the captured food label image to an OCR module to extract text information.

[0423] Input: Food label image

[0424] Specific operation: Send the image to an OCR module (e.g., Google Cloud Vision API) and extract text information.

[0425] Output: Extracted text data

[0426] Step 6:

[0427] The terminal transmits the extracted text data to the server.

[0428] Input: Extracted text data

[0429] Specific operation: The terminal generates an HTTP request and sends data to the server.

[0430] Output: Text data sent to the server

[0431] Step 7:

[0432] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[0433] Input: Text data

[0434] Specific operation: The server queries the food ingredient database and retrieves ingredient information.

[0435] Output: Detailed information for each component

[0436] Step 8:

[0437] The server uses generative AI to analyze the ingredient information and assess the risks associated with the user's health condition.

[0438] Input: Ingredient information, user profile

[0439] Specific behavior: Send an analytical prompt to an AI model (e.g., OpenAI's GPT-4) to perform risk assessment. Specific prompt example: "How does this ingredient affect high blood pressure?"

[0440] Output: Risk assessment results

[0441] Step 9:

[0442] The server generates the evaluated risk information as an alert and transmits it to the terminal for notification to the user.

[0443] Input: Risk assessment results

[0444] Specific operation: An alert message is generated and sent to the terminal as an HTTP response.

[0445] Output: Generated risk alerts

[0446] Step 10:

[0447] The device displays the alert information and detailed data received from the server on the app screen.

[0448] Enter: Risk Alert

[0449] Specific behavior: Parse the risk alert and display it in a UI component (e.g., a text view or an icon).

[0450] Output: Risk alert displayed on the app screen

[0451] Step 11:

[0452] The user reviews the displayed information and verbally asks additional questions if necessary.

[0453] Input: Risk assessment results displayed on the terminal

[0454] Specific operation: The user presses the microphone button and begins voice input.

[0455] Output: User's voice data

[0456] Step 12:

[0457] The terminal uses a voice recognition function to convert the user's voice into text and send it to the server.

[0458] Input: User's voice data

[0459] Specific operation: Uses a speech recognition API (e.g., Google Speech-to-Text) to convert voice data into text, and sends the converted text to the server.

[0460] Output: Textual question data

[0461] Step 13:

[0462] The server uses generative AI to generate the best answer based on the question and sends it to the device.

[0463] Input: Textual question data

[0464] Specific operation: Send a prompt (e.g., "What should I eat instead of this food?") to the generating AI and get an answer.

[0465] Output: Generated response data

[0466] Step 14:

[0467] The device will play the received answer aloud and also display it as text.

[0468] Input: Generated response data

[0469] Specific behavior: Converts text to speech using a speech synthesis API (e.g., Google Text-to-Speech), plays the converted speech, and displays the text in the UI.

[0470] Output: Answer information as audio and text

[0471] Step 15:

[0472] The emotion engine recognizes the user's emotional state from their voice and text input.

[0473] Input: User's voice data, text data

[0474] Specific behavior: Uses emotion recognition APIs (e.g. Microsoft Emotion API) to infer emotions from speech and text.

[0475] Output: Perceived emotional state

[0476] Step 16:

[0477] The server customizes the advice it provides depending on the perceived emotional state.

[0478] Input: perceived emotional state, user profile

[0479] Specific behavior: Send a prompt (e.g., "What should you recommend to a user who is feeling stressed?") to the generative AI to get customized advice.

[0480] Output: Customized advice

[0481] Step 17:

[0482] The server selects a method of expressing a risk alert that is appropriate for the emotional state and notifies the terminal.

[0483] Input: Customized advice, risk assessment results

[0484] Specific behavior: Adjust the tone and format of the text depending on the emotional state and send it to the device in an HTTP response.

[0485] Output: Adjusted risk alerts

[0486] Step 18:

[0487] The server stores the user's food selection data, received advice, and recognized emotional state in a database.

[0488] Input: User's food data, advice, perceived emotional state

[0489] Specific operation: Executes a data insertion query against the database.

[0490] Output: Information stored in the database

[0491] Step 19:

[0492] The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments.

[0493] Input: Accumulated data

[0494] Specific Tasks: Analyze data and improve algorithms using data analytics tools (e.g., Python's Pandas, Scikit-learn).

[0495] Output: Improved algorithm

[0496] (Application example 2)

[0497] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0498] Many consumers today are becoming increasingly health-conscious and understand the importance of choosing foods that are right for them. However, it is difficult to instantly evaluate food ingredients and health risks while shopping in a physical store, and to receive optimal advice based on one's emotional state at the time. Especially in today's world where a wide variety of foods are available, obtaining detailed information about each food item takes time and effort. Furthermore, existing systems are unable to provide advice that takes into account the user's emotional state, making it difficult to provide accurate support tailored to each user's individual situation.

[0499] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for generating personalized health advice based on information entered by a user, means for extracting text information from a photographed food label image using optical character recognition, means for comparing the extracted text information with a food ingredient database, analyzing the ingredient information using a generation AI, and evaluating risks associated with the user's health status, means for notifying the user of the evaluated risk information as an alert, means for analyzing the user's voice input using voice recognition and providing appropriate answers and advice using a generation AI, means for analyzing the user's emotional state and personalizing advice according to the emotional state, and means for providing recommended products and purchase plans suited to the user's health status and emotional status based on data from a physical store. This allows users to easily select foods suitable for them in a physical store and receive personalized advice tailored to their health status and emotional state.

[0500] "Means for generating personalized health advice based on user-entered information" refers to a method for collecting information entered by a user into the application, such as name, age, gender, medical history, and dietary restrictions, and generating and providing specific health advice based on that information.

[0501] "Means for extracting text information from photographed food label images using optical character recognition" refers to a method in which a user takes a photo of a food label using a camera on a smartphone or other device, and extracts text information from the image using optical character recognition technology.

[0502] "Means of comparing extracted text information with a food ingredient database, analyzing ingredient information using generative AI, and assessing risks related to the user's health" refers to a method of comparing text information obtained by optical character recognition with a food ingredient database, and then using a generative AI model to perform a detailed analysis of the ingredients and assess the user's health risks.

[0503] "Means for notifying the user of the evaluated risk information as an alert" refers to a method for evaluating the risk to the user's health based on the analyzed component information and notifying the user of the results as a warning.

[0504] "Means of analyzing a user's voice input through speech recognition and using generative AI to provide appropriate answers and advice" refers to a method of converting a user's spoken questions or requests into text using speech recognition technology, analyzing it using a generative AI model, and generating appropriate answers and advice.

[0505] "Means for analyzing the user's emotional state and personalizing advice according to that emotional state" refers to a method for analyzing emotional information obtained from the user's voice input or text input and providing customized advice according to that emotion.

[0506] "Means for providing recommended products and purchase plans suited to the user's health and emotional state based on data from physical stores" refers to a method that utilizes user behavior data and selected product data obtained from physical stores to suggest products and purchase plans that are optimal for the user's health and emotional state.

[0507] "Optical character recognition" is a technology that mechanically reads characters printed on paper or text contained in images and extracts it as digital data.

[0508] "Generative AI" is an artificial intelligence technology that generates new information and answers based on pre-trained data.

[0509] The present invention provides a system that allows users to easily select foods suitable for them in a physical store and receive personalized advice tailored to their health and emotional state. Specific embodiments of this system are described below.

[0510] User registration and initial settings

[0511] Users enter basic information such as name, age, gender, medical history, and dietary restrictions through an application installed on their smartphone. The device then sends this information to a server, which then generates a user profile based on the information received and prepares health advice. This profile is then used for future analysis and advice provision.

[0512] Food Identification

[0513] A user takes a photo of the label of a food item they are considering purchasing in a physical store using their smartphone camera. The device passes the image to an optical character recognition (OCR) module, which extracts text information from the image. The extracted text data is then sent to a server.

[0514] Ingredient analysis and risk assessment

[0515] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient. It then uses a generative AI model to analyze the ingredients and assess the risk associated with the user's health. For example, if the analysis reveals that a user with high blood pressure chooses a high-salt food, a risk assessment will be made, such as, "This food contains high salt and may have a negative impact on high blood pressure." The assessed risk information is generated as an alert and sent to the device to notify the user.

[0516] Viewing Alerts and Information

[0517] The device displays the alert information and detailed data received from the server on the app screen. For example, the risk assessment results can be displayed as text and icons to visually inform the user of the risk. If the user asks additional questions by voice, the voice is converted into text by a voice recognition module and sent to the server.

[0518] Voice recognition and additional information provision

[0519] The server uses a generative AI model to generate an optimal answer based on the content of the voice question and sends it to the device, which then plays the received answer back aloud and can also display it as text, allowing the user to obtain information both visually and audibly.

[0520] Utilizing the Emotion Engine

[0521] The emotion engine recognizes emotions from the user's voice and text input. For example, the emotion engine may determine that the user is "feeling stressed" based on the user's tone of voice and the content of the text. The server then customizes the advice it provides based on the recognized emotional state. Specifically, for a user who is feeling stressed, it provides advice on foods that have a relaxing effect and lifestyle habits to reduce stress. For users with unstable emotions, it displays risk alerts in a gentler tone.

[0522] Data accumulation and feedback

[0523] The server stores the user's food selections, received advice, and perceived emotional state in a database, which is used to improve the accuracy of future advice and risk assessments.

[0524] This allows users to easily choose the foods that are right for them in physical stores and receive personalized advice tailored to their health and emotional state.

[0525] Prompt Sentence Examples

[0526] Below are some examples of prompt sentences.

[0527] Food ingredient analysis prompt:

[0528] You are given the following list of food ingredients. Analyze the risks these ingredients pose to a user with high blood pressure and perform a risk assessment.

[0529] Ingredients: Salt

[0530] Ingredients: Sodium 1500mg

[0531] Emotion recognition prompts:

[0532] Recognize emotions from the following user voice inputs:

[0533] Text: "I'm feeling super stressed today. Is there a good way to relax?"

[0534] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0535] Step 1:

[0536] Users install the application on their smartphone and enter basic information such as their name, age, gender, medical history, and dietary restrictions. The application then collects the initial data to create a user profile. The entered information is sent from the device to a server. The server receives this information, generates a user profile, and stores it as the basis for health advice.

[0537] Step 2:

[0538] Users take a photo of the label of a food item they are considering purchasing in a physical store using their smartphone camera. The device then passes the image to an optical character recognition (OCR) module to extract text information from the image. This extraction process utilizes OpenCV and the Tesseract library. The extracted text data is then sent to a server.

[0539] Step 3:

[0540] The server compares the received text data with a food ingredient database, thereby obtaining detailed information about each ingredient. It then analyzes the ingredient information using a generative AI model (e.g., GPT-4) to assess the risk associated with the user's health. The results of the risk assessment are formatted as specific alert information, such as "not suitable for users with high blood pressure." The assessment results are then sent from the server to the device.

[0541] Step 4:

[0542] The terminal visually displays the alert information and detailed data received from the server to the user. The alert information is expressed using text and icons, and is used to intuitively convey risk information to the user. If the user has additional questions, their voice is converted into text using the application's speech recognition module. The Google Speech-to-Text API is used for speech recognition. The text data of the question is sent to the server.

[0543] Step 5:

[0544] The server generates an appropriate answer using a generative AI model based on the text data received as a speech recognition result. The generated answer is formatted as specific advice and sent from the server to the device. The device displays this advice visually to the user and plays it aloud using speech synthesis (Google Text-to-Speech API).

[0545] Step 6:

[0546] The emotional state of the user is analyzed using an emotion engine (e.g., Amazon Rekognition) based on the user's voice or text input. The server then personalizes advice according to the user's emotional state based on the emotional data received from the emotion engine. Advice is generated according to the emotion, and customization is performed, such as displaying risk alerts in a gentle tone. The customized advice is then sent from the server to the device.

[0547] Step 7:

[0548] The server stores the user's food selections, advice received, and perceived emotional state in a database. This data is used to improve the accuracy of future advice and risk assessments. This data is associated with the user's profile and analyzed to provide more appropriate advice.

[0549] These steps allow users to easily choose the foods that are right for them in a physical store and receive personalized advice based on their health and emotional state.

[0550] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0551] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0552] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0553] [Second embodiment]

[0554] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0555] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0556] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0557] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0558] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0559] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0560] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0561] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0562] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0563] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0564] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0565] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0566] The present invention provides a system that allows users to easily select healthy foods, and includes a program that operates by combining various technologies to achieve this. Specific embodiments of this system are described below.

[0567] User registration and initial settings

[0568] 1. The user installs the application and, when launching it for the first time, enters basic information such as name, age, gender, medical history, and dietary restrictions on the account creation screen.

[0569] 2. The terminal sends the information entered by the user to the server.

[0570] 3. The server generates a user profile based on the received information and tailors health advice to the user. This profile is stored in a database for future analysis and advice provision.

[0571] Food Identification

[0572] 1. The user opens the application and takes a photo of the label of a food item they are considering purchasing.

[0573] 2. The device passes the captured image to an OCR (Optical Character Recognition) module to extract text information from the image.

[0574] Example: If a food label contains information such as "phosphates," "sodium," and "sugars," these will be read as text.

[0575] 3. The terminal sends the extracted text data to the server.

[0576] Ingredient analysis and risk assessment

[0577] 1. The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[0578] 2. The server uses generative AI to analyze the ingredients and assess the risks associated with the user's health.

[0579] Example: If the analysis reveals that a user with high blood pressure selects a high-salt food, a risk assessment will be made, such as "This food contains a high amount of salt and may have a negative impact on high blood pressure."

[0580] 3. The server generates the results of the risk assessment as alert information and sends it to the terminal.

[0581] Viewing Alerts and Information

[0582] 1. The terminal displays the alert information and detailed data received from the server on the application screen.

[0583] Example: The application screen displays the results of the risk assessment using text and icons, visually communicating the risks to the user.

[0584] 2. The user reviews the information displayed and asks additional questions if necessary.

[0585] Voice recognition and additional information provision

[0586] 1. If the user wants more information, they can ask a question out loud. For example, "What should I eat instead of this food?"

[0587] 2. The device uses voice recognition to convert the user's question into text and send it to the server.

[0588] 3. The server uses generative AI to generate the best answer based on the question.

[0589] Example: Create a "list of low-salt foods" based on a food database and health guidelines and send it to your device.

[0590] 4. The device will play the received answer aloud and also display it as text, allowing the user to receive information both visually and audibly.

[0591] Data accumulation and feedback

[0592] 1. The server stores the user's food selection data and received advice in a database.

[0593] 2. The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments.

[0594] In this way, the present invention realizes a system that allows users to easily select healthy foods. By providing users with checked ingredient information and risk alerts, the system helps users make healthy food choices. This system allows users to make healthy food choices without spending time or effort.

[0595] The processing flow will be explained below.

[0596] Food selection system processing procedure

[0597] Step 1:

[0598] Users install the app and enter basic information such as their name, age, gender, medical history, and dietary restrictions when they first launch it.

[0599] Step 2:

[0600] The terminal transmits the input information to the server.

[0601] Step 3:

[0602] The server generates a user profile based on the received information and sets basic health advice tailored to the user. This profile is used for future analysis and advice provision.

[0603] Step 4:

[0604] The user launches the app and takes a photo of the label of the food item they plan to purchase.

[0605] Step 5:

[0606] The device passes the captured image to an OCR (optical character recognition) module, which extracts text information from the image.

[0607] Step 6:

[0608] The terminal transmits the extracted text data to the server.

[0609] Step 7:

[0610] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[0611] Step 8:

[0612] The server uses generative AI to analyze the ingredients and assess the risks associated with the user's health.

[0613] Step 9:

[0614] The server generates the evaluated risk information as an alert and transmits it to the terminal for notification to the user.

[0615] Step 10:

[0616] The device displays the alert information and detailed data received from the server on the app screen, using text and icons to visually communicate risk to the user.

[0617] Step 11:

[0618] The user reviews the displayed information and verbally asks additional questions if necessary.

[0619] Step 12:

[0620] The terminal uses a voice recognition function to convert the user's voice into text and send it to the server.

[0621] Step 13:

[0622] The server uses generative AI to generate the best answer based on the question and sends that answer to the device.

[0623] Step 14:

[0624] The device will play back the received answer aloud and also display it as text, allowing the user to receive information both visually and audibly.

[0625] Step 15:

[0626] The server stores the food data selected by the user and the advice received in a database, which is used to improve the accuracy of future advice and risk assessments.

[0627] This series of processes allows users to easily choose healthy foods and maintain and improve their own health.

[0628] Example 1

[0629] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0630] Choosing healthy foods is an important issue in modern society. However, many users have difficulty making appropriate choices because they do not fully understand the ingredients and health risks of foods. It is also difficult to obtain personalized advice tailored to their own health status. As a result, inappropriate food choices can have a negative impact on health. Furthermore, there is a lack of systems that can quickly respond to users' questions or requests for additional information. To address these issues, a system is needed that analyzes users' health information and food ingredients and provides individually customized advice and risk assessments.

[0631] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0632] In this invention, the server includes a means for generating personalized health advice based on information entered by the user, a means for extracting text information from a photographed food label image using OCR (optical character recognition), and a means for comparing the extracted text information with a food ingredient database, analyzing the ingredient information using a generation AI, and assessing risks associated with the user's health condition. This enables accurate advice and risk assessment that are individually customized based on the user's health condition.

[0633] "User" refers to an individual who uses the system to select healthy foods.

[0634] "Terminal" refers to a smartphone, tablet, or other electronic device used by a user to run applications and input, display, and communicate information.

[0635] "Server" refers to a remote computer system that processes data received from users and devices and provides advice and risk assessments.

[0636] "Health Advice" refers to specific recommendations to promote healthy living based on information entered by the user.

[0637] "Food label image" refers to an image of the label attached to the package of food that the user is considering purchasing, taken with a camera.

[0638] "OCR (Optical Character Recognition)" refers to a technology that analyzes the text information contained in an image and extracts it as machine-readable character data.

[0639] A "food composition database" refers to a collection of data that collects and stores information on the composition of various foods.

[0640] "Generative AI" refers to artificial intelligence that uses the technology of generative AI models (e.g., GPT-3) to generate appropriate answers and advice to human questions.

[0641] "Risk assessment" refers to analyzing the impact of ingredients contained in food on the user's health and determining the level of risk.

[0642] "Alert" refers to a warning message intended to notify the user of health risks or other important information.

[0643] "Speech recognition" refers to the technology that converts a user's voice input into text data.

[0644] A "profile" refers to a data set that records a user's basic information and health status.

[0645] "Historical Data" refers to a record of the foods a user selects and the advice they receive.

[0646] "Personalization accuracy" refers to the accuracy and suitability of the advice and information provided to an individual user.

[0647] The present invention provides a system that allows a user to easily select healthy foods. An embodiment of this system will now be described in detail.

[0648] First, the user installs the application and enters basic information such as name, age, gender, medical history, and dietary restrictions on the account creation screen when first starting the application. This basic information is used to create a user profile and is sent from the device to the server. The server generates a user profile based on the received information and stores it in a database. This allows for future analysis and provision of health advice tailored to the user.

[0649] Next, when the user takes a photo of the label of the food they are considering purchasing, an OCR (optical character recognition) module is used to extract text information from the image, using the Tesseract OCR library as an example. The extracted text data is sent from the device to a server and compared with a food ingredient database.

[0650] The server uses the received text data to compare it with a food ingredient database to obtain detailed information about each ingredient. It then analyzes the ingredients using a generative AI model (e.g., GPT-3) to assess the risk associated with the user's health. The assessed risk information is generated as an alert and sent to the device. The device displays the received alert information on the application screen and communicates the risk assessment results to the user visually and audibly.

[0651] If the user wants more detailed information, they can ask by voice. The device uses voice recognition to convert the user's question into text and send it to the server. The server then uses generative AI to generate the best answer based on the question, providing information such as a "list of low-salt foods" based on a food database and health guidelines. The device then plays back the received answer by voice and also displays it as text, allowing the user to receive information both visually and audibly.

[0652] In addition, all user-selected food data and advice received are stored in a database. The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments. Data mining algorithms can be used to extract patterns from the accumulated data and train models to improve the accuracy of advice.

[0653] (Example)

[0654] For example, if a user uses the application to scan a "potato chips" label, the OCR module extracts ingredients such as "sodium," "fat," and "sugar." The server analyzes these ingredients, and if the user is overweight and has a history of high blood pressure, an alert message will be displayed stating, "This food contains high amounts of sodium and fat, which may have adverse health effects." Furthermore, if the user asks "What are some alternative foods?" by voice, the server will list "low-sodium and low-calorie foods," and the device will provide that information via voice and text.

[0655] (Example of a prompt)

[0656] "Analyze the ingredients in this food and assess the risk."

[0657] "What healthy options can I use to replace this food?"

[0658] "Create a recommended food list for users with high blood pressure."

[0659] In this way, users can make healthy food choices without spending time or effort.

[0660] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0661] Step 1: Enter and submit basic user information

[0662] The user installs and launches the application, and enters basic information such as name, age, gender, medical history, and dietary restrictions on the account creation screen. The device then formats this information into a database and sends it to the server.

[0663] Input: Basic information entered by the user (name, age, gender, medical history, dietary restrictions)

[0664] Data processing: Convert input information into JSON format

[0665] Output: Send basic user information data in JSON format to the server

[0666] What it does: Parses data entered into form fields into JSON format and sends it via a REST API.

[0667] Step 2: Generate and save a profile on the server

[0668] The server generates a user profile based on the received user information and stores it in a database, and prepares to provide personalized health advice based on the profile.

[0669] Input: Received user basic information data

[0670] Data processing: Insert user information into the database

[0671] Output: User profile stored in the database

[0672] What it does: Runs a script that inserts user information into a MySQL database and generates profile data.

[0673] Step 3: Photograph food labels and extract text using OCR

[0674] The user launches the application and takes a photo of the label of a food item they are considering purchasing. The device inputs the image into the OCR module and extracts text information from the image.

[0675] Input: A photograph of a food label image

[0676] Data processing: Convert images to text using OCR processing

[0677] Output: Extracted text data

[0678] What it does: Uses the Tesseract OCR library to extract text from image data and save it as a string.

[0679] Step 4: Sending text data to the server

[0680] The terminal transmits the extracted text data to the server.

[0681] Input: Extracted text data

[0682] Data processing: Convert text data into JSON format

[0683] Output: Send JSON format text data to the server

[0684] Specific operation: The extracted text data is converted into JSON format and sent to the server via REST API.

[0685] Step 5: Obtaining ingredient information and risk assessment on the server

[0686] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient, then uses a generative AI model to analyze the ingredients and assess the risk associated with the user's health.

[0687] Input: Received text data

[0688] Data processing: Query the database using ingredient names as keys, and perform risk assessment using a generative AI model

[0689] Output: Risk assessment results

[0690] Specific operation: Query ingredient information based on text data, provide prompts to the generative AI model, and perform risk assessment.

[0691] Step 6: Generate and send alert information

[0692] The server generates alert information based on the results of the risk assessment and sends it to the terminal.

[0693] Input: Risk assessment results

[0694] Data processing: Converting risk assessment results into alert information

[0695] Output: Generated alert information

[0696] Specific operation: Based on the risk assessment, the alert information is parsed into JSON format and sent to the terminal.

[0697] Step 7: View alert information

[0698] The terminal displays the alert information and detailed data received from the server on the application screen.

[0699] Input: Received alert information

[0700] Data processing: Binding alert information to UI components

[0701] Output: Alert information displayed on the application screen

[0702] Specific behavior: Parses alert information and binds the data to UI components (e.g. TextView, ImageView).

[0703] Step 8: Request additional information via voice recognition

[0704] If the user wants additional information, they can ask a question by voice, and the device will use its voice recognition function to convert the question into text and send it to the server.

[0705] Input: User's voice question

[0706] Data processing: Converting voice data into text

[0707] Output: Textual question data

[0708] Specific operation: Activates the voice input function and converts the voice data into text using the Google Speech-to-Text API.

[0709] Step 9: Generate additional information on the server

[0710] The server uses generative AI to generate the best answer based on the question and sends it to the device.

[0711] Input: Textual question data

[0712] Data processing: Question data is used as a prompt and provided to the generative AI model to generate an answer

[0713] Output: Generated response data

[0714] Specific operation: The question content is provided as a prompt to the AI ​​model, an answer is generated, the answer data is converted to JSON format, and sent to the device.

[0715] Step 10: View and play answers

[0716] The terminal will play back the received answer aloud and also display it as text.

[0717] Input: Generated response data

[0718] Data processing: Converting response data into voice and text

[0719] Output: Spoken and textual answers

[0720] Specific operation: Text data is converted to audio using the Google Text-to-Speech API, played back to the user, and displayed in a text view.

[0721] Step 11: Data collection and feedback

[0722] The server stores the user's food selections and advice received in a database, accumulating feedback, and then analyzes the data to improve the accuracy of future advice and risk assessments.

[0723] Input: User food choice data, received advice

[0724] Data processing: Inserting data into databases, generating datasets for analysis

[0725] Output: Accumulated historical data, analysis results

[0726] Specific operation: User activity history is inserted into a database, and data mining algorithms are used to extract and analyze patterns from the accumulated data.

[0727] (Application example 1)

[0728] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0729] Conventional health management systems require users to spend a lot of time and effort selecting the right foods, making it difficult to provide appropriate health advice in real time. Furthermore, they are inadequate in terms of protecting user privacy and ensuring data security, making it difficult for users to manage their health data with peace of mind.

[0730] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0731] In this invention, the server includes means for generating personalized health advice based on information entered by the user, means for extracting text information from a photographed food label image using OCR (optical character recognition), means for comparing the extracted text information with a food ingredient database and analyzing the ingredient information using a generation AI to evaluate risks associated with the user's health status, means for visually and audibly notifying the user of the evaluated risk information in real time, and means for analyzing the user's voice input using voice recognition and providing appropriate answers and advice using a generation AI, thereby enabling users to make healthy food choices in real time while also ensuring privacy protection and data security.

[0732] A "user" is someone who uses the system to select healthy foods.

[0733] "Health advice" is a suggestion for appropriate food choices and lifestyle improvements based on the user's health condition.

[0734] "Food label images" are photographs or images containing ingredient and nutritional information found on food packaging.

[0735] "OCR (Optical Character Recognition)" is a technology that extracts character information from image data.

[0736] "Text information" refers to character data such as food ingredients and nutritional information extracted by OCR.

[0737] A "food composition database" is a collection of data that records the composition and nutritional information of various foods.

[0738] "Generative AI" is a system that uses machine learning and artificial intelligence technologies to analyze data, assess risks, and generate advice.

[0739] "Risk assessment" refers to the analysis and determination of the potential health risks posed by specific foods based on the user's health status and input information.

[0740] An "alert" is information that alerts or warns the user.

[0741] "Speech recognition" is a technology that analyzes a user's voice and converts it into text information.

[0742] A "smart device" is an electronic device with advanced functions, including internet connectivity and various sensors.

[0743] "Personalization accuracy" refers to the degree to which the advice and information provided is relevant to the characteristics and preferences of each individual user.

[0744] "Real-time" is a time concept that refers to data processing and information provision occurring almost instantaneously.

[0745] "Visual information" refers to information such as images and text that is presented through a display or screen.

[0746] "Audio information" refers to audio messages delivered through a speaker or speech synthesis system.

[0747] "Privacy protection" is a security measure to protect users' personal information and health data from third parties.

[0748] "Data security" means that data collected, stored and processed is protected from unauthorized access and destruction.

[0749] To realize the present invention, it is necessary to design a system using the following hardware and software.

[0750] First, the system uses a smart device (e.g., smart glasses or a smartphone) that includes a camera, display, microphone, and audio output device. A cloud-based computing service (e.g., AWS) is used as the server. For analysis and database management, an OCR module (e.g., Tesseract), generative AI (e.g., OpenAI GPT-3), and a database (e.g., MySQL) are used. For security software, Symantec Endpoint Protection is used.

[0751] 1. User registration and data management

[0752] The device presents a form for users to register their health data when they first launch the application. Users enter basic information such as name, age, gender, medical history, and dietary restrictions. This information is stored in a database and encrypted by endpoint protection software.

[0753] 2. Food Label Identification

[0754] Users take a photo of a food label using their smart device's camera. The image is sent to an OCR module to extract text information. The extracted text data is then sent to a cloud server where the generative AI analyzes the ingredient information.

[0755] 3. Component analysis and risk assessment

[0756] The cloud server analyzes the received text information by comparing it with a food composition database. The generating AI then compares it with the user's health profile and assesses health risks. For example, if a user with high blood pressure chooses a high-salt food, a risk assessment will be made, such as "This food contains a high amount of salt and may have a negative impact on high blood pressure," and the user will be notified in real time.

[0757] 4. Real-time alerts

[0758] The assessed risk information is not only visually displayed on the smart device display, but also audibly notified via the audio output device, allowing users to receive information both visually and audibly.

[0759] 5. User voice questions and feedback

[0760] If the user wants more detailed information, they can ask a question by voice. For example, they might ask, "What should I eat instead of this food?" The smart device uses its voice recognition function to convert the user's question into text and send it to a cloud server. Based on the question, the generative AI generates an answer from a food database and health guidelines, and provides feedback to the user in real time via text and voice.

[0761] Specific examples

[0762] A user selects a "low-sugar yogurt" at the supermarket and scans the label with the camera on the smart glasses. The OCR reads the ingredient information for the "low-sugar yogurt" and sends it to a cloud server. The generative AI analyzes it and displays advice on the smart glasses, such as "These ingredients are glycemic, but slightly higher in fat. Consider low-fat options."

[0763] Example prompts for generative AI models

[0764] User Profile:

[0765] Name: Tanaka Taro

[0766] Age: 45

[0767] Gender: Male

[0768] Medical history: high blood pressure

[0769] Dietary restrictions: Low salt

[0770] Input food: Low-sugar yogurt

[0771] Ingredients: Carbohydrates, fat, calcium

[0772] Questions asked:

[0773] "Is this food suitable for me?"

[0774] advice:

[0775] "These ingredients are blood sugar friendly, but they are a little higher in fat. Consider lower-fat options."

[0776] In this way, the present invention realizes a system that allows users to make healthy food choices in real time while ensuring privacy and data security.

[0777] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0778] Step 1: User registration and data management

[0779] When a user launches the application for the first time, they enter basic information such as their name, age, gender, medical history, dietary restrictions, etc. The device then sends this information to a database, where it is stored as encrypted data using endpoint protection software.

[0780] Input: User's basic information (name, age, gender, medical history, dietary restrictions)

[0781] Output: Encrypted user profile data

[0782] Specific operation: The information entered into the form on the terminal is sent to the server and stored in the destination database.

[0783] Step 2: Food Label Identification

[0784] The user takes a photo of a food label using the camera on their smart device, and the device passes the image to the OCR module to extract text information, which is then sent to the server.

[0785] Input: Food label image

[0786] Output: Extracted text information

[0787] Specific operation: The camera takes a photo of the food label, extracts text information using the OCR module (Tesseract), and sends it to the server.

[0788] Step 3: Ingredient analysis and risk assessment

[0789] The server compares the received text information with a food ingredient database to obtain detailed ingredient information, and uses generative AI to assess the risk based on the user's health status.

[0790] Input: Extracted text information, user profile

[0791] Output: Risk assessment results and advice on countermeasures

[0792] Specific operation: Using a generative AI model (GPT-3), ingredient information is analyzed and compared with the user's health profile to perform risk assessment.

[0793] Step 4: Real-time alerts

[0794] The terminal notifies the user of the risk assessment results received from the server visually and audibly, showing risk information on the display and explaining risk details via the audio output device.

[0795] Input: Risk assessment results

[0796] Output: Visual and audio risk information

[0797] Specific operation: The risk assessment results are displayed on the smart device display and notified to the user via voice output.

[0798] Step 5: User voice questions and feedback

[0799] The user asks a follow-up question by voice. The device uses voice recognition to convert the user's question into text and send it to the server. The server then uses generative AI to generate an answer and provides real-time feedback to the user in text and voice.

[0800] Input: User's voice question

[0801] Output: Answers and advice from generative AI

[0802] Specific operation: The question is converted into text using the voice recognition function, and the generative AI model (GPT-3) generates an appropriate answer based on the question. The generated answer is then fed back to the user via the device's display and voice output.

[0803] Step 6: Accumulating data and improving personalization accuracy

[0804] The server stores the food selections and advice received by the user in a database, and the accumulated data will be used to improve the accuracy of personalization in the future.

[0805] Input: Selected food data, received advice

[0806] Output: Updated user profile data

[0807] Specific operation: The server stores the selected food data and advice in a database and uses it as analytical data to improve the accuracy of future advice.

[0808] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0809] The present invention provides a system that allows users to easily select healthy foods and provides personalized advice taking into account the user's emotional state. Specific embodiments of this system are described below.

[0810] User registration and initial settings

[0811] 1. The user installs the application and enters basic information such as name, age, gender, medical history, and dietary restrictions when launching it for the first time.

[0812] 2. The terminal sends the entered information to the server.

[0813] 3. The server generates a user profile based on the received information and creates personalized health advice for the user. This profile is used for future analysis and advice provision.

[0814] Food Identification

[0815] 1. The user opens the application and takes a photo of the label of the food item they plan to purchase.

[0816] 2. The device passes the captured image to an OCR (Optical Character Recognition) module to extract text information from the image.

[0817] 3. The terminal sends the extracted text data to the server.

[0818] Ingredient analysis and risk assessment

[0819] 1. The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[0820] 2. The server uses generative AI to analyze the ingredients and assess the risks associated with the user's health.

[0821] Example: If the analysis reveals that a user with high blood pressure selects a high-salt food, a risk assessment will be made, such as "This food contains a high amount of salt and may have a negative impact on high blood pressure."

[0822] 3. The server generates the assessed risk information as an alert and sends it to the terminal to notify the user.

[0823] Viewing Alerts and Information

[0824] 1. The device displays the alert information and detailed data received from the server on the app screen.

[0825] Example: The app screen displays the risk assessment results using text and icons, visually communicating the risks to the user.

[0826] 2. The user reviews the displayed information and asks additional questions verbally if necessary.

[0827] Voice recognition and additional information provision

[0828] 1. If the user needs more information, they ask a question by voice. For example, "What should I eat instead of this food?"

[0829] 2. The device uses its voice recognition function to convert the user's voice into text and send it to the server.

[0830] 3. The server uses generative AI to generate the optimal answer based on the question and sends it to the device.

[0831] 4. The device will play back the received answer aloud and also display it as text, providing the user with both visual and auditory information.

[0832] Utilizing the Emotion Engine

[0833] 1. The emotion engine recognizes emotions from the user's voice or text input.

[0834] Example: Based on the user's tone of voice and text content, the emotion engine determines that the user is "stressed."

[0835] 2. The server customizes the advice it provides depending on the perceived emotional state.

[0836] Example: A user who is feeling stressed is offered advice on foods that have a relaxing effect and lifestyle habits to reduce stress.

[0837] 3. The server automatically selects a risk alert wording that is appropriate for the recognized emotion.

[0838] Example: Presenting risk alerts in a gentler tone to emotionally unstable users.

[0839] Data accumulation and feedback

[0840] 1. The server stores the user's food selection data, received advice, and recognized emotional state in a database.

[0841] 2. The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments.

[0842] In this way, the present invention provides a system that allows users to select appropriate foods taking into account their health and emotional state, thereby providing more personalized health support and helping users lead a healthier life with ease.

[0843] The processing flow will be explained below.

[0844] Processing procedure of a food selection system that combines an emotion engine

[0845] Step 1:

[0846] The user installs the application and enters basic information such as name, age, gender, medical history, and dietary restrictions when launching it for the first time.

[0847] Step 2:

[0848] The terminal transmits the input information to the server.

[0849] Step 3:

[0850] The server generates a user profile based on the received information, creates basic health advice tailored to the user, and stores it in a database.

[0851] Step 4:

[0852] The user launches the application and takes a photo of the label of the food item they plan to purchase.

[0853] Step 5:

[0854] The device passes the captured image to an OCR (optical character recognition) module, which extracts text information from the image.

[0855] How it works: The OCR module analyzes the image and extracts text such as ingredient information, calorie information, and additives.

[0856] Step 6:

[0857] The terminal transmits the extracted text data to the server.

[0858] Step 7:

[0859] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[0860] Step 8:

[0861] The server uses generative AI to analyze the ingredients and assess the risks associated with the user's health.

[0862] Specific operation: The health risk assessment module analyzes ingredient information and makes assessments such as "risk of high blood pressure due to high salt content."

[0863] Step 9:

[0864] The server generates the evaluated risk information as an alert and transmits it to the terminal for notification to the user.

[0865] Step 10:

[0866] The device displays the alert information and detailed data received from the server on the app screen.

[0867] Example: The app screen might say something like, "This food contains high salt content and may have a negative impact on high blood pressure."

[0868] Step 11:

[0869] The user reviews the displayed information and verbally asks additional questions if necessary.

[0870] Step 12:

[0871] The terminal uses a voice recognition function to convert the user's voice into text and send it to the server.

[0872] Specific operation: The speech recognition module analyzes the speech and converts it into text.

[0873] Step 13:

[0874] The server uses generative AI to generate the best answer based on the question and sends that answer to the device.

[0875] Example: Generate specific suggestions such as "List of low-sodium foods."

[0876] Step 14:

[0877] The device will play the received answer aloud and also display it as text.

[0878] Specific operation: Passes the text to the text-to-speech (TTS) module and reads it aloud.

[0879] Utilizing the Emotion Engine

[0880] Step 15:

[0881] The emotion engine recognizes emotions from the user's voice and text input.

[0882] Example: Recognizing emotions such as "anxiety" and "stress" from the user's voice.

[0883] Step 16:

[0884] The server customizes the advice it provides depending on the perceived emotional state.

[0885] Example: For users who are under stress, we suggest foods that have a relaxing effect and lifestyle habits that will help relieve stress.

[0886] Step 17:

[0887] The server automatically selects a risk alert presentation that is appropriate for the recognized emotion.

[0888] Specific behavior: For users who are emotionally anxious, a gentle tone of alert is displayed.

[0889] Data accumulation and feedback

[0890] Step 18:

[0891] The server stores the user's food selections, received advice, and recognized emotional state in a database.

[0892] Step 19:

[0893] The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments.

[0894] Example: A data analytics module analyzes accumulated data to improve the advice-giving algorithm.

[0895] Through this process, users can select the best foods based on their health and emotional state, providing more personalized health support and helping them live a healthier life.

[0896] Example 2

[0897] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0898] In recent years, with the increasing interest in health, there is a demand for personalized food selection support that takes into account individual health and emotional states. However, when users select healthy foods, it is not easy to accurately understand ingredient information and receive appropriate advice based on their current physical condition and emotional state. Furthermore, the advice provided is uniform and does not address individual emotional states, which may reduce user satisfaction.

[0899] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0900] In this invention, the server includes means for generating personalized health advice based on information entered by the user, means for extracting text information from a photographed food label image using optical character recognition, means for comparing the extracted text information with a food ingredient database and analyzing the ingredient information using a generation AI to evaluate risks associated with the user's health condition, means for notifying the user of the evaluated risk information as an alert, means for analyzing the user's voice input using voice recognition and providing appropriate answers and advice using a generation AI, means for analyzing the user's emotional state and customizing the advice to be provided depending on the emotional state, and means for selecting a method of expressing the risk alert appropriate to the emotional state. This enables the user to receive personalized advice that takes into account not only their health condition but also their emotional state.

[0901] "User" refers to an individual who uses the system and is the target of assistance in managing health conditions and food choices.

[0902] "Personalized health advice" refers to health guidance and recommendations tailored to a particular user based on user input.

[0903] Optical character recognition is a technology that extracts text information from images and photographs, and is used to incorporate food label information into the system.

[0904] The "Food Composition Database" is a database that collects information on the composition of various foods and provides basic data for composition analysis and risk assessment.

[0905] "Generative AI" refers to artificial intelligence technology that performs natural language processing and information generation based on large amounts of data, and is used to generate health advice and answers for users.

[0906] "Risk alert" refers to warning information that notifies users of health risks associated with food choices.

[0907] "Speech recognition" is a technology that converts a user's voice input into text data and is used to process spoken queries and commands.

[0908] "Emotional state" refers to the user's psychological and emotional state, and represents emotions such as stress or joy that can be inferred from voice and text input.

[0909] "Personalization" refers to providing optimized information and services that take into account the characteristics of each individual user.

[0910] "Food label" refers to the text and images containing ingredient and nutritional information that appear on food packaging.

[0911] MODE FOR CARRYING OUT THE INVENTION

[0912] The present invention provides a system that allows users to easily select healthy foods and provides personalized advice taking into account the user's emotional state. Specific embodiments of the system are described below.

[0913] User registration and initial settings

[0914] First, the user installs and launches the application on their device. When launching the application for the first time, the user enters basic information such as their name, age, gender, medical history, and dietary restrictions. The device then sends this information to the server. The server then creates a user profile based on the received information and creates initial health advice tailored to the user. For example, a user who needs to limit their fat intake may be provided with advice recommending low-fat foods.

[0915] Food Identification

[0916] The user uses the application to take a photo of the label of the food they plan to purchase with their camera. The device then passes the image to an optical character recognition (OCR) module, which extracts text information from the image. Specifically, the device uses an OCR service such as Google Cloud Vision API. The extracted text information is then sent from the device to a server.

[0917] Ingredient analysis and risk assessment

[0918] The server compares the received text information with a food ingredient database (e.g., USDA Nutrient Database) to obtain detailed information about each ingredient. Next, it uses generative AI (e.g., OpenAI's GPT-4) to analyze the ingredient information and evaluate the risk associated with the user's health. An example of a specific prompt is, "How does this ingredient affect high blood pressure?" The evaluated risk information is generated as an alert and sent to the device.

[0919] Viewing Alerts and Information

[0920] The device displays the alert information and detailed data received from the server on the app screen. For example, the app screen displays the risk assessment results using text and icons, visually communicating the risk to the user. The user can check the displayed information and ask additional questions by voice if necessary.

[0921] Voice recognition and additional information provision

[0922] If the user needs additional information, they can ask a question by voice. For example, they might ask, "What should I eat instead of this food?" The device uses a voice recognition function (e.g., Google Speech-to-Text) to convert the user's voice into text and send it to the server. The server then uses a generative AI to generate the optimal answer based on the question and sends it to the device. The device then plays the received answer aloud and also displays it as text.

[0923] Utilizing the Emotion Engine

[0924] An emotion engine (e.g., Microsoft Azure Emotion API) recognizes the user's emotional state from their voice or text input. Specifically, it determines whether the user is "feeling stressed" based on the user's tone of voice and the content of the text. The server then customizes the advice it provides based on the recognized emotional state. For example, a user who is feeling stressed could be offered advice about foods and lifestyle habits that have a relaxing effect. At the same time, the server selects a risk alert presentation style appropriate to the user's emotional state and sends it to the device.

[0925] Data accumulation and feedback

[0926] The server stores the user's food selections, advice received, and perceived emotional state in a database. This accumulated data is used to improve the accuracy of future advice and risk assessments. Specifically, the data is analyzed using data analytics tools (e.g., Python's Pandas and Scikit-learn), and the algorithm is improved based on the analysis results.

[0927] In this way, the present invention provides a system that allows users to select appropriate foods taking into account their health and emotional state, thereby providing more personalized health support and helping users lead a healthier life with ease.

[0928] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0929] Step 1:

[0930] Users install and launch the application on their device, then enter basic information such as their name, age, gender, medical history, and dietary restrictions.

[0931] Input: User's basic information (name, age, gender, medical history, dietary restrictions, etc.)

[0932] Specific action: The user enters information into an input form within the app and presses the "Submit" button.

[0933] Output: Request data with basic information entered

[0934] Step 2:

[0935] The terminal transmits the input basic information to the server.

[0936] Input: Request data (basic user information)

[0937] Specific operation: The terminal generates an HTTP request and sends data to the server.

[0938] Output: Basic information sent to the server

[0939] Step 3:

[0940] The server generates a user profile based on the received basic information, which also includes initial health advice for the user.

[0941] Input: User basic information

[0942] Specific operations: The server stores the information in a database and runs the profile generation algorithm.

[0943] Output: Generated user profile and initial advice

[0944] Step 4:

[0945] The user uses the application to take a photo of the label of the food item they plan to purchase with their camera.

[0946] Input: Food label image

[0947] Specific operation: The user activates the camera function and takes a picture of the food label.

[0948] Output: Photographed food label image

[0949] Step 5:

[0950] The device passes the captured food label image to an OCR module to extract text information.

[0951] Input: Food label image

[0952] Specific operation: Send the image to an OCR module (e.g., Google Cloud Vision API) and extract text information.

[0953] Output: Extracted text data

[0954] Step 6:

[0955] The terminal transmits the extracted text data to the server.

[0956] Input: Extracted text data

[0957] Specific operation: The terminal generates an HTTP request and sends data to the server.

[0958] Output: Text data sent to the server

[0959] Step 7:

[0960] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[0961] Input: Text data

[0962] Specific operation: The server queries the food ingredient database and retrieves ingredient information.

[0963] Output: Detailed information for each component

[0964] Step 8:

[0965] The server uses generative AI to analyze the ingredient information and assess the risks associated with the user's health condition.

[0966] Input: Ingredient information, user profile

[0967] Specific behavior: Send an analytical prompt to an AI model (e.g., OpenAI's GPT-4) to perform risk assessment. Specific prompt example: "How does this ingredient affect high blood pressure?"

[0968] Output: Risk assessment results

[0969] Step 9:

[0970] The server generates the evaluated risk information as an alert and transmits it to the terminal for notification to the user.

[0971] Input: Risk assessment results

[0972] Specific operation: An alert message is generated and sent to the terminal as an HTTP response.

[0973] Output: Generated risk alerts

[0974] Step 10:

[0975] The device displays the alert information and detailed data received from the server on the app screen.

[0976] Enter: Risk Alert

[0977] Specific behavior: Parse the risk alert and display it in a UI component (e.g., a text view or an icon).

[0978] Output: Risk alert displayed on the app screen

[0979] Step 11:

[0980] The user reviews the displayed information and verbally asks additional questions if necessary.

[0981] Input: Risk assessment results displayed on the terminal

[0982] Specific operation: The user presses the microphone button and begins voice input.

[0983] Output: User's voice data

[0984] Step 12:

[0985] The terminal uses a voice recognition function to convert the user's voice into text and send it to the server.

[0986] Input: User's voice data

[0987] Specific operation: Uses a speech recognition API (e.g., Google Speech-to-Text) to convert voice data into text, and sends the converted text to the server.

[0988] Output: Textual question data

[0989] Step 13:

[0990] The server uses generative AI to generate the best answer based on the question and sends it to the device.

[0991] Input: Textual question data

[0992] Specific operation: Send a prompt (e.g., "What should I eat instead of this food?") to the generating AI and get an answer.

[0993] Output: Generated response data

[0994] Step 14:

[0995] The device will play the received answer aloud and also display it as text.

[0996] Input: Generated response data

[0997] Specific behavior: Converts text to speech using a speech synthesis API (e.g., Google Text-to-Speech), plays the converted speech, and displays the text in the UI.

[0998] Output: Answer information as audio and text

[0999] Step 15:

[1000] The emotion engine recognizes the user's emotional state from their voice and text input.

[1001] Input: User's voice data, text data

[1002] Specific behavior: Uses emotion recognition APIs (e.g. Microsoft Emotion API) to infer emotions from speech and text.

[1003] Output: Perceived emotional state

[1004] Step 16:

[1005] The server customizes the advice it provides depending on the perceived emotional state.

[1006] Input: perceived emotional state, user profile

[1007] Specific behavior: Send a prompt (e.g., "What should you recommend to a user who is feeling stressed?") to the generative AI to get customized advice.

[1008] Output: Customized advice

[1009] Step 17:

[1010] The server selects a method of expressing a risk alert that is appropriate for the emotional state and notifies the terminal.

[1011] Input: Customized advice, risk assessment results

[1012] Specific behavior: Adjust the tone and format of the text depending on the emotional state and send it to the device in an HTTP response.

[1013] Output: Adjusted risk alerts

[1014] Step 18:

[1015] The server stores the user's food selection data, received advice, and recognized emotional state in a database.

[1016] Input: User's food data, advice, perceived emotional state

[1017] Specific operation: Executes a data insertion query against the database.

[1018] Output: Information stored in the database

[1019] Step 19:

[1020] The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments.

[1021] Input: Accumulated data

[1022] Specific Tasks: Analyze data and improve algorithms using data analytics tools (e.g., Python's Pandas, Scikit-learn).

[1023] Output: Improved algorithm

[1024] (Application example 2)

[1025] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1026] Many consumers today are becoming increasingly health-conscious and understand the importance of choosing foods that are right for them. However, it is difficult to instantly evaluate food ingredients and health risks while shopping in a physical store, and to receive optimal advice based on one's emotional state at the time. Especially in today's world where a wide variety of foods are available, obtaining detailed information about each food item takes time and effort. Furthermore, existing systems are unable to provide advice that takes into account the user's emotional state, making it difficult to provide accurate support tailored to each user's individual situation.

[1027] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for generating personalized health advice based on information entered by a user, means for extracting text information from a photographed food label image using optical character recognition, means for comparing the extracted text information with a food ingredient database, analyzing the ingredient information using a generation AI, and evaluating risks associated with the user's health status, means for notifying the user of the evaluated risk information as an alert, means for analyzing the user's voice input using voice recognition and providing appropriate answers and advice using a generation AI, means for analyzing the user's emotional state and personalizing advice according to the emotional state, and means for providing recommended products and purchase plans suited to the user's health status and emotional status based on data from a physical store. This allows users to easily select foods suitable for them in a physical store and receive personalized advice tailored to their health status and emotional state.

[1028] "Means for generating personalized health advice based on user-entered information" refers to a method for collecting information entered by a user into the application, such as name, age, gender, medical history, and dietary restrictions, and generating and providing specific health advice based on that information.

[1029] "Means for extracting text information from photographed food label images using optical character recognition" refers to a method in which a user takes a photo of a food label using a camera on a smartphone or other device, and extracts text information from the image using optical character recognition technology.

[1030] "Means of comparing extracted text information with a food ingredient database, analyzing ingredient information using generative AI, and assessing risks related to the user's health" refers to a method of comparing text information obtained by optical character recognition with a food ingredient database, and then using a generative AI model to perform a detailed analysis of the ingredients and assess the user's health risks.

[1031] "Means for notifying the user of the evaluated risk information as an alert" refers to a method for evaluating the risk to the user's health based on the analyzed component information and notifying the user of the results as a warning.

[1032] "Means of analyzing a user's voice input through speech recognition and using generative AI to provide appropriate answers and advice" refers to a method of converting a user's spoken questions or requests into text using speech recognition technology, analyzing it using a generative AI model, and generating appropriate answers and advice.

[1033] "Means for analyzing the user's emotional state and personalizing advice according to that emotional state" refers to a method for analyzing emotional information obtained from the user's voice input or text input and providing customized advice according to that emotion.

[1034] "Means for providing recommended products and purchase plans suited to the user's health and emotional state based on data from physical stores" refers to a method that utilizes user behavior data and selected product data obtained from physical stores to suggest products and purchase plans that are optimal for the user's health and emotional state.

[1035] "Optical character recognition" is a technology that mechanically reads characters printed on paper or text contained in images and extracts it as digital data.

[1036] "Generative AI" is an artificial intelligence technology that generates new information and answers based on pre-trained data.

[1037] The present invention provides a system that allows users to easily select foods suitable for them in a physical store and receive personalized advice tailored to their health and emotional state. Specific embodiments of this system are described below.

[1038] User registration and initial settings

[1039] Users enter basic information such as name, age, gender, medical history, and dietary restrictions through an application installed on their smartphone. The device then sends this information to a server, which then generates a user profile based on the information received and prepares health advice. This profile is then used for future analysis and advice provision.

[1040] Food Identification

[1041] A user takes a photo of the label of a food item they are considering purchasing in a physical store using their smartphone camera. The device passes the image to an optical character recognition (OCR) module, which extracts text information from the image. The extracted text data is then sent to a server.

[1042] Ingredient analysis and risk assessment

[1043] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient. It then uses a generative AI model to analyze the ingredients and assess the risk associated with the user's health. For example, if the analysis reveals that a user with high blood pressure chooses a high-salt food, a risk assessment will be made, such as, "This food contains high salt and may have a negative impact on high blood pressure." The assessed risk information is generated as an alert and sent to the device to notify the user.

[1044] Viewing Alerts and Information

[1045] The device displays the alert information and detailed data received from the server on the app screen. For example, the risk assessment results can be displayed as text and icons to visually inform the user of the risk. If the user asks additional questions by voice, the voice is converted into text by a voice recognition module and sent to the server.

[1046] Voice recognition and additional information provision

[1047] The server uses a generative AI model to generate an optimal answer based on the content of the voice question and sends it to the device, which then plays the received answer back aloud and can also display it as text, allowing the user to obtain information both visually and audibly.

[1048] Utilizing the Emotion Engine

[1049] The emotion engine recognizes emotions from the user's voice and text input. For example, the emotion engine may determine that the user is "feeling stressed" based on the user's tone of voice and the content of the text. The server then customizes the advice it provides based on the recognized emotional state. Specifically, for a user who is feeling stressed, it provides advice on foods that have a relaxing effect and lifestyle habits to reduce stress. For users with unstable emotions, it displays risk alerts in a gentler tone.

[1050] Data accumulation and feedback

[1051] The server stores the user's food selections, received advice, and perceived emotional state in a database, which is used to improve the accuracy of future advice and risk assessments.

[1052] This allows users to easily choose the foods that are right for them in physical stores and receive personalized advice tailored to their health and emotional state.

[1053] Prompt Sentence Examples

[1054] Below are some examples of prompt sentences.

[1055] Food ingredient analysis prompt:

[1056] You are given the following list of food ingredients. Analyze the risks these ingredients pose to a user with high blood pressure and perform a risk assessment.

[1057] Ingredients: Salt

[1058] Ingredients: Sodium 1500mg

[1059] Emotion recognition prompts:

[1060] Recognize emotions from the following user voice inputs:

[1061] Text: "I'm feeling super stressed today. Is there a good way to relax?"

[1062] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1063] Step 1:

[1064] Users install the application on their smartphone and enter basic information such as their name, age, gender, medical history, and dietary restrictions. The application then collects the initial data to create a user profile. The entered information is sent from the device to a server. The server receives this information, generates a user profile, and stores it as the basis for health advice.

[1065] Step 2:

[1066] Users take a photo of the label of a food item they are considering purchasing in a physical store using their smartphone camera. The device then passes the image to an optical character recognition (OCR) module to extract text information from the image. This extraction process utilizes OpenCV and the Tesseract library. The extracted text data is then sent to a server.

[1067] Step 3:

[1068] The server compares the received text data with a food ingredient database, thereby obtaining detailed information about each ingredient. It then analyzes the ingredient information using a generative AI model (e.g., GPT-4) to assess the risk associated with the user's health. The results of the risk assessment are formatted as specific alert information, such as "not suitable for users with high blood pressure." The assessment results are then sent from the server to the device.

[1069] Step 4:

[1070] The terminal visually displays the alert information and detailed data received from the server to the user. The alert information is expressed using text and icons, and is used to intuitively convey risk information to the user. If the user has additional questions, their voice is converted into text using the application's speech recognition module. The Google Speech-to-Text API is used for speech recognition. The text data of the question is sent to the server.

[1071] Step 5:

[1072] The server generates an appropriate answer using a generative AI model based on the text data received as a speech recognition result. The generated answer is formatted as specific advice and sent from the server to the device. The device displays this advice visually to the user and plays it aloud using speech synthesis (Google Text-to-Speech API).

[1073] Step 6:

[1074] The emotional state of the user is analyzed using an emotion engine (e.g., Amazon Rekognition) based on the user's voice or text input. The server then personalizes advice according to the user's emotional state based on the emotional data received from the emotion engine. Advice is generated according to the emotion, and customization is performed, such as displaying risk alerts in a gentle tone. The customized advice is then sent from the server to the device.

[1075] Step 7:

[1076] The server stores the user's food selections, advice received, and perceived emotional state in a database. This data is used to improve the accuracy of future advice and risk assessments. This data is associated with the user's profile and analyzed to provide more appropriate advice.

[1077] These steps allow users to easily choose the foods that are right for them in a physical store and receive personalized advice based on their health and emotional state.

[1078] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1079] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1080] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1081] [Third embodiment]

[1082] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1083] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1084] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1085] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1086] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1087] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1088] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1089] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1090] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1091] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1092] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1093] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1094] The present invention provides a system that allows users to easily select healthy foods, and includes a program that operates by combining various technologies to achieve this. Specific embodiments of this system are described below.

[1095] User registration and initial settings

[1096] 1. The user installs the application and, when launching it for the first time, enters basic information such as name, age, gender, medical history, and dietary restrictions on the account creation screen.

[1097] 2. The terminal sends the information entered by the user to the server.

[1098] 3. The server generates a user profile based on the received information and tailors health advice to the user. This profile is stored in a database for future analysis and advice provision.

[1099] Food Identification

[1100] 1. The user opens the application and takes a photo of the label of a food item they are considering purchasing.

[1101] 2. The device passes the captured image to an OCR (Optical Character Recognition) module to extract text information from the image.

[1102] Example: If a food label contains information such as "phosphates," "sodium," and "sugars," these will be read as text.

[1103] 3. The terminal sends the extracted text data to the server.

[1104] Ingredient analysis and risk assessment

[1105] 1. The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[1106] 2. The server uses generative AI to analyze the ingredients and assess the risks associated with the user's health.

[1107] Example: If the analysis reveals that a user with high blood pressure selects a high-salt food, a risk assessment will be made, such as "This food contains a high amount of salt and may have a negative impact on high blood pressure."

[1108] 3. The server generates the results of the risk assessment as alert information and sends it to the terminal.

[1109] Viewing Alerts and Information

[1110] 1. The terminal displays the alert information and detailed data received from the server on the application screen.

[1111] Example: The application screen displays the results of the risk assessment using text and icons, visually communicating the risks to the user.

[1112] 2. The user reviews the information displayed and asks additional questions if necessary.

[1113] Voice recognition and additional information provision

[1114] 1. If the user wants more information, they can ask a question out loud. For example, "What should I eat instead of this food?"

[1115] 2. The device uses voice recognition to convert the user's question into text and send it to the server.

[1116] 3. The server uses generative AI to generate the best answer based on the question.

[1117] Example: Create a "list of low-salt foods" based on a food database and health guidelines and send it to your device.

[1118] 4. The device will play the received answer aloud and also display it as text, allowing the user to receive information both visually and audibly.

[1119] Data accumulation and feedback

[1120] 1. The server stores the user's food selection data and received advice in a database.

[1121] 2. The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments.

[1122] In this way, the present invention realizes a system that allows users to easily select healthy foods. By providing users with checked ingredient information and risk alerts, the system helps users make healthy food choices. This system allows users to make healthy food choices without spending time or effort.

[1123] The processing flow will be explained below.

[1124] Food selection system processing procedure

[1125] Step 1:

[1126] Users install the app and enter basic information such as their name, age, gender, medical history, and dietary restrictions when they first launch it.

[1127] Step 2:

[1128] The terminal transmits the input information to the server.

[1129] Step 3:

[1130] The server generates a user profile based on the received information and sets basic health advice tailored to the user. This profile is used for future analysis and advice provision.

[1131] Step 4:

[1132] The user launches the app and takes a photo of the label of the food item they plan to purchase.

[1133] Step 5:

[1134] The device passes the captured image to an OCR (optical character recognition) module, which extracts text information from the image.

[1135] Step 6:

[1136] The terminal transmits the extracted text data to the server.

[1137] Step 7:

[1138] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[1139] Step 8:

[1140] The server uses generative AI to analyze the ingredients and assess the risks associated with the user's health.

[1141] Step 9:

[1142] The server generates the evaluated risk information as an alert and transmits it to the terminal for notification to the user.

[1143] Step 10:

[1144] The device displays the alert information and detailed data received from the server on the app screen, using text and icons to visually communicate risk to the user.

[1145] Step 11:

[1146] The user reviews the displayed information and verbally asks additional questions if necessary.

[1147] Step 12:

[1148] The terminal uses a voice recognition function to convert the user's voice into text and send it to the server.

[1149] Step 13:

[1150] The server uses generative AI to generate the best answer based on the question and sends that answer to the device.

[1151] Step 14:

[1152] The device will play back the received answer aloud and also display it as text, allowing the user to receive information both visually and audibly.

[1153] Step 15:

[1154] The server stores the food data selected by the user and the advice received in a database, which is used to improve the accuracy of future advice and risk assessments.

[1155] This series of processes allows users to easily choose healthy foods and maintain and improve their own health.

[1156] Example 1

[1157] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1158] Choosing healthy foods is an important issue in modern society. However, many users have difficulty making appropriate choices because they do not fully understand the ingredients and health risks of foods. It is also difficult to obtain personalized advice tailored to their own health status. As a result, inappropriate food choices can have a negative impact on health. Furthermore, there is a lack of systems that can quickly respond to users' questions or requests for additional information. To address these issues, a system is needed that analyzes users' health information and food ingredients and provides individually customized advice and risk assessments.

[1159] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1160] In this invention, the server includes a means for generating personalized health advice based on information entered by the user, a means for extracting text information from a photographed food label image using OCR (optical character recognition), and a means for comparing the extracted text information with a food ingredient database, analyzing the ingredient information using a generation AI, and assessing risks associated with the user's health condition. This enables accurate advice and risk assessment that are individually customized based on the user's health condition.

[1161] "User" refers to an individual who uses the system to select healthy foods.

[1162] "Terminal" refers to a smartphone, tablet, or other electronic device used by a user to run applications and input, display, and communicate information.

[1163] "Server" refers to a remote computer system that processes data received from users and devices and provides advice and risk assessments.

[1164] "Health Advice" refers to specific recommendations to promote healthy living based on information entered by the user.

[1165] "Food label image" refers to an image of the label attached to the package of food that the user is considering purchasing, taken with a camera.

[1166] "OCR (Optical Character Recognition)" refers to a technology that analyzes the text information contained in an image and extracts it as machine-readable character data.

[1167] A "food composition database" refers to a collection of data that collects and stores information on the composition of various foods.

[1168] "Generative AI" refers to artificial intelligence that uses the technology of generative AI models (e.g., GPT-3) to generate appropriate answers and advice to human questions.

[1169] "Risk assessment" refers to analyzing the impact of ingredients contained in food on the user's health and determining the level of risk.

[1170] "Alert" refers to a warning message intended to notify the user of health risks or other important information.

[1171] "Speech recognition" refers to the technology that converts a user's voice input into text data.

[1172] A "profile" refers to a data set that records a user's basic information and health status.

[1173] "Historical Data" refers to a record of the foods a user selects and the advice they receive.

[1174] "Personalization accuracy" refers to the accuracy and suitability of the advice and information provided to an individual user.

[1175] The present invention provides a system that allows a user to easily select healthy foods. An embodiment of this system will now be described in detail.

[1176] First, the user installs the application and enters basic information such as name, age, gender, medical history, and dietary restrictions on the account creation screen when first starting the application. This basic information is used to create a user profile and is sent from the device to the server. The server generates a user profile based on the received information and stores it in a database. This allows for future analysis and provision of health advice tailored to the user.

[1177] Next, when the user takes a photo of the label of the food they are considering purchasing, an OCR (optical character recognition) module is used to extract text information from the image, using the Tesseract OCR library as an example. The extracted text data is sent from the device to a server and compared with a food ingredient database.

[1178] The server uses the received text data to compare it with a food ingredient database to obtain detailed information about each ingredient. It then analyzes the ingredients using a generative AI model (e.g., GPT-3) to assess the risk associated with the user's health. The assessed risk information is generated as an alert and sent to the device. The device displays the received alert information on the application screen and communicates the risk assessment results to the user visually and audibly.

[1179] If the user wants more detailed information, they can ask by voice. The device uses voice recognition to convert the user's question into text and send it to the server. The server then uses generative AI to generate the best answer based on the question, providing information such as a "list of low-salt foods" based on a food database and health guidelines. The device then plays back the received answer by voice and also displays it as text, allowing the user to receive information both visually and audibly.

[1180] In addition, all user-selected food data and advice received are stored in a database. The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments. Data mining algorithms can be used to extract patterns from the accumulated data and train models to improve the accuracy of advice.

[1181] (Example)

[1182] For example, if a user uses the application to scan a "potato chips" label, the OCR module extracts ingredients such as "sodium," "fat," and "sugar." The server analyzes these ingredients, and if the user is overweight and has a history of high blood pressure, an alert message will be displayed stating, "This food contains high amounts of sodium and fat, which may have adverse health effects." Furthermore, if the user asks "What are some alternative foods?" by voice, the server will list "low-sodium and low-calorie foods," and the device will provide that information via voice and text.

[1183] (Example of a prompt)

[1184] "Analyze the ingredients in this food and assess the risk."

[1185] "What healthy options can I use to replace this food?"

[1186] "Create a recommended food list for users with high blood pressure."

[1187] In this way, users can make healthy food choices without spending time or effort.

[1188] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1189] Step 1: Enter and submit basic user information

[1190] The user installs and launches the application, and enters basic information such as name, age, gender, medical history, and dietary restrictions on the account creation screen. The device then formats this information into a database and sends it to the server.

[1191] Input: Basic information entered by the user (name, age, gender, medical history, dietary restrictions)

[1192] Data processing: Convert input information into JSON format

[1193] Output: Send basic user information data in JSON format to the server

[1194] What it does: Parses data entered into form fields into JSON format and sends it via a REST API.

[1195] Step 2: Generate and save a profile on the server

[1196] The server generates a user profile based on the received user information and stores it in a database, and prepares to provide personalized health advice based on the profile.

[1197] Input: Received user basic information data

[1198] Data processing: Insert user information into the database

[1199] Output: User profile stored in the database

[1200] What it does: Runs a script that inserts user information into a MySQL database and generates profile data.

[1201] Step 3: Photograph food labels and extract text using OCR

[1202] The user launches the application and takes a photo of the label of a food item they are considering purchasing. The device inputs the image into the OCR module and extracts text information from the image.

[1203] Input: A photograph of a food label image

[1204] Data processing: Convert images to text using OCR processing

[1205] Output: Extracted text data

[1206] What it does: Uses the Tesseract OCR library to extract text from image data and save it as a string.

[1207] Step 4: Sending text data to the server

[1208] The terminal transmits the extracted text data to the server.

[1209] Input: Extracted text data

[1210] Data processing: Convert text data into JSON format

[1211] Output: Send JSON format text data to the server

[1212] Specific operation: The extracted text data is converted into JSON format and sent to the server via REST API.

[1213] Step 5: Obtaining ingredient information and risk assessment on the server

[1214] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient, then uses a generative AI model to analyze the ingredients and assess the risk associated with the user's health.

[1215] Input: Received text data

[1216] Data processing: Query the database using ingredient names as keys, and perform risk assessment using a generative AI model

[1217] Output: Risk assessment results

[1218] Specific operation: Query ingredient information based on text data, provide prompts to the generative AI model, and perform risk assessment.

[1219] Step 6: Generate and send alert information

[1220] The server generates alert information based on the results of the risk assessment and sends it to the terminal.

[1221] Input: Risk assessment results

[1222] Data processing: Converting risk assessment results into alert information

[1223] Output: Generated alert information

[1224] Specific operation: Based on the risk assessment, the alert information is parsed into JSON format and sent to the terminal.

[1225] Step 7: View alert information

[1226] The terminal displays the alert information and detailed data received from the server on the application screen.

[1227] Input: Received alert information

[1228] Data processing: Binding alert information to UI components

[1229] Output: Alert information displayed on the application screen

[1230] Specific behavior: Parses alert information and binds the data to UI components (e.g. TextView, ImageView).

[1231] Step 8: Request additional information via voice recognition

[1232] If the user wants additional information, they can ask a question by voice, and the device will use its voice recognition function to convert the question into text and send it to the server.

[1233] Input: User's voice question

[1234] Data processing: Converting voice data into text

[1235] Output: Textual question data

[1236] Specific operation: Activates the voice input function and converts the voice data into text using the Google Speech-to-Text API.

[1237] Step 9: Generate additional information on the server

[1238] The server uses generative AI to generate the best answer based on the question and sends it to the device.

[1239] Input: Textual question data

[1240] Data processing: Question data is used as a prompt and provided to the generative AI model to generate an answer

[1241] Output: Generated response data

[1242] Specific operation: The question content is provided as a prompt to the AI ​​model, an answer is generated, the answer data is converted to JSON format, and sent to the device.

[1243] Step 10: View and play answers

[1244] The terminal will play back the received answer aloud and also display it as text.

[1245] Input: Generated response data

[1246] Data processing: Converting response data into voice and text

[1247] Output: Spoken and textual answers

[1248] Specific operation: Text data is converted to audio using the Google Text-to-Speech API, played back to the user, and displayed in a text view.

[1249] Step 11: Data collection and feedback

[1250] The server stores the user's food selections and advice received in a database, accumulating feedback, and then analyzes the data to improve the accuracy of future advice and risk assessments.

[1251] Input: User food choice data, received advice

[1252] Data processing: Inserting data into databases, generating datasets for analysis

[1253] Output: Accumulated historical data, analysis results

[1254] Specific operation: User activity history is inserted into a database, and data mining algorithms are used to extract and analyze patterns from the accumulated data.

[1255] (Application example 1)

[1256] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1257] Conventional health management systems require users to spend a lot of time and effort selecting the right foods, making it difficult to provide appropriate health advice in real time. Furthermore, they are inadequate in terms of protecting user privacy and ensuring data security, making it difficult for users to manage their health data with peace of mind.

[1258] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1259] In this invention, the server includes means for generating personalized health advice based on information entered by the user, means for extracting text information from a photographed food label image using OCR (optical character recognition), means for comparing the extracted text information with a food ingredient database and analyzing the ingredient information using a generation AI to evaluate risks associated with the user's health status, means for visually and audibly notifying the user of the evaluated risk information in real time, and means for analyzing the user's voice input using voice recognition and providing appropriate answers and advice using a generation AI, thereby enabling users to make healthy food choices in real time while also ensuring privacy protection and data security.

[1260] A "user" is someone who uses the system to select healthy foods.

[1261] "Health advice" is a suggestion for appropriate food choices and lifestyle improvements based on the user's health condition.

[1262] "Food label images" are photographs or images containing ingredient and nutritional information found on food packaging.

[1263] "OCR (Optical Character Recognition)" is a technology that extracts character information from image data.

[1264] "Text information" refers to character data such as food ingredients and nutritional information extracted by OCR.

[1265] A "food composition database" is a collection of data that records the composition and nutritional information of various foods.

[1266] "Generative AI" is a system that uses machine learning and artificial intelligence technologies to analyze data, assess risks, and generate advice.

[1267] "Risk assessment" refers to the analysis and determination of the potential health risks posed by specific foods based on the user's health status and input information.

[1268] An "alert" is information that alerts or warns the user.

[1269] "Speech recognition" is a technology that analyzes a user's voice and converts it into text information.

[1270] A "smart device" is an electronic device with advanced functions, including internet connectivity and various sensors.

[1271] "Personalization accuracy" refers to the degree to which the advice and information provided is relevant to the characteristics and preferences of each individual user.

[1272] "Real-time" is a time concept that refers to data processing and information provision occurring almost instantaneously.

[1273] "Visual information" refers to information such as images and text that is presented through a display or screen.

[1274] "Audio information" refers to audio messages delivered through a speaker or speech synthesis system.

[1275] "Privacy protection" is a security measure to protect users' personal information and health data from third parties.

[1276] "Data security" means that data collected, stored and processed is protected from unauthorized access and destruction.

[1277] To realize the present invention, it is necessary to design a system using the following hardware and software.

[1278] First, the system uses a smart device (e.g., smart glasses or a smartphone) that includes a camera, display, microphone, and audio output device. A cloud-based computing service (e.g., AWS) is used as the server. For analysis and database management, an OCR module (e.g., Tesseract), generative AI (e.g., OpenAI GPT-3), and a database (e.g., MySQL) are used. For security software, Symantec Endpoint Protection is used.

[1279] 1. User registration and data management

[1280] The device presents a form for users to register their health data when they first launch the application. Users enter basic information such as name, age, gender, medical history, and dietary restrictions. This information is stored in a database and encrypted by endpoint protection software.

[1281] 2. Food Label Identification

[1282] Users take a photo of a food label using their smart device's camera. The image is sent to an OCR module to extract text information. The extracted text data is then sent to a cloud server where the generative AI analyzes the ingredient information.

[1283] 3. Component analysis and risk assessment

[1284] The cloud server analyzes the received text information by comparing it with a food composition database. The generating AI then compares it with the user's health profile and assesses health risks. For example, if a user with high blood pressure chooses a high-salt food, a risk assessment will be made, such as "This food contains a high amount of salt and may have a negative impact on high blood pressure," and the user will be notified in real time.

[1285] 4. Real-time alerts

[1286] The assessed risk information is not only visually displayed on the smart device display, but also audibly notified via the audio output device, allowing users to receive information both visually and audibly.

[1287] 5. User voice questions and feedback

[1288] If the user wants more detailed information, they can ask a question by voice. For example, they might ask, "What should I eat instead of this food?" The smart device uses its voice recognition function to convert the user's question into text and send it to a cloud server. Based on the question, the generative AI generates an answer from a food database and health guidelines, and provides feedback to the user in real time via text and voice.

[1289] Specific examples

[1290] A user selects a "low-sugar yogurt" at the supermarket and scans the label with the camera on the smart glasses. The OCR reads the ingredient information for the "low-sugar yogurt" and sends it to a cloud server. The generative AI analyzes it and displays advice on the smart glasses, such as "These ingredients are glycemic, but slightly higher in fat. Consider low-fat options."

[1291] Example prompts for generative AI models

[1292] User Profile:

[1293] Name: Tanaka Taro

[1294] Age: 45

[1295] Gender: Male

[1296] Medical history: high blood pressure

[1297] Dietary restrictions: Low salt

[1298] Input food: Low-sugar yogurt

[1299] Ingredients: Carbohydrates, fat, calcium

[1300] Questions asked:

[1301] "Is this food suitable for me?"

[1302] advice:

[1303] "These ingredients are blood sugar friendly, but they are a little higher in fat. Consider lower-fat options."

[1304] In this way, the present invention realizes a system that allows users to make healthy food choices in real time while ensuring privacy and data security.

[1305] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1306] Step 1: User registration and data management

[1307] When a user launches the application for the first time, they enter basic information such as their name, age, gender, medical history, dietary restrictions, etc. The device then sends this information to a database, where it is stored as encrypted data using endpoint protection software.

[1308] Input: User's basic information (name, age, gender, medical history, dietary restrictions)

[1309] Output: Encrypted user profile data

[1310] Specific operation: The information entered into the form on the terminal is sent to the server and stored in the destination database.

[1311] Step 2: Food Label Identification

[1312] The user takes a photo of a food label using the camera on their smart device, and the device passes the image to the OCR module to extract text information, which is then sent to the server.

[1313] Input: Food label image

[1314] Output: Extracted text information

[1315] Specific operation: The camera takes a photo of the food label, extracts text information using the OCR module (Tesseract), and sends it to the server.

[1316] Step 3: Ingredient analysis and risk assessment

[1317] The server compares the received text information with a food ingredient database to obtain detailed ingredient information, and uses generative AI to assess the risk based on the user's health status.

[1318] Input: Extracted text information, user profile

[1319] Output: Risk assessment results and advice on countermeasures

[1320] Specific operation: Using a generative AI model (GPT-3), ingredient information is analyzed and compared with the user's health profile to perform risk assessment.

[1321] Step 4: Real-time alerts

[1322] The terminal notifies the user of the risk assessment results received from the server visually and audibly, showing risk information on the display and explaining risk details via the audio output device.

[1323] Input: Risk assessment results

[1324] Output: Visual and audio risk information

[1325] Specific operation: The risk assessment results are displayed on the smart device display and notified to the user via voice output.

[1326] Step 5: User voice questions and feedback

[1327] The user asks a follow-up question by voice. The device uses voice recognition to convert the user's question into text and send it to the server. The server then uses generative AI to generate an answer and provides real-time feedback to the user in text and voice.

[1328] Input: User's voice question

[1329] Output: Answers and advice from generative AI

[1330] Specific operation: The question is converted into text using the voice recognition function, and the generative AI model (GPT-3) generates an appropriate answer based on the question. The generated answer is then fed back to the user via the device's display and voice output.

[1331] Step 6: Accumulating data and improving personalization accuracy

[1332] The server stores the food selections and advice received by the user in a database, and the accumulated data will be used to improve the accuracy of personalization in the future.

[1333] Input: Selected food data, received advice

[1334] Output: Updated user profile data

[1335] Specific operation: The server stores the selected food data and advice in a database and uses it as analytical data to improve the accuracy of future advice.

[1336] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1337] The present invention provides a system that allows users to easily select healthy foods and provides personalized advice taking into account the user's emotional state. Specific embodiments of this system are described below.

[1338] User registration and initial settings

[1339] 1. The user installs the application and enters basic information such as name, age, gender, medical history, and dietary restrictions when launching it for the first time.

[1340] 2. The terminal sends the entered information to the server.

[1341] 3. The server generates a user profile based on the received information and creates personalized health advice for the user. This profile is used for future analysis and advice provision.

[1342] Food Identification

[1343] 1. The user opens the application and takes a photo of the label of the food item they plan to purchase.

[1344] 2. The device passes the captured image to an OCR (Optical Character Recognition) module to extract text information from the image.

[1345] 3. The terminal sends the extracted text data to the server.

[1346] Ingredient analysis and risk assessment

[1347] 1. The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[1348] 2. The server uses generative AI to analyze the ingredients and assess the risks associated with the user's health.

[1349] Example: If the analysis reveals that a user with high blood pressure selects a high-salt food, a risk assessment will be made, such as "This food contains a high amount of salt and may have a negative impact on high blood pressure."

[1350] 3. The server generates the assessed risk information as an alert and sends it to the terminal to notify the user.

[1351] Viewing Alerts and Information

[1352] 1. The device displays the alert information and detailed data received from the server on the app screen.

[1353] Example: The app screen displays the risk assessment results using text and icons, visually communicating the risks to the user.

[1354] 2. The user reviews the displayed information and asks additional questions verbally if necessary.

[1355] Voice recognition and additional information provision

[1356] 1. If the user needs more information, they ask a question by voice. For example, "What should I eat instead of this food?"

[1357] 2. The device uses its voice recognition function to convert the user's voice into text and send it to the server.

[1358] 3. The server uses generative AI to generate the optimal answer based on the question and sends it to the device.

[1359] 4. The device will play back the received answer aloud and also display it as text, providing the user with both visual and auditory information.

[1360] Utilizing the Emotion Engine

[1361] 1. The emotion engine recognizes emotions from the user's voice or text input.

[1362] Example: Based on the user's tone of voice and text content, the emotion engine determines that the user is "stressed."

[1363] 2. The server customizes the advice it provides depending on the perceived emotional state.

[1364] Example: A user who is feeling stressed is offered advice on foods that have a relaxing effect and lifestyle habits to reduce stress.

[1365] 3. The server automatically selects a risk alert wording that is appropriate for the recognized emotion.

[1366] Example: Presenting risk alerts in a gentler tone to emotionally unstable users.

[1367] Data accumulation and feedback

[1368] 1. The server stores the user's food selection data, received advice, and recognized emotional state in a database.

[1369] 2. The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments.

[1370] In this way, the present invention provides a system that allows users to select appropriate foods taking into account their health and emotional state, thereby providing more personalized health support and helping users lead a healthier life with ease.

[1371] The processing flow will be explained below.

[1372] Processing procedure of a food selection system that combines an emotion engine

[1373] Step 1:

[1374] The user installs the application and enters basic information such as name, age, gender, medical history, and dietary restrictions when launching it for the first time.

[1375] Step 2:

[1376] The terminal transmits the input information to the server.

[1377] Step 3:

[1378] The server generates a user profile based on the received information, creates basic health advice tailored to the user, and stores it in a database.

[1379] Step 4:

[1380] The user launches the application and takes a photo of the label of the food item they plan to purchase.

[1381] Step 5:

[1382] The device passes the captured image to an OCR (optical character recognition) module, which extracts text information from the image.

[1383] How it works: The OCR module analyzes the image and extracts text such as ingredient information, calorie information, and additives.

[1384] Step 6:

[1385] The terminal transmits the extracted text data to the server.

[1386] Step 7:

[1387] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[1388] Step 8:

[1389] The server uses generative AI to analyze the ingredients and assess the risks associated with the user's health.

[1390] Specific operation: The health risk assessment module analyzes ingredient information and makes assessments such as "risk of high blood pressure due to high salt content."

[1391] Step 9:

[1392] The server generates the evaluated risk information as an alert and transmits it to the terminal for notification to the user.

[1393] Step 10:

[1394] The device displays the alert information and detailed data received from the server on the app screen.

[1395] Example: The app screen might say something like, "This food contains high salt content and may have a negative impact on high blood pressure."

[1396] Step 11:

[1397] The user reviews the displayed information and verbally asks additional questions if necessary.

[1398] Step 12:

[1399] The terminal uses a voice recognition function to convert the user's voice into text and send it to the server.

[1400] Specific operation: The speech recognition module analyzes the speech and converts it into text.

[1401] Step 13:

[1402] The server uses generative AI to generate the best answer based on the question and sends that answer to the device.

[1403] Example: Generate specific suggestions such as "List of low-sodium foods."

[1404] Step 14:

[1405] The device will play the received answer aloud and also display it as text.

[1406] Specific operation: Passes the text to the text-to-speech (TTS) module and reads it aloud.

[1407] Utilizing the Emotion Engine

[1408] Step 15:

[1409] The emotion engine recognizes emotions from the user's voice and text input.

[1410] Example: Recognizing emotions such as "anxiety" and "stress" from the user's voice.

[1411] Step 16:

[1412] The server customizes the advice it provides depending on the perceived emotional state.

[1413] Example: For users who are under stress, we suggest foods that have a relaxing effect and lifestyle habits that will help relieve stress.

[1414] Step 17:

[1415] The server automatically selects a risk alert presentation that is appropriate for the recognized emotion.

[1416] Specific behavior: For users who are emotionally anxious, a gentle tone of alert is displayed.

[1417] Data accumulation and feedback

[1418] Step 18:

[1419] The server stores the user's food selections, received advice, and recognized emotional state in a database.

[1420] Step 19:

[1421] The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments.

[1422] Example: A data analytics module analyzes accumulated data to improve the advice-giving algorithm.

[1423] Through this process, users can select the best foods based on their health and emotional state, providing more personalized health support and helping them live a healthier life.

[1424] Example 2

[1425] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1426] In recent years, with the increasing interest in health, there is a demand for personalized food selection support that takes into account individual health and emotional states. However, when users select healthy foods, it is not easy to accurately understand ingredient information and receive appropriate advice based on their current physical condition and emotional state. Furthermore, the advice provided is uniform and does not address individual emotional states, which may reduce user satisfaction.

[1427] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1428] In this invention, the server includes means for generating personalized health advice based on information entered by the user, means for extracting text information from a photographed food label image using optical character recognition, means for comparing the extracted text information with a food ingredient database and analyzing the ingredient information using a generation AI to evaluate risks associated with the user's health condition, means for notifying the user of the evaluated risk information as an alert, means for analyzing the user's voice input using voice recognition and providing appropriate answers and advice using a generation AI, means for analyzing the user's emotional state and customizing the advice to be provided depending on the emotional state, and means for selecting a method of expressing the risk alert appropriate to the emotional state. This enables the user to receive personalized advice that takes into account not only their health condition but also their emotional state.

[1429] "User" refers to an individual who uses the system and is the target of assistance in managing health conditions and food choices.

[1430] "Personalized health advice" refers to health guidance and recommendations tailored to a particular user based on user input.

[1431] Optical character recognition is a technology that extracts text information from images and photographs, and is used to incorporate food label information into the system.

[1432] The "Food Composition Database" is a database that collects information on the composition of various foods and provides basic data for composition analysis and risk assessment.

[1433] "Generative AI" refers to artificial intelligence technology that performs natural language processing and information generation based on large amounts of data, and is used to generate health advice and answers for users.

[1434] "Risk alert" refers to warning information that notifies users of health risks associated with food choices.

[1435] "Speech recognition" is a technology that converts a user's voice input into text data and is used to process spoken queries and commands.

[1436] "Emotional state" refers to the user's psychological and emotional state, and represents emotions such as stress or joy that can be inferred from voice and text input.

[1437] "Personalization" refers to providing optimized information and services that take into account the characteristics of each individual user.

[1438] "Food label" refers to the text and images containing ingredient and nutritional information that appear on food packaging.

[1439] MODE FOR CARRYING OUT THE INVENTION

[1440] The present invention provides a system that allows users to easily select healthy foods and provides personalized advice taking into account the user's emotional state. Specific embodiments of the system are described below.

[1441] User registration and initial settings

[1442] First, the user installs and launches the application on their device. When launching the application for the first time, the user enters basic information such as their name, age, gender, medical history, and dietary restrictions. The device then sends this information to the server. The server then creates a user profile based on the received information and creates initial health advice tailored to the user. For example, a user who needs to limit their fat intake may be provided with advice recommending low-fat foods.

[1443] Food Identification

[1444] The user uses the application to take a photo of the label of the food they plan to purchase with their camera. The device then passes the image to an optical character recognition (OCR) module, which extracts text information from the image. Specifically, the device uses an OCR service such as Google Cloud Vision API. The extracted text information is then sent from the device to a server.

[1445] Ingredient analysis and risk assessment

[1446] The server compares the received text information with a food ingredient database (e.g., USDA Nutrient Database) to obtain detailed information about each ingredient. Next, it uses generative AI (e.g., OpenAI's GPT-4) to analyze the ingredient information and evaluate the risk associated with the user's health. An example of a specific prompt is, "How does this ingredient affect high blood pressure?" The evaluated risk information is generated as an alert and sent to the device.

[1447] Viewing Alerts and Information

[1448] The device displays the alert information and detailed data received from the server on the app screen. For example, the app screen displays the risk assessment results using text and icons, visually communicating the risk to the user. The user can check the displayed information and ask additional questions by voice if necessary.

[1449] Voice recognition and additional information provision

[1450] If the user needs additional information, they can ask a question by voice. For example, they might ask, "What should I eat instead of this food?" The device uses a voice recognition function (e.g., Google Speech-to-Text) to convert the user's voice into text and send it to the server. The server then uses a generative AI to generate the optimal answer based on the question and sends it to the device. The device then plays the received answer aloud and also displays it as text.

[1451] Utilizing the Emotion Engine

[1452] An emotion engine (e.g., Microsoft Azure Emotion API) recognizes the user's emotional state from their voice or text input. Specifically, it determines whether the user is "feeling stressed" based on the user's tone of voice and the content of the text. The server then customizes the advice it provides based on the recognized emotional state. For example, a user who is feeling stressed could be offered advice about foods and lifestyle habits that have a relaxing effect. At the same time, the server selects a risk alert presentation style appropriate to the user's emotional state and sends it to the device.

[1453] Data accumulation and feedback

[1454] The server stores the user's food selections, advice received, and perceived emotional state in a database. This accumulated data is used to improve the accuracy of future advice and risk assessments. Specifically, the data is analyzed using data analytics tools (e.g., Python's Pandas and Scikit-learn), and the algorithm is improved based on the analysis results.

[1455] In this way, the present invention provides a system that allows users to select appropriate foods taking into account their health and emotional state, thereby providing more personalized health support and helping users lead a healthier life with ease.

[1456] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1457] Step 1:

[1458] Users install and launch the application on their device, then enter basic information such as their name, age, gender, medical history, and dietary restrictions.

[1459] Input: User's basic information (name, age, gender, medical history, dietary restrictions, etc.)

[1460] Specific action: The user enters information into an input form within the app and presses the "Submit" button.

[1461] Output: Request data with basic information entered

[1462] Step 2:

[1463] The terminal transmits the input basic information to the server.

[1464] Input: Request data (basic user information)

[1465] Specific operation: The terminal generates an HTTP request and sends data to the server.

[1466] Output: Basic information sent to the server

[1467] Step 3:

[1468] The server generates a user profile based on the received basic information, which also includes initial health advice for the user.

[1469] Input: User basic information

[1470] Specific operations: The server stores the information in a database and runs the profile generation algorithm.

[1471] Output: Generated user profile and initial advice

[1472] Step 4:

[1473] The user uses the application to take a photo of the label of the food item they plan to purchase with their camera.

[1474] Input: Food label image

[1475] Specific operation: The user activates the camera function and takes a picture of the food label.

[1476] Output: Photographed food label image

[1477] Step 5:

[1478] The device passes the captured food label image to an OCR module to extract text information.

[1479] Input: Food label image

[1480] Specific operation: Send the image to an OCR module (e.g., Google Cloud Vision API) and extract text information.

[1481] Output: Extracted text data

[1482] Step 6:

[1483] The terminal transmits the extracted text data to the server.

[1484] Input: Extracted text data

[1485] Specific operation: The terminal generates an HTTP request and sends data to the server.

[1486] Output: Text data sent to the server

[1487] Step 7:

[1488] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[1489] Input: Text data

[1490] Specific operation: The server queries the food ingredient database and retrieves ingredient information.

[1491] Output: Detailed information for each component

[1492] Step 8:

[1493] The server uses generative AI to analyze the ingredient information and assess the risks associated with the user's health condition.

[1494] Input: Ingredient information, user profile

[1495] Specific behavior: Send an analytical prompt to an AI model (e.g., OpenAI's GPT-4) to perform risk assessment. Specific prompt example: "How does this ingredient affect high blood pressure?"

[1496] Output: Risk assessment results

[1497] Step 9:

[1498] The server generates the evaluated risk information as an alert and transmits it to the terminal for notification to the user.

[1499] Input: Risk assessment results

[1500] Specific operation: An alert message is generated and sent to the terminal as an HTTP response.

[1501] Output: Generated risk alerts

[1502] Step 10:

[1503] The device displays the alert information and detailed data received from the server on the app screen.

[1504] Enter: Risk Alert

[1505] Specific behavior: Parse the risk alert and display it in a UI component (e.g., a text view or an icon).

[1506] Output: Risk alert displayed on the app screen

[1507] Step 11:

[1508] The user reviews the displayed information and verbally asks additional questions if necessary.

[1509] Input: Risk assessment results displayed on the terminal

[1510] Specific operation: The user presses the microphone button and begins voice input.

[1511] Output: User's voice data

[1512] Step 12:

[1513] The terminal uses a voice recognition function to convert the user's voice into text and send it to the server.

[1514] Input: User's voice data

[1515] Specific operation: Uses a speech recognition API (e.g., Google Speech-to-Text) to convert voice data into text, and sends the converted text to the server.

[1516] Output: Textual question data

[1517] Step 13:

[1518] The server uses generative AI to generate the best answer based on the question and sends it to the device.

[1519] Input: Textual question data

[1520] Specific operation: Send a prompt (e.g., "What should I eat instead of this food?") to the generating AI and get an answer.

[1521] Output: Generated response data

[1522] Step 14:

[1523] The device will play the received answer aloud and also display it as text.

[1524] Input: Generated response data

[1525] Specific behavior: Converts text to speech using a speech synthesis API (e.g., Google Text-to-Speech), plays the converted speech, and displays the text in the UI.

[1526] Output: Answer information as audio and text

[1527] Step 15:

[1528] The emotion engine recognizes the user's emotional state from their voice and text input.

[1529] Input: User's voice data, text data

[1530] Specific behavior: Uses emotion recognition APIs (e.g. Microsoft Emotion API) to infer emotions from speech and text.

[1531] Output: Perceived emotional state

[1532] Step 16:

[1533] The server customizes the advice it provides depending on the perceived emotional state.

[1534] Input: perceived emotional state, user profile

[1535] Specific behavior: Send a prompt (e.g., "What should you recommend to a user who is feeling stressed?") to the generative AI to get customized advice.

[1536] Output: Customized advice

[1537] Step 17:

[1538] The server selects a method of expressing a risk alert that is appropriate for the emotional state and notifies the terminal.

[1539] Input: Customized advice, risk assessment results

[1540] Specific behavior: Adjust the tone and format of the text depending on the emotional state and send it to the device in an HTTP response.

[1541] Output: Adjusted risk alerts

[1542] Step 18:

[1543] The server stores the user's food selection data, received advice, and recognized emotional state in a database.

[1544] Input: User's food data, advice, perceived emotional state

[1545] Specific operation: Executes a data insertion query against the database.

[1546] Output: Information stored in the database

[1547] Step 19:

[1548] The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments.

[1549] Input: Accumulated data

[1550] Specific Tasks: Analyze data and improve algorithms using data analytics tools (e.g., Python's Pandas, Scikit-learn).

[1551] Output: Improved algorithm

[1552] (Application example 2)

[1553] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1554] Many consumers today are becoming increasingly health-conscious and understand the importance of choosing foods that are right for them. However, it is difficult to instantly evaluate food ingredients and health risks while shopping in a physical store, and to receive optimal advice based on one's emotional state at the time. Especially in today's world where a wide variety of foods are available, obtaining detailed information about each food item takes time and effort. Furthermore, existing systems are unable to provide advice that takes into account the user's emotional state, making it difficult to provide accurate support tailored to each user's individual situation.

[1555] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for generating personalized health advice based on information entered by a user, means for extracting text information from a photographed food label image using optical character recognition, means for comparing the extracted text information with a food ingredient database, analyzing the ingredient information using a generation AI, and evaluating risks associated with the user's health status, means for notifying the user of the evaluated risk information as an alert, means for analyzing the user's voice input using voice recognition and providing appropriate answers and advice using a generation AI, means for analyzing the user's emotional state and personalizing advice according to the emotional state, and means for providing recommended products and purchase plans suited to the user's health status and emotional status based on data from a physical store. This allows users to easily select foods suitable for them in a physical store and receive personalized advice tailored to their health status and emotional state.

[1556] "Means for generating personalized health advice based on user-entered information" refers to a method for collecting information entered by a user into the application, such as name, age, gender, medical history, and dietary restrictions, and generating and providing specific health advice based on that information.

[1557] "Means for extracting text information from photographed food label images using optical character recognition" refers to a method in which a user takes a photo of a food label using a camera on a smartphone or other device, and extracts text information from the image using optical character recognition technology.

[1558] "Means of comparing extracted text information with a food ingredient database, analyzing ingredient information using generative AI, and assessing risks related to the user's health" refers to a method of comparing text information obtained by optical character recognition with a food ingredient database, and then using a generative AI model to perform a detailed analysis of the ingredients and assess the user's health risks.

[1559] "Means for notifying the user of the evaluated risk information as an alert" refers to a method for evaluating the risk to the user's health based on the analyzed component information and notifying the user of the results as a warning.

[1560] "Means of analyzing a user's voice input through speech recognition and using generative AI to provide appropriate answers and advice" refers to a method of converting a user's spoken questions or requests into text using speech recognition technology, analyzing it using a generative AI model, and generating appropriate answers and advice.

[1561] "Means for analyzing the user's emotional state and personalizing advice according to that emotional state" refers to a method for analyzing emotional information obtained from the user's voice input or text input and providing customized advice according to that emotion.

[1562] "Means for providing recommended products and purchase plans suited to the user's health and emotional state based on data from physical stores" refers to a method that utilizes user behavior data and selected product data obtained from physical stores to suggest products and purchase plans that are optimal for the user's health and emotional state.

[1563] "Optical character recognition" is a technology that mechanically reads characters printed on paper or text contained in images and extracts it as digital data.

[1564] "Generative AI" is an artificial intelligence technology that generates new information and answers based on pre-trained data.

[1565] The present invention provides a system that allows users to easily select foods suitable for them in a physical store and receive personalized advice tailored to their health and emotional state. Specific embodiments of this system are described below.

[1566] User registration and initial settings

[1567] Users enter basic information such as name, age, gender, medical history, and dietary restrictions through an application installed on their smartphone. The device then sends this information to a server, which then generates a user profile based on the information received and prepares health advice. This profile is then used for future analysis and advice provision.

[1568] Food Identification

[1569] A user takes a photo of the label of a food item they are considering purchasing in a physical store using their smartphone camera. The device passes the image to an optical character recognition (OCR) module, which extracts text information from the image. The extracted text data is then sent to a server.

[1570] Ingredient analysis and risk assessment

[1571] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient. It then uses a generative AI model to analyze the ingredients and assess the risk associated with the user's health. For example, if the analysis reveals that a user with high blood pressure chooses a high-salt food, a risk assessment will be made, such as, "This food contains high salt and may have a negative impact on high blood pressure." The assessed risk information is generated as an alert and sent to the device to notify the user.

[1572] Viewing Alerts and Information

[1573] The device displays the alert information and detailed data received from the server on the app screen. For example, the risk assessment results can be displayed as text and icons to visually inform the user of the risk. If the user asks additional questions by voice, the voice is converted into text by a voice recognition module and sent to the server.

[1574] Voice recognition and additional information provision

[1575] The server uses a generative AI model to generate an optimal answer based on the content of the voice question and sends it to the device, which then plays the received answer back aloud and can also display it as text, allowing the user to obtain information both visually and audibly.

[1576] Utilizing the Emotion Engine

[1577] The emotion engine recognizes emotions from the user's voice and text input. For example, the emotion engine may determine that the user is "feeling stressed" based on the user's tone of voice and the content of the text. The server then customizes the advice it provides based on the recognized emotional state. Specifically, for a user who is feeling stressed, it provides advice on foods that have a relaxing effect and lifestyle habits to reduce stress. For users with unstable emotions, it displays risk alerts in a gentler tone.

[1578] Data accumulation and feedback

[1579] The server stores the user's food selections, received advice, and perceived emotional state in a database, which is used to improve the accuracy of future advice and risk assessments.

[1580] This allows users to easily choose the foods that are right for them in physical stores and receive personalized advice tailored to their health and emotional state.

[1581] Prompt Sentence Examples

[1582] Below are some examples of prompt sentences.

[1583] Food ingredient analysis prompt:

[1584] You are given the following list of food ingredients. Analyze the risks these ingredients pose to a user with high blood pressure and perform a risk assessment.

[1585] Ingredients: Salt

[1586] Ingredients: Sodium 1500mg

[1587] Emotion recognition prompts:

[1588] Recognize emotions from the following user voice inputs:

[1589] Text: "I'm feeling super stressed today. Is there a good way to relax?"

[1590] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1591] Step 1:

[1592] Users install the application on their smartphone and enter basic information such as their name, age, gender, medical history, and dietary restrictions. The application then collects the initial data to create a user profile. The entered information is sent from the device to a server. The server receives this information, generates a user profile, and stores it as the basis for health advice.

[1593] Step 2:

[1594] Users take a photo of the label of a food item they are considering purchasing in a physical store using their smartphone camera. The device then passes the image to an optical character recognition (OCR) module to extract text information from the image. This extraction process utilizes OpenCV and the Tesseract library. The extracted text data is then sent to a server.

[1595] Step 3:

[1596] The server compares the received text data with a food ingredient database, thereby obtaining detailed information about each ingredient. It then analyzes the ingredient information using a generative AI model (e.g., GPT-4) to assess the risk associated with the user's health. The results of the risk assessment are formatted as specific alert information, such as "not suitable for users with high blood pressure." The assessment results are then sent from the server to the device.

[1597] Step 4:

[1598] The terminal visually displays the alert information and detailed data received from the server to the user. The alert information is expressed using text and icons, and is used to intuitively convey risk information to the user. If the user has additional questions, their voice is converted into text using the application's speech recognition module. The Google Speech-to-Text API is used for speech recognition. The text data of the question is sent to the server.

[1599] Step 5:

[1600] The server generates an appropriate answer using a generative AI model based on the text data received as a speech recognition result. The generated answer is formatted as specific advice and sent from the server to the device. The device displays this advice visually to the user and plays it aloud using speech synthesis (Google Text-to-Speech API).

[1601] Step 6:

[1602] The emotional state of the user is analyzed using an emotion engine (e.g., Amazon Rekognition) based on the user's voice or text input. The server then personalizes advice according to the user's emotional state based on the emotional data received from the emotion engine. Advice is generated according to the emotion, and customization is performed, such as displaying risk alerts in a gentle tone. The customized advice is then sent from the server to the device.

[1603] Step 7:

[1604] The server stores the user's food selections, advice received, and perceived emotional state in a database. This data is used to improve the accuracy of future advice and risk assessments. This data is associated with the user's profile and analyzed to provide more appropriate advice.

[1605] These steps allow users to easily choose the foods that are right for them in a physical store and receive personalized advice based on their health and emotional state.

[1606] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1607] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1608] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1609] [Fourth embodiment]

[1610] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1611] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1612] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1613] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1614] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1615] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1616] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1617] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1618] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1619] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1620] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1621] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1622] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1623] The present invention provides a system that allows users to easily select healthy foods, and includes a program that operates by combining various technologies to achieve this. Specific embodiments of this system are described below.

[1624] User registration and initial settings

[1625] 1. The user installs the application and, when launching it for the first time, enters basic information such as name, age, gender, medical history, and dietary restrictions on the account creation screen.

[1626] 2. The terminal sends the information entered by the user to the server.

[1627] 3. The server generates a user profile based on the received information and tailors health advice to the user. This profile is stored in a database for future analysis and advice provision.

[1628] Food Identification

[1629] 1. The user opens the application and takes a photo of the label of a food item they are considering purchasing.

[1630] 2. The device passes the captured image to an OCR (Optical Character Recognition) module to extract text information from the image.

[1631] Example: If a food label contains information such as "phosphates," "sodium," and "sugars," these will be read as text.

[1632] 3. The terminal sends the extracted text data to the server.

[1633] Ingredient analysis and risk assessment

[1634] 1. The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[1635] 2. The server uses generative AI to analyze the ingredients and assess the risks associated with the user's health.

[1636] Example: If the analysis reveals that a user with high blood pressure selects a high-salt food, a risk assessment will be made, such as "This food contains a high amount of salt and may have a negative impact on high blood pressure."

[1637] 3. The server generates the results of the risk assessment as alert information and sends it to the terminal.

[1638] Viewing Alerts and Information

[1639] 1. The terminal displays the alert information and detailed data received from the server on the application screen.

[1640] Example: The application screen displays the results of the risk assessment using text and icons, visually communicating the risks to the user.

[1641] 2. The user reviews the information displayed and asks additional questions if necessary.

[1642] Voice recognition and additional information provision

[1643] 1. If the user wants more information, they can ask a question out loud. For example, "What should I eat instead of this food?"

[1644] 2. The device uses voice recognition to convert the user's question into text and send it to the server.

[1645] 3. The server uses generative AI to generate the best answer based on the question.

[1646] Example: Create a "list of low-salt foods" based on a food database and health guidelines and send it to your device.

[1647] 4. The device will play the received answer aloud and also display it as text, allowing the user to receive information both visually and audibly.

[1648] Data accumulation and feedback

[1649] 1. The server stores the user's food selection data and received advice in a database.

[1650] 2. The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments.

[1651] In this way, the present invention realizes a system that allows users to easily select healthy foods. By providing users with checked ingredient information and risk alerts, the system helps users make healthy food choices. This system allows users to make healthy food choices without spending time or effort.

[1652] The processing flow will be explained below.

[1653] Food selection system processing procedure

[1654] Step 1:

[1655] Users install the app and enter basic information such as their name, age, gender, medical history, and dietary restrictions when they first launch it.

[1656] Step 2:

[1657] The terminal transmits the input information to the server.

[1658] Step 3:

[1659] The server generates a user profile based on the received information and sets basic health advice tailored to the user. This profile is used for future analysis and advice provision.

[1660] Step 4:

[1661] The user launches the app and takes a photo of the label of the food item they plan to purchase.

[1662] Step 5:

[1663] The device passes the captured image to an OCR (optical character recognition) module, which extracts text information from the image.

[1664] Step 6:

[1665] The terminal transmits the extracted text data to the server.

[1666] Step 7:

[1667] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[1668] Step 8:

[1669] The server uses generative AI to analyze the ingredients and assess the risks associated with the user's health.

[1670] Step 9:

[1671] The server generates the evaluated risk information as an alert and transmits it to the terminal for notification to the user.

[1672] Step 10:

[1673] The device displays the alert information and detailed data received from the server on the app screen, using text and icons to visually communicate risk to the user.

[1674] Step 11:

[1675] The user reviews the displayed information and verbally asks additional questions if necessary.

[1676] Step 12:

[1677] The terminal uses a voice recognition function to convert the user's voice into text and send it to the server.

[1678] Step 13:

[1679] The server uses generative AI to generate the best answer based on the question and sends that answer to the device.

[1680] Step 14:

[1681] The device will play back the received answer aloud and also display it as text, allowing the user to receive information both visually and audibly.

[1682] Step 15:

[1683] The server stores the food data selected by the user and the advice received in a database, which is used to improve the accuracy of future advice and risk assessments.

[1684] This series of processes allows users to easily choose healthy foods and maintain and improve their own health.

[1685] Example 1

[1686] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1687] Choosing healthy foods is an important issue in modern society. However, many users have difficulty making appropriate choices because they do not fully understand the ingredients and health risks of foods. It is also difficult to obtain personalized advice tailored to their own health status. As a result, inappropriate food choices can have a negative impact on health. Furthermore, there is a lack of systems that can quickly respond to users' questions or requests for additional information. To address these issues, a system is needed that analyzes users' health information and food ingredients and provides individually customized advice and risk assessments.

[1688] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1689] In this invention, the server includes a means for generating personalized health advice based on information entered by the user, a means for extracting text information from a photographed food label image using OCR (optical character recognition), and a means for comparing the extracted text information with a food ingredient database, analyzing the ingredient information using a generation AI, and assessing risks associated with the user's health condition. This enables accurate advice and risk assessment that are individually customized based on the user's health condition.

[1690] "User" refers to an individual who uses the system to select healthy foods.

[1691] "Terminal" refers to a smartphone, tablet, or other electronic device used by a user to run applications and input, display, and communicate information.

[1692] "Server" refers to a remote computer system that processes data received from users and devices and provides advice and risk assessments.

[1693] "Health Advice" refers to specific recommendations to promote healthy living based on information entered by the user.

[1694] "Food label image" refers to an image of the label attached to the package of food that the user is considering purchasing, taken with a camera.

[1695] "OCR (Optical Character Recognition)" refers to a technology that analyzes the text information contained in an image and extracts it as machine-readable character data.

[1696] A "food composition database" refers to a collection of data that collects and stores information on the composition of various foods.

[1697] "Generative AI" refers to artificial intelligence that uses the technology of generative AI models (e.g., GPT-3) to generate appropriate answers and advice to human questions.

[1698] "Risk assessment" refers to analyzing the impact of ingredients contained in food on the user's health and determining the level of risk.

[1699] "Alert" refers to a warning message intended to notify the user of health risks or other important information.

[1700] "Speech recognition" refers to the technology that converts a user's voice input into text data.

[1701] A "profile" refers to a data set that records a user's basic information and health status.

[1702] "Historical Data" refers to a record of the foods a user selects and the advice they receive.

[1703] "Personalization accuracy" refers to the accuracy and suitability of the advice and information provided to an individual user.

[1704] The present invention provides a system that allows a user to easily select healthy foods. An embodiment of this system will now be described in detail.

[1705] First, the user installs the application and enters basic information such as name, age, gender, medical history, and dietary restrictions on the account creation screen when first starting the application. This basic information is used to create a user profile and is sent from the device to the server. The server generates a user profile based on the received information and stores it in a database. This allows for future analysis and provision of health advice tailored to the user.

[1706] Next, when the user takes a photo of the label of the food they are considering purchasing, an OCR (optical character recognition) module is used to extract text information from the image, using the Tesseract OCR library as an example. The extracted text data is sent from the device to a server and compared with a food ingredient database.

[1707] The server uses the received text data to compare it with a food ingredient database to obtain detailed information about each ingredient. It then analyzes the ingredients using a generative AI model (e.g., GPT-3) to assess the risk associated with the user's health. The assessed risk information is generated as an alert and sent to the device. The device displays the received alert information on the application screen and communicates the risk assessment results to the user visually and audibly.

[1708] If the user wants more detailed information, they can ask by voice. The device uses voice recognition to convert the user's question into text and send it to the server. The server then uses generative AI to generate the best answer based on the question, providing information such as a "list of low-salt foods" based on a food database and health guidelines. The device then plays back the received answer by voice and also displays it as text, allowing the user to receive information both visually and audibly.

[1709] In addition, all user-selected food data and advice received are stored in a database. The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments. Data mining algorithms can be used to extract patterns from the accumulated data and train models to improve the accuracy of advice.

[1710] (Example)

[1711] For example, if a user uses the application to scan a "potato chips" label, the OCR module extracts ingredients such as "sodium," "fat," and "sugar." The server analyzes these ingredients, and if the user is overweight and has a history of high blood pressure, an alert message will be displayed stating, "This food contains high amounts of sodium and fat, which may have adverse health effects." Furthermore, if the user asks "What are some alternative foods?" by voice, the server will list "low-sodium and low-calorie foods," and the device will provide that information via voice and text.

[1712] (Example of a prompt)

[1713] "Analyze the ingredients in this food and assess the risk."

[1714] "What healthy options can I use to replace this food?"

[1715] "Create a recommended food list for users with high blood pressure."

[1716] In this way, users can make healthy food choices without spending time or effort.

[1717] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1718] Step 1: Enter and submit basic user information

[1719] The user installs and launches the application, and enters basic information such as name, age, gender, medical history, and dietary restrictions on the account creation screen. The device then formats this information into a database and sends it to the server.

[1720] Input: Basic information entered by the user (name, age, gender, medical history, dietary restrictions)

[1721] Data processing: Convert input information into JSON format

[1722] Output: Send basic user information data in JSON format to the server

[1723] What it does: Parses data entered into form fields into JSON format and sends it via a REST API.

[1724] Step 2: Generate and save a profile on the server

[1725] The server generates a user profile based on the received user information and stores it in a database, and prepares to provide personalized health advice based on the profile.

[1726] Input: Received user basic information data

[1727] Data processing: Insert user information into the database

[1728] Output: User profile stored in the database

[1729] What it does: Runs a script that inserts user information into a MySQL database and generates profile data.

[1730] Step 3: Photograph food labels and extract text using OCR

[1731] The user launches the application and takes a photo of the label of a food item they are considering purchasing. The device inputs the image into the OCR module and extracts text information from the image.

[1732] Input: A photograph of a food label image

[1733] Data processing: Convert images to text using OCR processing

[1734] Output: Extracted text data

[1735] What it does: Uses the Tesseract OCR library to extract text from image data and save it as a string.

[1736] Step 4: Sending text data to the server

[1737] The terminal transmits the extracted text data to the server.

[1738] Input: Extracted text data

[1739] Data processing: Convert text data into JSON format

[1740] Output: Send JSON format text data to the server

[1741] Specific operation: The extracted text data is converted into JSON format and sent to the server via REST API.

[1742] Step 5: Obtaining ingredient information and risk assessment on the server

[1743] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient, then uses a generative AI model to analyze the ingredients and assess the risk associated with the user's health.

[1744] Input: Received text data

[1745] Data processing: Query the database using ingredient names as keys, and perform risk assessment using a generative AI model

[1746] Output: Risk assessment results

[1747] Specific operation: Query ingredient information based on text data, provide prompts to the generative AI model, and perform risk assessment.

[1748] Step 6: Generate and send alert information

[1749] The server generates alert information based on the results of the risk assessment and sends it to the terminal.

[1750] Input: Risk assessment results

[1751] Data processing: Converting risk assessment results into alert information

[1752] Output: Generated alert information

[1753] Specific operation: Based on the risk assessment, the alert information is parsed into JSON format and sent to the terminal.

[1754] Step 7: View alert information

[1755] The terminal displays the alert information and detailed data received from the server on the application screen.

[1756] Input: Received alert information

[1757] Data processing: Binding alert information to UI components

[1758] Output: Alert information displayed on the application screen

[1759] Specific behavior: Parses alert information and binds the data to UI components (e.g. TextView, ImageView).

[1760] Step 8: Request additional information via voice recognition

[1761] If the user wants additional information, they can ask a question by voice, and the device will use its voice recognition function to convert the question into text and send it to the server.

[1762] Input: User's voice question

[1763] Data processing: Converting voice data into text

[1764] Output: Textual question data

[1765] Specific operation: Activates the voice input function and converts the voice data into text using the Google Speech-to-Text API.

[1766] Step 9: Generate additional information on the server

[1767] The server uses generative AI to generate the best answer based on the question and sends it to the device.

[1768] Input: Textual question data

[1769] Data processing: Question data is used as a prompt and provided to the generative AI model to generate an answer

[1770] Output: Generated response data

[1771] Specific operation: The question content is provided as a prompt to the AI ​​model, an answer is generated, the answer data is converted to JSON format, and sent to the device.

[1772] Step 10: View and play answers

[1773] The terminal will play back the received answer aloud and also display it as text.

[1774] Input: Generated response data

[1775] Data processing: Converting response data into voice and text

[1776] Output: Spoken and textual answers

[1777] Specific operation: Text data is converted to audio using the Google Text-to-Speech API, played back to the user, and displayed in a text view.

[1778] Step 11: Data collection and feedback

[1779] The server stores the user's food selections and advice received in a database, accumulating feedback, and then analyzes the data to improve the accuracy of future advice and risk assessments.

[1780] Input: User food choice data, received advice

[1781] Data processing: Inserting data into databases, generating datasets for analysis

[1782] Output: Accumulated historical data, analysis results

[1783] Specific operation: User activity history is inserted into a database, and data mining algorithms are used to extract and analyze patterns from the accumulated data.

[1784] (Application example 1)

[1785] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1786] Conventional health management systems require users to spend a lot of time and effort selecting the right foods, making it difficult to provide appropriate health advice in real time. Furthermore, they are inadequate in terms of protecting user privacy and ensuring data security, making it difficult for users to manage their health data with peace of mind.

[1787] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1788] In this invention, the server includes means for generating personalized health advice based on information entered by the user, means for extracting text information from a photographed food label image using OCR (optical character recognition), means for comparing the extracted text information with a food ingredient database and analyzing the ingredient information using a generation AI to evaluate risks associated with the user's health status, means for visually and audibly notifying the user of the evaluated risk information in real time, and means for analyzing the user's voice input using voice recognition and providing appropriate answers and advice using a generation AI, thereby enabling users to make healthy food choices in real time while also ensuring privacy protection and data security.

[1789] A "user" is someone who uses the system to select healthy foods.

[1790] "Health advice" is a suggestion for appropriate food choices and lifestyle improvements based on the user's health condition.

[1791] "Food label images" are photographs or images containing ingredient and nutritional information found on food packaging.

[1792] "OCR (Optical Character Recognition)" is a technology that extracts character information from image data.

[1793] "Text information" refers to character data such as food ingredients and nutritional information extracted by OCR.

[1794] A "food composition database" is a collection of data that records the composition and nutritional information of various foods.

[1795] "Generative AI" is a system that uses machine learning and artificial intelligence technologies to analyze data, assess risks, and generate advice.

[1796] "Risk assessment" refers to the analysis and determination of the potential health risks posed by specific foods based on the user's health status and input information.

[1797] An "alert" is information that alerts or warns the user.

[1798] "Speech recognition" is a technology that analyzes a user's voice and converts it into text information.

[1799] A "smart device" is an electronic device with advanced functions, including internet connectivity and various sensors.

[1800] "Personalization accuracy" refers to the degree to which the advice and information provided is relevant to the characteristics and preferences of each individual user.

[1801] "Real-time" is a time concept that refers to data processing and information provision occurring almost instantaneously.

[1802] "Visual information" refers to information such as images and text that is presented through a display or screen.

[1803] "Audio information" refers to audio messages delivered through a speaker or speech synthesis system.

[1804] "Privacy protection" is a security measure to protect users' personal information and health data from third parties.

[1805] "Data security" means that data collected, stored and processed is protected from unauthorized access and destruction.

[1806] To realize the present invention, it is necessary to design a system using the following hardware and software.

[1807] First, the system uses a smart device (e.g., smart glasses or a smartphone) that includes a camera, display, microphone, and audio output device. A cloud-based computing service (e.g., AWS) is used as the server. For analysis and database management, an OCR module (e.g., Tesseract), generative AI (e.g., OpenAI GPT-3), and a database (e.g., MySQL) are used. For security software, Symantec Endpoint Protection is used.

[1808] 1. User registration and data management

[1809] The device presents a form for users to register their health data when they first launch the application. Users enter basic information such as name, age, gender, medical history, and dietary restrictions. This information is stored in a database and encrypted by endpoint protection software.

[1810] 2. Food Label Identification

[1811] Users take a photo of a food label using their smart device's camera. The image is sent to an OCR module to extract text information. The extracted text data is then sent to a cloud server where the generative AI analyzes the ingredient information.

[1812] 3. Component analysis and risk assessment

[1813] The cloud server analyzes the received text information by comparing it with a food composition database. The generating AI then compares it with the user's health profile and assesses health risks. For example, if a user with high blood pressure chooses a high-salt food, a risk assessment will be made, such as "This food contains a high amount of salt and may have a negative impact on high blood pressure," and the user will be notified in real time.

[1814] 4. Real-time alerts

[1815] The assessed risk information is not only visually displayed on the smart device display, but also audibly notified via the audio output device, allowing users to receive information both visually and audibly.

[1816] 5. User voice questions and feedback

[1817] If the user wants more detailed information, they can ask a question by voice. For example, they might ask, "What should I eat instead of this food?" The smart device uses its voice recognition function to convert the user's question into text and send it to a cloud server. Based on the question, the generative AI generates an answer from a food database and health guidelines, and provides feedback to the user in real time via text and voice.

[1818] Specific examples

[1819] A user selects a "low-sugar yogurt" at the supermarket and scans the label with the camera on the smart glasses. The OCR reads the ingredient information for the "low-sugar yogurt" and sends it to a cloud server. The generative AI analyzes it and displays advice on the smart glasses, such as "These ingredients are glycemic, but slightly higher in fat. Consider low-fat options."

[1820] Example prompts for generative AI models

[1821] User Profile:

[1822] Name: Tanaka Taro

[1823] Age: 45

[1824] Gender: Male

[1825] Medical history: high blood pressure

[1826] Dietary restrictions: Low salt

[1827] Input food: Low-sugar yogurt

[1828] Ingredients: Carbohydrates, fat, calcium

[1829] Questions asked:

[1830] "Is this food suitable for me?"

[1831] advice:

[1832] "These ingredients are blood sugar friendly, but they are a little higher in fat. Consider lower-fat options."

[1833] In this way, the present invention realizes a system that allows users to make healthy food choices in real time while ensuring privacy and data security.

[1834] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1835] Step 1: User registration and data management

[1836] When a user launches the application for the first time, they enter basic information such as their name, age, gender, medical history, dietary restrictions, etc. The device then sends this information to a database, where it is stored as encrypted data using endpoint protection software.

[1837] Input: User's basic information (name, age, gender, medical history, dietary restrictions)

[1838] Output: Encrypted user profile data

[1839] Specific operation: The information entered into the form on the terminal is sent to the server and stored in the destination database.

[1840] Step 2: Food Label Identification

[1841] The user takes a photo of a food label using the camera on their smart device, and the device passes the image to the OCR module to extract text information, which is then sent to the server.

[1842] Input: Food label image

[1843] Output: Extracted text information

[1844] Specific operation: The camera takes a photo of the food label, extracts text information using the OCR module (Tesseract), and sends it to the server.

[1845] Step 3: Ingredient analysis and risk assessment

[1846] The server compares the received text information with a food ingredient database to obtain detailed ingredient information, and uses generative AI to assess the risk based on the user's health status.

[1847] Input: Extracted text information, user profile

[1848] Output: Risk assessment results and advice on countermeasures

[1849] Specific operation: Using a generative AI model (GPT-3), ingredient information is analyzed and compared with the user's health profile to perform risk assessment.

[1850] Step 4: Real-time alerts

[1851] The terminal notifies the user of the risk assessment results received from the server visually and audibly, showing risk information on the display and explaining risk details via the audio output device.

[1852] Input: Risk assessment results

[1853] Output: Visual and audio risk information

[1854] Specific operation: The risk assessment results are displayed on the smart device display and notified to the user via voice output.

[1855] Step 5: User voice questions and feedback

[1856] The user asks a follow-up question by voice. The device uses voice recognition to convert the user's question into text and send it to the server. The server then uses generative AI to generate an answer and provides real-time feedback to the user in text and voice.

[1857] Input: User's voice question

[1858] Output: Answers and advice from generative AI

[1859] Specific operation: The question is converted into text using the voice recognition function, and the generative AI model (GPT-3) generates an appropriate answer based on the question. The generated answer is then fed back to the user via the device's display and voice output.

[1860] Step 6: Accumulating data and improving personalization accuracy

[1861] The server stores the food selections and advice received by the user in a database, and the accumulated data will be used to improve the accuracy of personalization in the future.

[1862] Input: Selected food data, received advice

[1863] Output: Updated user profile data

[1864] Specific operation: The server stores the selected food data and advice in a database and uses it as analytical data to improve the accuracy of future advice.

[1865] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1866] The present invention provides a system that allows users to easily select healthy foods and provides personalized advice taking into account the user's emotional state. Specific embodiments of this system are described below.

[1867] User registration and initial settings

[1868] 1. The user installs the application and enters basic information such as name, age, gender, medical history, and dietary restrictions when launching it for the first time.

[1869] 2. The terminal sends the entered information to the server.

[1870] 3. The server generates a user profile based on the received information and creates personalized health advice for the user. This profile is used for future analysis and advice provision.

[1871] Food Identification

[1872] 1. The user opens the application and takes a photo of the label of the food item they plan to purchase.

[1873] 2. The device passes the captured image to an OCR (Optical Character Recognition) module to extract text information from the image.

[1874] 3. The terminal sends the extracted text data to the server.

[1875] Ingredient analysis and risk assessment

[1876] 1. The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[1877] 2. The server uses generative AI to analyze the ingredients and assess the risks associated with the user's health.

[1878] Example: If the analysis reveals that a user with high blood pressure selects a high-salt food, a risk assessment will be made, such as "This food contains a high amount of salt and may have a negative impact on high blood pressure."

[1879] 3. The server generates the assessed risk information as an alert and sends it to the terminal to notify the user.

[1880] Viewing Alerts and Information

[1881] 1. The device displays the alert information and detailed data received from the server on the app screen.

[1882] Example: The app screen displays the risk assessment results using text and icons, visually communicating the risks to the user.

[1883] 2. The user reviews the displayed information and asks additional questions verbally if necessary.

[1884] Voice recognition and additional information provision

[1885] 1. If the user needs more information, they ask a question by voice. For example, "What should I eat instead of this food?"

[1886] 2. The device uses its voice recognition function to convert the user's voice into text and send it to the server.

[1887] 3. The server uses generative AI to generate the optimal answer based on the question and sends it to the device.

[1888] 4. The device will play back the received answer aloud and also display it as text, providing the user with both visual and auditory information.

[1889] Utilizing the Emotion Engine

[1890] 1. The emotion engine recognizes emotions from the user's voice or text input.

[1891] Example: Based on the user's tone of voice and text content, the emotion engine determines that the user is "stressed."

[1892] 2. The server customizes the advice it provides depending on the perceived emotional state.

[1893] Example: A user who is feeling stressed is offered advice on foods that have a relaxing effect and lifestyle habits to reduce stress.

[1894] 3. The server automatically selects a risk alert wording that is appropriate for the recognized emotion.

[1895] Example: Presenting risk alerts in a gentler tone to emotionally unstable users.

[1896] Data accumulation and feedback

[1897] 1. The server stores the user's food selection data, received advice, and recognized emotional state in a database.

[1898] 2. The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments.

[1899] In this way, the present invention provides a system that allows users to select appropriate foods taking into account their health and emotional state, thereby providing more personalized health support and helping users lead a healthier life with ease.

[1900] The processing flow will be explained below.

[1901] Processing procedure of a food selection system that combines an emotion engine

[1902] Step 1:

[1903] The user installs the application and enters basic information such as name, age, gender, medical history, and dietary restrictions when launching it for the first time.

[1904] Step 2:

[1905] The terminal transmits the input information to the server.

[1906] Step 3:

[1907] The server generates a user profile based on the received information, creates basic health advice tailored to the user, and stores it in a database.

[1908] Step 4:

[1909] The user launches the application and takes a photo of the label of the food item they plan to purchase.

[1910] Step 5:

[1911] The device passes the captured image to an OCR (optical character recognition) module, which extracts text information from the image.

[1912] How it works: The OCR module analyzes the image and extracts text such as ingredient information, calorie information, and additives.

[1913] Step 6:

[1914] The terminal transmits the extracted text data to the server.

[1915] Step 7:

[1916] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[1917] Step 8:

[1918] The server uses generative AI to analyze the ingredients and assess the risks associated with the user's health.

[1919] Specific operation: The health risk assessment module analyzes ingredient information and makes assessments such as "risk of high blood pressure due to high salt content."

[1920] Step 9:

[1921] The server generates the evaluated risk information as an alert and transmits it to the terminal for notification to the user.

[1922] Step 10:

[1923] The device displays the alert information and detailed data received from the server on the app screen.

[1924] Example: The app screen might say something like, "This food contains high salt content and may have a negative impact on high blood pressure."

[1925] Step 11:

[1926] The user reviews the displayed information and verbally asks additional questions if necessary.

[1927] Step 12:

[1928] The terminal uses a voice recognition function to convert the user's voice into text and send it to the server.

[1929] Specific operation: The speech recognition module analyzes the speech and converts it into text.

[1930] Step 13:

[1931] The server uses generative AI to generate the best answer based on the question and sends that answer to the device.

[1932] Example: Generate specific suggestions such as "List of low-sodium foods."

[1933] Step 14:

[1934] The device will play the received answer aloud and also display it as text.

[1935] Specific operation: Passes the text to the text-to-speech (TTS) module and reads it aloud.

[1936] Utilizing the Emotion Engine

[1937] Step 15:

[1938] The emotion engine recognizes emotions from the user's voice and text input.

[1939] Example: Recognizing emotions such as "anxiety" and "stress" from the user's voice.

[1940] Step 16:

[1941] The server customizes the advice it provides depending on the perceived emotional state.

[1942] Example: For users who are under stress, we suggest foods that have a relaxing effect and lifestyle habits that will help relieve stress.

[1943] Step 17:

[1944] The server automatically selects a risk alert presentation that is appropriate for the recognized emotion.

[1945] Specific behavior: For users who are emotionally anxious, a gentle tone of alert is displayed.

[1946] Data accumulation and feedback

[1947] Step 18:

[1948] The server stores the user's food selections, received advice, and recognized emotional state in a database.

[1949] Step 19:

[1950] The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments.

[1951] Example: A data analytics module analyzes accumulated data to improve the advice-giving algorithm.

[1952] Through this process, users can select the best foods based on their health and emotional state, providing more personalized health support and helping them live a healthier life.

[1953] Example 2

[1954] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1955] In recent years, with the increasing interest in health, there is a demand for personalized food selection support that takes into account individual health and emotional states. However, when users select healthy foods, it is not easy to accurately understand ingredient information and receive appropriate advice based on their current physical condition and emotional state. Furthermore, the advice provided is uniform and does not address individual emotional states, which may reduce user satisfaction.

[1956] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1957] In this invention, the server includes means for generating personalized health advice based on information entered by the user, means for extracting text information from a photographed food label image using optical character recognition, means for comparing the extracted text information with a food ingredient database and analyzing the ingredient information using a generation AI to evaluate risks associated with the user's health condition, means for notifying the user of the evaluated risk information as an alert, means for analyzing the user's voice input using voice recognition and providing appropriate answers and advice using a generation AI, means for analyzing the user's emotional state and customizing the advice to be provided depending on the emotional state, and means for selecting a method of expressing the risk alert appropriate to the emotional state. This enables the user to receive personalized advice that takes into account not only their health condition but also their emotional state.

[1958] "User" refers to an individual who uses the system and is the target of assistance in managing health conditions and food choices.

[1959] "Personalized health advice" refers to health guidance and recommendations tailored to a particular user based on user input.

[1960] Optical character recognition is a technology that extracts text information from images and photographs, and is used to incorporate food label information into the system.

[1961] The "Food Composition Database" is a database that collects information on the composition of various foods and provides basic data for composition analysis and risk assessment.

[1962] "Generative AI" refers to artificial intelligence technology that performs natural language processing and information generation based on large amounts of data, and is used to generate health advice and answers for users.

[1963] "Risk alert" refers to warning information that notifies users of health risks associated with food choices.

[1964] "Speech recognition" is a technology that converts a user's voice input into text data and is used to process spoken queries and commands.

[1965] "Emotional state" refers to the user's psychological and emotional state, and represents emotions such as stress or joy that can be inferred from voice and text input.

[1966] "Personalization" refers to providing optimized information and services that take into account the characteristics of each individual user.

[1967] "Food label" refers to the text and images containing ingredient and nutritional information that appear on food packaging.

[1968] MODE FOR CARRYING OUT THE INVENTION

[1969] The present invention provides a system that allows users to easily select healthy foods and provides personalized advice taking into account the user's emotional state. Specific embodiments of the system are described below.

[1970] User registration and initial settings

[1971] First, the user installs and launches the application on their device. When launching the application for the first time, the user enters basic information such as their name, age, gender, medical history, and dietary restrictions. The device then sends this information to the server. The server then creates a user profile based on the received information and creates initial health advice tailored to the user. For example, a user who needs to limit their fat intake may be provided with advice recommending low-fat foods.

[1972] Food Identification

[1973] The user uses the application to take a photo of the label of the food they plan to purchase with their camera. The device then passes the image to an optical character recognition (OCR) module, which extracts text information from the image. Specifically, the device uses an OCR service such as Google Cloud Vision API. The extracted text information is then sent from the device to a server.

[1974] Ingredient analysis and risk assessment

[1975] The server compares the received text information with a food ingredient database (e.g., USDA Nutrient Database) to obtain detailed information about each ingredient. Next, it uses generative AI (e.g., OpenAI's GPT-4) to analyze the ingredient information and evaluate the risk associated with the user's health. An example of a specific prompt is, "How does this ingredient affect high blood pressure?" The evaluated risk information is generated as an alert and sent to the device.

[1976] Viewing Alerts and Information

[1977] The device displays the alert information and detailed data received from the server on the app screen. For example, the app screen displays the risk assessment results using text and icons, visually communicating the risk to the user. The user can check the displayed information and ask additional questions by voice if necessary.

[1978] Voice recognition and additional information provision

[1979] If the user needs additional information, they can ask a question by voice. For example, they might ask, "What should I eat instead of this food?" The device uses a voice recognition function (e.g., Google Speech-to-Text) to convert the user's voice into text and send it to the server. The server then uses a generative AI to generate the optimal answer based on the question and sends it to the device. The device then plays the received answer aloud and also displays it as text.

[1980] Utilizing the Emotion Engine

[1981] An emotion engine (e.g., Microsoft Azure Emotion API) recognizes the user's emotional state from their voice or text input. Specifically, it determines whether the user is "feeling stressed" based on the user's tone of voice and the content of the text. The server then customizes the advice it provides based on the recognized emotional state. For example, a user who is feeling stressed could be offered advice about foods and lifestyle habits that have a relaxing effect. At the same time, the server selects a risk alert presentation style appropriate to the user's emotional state and sends it to the device.

[1982] Data accumulation and feedback

[1983] The server stores the user's food selections, advice received, and perceived emotional state in a database. This accumulated data is used to improve the accuracy of future advice and risk assessments. Specifically, the data is analyzed using data analytics tools (e.g., Python's Pandas and Scikit-learn), and the algorithm is improved based on the analysis results.

[1984] In this way, the present invention provides a system that allows users to select appropriate foods taking into account their health and emotional state, thereby providing more personalized health support and helping users lead a healthier life with ease.

[1985] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1986] Step 1:

[1987] Users install and launch the application on their device, then enter basic information such as their name, age, gender, medical history, and dietary restrictions.

[1988] Input: User's basic information (name, age, gender, medical history, dietary restrictions, etc.)

[1989] Specific action: The user enters information into an input form within the app and presses the "Submit" button.

[1990] Output: Request data with basic information entered

[1991] Step 2:

[1992] The terminal transmits the input basic information to the server.

[1993] Input: Request data (basic user information)

[1994] Specific operation: The terminal generates an HTTP request and sends data to the server.

[1995] Output: Basic information sent to the server

[1996] Step 3:

[1997] The server generates a user profile based on the received basic information, which also includes initial health advice for the user.

[1998] Input: User basic information

[1999] Specific operations: The server stores the information in a database and runs the profile generation algorithm.

[2000] Output: Generated user profile and initial advice

[2001] Step 4:

[2002] The user uses the application to take a photo of the label of the food item they plan to purchase with their camera.

[2003] Input: Food label image

[2004] Specific operation: The user activates the camera function and takes a picture of the food label.

[2005] Output: Photographed food label image

[2006] Step 5:

[2007] The device passes the captured food label image to an OCR module to extract text information.

[2008] Input: Food label image

[2009] Specific operation: Send the image to an OCR module (e.g., Google Cloud Vision API) and extract text information.

[2010] Output: Extracted text data

[2011] Step 6:

[2012] The terminal transmits the extracted text data to the server.

[2013] Input: Extracted text data

[2014] Specific operation: The terminal generates an HTTP request and sends data to the server.

[2015] Output: Text data sent to the server

[2016] Step 7:

[2017] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient.

[2018] Input: Text data

[2019] Specific operation: The server queries the food ingredient database and retrieves ingredient information.

[2020] Output: Detailed information for each component

[2021] Step 8:

[2022] The server uses generative AI to analyze the ingredient information and assess the risks associated with the user's health condition.

[2023] Input: Ingredient information, user profile

[2024] Specific behavior: Send an analytical prompt to an AI model (e.g., OpenAI's GPT-4) to perform risk assessment. Specific prompt example: "How does this ingredient affect high blood pressure?"

[2025] Output: Risk assessment results

[2026] Step 9:

[2027] The server generates the evaluated risk information as an alert and transmits it to the terminal for notification to the user.

[2028] Input: Risk assessment results

[2029] Specific operation: An alert message is generated and sent to the terminal as an HTTP response.

[2030] Output: Generated risk alerts

[2031] Step 10:

[2032] The device displays the alert information and detailed data received from the server on the app screen.

[2033] Enter: Risk Alert

[2034] Specific behavior: Parse the risk alert and display it in a UI component (e.g., a text view or an icon).

[2035] Output: Risk alert displayed on the app screen

[2036] Step 11:

[2037] The user reviews the displayed information and verbally asks additional questions if necessary.

[2038] Input: Risk assessment results displayed on the terminal

[2039] Specific operation: The user presses the microphone button and begins voice input.

[2040] Output: User's voice data

[2041] Step 12:

[2042] The terminal uses a voice recognition function to convert the user's voice into text and send it to the server.

[2043] Input: User's voice data

[2044] Specific operation: Uses a speech recognition API (e.g., Google Speech-to-Text) to convert voice data into text, and sends the converted text to the server.

[2045] Output: Textual question data

[2046] Step 13:

[2047] The server uses generative AI to generate the best answer based on the question and sends it to the device.

[2048] Input: Textual question data

[2049] Specific operation: Send a prompt (e.g., "What should I eat instead of this food?") to the generating AI and get an answer.

[2050] Output: Generated response data

[2051] Step 14:

[2052] The device will play the received answer aloud and also display it as text.

[2053] Input: Generated response data

[2054] Specific behavior: Converts text to speech using a speech synthesis API (e.g., Google Text-to-Speech), plays the converted speech, and displays the text in the UI.

[2055] Output: Answer information as audio and text

[2056] Step 15:

[2057] The emotion engine recognizes the user's emotional state from their voice and text input.

[2058] Input: User's voice data, text data

[2059] Specific behavior: Uses emotion recognition APIs (e.g. Microsoft Emotion API) to infer emotions from speech and text.

[2060] Output: Perceived emotional state

[2061] Step 16:

[2062] The server customizes the advice it provides depending on the perceived emotional state.

[2063] Input: perceived emotional state, user profile

[2064] Specific behavior: Send a prompt (e.g., "What should you recommend to a user who is feeling stressed?") to the generative AI to get customized advice.

[2065] Output: Customized advice

[2066] Step 17:

[2067] The server selects a method of expressing a risk alert that is appropriate for the emotional state and notifies the terminal.

[2068] Input: Customized advice, risk assessment results

[2069] Specific behavior: Adjust the tone and format of the text depending on the emotional state and send it to the device in an HTTP response.

[2070] Output: Adjusted risk alerts

[2071] Step 18:

[2072] The server stores the user's food selection data, received advice, and recognized emotional state in a database.

[2073] Input: User's food data, advice, perceived emotional state

[2074] Specific operation: Executes a data insertion query against the database.

[2075] Output: Information stored in the database

[2076] Step 19:

[2077] The server analyzes the accumulated data and uses it to improve the accuracy of future advice and risk assessments.

[2078] Input: Accumulated data

[2079] Specific Tasks: Analyze data and improve algorithms using data analytics tools (e.g., Python's Pandas, Scikit-learn).

[2080] Output: Improved algorithm

[2081] (Application example 2)

[2082] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2083] Many consumers today are becoming increasingly health-conscious and understand the importance of choosing foods that are right for them. However, it is difficult to instantly evaluate food ingredients and health risks while shopping in a physical store, and to receive optimal advice based on one's emotional state at the time. Especially in today's world where a wide variety of foods are available, obtaining detailed information about each food item takes time and effort. Furthermore, existing systems are unable to provide advice that takes into account the user's emotional state, making it difficult to provide accurate support tailored to each user's individual situation.

[2084] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for generating personalized health advice based on information entered by a user, means for extracting text information from a photographed food label image using optical character recognition, means for comparing the extracted text information with a food ingredient database, analyzing the ingredient information using a generation AI, and evaluating risks associated with the user's health status, means for notifying the user of the evaluated risk information as an alert, means for analyzing the user's voice input using voice recognition and providing appropriate answers and advice using a generation AI, means for analyzing the user's emotional state and personalizing advice according to the emotional state, and means for providing recommended products and purchase plans suited to the user's health status and emotional status based on data from a physical store. This allows users to easily select foods suitable for them in a physical store and receive personalized advice tailored to their health status and emotional state.

[2085] "Means for generating personalized health advice based on user-entered information" refers to a method for collecting information entered by a user into the application, such as name, age, gender, medical history, and dietary restrictions, and generating and providing specific health advice based on that information.

[2086] "Means for extracting text information from photographed food label images using optical character recognition" refers to a method in which a user takes a photo of a food label using a camera on a smartphone or other device, and extracts text information from the image using optical character recognition technology.

[2087] "Means of comparing extracted text information with a food ingredient database, analyzing ingredient information using generative AI, and assessing risks related to the user's health" refers to a method of comparing text information obtained by optical character recognition with a food ingredient database, and then using a generative AI model to perform a detailed analysis of the ingredients and assess the user's health risks.

[2088] "Means for notifying the user of the evaluated risk information as an alert" refers to a method for evaluating the risk to the user's health based on the analyzed component information and notifying the user of the results as a warning.

[2089] "Means of analyzing a user's voice input through speech recognition and using generative AI to provide appropriate answers and advice" refers to a method of converting a user's spoken questions or requests into text using speech recognition technology, analyzing it using a generative AI model, and generating appropriate answers and advice.

[2090] "Means for analyzing the user's emotional state and personalizing advice according to that emotional state" refers to a method for analyzing emotional information obtained from the user's voice input or text input and providing customized advice according to that emotion.

[2091] "Means for providing recommended products and purchase plans suited to the user's health and emotional state based on data from physical stores" refers to a method that utilizes user behavior data and selected product data obtained from physical stores to suggest products and purchase plans that are optimal for the user's health and emotional state.

[2092] "Optical character recognition" is a technology that mechanically reads characters printed on paper or text contained in images and extracts it as digital data.

[2093] "Generative AI" is an artificial intelligence technology that generates new information and answers based on pre-trained data.

[2094] The present invention provides a system that allows users to easily select foods suitable for them in a physical store and receive personalized advice tailored to their health and emotional state. Specific embodiments of this system are described below.

[2095] User registration and initial settings

[2096] Users enter basic information such as name, age, gender, medical history, and dietary restrictions through an application installed on their smartphone. The device then sends this information to a server, which then generates a user profile based on the information received and prepares health advice. This profile is then used for future analysis and advice provision.

[2097] Food Identification

[2098] A user takes a photo of the label of a food item they are considering purchasing in a physical store using their smartphone camera. The device passes the image to an optical character recognition (OCR) module, which extracts text information from the image. The extracted text data is then sent to a server.

[2099] Ingredient analysis and risk assessment

[2100] The server compares the received text data with a food ingredient database to obtain detailed information about each ingredient. It then uses a generative AI model to analyze the ingredients and assess the risk associated with the user's health. For example, if the analysis reveals that a user with high blood pressure chooses a high-salt food, a risk assessment will be made, such as, "This food contains high salt and may have a negative impact on high blood pressure." The assessed risk information is generated as an alert and sent to the device to notify the user.

[2101] Viewing Alerts and Information

[2102] The device displays the alert information and detailed data received from the server on the app screen. For example, the risk assessment results can be displayed as text and icons to visually inform the user of the risk. If the user asks additional questions by voice, the voice is converted into text by a voice recognition module and sent to the server.

[2103] Voice recognition and additional information provision

[2104] The server uses a generative AI model to generate an optimal answer based on the content of the voice question and sends it to the device, which then plays the received answer back aloud and can also display it as text, allowing the user to obtain information both visually and audibly.

[2105] Utilizing the Emotion Engine

[2106] The emotion engine recognizes emotions from the user's voice and text input. For example, the emotion engine may determine that the user is "feeling stressed" based on the user's tone of voice and the content of the text. The server then customizes the advice it provides based on the recognized emotional state. Specifically, for a user who is feeling stressed, it provides advice on foods that have a relaxing effect and lifestyle habits to reduce stress. For users with unstable emotions, it displays risk alerts in a gentler tone.

[2107] Data accumulation and feedback

[2108] The server stores the user's food selections, received advice, and perceived emotional state in a database, which is used to improve the accuracy of future advice and risk assessments.

[2109] This allows users to easily choose the foods that are right for them in physical stores and receive personalized advice tailored to their health and emotional state.

[2110] Prompt Sentence Examples

[2111] Below are some examples of prompt sentences.

[2112] Food ingredient analysis prompt:

[2113] You are given the following list of food ingredients. Analyze the risks these ingredients pose to a user with high blood pressure and perform a risk assessment.

[2114] Ingredients: Salt

[2115] Ingredients: Sodium 1500mg

[2116] Emotion recognition prompts:

[2117] Recognize emotions from the following user voice inputs:

[2118] Text: "I'm feeling super stressed today. Is there a good way to relax?"

[2119] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2120] Step 1:

[2121] Users install the application on their smartphone and enter basic information such as their name, age, gender, medical history, and dietary restrictions. The application then collects the initial data to create a user profile. The entered information is sent from the device to a server. The server receives this information, generates a user profile, and stores it as the basis for health advice.

[2122] Step 2:

[2123] Users take a photo of the label of a food item...

Claims

1. means for generating personalized health advice based on information entered by the user; A method for extracting text information from photographed food label images using OCR (Optical Character Recognition), A means for comparing the extracted text information with a food ingredient database and analyzing the ingredient information using a generating AI to assess the risk associated with the user's health condition; a means for notifying a user of the evaluated risk information as an alert; A means of analyzing a user's voice input through speech recognition and using generative AI to provide appropriate answers and advice A system including:

2. The system according to claim 1, further comprising means for storing data on food items selected by the user and advice received in a database and utilizing the data to improve the accuracy of personalization in the future.

3. 10. The system of claim 1, wherein the information communicated as an alert includes a detailed description of the ingredients and their health risks and is provided to the user visually and audibly.

Images