System

Abstract

A system is provided.SOLUTION: A system comprising: means for a user to input his / her own allergy information; means for photographing a food package; means for transmitting an image to a server; means for the server to analyze the image and extract an ingredient list of the food; means for collating the extracted ingredient list with an allergy database and determining the presence or absence of an allergic component; means for notifying the user of a determination result; and means for accumulating the determination result in the database.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] People with food allergies must always carefully check the ingredients and components of the food they purchase. However, commercially available foods contain a wide variety of ingredients, making analyzing and verifying them extremely time-consuming and difficult. Checking the small print and complex ingredient lists on food packages one by one is particularly time-consuming and labor-intensive. Furthermore, when the information on food packages is written in a different language, verification becomes even more difficult. Given this background, there is a need for a simple and quick method that allows people with food allergies to purchase food and enjoy their meals with peace of mind. [Means for solving the problem]

[0005] To address this issue, we developed a system that provides the following: a means for users to input their own allergy information, a means for photographing food packaging, a means for sending the image to a server, a means for the server to analyze the image and extract a list of the food's ingredients, a means for the server to compare the extracted ingredient list with an allergy database to determine whether or not an allergen is present, a means for notifying the user of the determination result, and a means for storing the determination result in the database. The system also includes a means for displaying a warning message to the user if an allergen is present, and a means for the server to use optical character recognition (OCR) technology to extract text from the image, enabling the system to quickly determine whether or not an ingredient to which the user has an allergy is present and efficiently notify the user of the result. This will enable people with food allergies to safely purchase foods and enjoy their daily meals.

[0006] "User" refers to a person who uses this system to check the allergens in food.

[0007] "Allergy information" refers to information about ingredients that a user is allergic to, which is entered and saved in the system.

[0008] "Food packaging" refers to the label, bag, etc. containing information printed on the packaging of food.

[0009] "Photographing" refers to the act of taking an image of an object using a smartphone or camera.

[0010] The term "server" refers to a computer that performs the central data processing of the system, analyzing received images, comparing them with a database, and generating judgment results.

[0011] "Sending an image" refers to the act of transferring image data captured by a terminal to a server via a network.

[0012] "Analyzing an image" refers to the process of extracting text information from image data received by the server using optical character recognition (OCR) technology.

[0013] "Ingredient list" refers to the list of ingredients or materials contained in a food product that is printed on the food package.

[0014] "Database" refers to a data storage system for storing and making available allergy information and food ingredient information.

[0015] "Matching" refers to the process of comparing the extracted ingredient list with the stored allergy information to see if there are any matching ingredients.

[0016] "Determine" refers to the process of determining whether or not an allergen is present based on the collation results.

[0017] "Notifying" refers to the act of conveying the judgment result from the server to the user, which is mainly done through an application.

[0018] The "warning message" refers to alert information that notifies the user that an allergic ingredient is included.

[0019] "Optical character recognition (OCR) technology" refers to technology that automatically reads character information from image data.

[0020] "Storing" refers to the act of saving the results of the determination in a database and making them available for future reference. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] The present invention is a system that enables users with food allergies to safely purchase food and enjoy their meals, and is realized through a series of processes: registering allergy information, photographing food packaging, analyzing the images, extracting and comparing ingredient lists, notifying the user of the results, and storing the data. Specific embodiments of the system are described below.

[0043] Registering allergy information

[0044] User: Launches the app and opens the user profile settings screen. The user enters their allergy information (e.g., "wheat," "peanuts," "dairy," etc.). This information can be added, deleted, or edited at any time from the application's settings menu.

[0045] Terminal: The entered allergy information is saved in a local database and the data is prepared for transmission to the server. The following is the process for registering the user's allergy information.

[0046] Server: Receives allergy information sent from the device and stores it in a user-specific allergy database. This data is later used to match food ingredients against a food ingredient list.

[0047] Food packaging photography

[0048] User: Take a photo of the packaging of the food item you are considering purchasing using your smartphone camera. When taking the photo, make sure to adjust it so that the entire package is clearly visible.

[0049] Terminal: Temporarily stores the captured image data and prepares a request to send it to the server. After the user presses the capture button, the image is immediately uploaded to the server.

[0050] Parsing package information

[0051] Server: The server analyzes the received image data. As a first step in the analysis, it uses optical character recognition (OCR) technology to extract text from the image. The extracted text is then processed into a list of raw materials. During this process, preprocessing is performed to remove noise and improve character recognition accuracy.

[0052] Server: Normalizes the extracted ingredient list to match a standard format in the database, and translates it if necessary based on specific language codes.

[0053] Database Matching

[0054] Server: Compares the normalized ingredient list with the user's allergy database. Checks whether the ingredients match the allergy information in the database. If a match is found, identifies the ingredient and passes the result on to further processes.

[0055] Notification of the results

[0056] Server: Generates a warning message if allergens are identified. The result includes a list of identified allergens.

[0057] Server: Sends the result of the judgment to the user's device. A notification message is generated so that the user can view it.

[0058] Device: Receives the sent judgment result and displays it to the user as an alert or pop-up message within the app, allowing the user to make safe food choices.

[0059] Data accumulation

[0060] Server: Each judgment result is stored in a database. This allows the system to provide quick results based on existing data the next time the same food is analyzed. The stored data is used to improve the efficiency of the entire system.

[0061] Specific examples

[0062] A case where a user checks a package of "chocolate" will be described.

[0063] 1. User: Registers in the app that they are allergic to wheat, peanuts, and dairy products.

[0064] 2. Device: Stores allergy information locally and sends it to the server.

[0065] 3. Server: Stores allergy information in a database.

[0066] 4. User: Takes a photo of the chocolate packaging with their smartphone.

[0067] 5. Terminal: Sends image data to the server.

[0068] 6. Server: Performs image analysis and extracts text information, such as "cocoa, sugar, milk components, lecithin (derived from soybeans)."

[0069] 7. Server: Checks the ingredients list and identifies that it contains the user's allergic ingredient, "dairy."

[0070] 8. Server: Sends the judgment result to the terminal and generates a warning message.

[0071] 9. Device: Display a warning to the user that "This product contains milk ingredients."

[0072] 10. Server: Stores the judgment results in a database.

[0073] Thus, the present invention provides fast and efficient assistance to users with food allergies in making safe food choices.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] User: Launch the app and open the "Register Allergy Information" screen from the settings menu. The user selects and enters the ingredients they are allergic to (e.g., "wheat," "peanuts," "dairy products") from a list.

[0077] Step 2:

[0078] Terminal: Stores the entered allergy information in a local database, prepares the data for synchronization with the server, and creates a transmission request.

[0079] Step 3:

[0080] Server: Receives allergy information sent from the device. Stores the received data in a user-specific allergy database. At this stage, the data is formatted for matching.

[0081] Step 4:

[0082] User: Using their smartphone camera, they take a picture of the packaging of a food item they are considering purchasing. They frame the photo so that the entire label is visible.

[0083] Step 5:

[0084] Terminal: Temporarily stores the captured image and prepares a request to send it to the server. When the user presses the send button, the image data is uploaded to the server.

[0085] Step 6:

[0086] Server: Receives image data sent from the device. The received data is input into an optical character recognition (OCR) engine for image analysis. During this process, text information is extracted from the image.

[0087] Step 7:

[0088] Server: Preprocesses the text data extracted by the OCR engine, including noise removal and text normalization. For example, ingredients such as "sugar" and "dairy" are listed.

[0089] Step 8:

[0090] Server: Compares the preprocessed text data with the allergy information stored in the database, and verifies that the allergens registered by the user in advance match the analyzed ingredient list.

[0091] Step 9:

[0092] Server: Makes a decision based on the match. If a matching allergen is found, identifies the allergen and adds it to the list. Generates a warning message and prepares a notification to the user.

[0093] Step 10:

[0094] Server: Sends the judgment result to the user's device, which includes the identified allergens and a warning message.

[0095] Step 11:

[0096] Device: Notify the user of the received test result by displaying a message such as "This product contains milk ingredients" in a pop-up or notification bar within the app.

[0097] Step 12:

[0098] Server: Each judgment result is stored in a database. This allows for a quick response the next time the same product is analyzed. The data is saved for future reference and analysis, contributing to the efficiency of the entire system.

[0099] The above are the specific processing steps and operations of the proposed system.

[0100] Example 1

[0101] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0102] For consumers with food allergies, the process of checking food safety when making everyday food purchases is extremely cumbersome. Manually checking ingredient lists is time-consuming and laborious, and there is a risk of overlooking or misreading information. Furthermore, in many markets, product information is displayed in different languages, which can be difficult to understand. Therefore, there is a need for a system that allows users with food allergies to efficiently and accurately check food safety and purchase and consume with peace of mind.

[0103] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0104] In this invention, the server includes means for a user to input their own allergy information, means for photographing a food package, means for transmitting the image to the server, means for the server to analyze the image and extract a list of ingredients for the food, means for comparing the extracted ingredient list with an allergy database to determine whether or not an allergic ingredient is present, means for notifying the user of the determination result, means for storing the determination result in a database, means for extracting text from the image using optical character recognition technology, means for temporarily saving image data photographed by the user, means for preparing data for the user to transmit their allergy information to the server, means for saving the user's allergy information, and means for displaying the determination result within the app. This allows users to efficiently and accurately confirm the safety of foods and purchase or consume them with peace of mind.

[0105] "Users" are consumers who have food allergies and want to make safe food choices.

[0106] "Allergy information" is a list of ingredients that cause allergic reactions in specific foods, and is entered by the user.

[0107] "Food packaging" refers to the packaging or label that lists the product information and ingredients of a food product.

[0108] "Server" means a central computer system that receives and processes data submitted by users.

[0109] "Image" refers to photographic data of food packaging taken with a smartphone or other device.

[0110] "Optical character recognition (OCR) technology" is a technology for recognizing and extracting text data from images.

[0111] A "database" is a system for structuring and storing data such as user allergy information and food ingredient lists.

[0112] An "ingredient list" is a list of the ingredients and materials contained in a food product, and is text data extracted from the packaging.

[0113] "Notification" refers to a message or warning that the server sends to the user to inform them of the judgment result.

[0114] An "alert" is a warning message that is displayed to prompt the user to take a specific action.

[0115] A "pop-up message" is a message that appears temporarily on the screen to attract the user's attention.

[0116] The "determination result" refers to the result of determining whether or not an allergy ingredient is present based on a comparison between the extracted ingredient list and the user's allergy database.

[0117] The "user profile setting screen" is the screen within the app where users can enter, edit, and save their personal information and allergy information.

[0118] The present invention provides a system that enables users with food allergies to safely purchase food and enjoy meals. This system executes a series of processes using the following hardware and software:

[0119] Hardware

[0120] Device: A mobile information device such as a smartphone or tablet.

[0121] Server: A centralized computer system.

[0122] software

[0123] Local database: Manage user allergy information stored on the device using a lightweight database such as SQLite.

[0124] Optical Character Recognition (OCR) technology: Using libraries such as OpenCV and Tesseract, the server extracts text from images.

[0125] Translation API: Using the Google Cloud Translation API or similar, translate the ingredient list into a user-understandable language as needed.

[0126] Notification system: The result of the judgment is notified to the user using GCM (Google Cloud Messaging) or APNs (Apple Push Notification service).

[0127] Operating Procedures and Processes

[0128] 1. Allergy Information Registration:

[0129] User: Launch the app and open the user profile settings screen. Enter your allergy information (e.g., "wheat," "peanuts," "dairy products," etc.).

[0130] Terminal: Stores the entered allergy information in a local database and prepares the data for transmission to the server.

[0131] Server: Receives allergy information sent from the device and stores it in an allergy database for each user.

[0132] 2. Food packaging photography:

[0133] User: Take a photo of the packaging of a food item they are considering purchasing with their smartphone camera, adjusting the camera so that the entire package is clearly visible.

[0134] Terminal: Temporarily stores the captured image data and prepares a request to send it to the server.

[0135] 3. Parse package information:

[0136] Server: Analyzes the received image data and extracts the text within the image using optical character recognition (OCR) technology.

[0137] Specific examples

[0138] A case where a user checks a package of "chocolate" will be described.

[0139] 1. User: Registers in the app that they are allergic to wheat, peanuts, and dairy products.

[0140] 2. Device: Stores allergy information locally and sends it to the server.

[0141] 3. Server: Stores allergy information in a database.

[0142] 4. User: Takes a photo of the chocolate packaging with their smartphone.

[0143] 5. Terminal: Sends image data to the server.

[0144] 6. Server: Performs image analysis and extracts text information. For example, "cocoa, sugar, milk components, lecithin (derived from soybeans)" is extracted.

[0145] 7. Server: Checks the ingredients list and identifies that it contains the user's allergic ingredient, "dairy."

[0146] 8. Server: Sends the judgment result to the terminal and generates a warning message.

[0147] 9. Device: Display a warning to the user that "This product contains milk ingredients."

[0148] Examples of prompt statements

[0149] "Open your user profile settings screen and enter your allergy information."

[0150] "Take a photo of the packaging of the food you're considering purchasing."

[0151] "The captured image data is being sent to the server. Please wait."

[0152] "The ingredients list has been analyzed. We are currently reviewing the results."

[0153] "Avoid this food because it contains dairy."

[0154] This system provides support to users with food allergies by efficiently and accurately checking food safety and enabling them to select foods with peace of mind.

[0155] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0156] Step 1:

[0157] User: Launches the app and opens the user profile setting screen. The user enters their allergy information (e.g., "wheat," "peanuts," "dairy products," etc.). This is the input data.

[0158] Terminal: The entered allergy information is saved in a local database (e.g., SQLite). Data calculations at this stage involve converting the information entered by the user into a database format and saving it. Data is also prepared for sending to the server. Specific operations include writing to the database and converting the data format for sending to the server. The output is the allergy information saved locally and the data to be sent to the server.

[0159] Server: Receives allergy information sent from the device. Stores the received data in an allergy database for each user. Specifically, the data is written to the allergy database (e.g., MySQL or PostgreSQL). This is the output data.

[0160] Step 2:

[0161] User: Take a photo of the packaging of a food item they are considering purchasing with their smartphone camera. When taking the photo, they need to adjust the camera so that the entire package is clearly visible. This is the input data.

[0162] Terminal: Temporarily stores the captured image data. Data calculation at this stage involves converting the image data into an appropriate format (e.g., JPEG or PNG) and saving it. After the user presses the capture button, a request is immediately prepared to upload the image to the server. The specific operation is to convert the image data for network transmission. The output is image data ready for transmission to the server.

[0163] Step 3:

[0164] Server: The server receives the image data sent from the terminal. It analyzes the received image data and extracts the text within the image using optical character recognition (OCR) technology (e.g., Tesseract). This is the input data. Data processing at this stage involves converting the image data into text data, including noise removal and image preprocessing. The output is the extracted text data of the ingredient list.

[0165] Step 4:

[0166] Server: The extracted text data is normalized and translated into a specific language using a translation API (e.g., Google Cloud Translation API) if necessary to conform to a standard format. This is the input data. Data operations at this stage involve cleaning and normalizing the text data, and translating it if necessary. The output is the normalized and translated text data of the ingredient list.

[0167] Step 5:

[0168] Server: Compares the normalized ingredient list with the user's allergy database. The specific operation is to compare the allergy information in the database with the ingredient list. This is the input data. The data operation at this stage is to identify matching ingredients using a comparison algorithm. The output is a judgment result on the presence or absence of allergens.

[0169] Step 6:

[0170] Server: If an allergen is identified, a warning message is generated. This is the input data. Data calculation at this stage generates a warning message based on the judgment result. The output is a warning message and the judgment result.

[0171] Server: Sends the result of the judgment to the user's device. Specifically, it sends a message in real time via a notification system such as GCM or APNs. This is the input data. The output is the sent notification message.

[0172] Device: Receives the sent judgment result and displays it to the user as an alert or pop-up message within the app. The specific behavior is to display a notification on the device's user interface. This is the input data, and the output is the displayed warning message.

[0173] Step 7:

[0174] Server: Each decision result is stored in a database. This is the input data. Data calculations at this stage are to save the decision results in a database (e.g. MySQL or PostgreSQL). The output is the updated database.

[0175] The above are the specific processing steps in the system of the present invention, which allow users with food allergies to select foods efficiently and safely.

[0176] (Application example 1)

[0177] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0178] When consumers with food allergies purchase food at physical stores, they need a method to quickly and accurately identify allergens and safely select foods. Conventional methods often require complicated and time-consuming food safety checks, making it difficult to ensure consumer peace of mind.

[0179] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0180] In this invention, the server includes means for a user to input their own allergy information, means for photographing the food package, means for transmitting the image to the server, means for the server to analyze the image and extract a list of the food's ingredients, means for comparing the extracted ingredient list with an allergy database to determine whether or not an allergen is present, means for notifying the user of the determination result, means for storing the determination result in the database, and means for displaying the allergen determination result in real time using a smart device at a physical store. This allows users to instantly check whether or not a food product contains an allergen by scanning the food package at the physical store, enabling them to select safe foods.

[0181] "User" refers to an individual who uses the system and is the entity that inputs allergy information and photographs food packaging.

[0182] "Allergy information" refers to information that indicates the specific ingredient names and food names for which the user has a food allergy.

[0183] "Food packaging" refers to the packaging in which food is packaged, and is the target from which ingredient information written inside is extracted.

[0184] "Smart devices" refer to electronic devices that can be carried by users, such as smartphones, smart glasses, and head-mounted displays.

[0185] "Server" refers to a computer system that processes information for the entire system, and performs image analysis and database matching.

[0186] "Image analysis" is the process of processing images of food packaging to extract useful textual information from them.

[0187] "Optical Character Recognition (OCR) technology" is a technology that recognizes characters from images or printed text as electronic data.

[0188] "Allergy database" refers to a database that stores the user's allergy information and is used for matching purposes.

[0189] The "determination result" is a conclusion as to whether or not an allergen is present, obtained by comparing the extracted ingredient list with the allergy database.

[0190] "Notification" refers to the process of communicating the results of a determination to the user, which is done through a warning message or a pop-up message.

[0191] MODE FOR CARRYING OUT THE INVENTION

[0192] This invention is a system that allows users with food allergies to safely select food in physical stores. How this system is implemented will be described below in detail.

[0193] System Program Overview

[0194] The server executes a series of processes, including setting up a user profile, photographing the food packaging, analyzing the image, extracting ingredients, comparing it with a database, notifying the user of the results, and storing the results, allowing users to quickly determine the safety of food in physical stores.

[0195] Hardware / Software used

[0196] Hardware:

[0197] Smartphone (Android or iOS)

[0198] server

[0199] software:

[0200] For server-side processing, the Flask framework is used.

[0201] For image analysis, pytesseract is used as an optical character recognition (OCR) technology.

[0202] For image processing, we use PIL, a Python image processing library.

[0203] SQLite is used as the database.

[0204] Data processing and calculation

[0205] User:

[0206] 1. The user launches the app on their smartphone, enters their allergy information (e.g., "wheat," "peanuts," "dairy products," etc.), and registers it.

[0207] 2. The registered allergy information is stored in the smartphone's local database and also sent to the server.

[0208] server:

[0209] 1. The server receives allergy information sent by the user and stores it in a dedicated allergy database.

[0210] 2. When a user takes a photo of a food package in a physical store with their smartphone camera, the image data is sent to the server.

[0211] 3. The server analyzes the received image data using optical character recognition (OCR) technology to extract the ingredient list from the package. This analysis involves preprocessing to remove noise and improve character recognition accuracy.

[0212] 4. The extracted ingredient list is normalized to a standard format and compared with the user's allergy database to determine whether or not an allergen is present.

[0213] Notification of decision:

[0214] 1. The server generates a warning message if an allergen is found. The result includes a list of identified allergens.

[0215] 2. The judgment result is sent to the user's smartphone, and a notification message is displayed so that the user can view it immediately.

[0216] Data accumulation:

[0217] 1. The results of the assessment are stored in a server database. This data is used to speed up future analysis processes.

[0218] Specific examples

[0219] A case will be described where the user wishes to check the packaging of "Chocolate".

[0220] 1. The user registers in a smartphone app that they are allergic to wheat, peanuts, and dairy products.

[0221] 2. The smartphone stores the allergy information locally and sends it to the server.

[0222] 3. The server stores this information in a dedicated database.

[0223] 4. The user takes a photo of the chocolate packaging with their smartphone in a physical store.

[0224] 5. The captured image data is sent to the server.

[0225] 6. The server analyzes the image and extracts text information, such as "cocoa, sugar, milk components, lecithin (derived from soybeans)."

[0226] 7. The extracted ingredient list is compared and it is determined that the product contains the user's allergic ingredient, "dairy products."

[0227] 8. The server sends the result of the judgment to the smartphone and generates a warning message.

[0228] 9. Display a "This product contains dairy" warning on your smartphone.

[0229] Prompt Sentence Examples

[0230] "Create an application that, when a user scans a food package in a physical store with their smartphone camera, analyzes the ingredients contained in the package in real time and compares them with the allergy information registered by the user to detect allergens."

[0231] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0232] Step 1:

[0233] The user launches the app on their smartphone and enters their allergy information (e.g., "wheat," "peanuts," "dairy products," etc.). The entered allergy information is stored in a local database and is also prepared as data to be sent to the server.

[0234] Input: Allergy information entered by the user

[0235] Output: Allergy information stored in the local database and sent to the server

[0236] Step 2:

[0237] The terminal receives the allergy information entered by the user, stores it in a local database, and then transmits it to the server.

[0238] Input: Allergy information entered by the user

[0239] Output: Allergy information sent to the server

[0240] Step 3:

[0241] The server receives the allergy information sent from the terminal and stores the information in an allergy database.

[0242] Input: Allergy information sent from the device

[0243] Output: Allergy information stored in the allergy database

[0244] Step 4:

[0245] The user takes a photo of the packaging of a food item they are considering purchasing at a physical store using their smartphone camera, adjusting the camera so that the entire package is clearly visible.

[0246] Input: Food packaging you're considering buying

[0247] Output: Package image saved on your smartphone

[0248] Step 5:

[0249] The device temporarily stores the captured image data and prepares a request to send it to the server. When the user presses the capture button, the image is immediately uploaded to the server.

[0250] Input: Photographed package image

[0251] Output: Image data sent to the server

[0252] Step 6:

[0253] The server analyzes the received image data, extracts the text within the image using optical character recognition (OCR) technology, and processes it as an ingredient list.

[0254] Input: Received image data

[0255] Output: Extracted ingredient list

[0256] Step 7:

[0257] The server normalizes the extracted ingredient list, matches it with a standard format in an allergy database, and may translate it into specific languages ​​if necessary.

[0258] Input: Extracted ingredient list

[0259] Output: Normalized ingredient list

[0260] Step 8:

[0261] The server checks the normalized ingredient list against the user's allergy database to see if the product contains any allergens.

[0262] Input: Normalized ingredient list, allergy database

[0263] Output: Judgment result regarding the presence or absence of allergens

[0264] Step 9:

[0265] If an allergy ingredient is identified, the server generates a warning message and creates a message to notify the user based on the generated judgment result.

[0266] Input: Judgment result regarding the presence or absence of allergens

[0267] Output: Warning message

[0268] Step 10:

[0269] The device receives the judgment results sent from the server and displays them to the user as an alert within the app, allowing the user to select safe foods.

[0270] Input: Verification result sent from the server

[0271] Output: The alert message that is displayed to the user

[0272] Step 11:

[0273] The server stores each judgment result in a database, which allows it to quickly provide results based on existing data the next time the same food is analyzed.

[0274] Input: Judgment result

[0275] Output: Judgment results stored in the database

[0276] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0277] The present invention provides a new approach to a system that allows users with food allergies to safely purchase food and enjoy meals, taking into account the emotional state of the user. Specific embodiments of this system are described in detail below.

[0278] Registering allergy information

[0279] User: Launch the app and open the "Register Allergy Information" screen from the settings menu. The user selects and enters the ingredients they are allergic to (e.g., "wheat," "peanuts," "dairy products") from a list.

[0280] Terminal: Stores the entered allergy information in a local database, prepares the data for synchronization with the server, and creates a transmission request.

[0281] Server: Receives allergy information sent from the device and stores it in a user-specific allergy database. This data is later used to match food ingredients against a food ingredient list.

[0282] Food packaging photography

[0283] User: Using their smartphone camera, they take a picture of the packaging of a food item they are considering purchasing. They frame the photo so that the entire label is visible.

[0284] Terminal: Temporarily stores the captured image data and prepares a request to send it to the server. When the user presses the send button, the image data is uploaded to the server.

[0285] Parsing package information

[0286] Server: Receives image data sent from the device. The received data is input into an optical character recognition (OCR) engine for image analysis. During this process, text information is extracted from the image.

[0287] Server: Preprocesses the text data extracted by the OCR engine, including noise removal and text normalization. For example, ingredients such as "sugar" and "dairy" are listed.

[0288] Database Matching

[0289] Server: Compares the preprocessed text data with the allergy information stored in the database, and verifies that the allergens registered by the user in advance match the analyzed ingredient list.

[0290] Notification of judgment results and emotion engine

[0291] Server: Makes a decision based on the match. If a matching allergen is found, identifies the allergen and adds it to the list. Generates a warning message and prepares a notification to the user.

[0292] Device: Notify the user of the received test result by displaying a message such as "This product contains milk ingredients" in a pop-up or notification bar within the app.

[0293] Emotion engine: Collects user reactions through cameras and sensors and analyzes the user's emotional state. For example, it can read emotions from facial expressions and tone of voice after a user sees a warning message.

[0294] On the device: If the emotion engine analyzes the user's emotional state and determines they are anxious or stressed, the display of the warning message can be adjusted, for example, by making the message more friendly or by providing additional support information.

[0295] Data accumulation

[0296] Server: Each judgment result and the emotion engine's analysis results are stored in a database. This allows for a quick response the next time the same product is analyzed. Emotion data is also stored, and the server learns how to display the optimal warning message for each user.

[0297] Specific examples

[0298] Let us consider the case where a user wants to purchase "chocolate." The system operates as follows:

[0299] 1. User: Registers in the app that they are allergic to wheat, peanuts, and dairy products.

[0300] 2. Device: Stores allergy information locally and sends it to the server.

[0301] 3. Server: Stores allergy information in a database.

[0302] 4. User: Takes a photo of the chocolate packaging with their smartphone.

[0303] 5. Terminal: Sends image data to the server.

[0304] 6. Server: Performs image analysis and extracts text information, such as "cocoa, sugar, milk components, lecithin (derived from soybeans)."

[0305] 7. Server: Checks the ingredients list and identifies that it contains the user's allergic ingredient, "dairy."

[0306] 8. Server: Sends the judgment result to the terminal and generates a warning message.

[0307] 9. Device: Display a warning to the user that "This product contains milk ingredients."

[0308] 10. Emotion Engine: Detects the user's reaction to receiving a warning message and analyzes their emotional state.

[0309] 11. Device: Based on the results of the emotion engine, adjust the message display as needed (e.g., add a reassuring message).

[0310] 12. Server: Stores the judgment results and emotion data in a database.

[0311] In this way, the present invention provides fast and efficient assistance to users with food allergies in making safe food choices, while also providing more appropriate warnings that take into account the user's emotional state.

[0312] The processing flow will be explained below.

[0313] Step 1:

[0314] User: Start the app and open the "Register Allergy Information" screen from the settings menu. Select and enter the ingredients to which you are allergic (e.g., "wheat," "peanuts," "dairy products") from the list.

[0315] Step 2:

[0316] Terminal: Stores the entered allergy information in a local database, prepares the data for synchronization with the server, and creates a transmission request.

[0317] Step 3:

[0318] Server: Receives allergy information sent from the device and stores it in a user-specific allergy database. This data is later used to match food ingredients against a food ingredient list.

[0319] Step 4:

[0320] User: Uses the smartphone camera to take a picture of the packaging of a food item they are considering purchasing, adjusting the camera so that the entire package is visible within the frame.

[0321] Step 5:

[0322] Terminal: Temporarily stores the captured image data and prepares a request to send it to the server. When the user presses the send button, the image data is uploaded to the server.

[0323] Step 6:

[0324] Server: Receives image data sent from the device. The received image data is input into the OCR engine and text information within the image is extracted. During this process, preprocessing is performed to remove noise and improve character recognition accuracy.

[0325] Step 7:

[0326] Server: Preprocesses the extracted text data and normalizes it into an ingredient list, for example, listing ingredients such as "sugar" and "dairy."

[0327] Step 8:

[0328] Server: Compares the normalized ingredient list with the user's allergy database. Checks for a match against the user's allergy ingredients.

[0329] Step 9:

[0330] Server: Based on the matching result, if there is a matching allergy ingredient, identify the ingredient and add it to the list. Generate a warning message and prepare the content to notify the user.

[0331] Step 10:

[0332] Server: Sends the judgment result and warning message to the user's device. The judgment result includes the identified allergen and the warning message.

[0333] Step 11:

[0334] Device: Notify the user of the received test result by displaying a message such as "This product contains milk ingredients" in a pop-up or notification bar within the app.

[0335] Step 12:

[0336] User: Checks the warning message. Their facial expressions, tone of voice, and other reactions are collected by the emotion engine.

[0337] Step 13:

[0338] On the device: The emotion engine analyzes the user's reactions and determines their emotional state (e.g., anxiety, surprise, relief, etc.), for example by analyzing the user's facial expressions and tone of voice via a camera or microphone.

[0339] Step 14:

[0340] On the device: If the emotion engine determines that the user's emotional state indicates anxiety or stress, the display of the warning message will be adjusted to be more friendly and provide additional reassurance or support information.

[0341] Step 15:

[0342] Server: The judgment results and the emotion engine analysis results are stored in a database for future reference and system improvement.

[0343] Step 16:

[0344] Server: Based on the information accumulated in the database, the server learns to provide appropriate warning messages that take the user's emotional state into account from the next time onwards.

[0345] These are the specific processing steps and operations of the system that combines the emotion engine. This not only allows users to select foods with peace of mind, but also provides appropriate support that takes into account their emotional state.

[0346] Example 2

[0347] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0348] Existing food allergy prevention systems provide basic functions to reduce the risk of users purchasing foods containing allergens, but they do not provide warning messages that take into account the user's emotional state. This leaves users with a problem: the anxiety and stress they feel when receiving a warning message is not alleviated, making it difficult for them to make safe and secure food choices.

[0349] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for using optical character recognition technology to extract text from an image, means for comparing the extracted ingredient list with an allergy database to determine whether or not an allergic ingredient is present, means for analyzing the user's emotional state, and means for adjusting the display method of the warning message based on the user's emotional state. This not only enables the user to safely select foods to which they have allergies, but also gives them a sense of security when receiving a warning message, thereby reducing stress.

[0350] "User" refers to an individual who uses the system, specifically someone who inputs food allergy information and makes safe food choices.

[0351] "Allergy information" is data about allergens that the user knows about, and the system uses this information to determine the safety of food.

[0352] "Food packaging" refers to packaging material on which food product information is printed, and is the object that a user photographs using a camera.

[0353] "Server" refers to a computer device that receives and processes data sent from user terminals and performs database management and analysis.

[0354] "Image analysis" refers to the process of extracting information from photographed images of food packaging and recognizing it as text data.

[0355] "Optical character recognition technology" refers to technology that digitizes and recognizes characters in an image and extracts them as text data.

[0356] An "ingredient list" is a list of ingredients contained in a food product, and is used to check whether it contains specific allergens.

[0357] "Allergy database" refers to a database that stores and manages allergy information registered by users.

[0358] "Determination" refers to the process of checking the ingredient list against an allergy database to determine whether it contains any ingredients that may be dangerous to the user.

[0359] "Emotional state" refers to the psychological reaction (e.g., anxiety, relief, stress) of the user when they receive a warning message, and is analyzed by the emotion engine.

[0360] "Emotion engine" refers to software or hardware for detecting and analyzing a user's emotional state.

[0361] A "warning message" is a notification that informs the user that a particular food contains an allergic ingredient, and is used to ensure the user's safety.

[0362] "Database" refers to an electronic information storage medium that organizes large amounts of data processed by a system and efficiently stores, searches, and manages them.

[0363] "Adjustment" refers to the act of providing a more appropriate user experience by changing the content and display method of a warning message based on the user's emotional state.

[0364] This invention provides a system for safely purchasing and consuming food for users with food allergies, and offers a new approach that takes into account the emotional state of the user. The system has specific roles for the user, the terminal, and the server.

[0365] Registering allergy information

[0366] User: Launch the app and open the "Register Allergy Information" screen from the settings menu. The user selects and enters the ingredients to which they are allergic (e.g., "wheat," "peanuts," "dairy products") from a list.

[0367] Terminal: Save the allergy information entered by the user to a local database. After saving, generate JSON data to be synchronized with the server and create a send request.

[0368] Server: Receives allergy information sent from the device and stores it in a personal allergy database. This data is used to match food ingredients with the food's ingredient list in a later step.

[0369] Food packaging photography

[0370] User: Use your smartphone camera to take a photo of the entire package of a food item you are considering purchasing, using the frame guidelines to guide you in the best position to take the photo.

[0371] Device: The captured image data is temporarily stored in local storage, and a thumbnail is displayed for the user to confirm. When the user confirms and presses the send button, a request to send the image data to the server is prepared.

[0372] Parsing package information

[0373] Server: Receives image data sent from the device. Inputs the received data into an OCR (Optical Character Recognition) engine. The OCR engine extracts text information from the image.

[0374] Server: Preprocesses the text data extracted by the OCR engine, including noise reduction and text normalization, such as removing symbols and unnecessary spaces, and listing important ingredient information (e.g., "sugar," "dairy").

[0375] Database Matching

[0376] Server: Matches the preprocessed text data with the allergy information stored in the database. It uses SQL queries to match the allergens registered by the user with the extracted ingredient list.

[0377] Server: If a matching allergen exists, identify and list the ingredient. For example, if "dairy" is included, add this information to the warning list.

[0378] Notification of judgment results and emotion engine

[0379] Server: Generates a warning message based on the matching result. For example, if the ingredient to which the user is allergic is "dairy products," prepare a warning message stating "This product contains dairy ingredients."

[0380] Device: Notify the user of received alert messages. Display alert messages within the app using a pop-up window or notification bar.

[0381] Emotion engine: The system uses camera and microphone sensors to collect data on how users view warning messages. For example, it analyzes changes in facial expressions and speech in real time the moment the user sees the message.

[0382] On the device: Based on the results of the emotion engine, the message display can be adjusted depending on the user's emotional state. For example, if the user expresses anxiety, more friendly language or additional support information can be displayed.

[0383] Data accumulation

[0384] Server: Stores each judgment result and the emotion engine's analysis results in a database. This allows for quick response when the same product is scanned in the future. It also continuously learns how to display the most appropriate warning message to the user based on emotion data.

[0385] Specific examples

[0386] As an example, a case where a user wants to purchase "chocolate" will be described.

[0387] 1. User: Registers in the app that they have allergies to "wheat," "peanuts," and "dairy products." The user opens the "Register Allergy Information" screen from the settings menu, selects each item, and confirms.

[0388] 2. Terminal: Stores this information in a local database, packages the allergy information in JSON format, and generates a request to send to the server.

[0389] 3. Server: Receives the submitted allergy information and stores it in a personal allergy database for each user. This information is stored in an SQL database and saved in a format that can be easily verified later.

[0390] 4. User: Use the smartphone camera to take a photo of the entire chocolate package, following the photography guidelines and centering the food label.

[0391] 5. Terminal: Displays thumbnails of the captured images for the user to check. When the user checks the image and presses the send button, a request to send the image data to the server is prepared.

[0392] 6. Server: Passes the received image data to the OCR engine, which extracts text information such as "cocoa, sugar, milk components, lecithin (soybean-derived)" from the image.

[0393] 7. Server: Preprocesses the text data, removing noise and normalizing the necessary raw material information into a list.

[0394] 8. Server: Checks the normalized ingredient list against the user's allergy database, using an SQL query to ensure "dairy" is included.

[0395] 9. Server: Based on the matching results, a warning message such as "This product contains milk ingredients" is generated and sent to the terminal.

[0396] 10. Terminal: Displays a popup to alert the user of the received message.

[0397] 11. Emotion Engine: User reactions are collected using cameras and sensors and emotion analysis is performed. If the user shows surprise or anxiety in response to a displayed message, the emotional data is analyzed.

[0398] 12. On the device: Based on the results of the emotion engine, an additional message is displayed to reassure the user. For example, a friendly message such as "Please avoid purchasing this product as it contains dairy ingredients" is displayed.

[0399] 13. Server: The final judgment result and the user's emotional data are stored in a database, which will enable faster and more appropriate responses when the same situation occurs in the future.

[0400] In this way, the present invention helps users with food allergies make safe food choices and further enhances the user experience by adjusting the display of warning messages taking into account the user's emotional state in real time.

[0401] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0402] Step 1: The user launches the app and opens the "Register Allergy Information" screen from the settings menu. The user selects and enters the ingredients to which they are allergic (e.g., "wheat," "peanuts," "dairy products") from a list. The entered allergy information is saved in JSON format on the device. This becomes the input data.

[0403] Step 2: The device generates a request to send the saved JSON data to the server. Specifically, the created JSON data is sent to the server as an HTTP POST request. The input is the allergy information data in JSON format, and the output is the request sent to the server.

[0404] Step 3: The server parses the received allergy information and stores it in a per-user allergy database. Specifically, it executes a SQL query to parse the received data and insert it into the database. The input is the JSON-formatted allergy information sent to the server, and the output is the updated database state.

[0405] Step 4: The user uses the smartphone camera to take a picture of the entire food package they are considering purchasing. Guidelines are displayed to guide the user in taking the picture so that the entire label fits in the picture. This results in a captured image. The input is a physical image of the food package, and the output is a captured digital image.

[0406] Step 5: The device temporarily saves the captured image in local storage, generates a thumbnail, and asks the user to confirm it. When the user confirms and presses the send button, a request is generated to send the image data to the server. The input is the captured digital image, and the output is an image data request sent to the server.

[0407] Step 6: The server inputs the received image data into the OCR engine, which extracts the text information in the image and stores it as text data. Specifically, it performs binary analysis on the image data and applies a character recognition algorithm. The input is the digital image sent to the server, and the output is the extracted text data.

[0408] Step 7: The server preprocesses the text data extracted by the OCR engine. This includes noise removal and normalization. Specifically, it removes symbols and unnecessary spaces, extracts important keywords, and compiles them into a list. The input is the extracted text data, and the output is a preprocessed raw material list.

[0409] Step 8: The server compares the preprocessed text data with the allergy information stored in the database. Specifically, it executes an SQL query to retrieve the user's allergy ingredients from the database and compare them with the ingredient list. The input is the preprocessed ingredient list and the allergy information in the database, and the output is the comparison result.

[0410] Step 9: The server generates a warning message based on the matching result. For example, if the ingredient "dairy" is matched, it prepares a warning message saying "This product contains dairy ingredients." The input is the matching result, and the output is the generated warning message.

[0411] Step 10: The device notifies the user of the received warning message. Specifically, it displays the message in the app using a popup or notification bar. The input is the warning message, and the output is the notification displayed to the user.

[0412] Step 11: The emotion engine collects and analyzes the user's reaction to the warning message through the camera and microphone sensors. It analyzes the user's facial expressions and tone of voice to evaluate their emotional state in real time. The input is the user's reaction data, and the output is the analyzed emotional state.

[0413] Step 12: The device adjusts the display of the warning message based on the results of the emotion engine. Specifically, it provides appropriate information to the user by adding reassuring and friendly language. The input is the analyzed emotional state, and the output is the adjusted warning message.

[0414] Step 13: The server saves the final judgment result and the user's emotion data in the database. This allows for quick response when the same product is scanned in the future and enables learning based on the emotion data. The input is the final judgment result and emotion data, and the output is the updated database.

[0415] (Application example 2)

[0416] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0417] Previously, there were systems that allowed users to register allergy information and check food ingredients. However, these systems did not take into account the user's emotional state and lacked measures to address the anxiety and stress users felt when choosing foods containing allergens. This limited the effectiveness of these systems in increasing users' sense of security and created a psychological burden when choosing food.

[0418] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting and analyzing the user's emotional state, means for adjusting the display method of the warning message based on the analyzed emotional state, and means for storing the determination result and the emotional state in a database. This makes it possible to provide flexible warning messages that take the user's emotional state into consideration.

[0419] "User" refers to an individual who uses the system to register their own allergy information and confirm food safety.

[0420] "Allergy information" is data registered by users regarding their allergens. This information is used to confirm food safety when selecting foods.

[0421] "Food packaging" refers to materials such as paper, plastic, and metal used to pack or wrap food, and which bear the name of the food, as well as information about its ingredients and components.

[0422] "Means for taking photographs" refers to a function that allows a user to take a photograph of a food package using a smartphone or camera.

[0423] A "server" is a computer system that receives, analyzes, and stores data sent by users.

[0424] The "means for transmitting images" is a function that allows a user to upload an image of a food package taken by the user to a server.

[0425] The "means for the server to analyze the image" refers to the function of analyzing the received image data and extracting the list of ingredients for the food. Optical character recognition (OCR) technology is generally used.

[0426] A "food ingredient list" is a list of ingredient names listed on a food package.

[0427] The "allergy database" is a collection of numerical information that stores allergy ingredient information registered by the user.

[0428] The "means for determining" is a function that checks the extracted ingredient list against an allergy database to determine whether or not an allergic ingredient is present.

[0429] "Means for notifying" refers to a function for displaying or sending a message to inform the user of the analysis results.

[0430] The "user's emotional state" refers to the user's psychological reaction or emotions when seeing food containing an allergic ingredient.

[0431] "Means of collection and analysis" refers to the function of collecting the user's emotional state through cameras and sensors and analyzing it using dedicated software and algorithms.

[0432] The "means for adjusting the display method of the warning message" is a function for changing the content and display method of the warning message depending on the emotional state of the user.

[0433] "Means for storing the judgment results and emotional state in a database" is a function that stores the analysis results and the user's emotional data in a database and makes them useful for future analysis and responses.

[0434] This system allows users to register allergy information, photograph food packages, and analyze their ingredients to help them choose foods safely. It also collects and analyzes the user's emotional state and provides appropriate messages to enhance the user's sense of security.

[0435] 1. Registering allergy information

[0436] The user starts the application and opens the "Register Allergy Information" screen from the settings menu. Here, the user selects and enters the ingredients to which they are allergic (e.g., "wheat," "peanuts," "dairy products") from a list. The device saves this allergy information locally and then sends the data to the cloud server.

[0437] 2. Take a photo of the food package and send it

[0438] A user takes a photo of the packaging of a food item they are considering purchasing in a physical store with their smartphone camera. The device temporarily stores the captured image data and prepares a request to send it to the server. When the user presses the send button, the image is uploaded to the cloud server.

[0439] 3. Package information analysis

[0440] The server uses optical character recognition (OCR) technology to extract text information from the received image data. This is done using Google Cloud Vision API, among other tools. The server then preprocesses the extracted text data and lists the ingredients. For example, ingredients such as "cocoa, sugar, milk components, lecithin (soybean-derived)" are identified here.

[0441] 4. Database Matching

[0442] The server compares the preprocessed text data with the allergy information stored in the database, checks whether the allergens registered by the user beforehand match the analyzed ingredient list, and generates a warning message if there is a match.

[0443] 5. Notification of judgment results and emotion analysis

[0444] The device will then notify the user of a warning message, such as "This product contains milk ingredients." The device will then collect the user's reactions through its camera and sensors and analyze their emotional state. This analysis utilizes Azure Emotion API and other tools. Based on the analysis results, if the user's emotional state indicates anxiety or stress, the server will adjust the message display and provide additional support information.

[0445] 6. Data accumulation

[0446] The server stores the results of each judgment and the emotion engine's analysis in a database. This allows for a quick response the next time the same product is analyzed. Emotion data is also stored, and the server learns how to display the optimal warning message for each user.

[0447] Specific examples

[0448] Let's consider the case where a user is about to purchase "chocolate." First, the user registers in the app that they are allergic to "wheat," "peanuts," and "dairy products." The device saves the allergy information locally and sends it to the server. The server saves the allergy information in a database. The user takes a photo of the "chocolate" packaging with their smartphone, and the device sends the image data to the server. The server analyzes the image and extracts text information. Ingredients such as "cocoa, sugar, milk ingredients, and lecithin (soybean-derived)" are extracted, and the information is then compared with the ingredient list to determine that the product contains "dairy products," the user's allergic ingredient. The server sends the result of the assessment to the device and generates a warning message. The device displays a warning to the user stating "This product contains milk ingredients," and the emotion engine detects the user's reaction and analyzes their emotional state. The device adjusts the message display as necessary, and the server saves the assessment result and emotion data in a database.

[0449] Prompt Sentence Examples

[0450] "Just enter your allergy information (e.g., wheat, peanuts, dairy) and take a photo of a peanut butter package. The system will detect the allergen and notify you. It will also provide additional support information as needed based on your emotional state."

[0451] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0452] Step 1: Register your allergy information

[0453] The user starts the application and opens the "Register Allergy Information" screen. As input, the user selects and enters the ingredients to which they are allergic (e.g., "wheat," "peanuts," "dairy products") from a list. The device saves this allergy information in local storage and generates a request to send to the server. The input data is the user's allergy information, and the output data is the data request sent to the server. Specifically, the device sends the data to the server using a REST API.

[0454] Step 2: Photograph and send the food package

[0455] A user takes a photo of the packaging of a food item they are considering purchasing with their smartphone camera. The input is the food packaging to be photographed, and the output is the captured image data. The device temporarily stores this image data and prepares a request to send to the server. Specifically, when the user presses the send button, the device uploads the image data to the server. An HTTP POST request is used to upload the image data.

[0456] Step 3: Parse package information

[0457] The server extracts text information from the received image data using optical character recognition (OCR) technology. The input is image data and the output is extracted text data. The OCR engine uses services such as Google Cloud Vision API. Specifically, the server sends data to the OCR engine and receives text data as the analysis result.

[0458] Step 4: Check against the database

[0459] The server compares the preprocessed text data with the allergy database. The input is the extracted text data and the user's allergy information, and the output is the comparison result. The comparison process identifies ingredients that match the allergens in the database. Specifically, the text data is used as a search query to query the database to see if there are any matching ingredients.

[0460] Step 5: Notification of decision

[0461] The server generates a warning message based on the matching result and sends it to the terminal. The input is the matching result and the output is the warning message. Specifically, the server analyzes the matching result and generates a warning message if the product contains an ingredient to which the user is allergic. The terminal displays this message to the user via a pop-up or notification bar.

[0462] Step 6: Sentiment Analysis

[0463] The device collects user reactions through cameras and sensors and analyzes their emotional state. The input is user reaction data, and the output is emotional state data. The emotion engine uses the Azure Emotion API, etc. Specifically, it sends reaction data to the emotion engine and receives the emotional state as the analysis result.

[0464] Step 7: Adjust message display

[0465] The server adjusts the display of warning messages based on the emotional state. The input is the emotional state data and the current warning message, and the output is the adjusted warning message. Specifically, it analyzes the emotional state data and provides friendly messages or additional support information as needed.

[0466] Step 8: Accumulate data

[0467] The server accumulates the judgment results and emotion analysis results in a database. The input is the judgment results and emotional state data, and the output is an updated database. Specifically, the server saves this data in the database and makes it available for future analysis.

[0468] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0469] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0470] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0471] [Second embodiment]

[0472] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0473] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0474] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0475] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0476] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0477] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0478] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0479] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0480] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0481] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0482] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0483] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0484] The present invention is a system that enables users with food allergies to safely purchase food and enjoy their meals, and is realized through a series of processes: registering allergy information, photographing food packaging, analyzing the images, extracting and comparing ingredient lists, notifying the user of the results, and storing the data. Specific embodiments of the system are described below.

[0485] Registering allergy information

[0486] User: Launches the app and opens the user profile settings screen. The user enters their allergy information (e.g., "wheat," "peanuts," "dairy," etc.). This information can be added, deleted, or edited at any time from the application's settings menu.

[0487] Terminal: The entered allergy information is saved in a local database and the data is prepared for transmission to the server. The following is the process for registering the user's allergy information.

[0488] Server: Receives allergy information sent from the device and stores it in a user-specific allergy database. This data is later used to match food ingredients against a food ingredient list.

[0489] Food packaging photography

[0490] User: Take a photo of the packaging of the food item you are considering purchasing using your smartphone camera. When taking the photo, make sure to adjust it so that the entire package is clearly visible.

[0491] Terminal: Temporarily stores the captured image data and prepares a request to send it to the server. After the user presses the capture button, the image is immediately uploaded to the server.

[0492] Parsing package information

[0493] Server: The server analyzes the received image data. As a first step in the analysis, it uses optical character recognition (OCR) technology to extract text from the image. The extracted text is then processed into a list of raw materials. During this process, preprocessing is performed to remove noise and improve character recognition accuracy.

[0494] Server: Normalizes the extracted ingredient list to match a standard format in the database, and translates it if necessary based on specific language codes.

[0495] Database Matching

[0496] Server: Compares the normalized ingredient list with the user's allergy database. Checks whether the ingredients match the allergy information in the database. If a match is found, identifies the ingredient and passes the result on to further processes.

[0497] Notification of the results

[0498] Server: Generates a warning message if allergens are identified. The result includes a list of identified allergens.

[0499] Server: Sends the result of the judgment to the user's device. A notification message is generated so that the user can view it.

[0500] Device: Receives the sent judgment result and displays it to the user as an alert or pop-up message within the app, allowing the user to make safe food choices.

[0501] Data accumulation

[0502] Server: Each judgment result is stored in a database. This allows the system to provide quick results based on existing data the next time the same food is analyzed. The stored data is used to improve the efficiency of the entire system.

[0503] Specific examples

[0504] A case where a user checks a package of "chocolate" will be described.

[0505] 1. User: Registers in the app that they are allergic to wheat, peanuts, and dairy products.

[0506] 2. Device: Stores allergy information locally and sends it to the server.

[0507] 3. Server: Stores allergy information in a database.

[0508] 4. User: Takes a photo of the chocolate packaging with their smartphone.

[0509] 5. Terminal: Sends image data to the server.

[0510] 6. Server: Performs image analysis and extracts text information, such as "cocoa, sugar, milk components, lecithin (derived from soybeans)."

[0511] 7. Server: Checks the ingredients list and identifies that it contains the user's allergic ingredient, "dairy."

[0512] 8. Server: Sends the judgment result to the terminal and generates a warning message.

[0513] 9. Device: Display a warning to the user that "This product contains milk ingredients."

[0514] 10. Server: Stores the judgment results in a database.

[0515] Thus, the present invention provides fast and efficient assistance to users with food allergies in making safe food choices.

[0516] The processing flow will be explained below.

[0517] Step 1:

[0518] User: Launch the app and open the "Register Allergy Information" screen from the settings menu. The user selects and enters the ingredients they are allergic to (e.g., "wheat," "peanuts," "dairy products") from a list.

[0519] Step 2:

[0520] Terminal: Stores the entered allergy information in a local database, prepares the data for synchronization with the server, and creates a transmission request.

[0521] Step 3:

[0522] Server: Receives allergy information sent from the device. Stores the received data in a user-specific allergy database. At this stage, the data is formatted for matching.

[0523] Step 4:

[0524] User: Using their smartphone camera, they take a picture of the packaging of a food item they are considering purchasing. They frame the photo so that the entire label is visible.

[0525] Step 5:

[0526] Terminal: Temporarily stores the captured image and prepares a request to send it to the server. When the user presses the send button, the image data is uploaded to the server.

[0527] Step 6:

[0528] Server: Receives image data sent from the device. The received data is input into an optical character recognition (OCR) engine for image analysis. During this process, text information is extracted from the image.

[0529] Step 7:

[0530] Server: Preprocesses the text data extracted by the OCR engine, including noise removal and text normalization. For example, ingredients such as "sugar" and "dairy" are listed.

[0531] Step 8:

[0532] Server: Compares the preprocessed text data with the allergy information stored in the database, and verifies that the allergens registered by the user in advance match the analyzed ingredient list.

[0533] Step 9:

[0534] Server: Makes a decision based on the match. If a matching allergen is found, identifies the allergen and adds it to the list. Generates a warning message and prepares a notification to the user.

[0535] Step 10:

[0536] Server: Sends the judgment result to the user's device, which includes the identified allergens and a warning message.

[0537] Step 11:

[0538] Device: Notify the user of the received test result by displaying a message such as "This product contains milk ingredients" in a pop-up or notification bar within the app.

[0539] Step 12:

[0540] Server: Each judgment result is stored in a database. This allows for a quick response the next time the same product is analyzed. The data is saved for future reference and analysis, contributing to the efficiency of the entire system.

[0541] The above are the specific processing steps and operations of the proposed system.

[0542] Example 1

[0543] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0544] For consumers with food allergies, the process of checking food safety when making everyday food purchases is extremely cumbersome. Manually checking ingredient lists is time-consuming and laborious, and there is a risk of overlooking or misreading information. Furthermore, in many markets, product information is displayed in different languages, which can be difficult to understand. Therefore, there is a need for a system that allows users with food allergies to efficiently and accurately check food safety and purchase and consume with peace of mind.

[0545] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0546] In this invention, the server includes means for a user to input their own allergy information, means for photographing a food package, means for transmitting the image to the server, means for the server to analyze the image and extract a list of ingredients for the food, means for comparing the extracted ingredient list with an allergy database to determine whether or not an allergic ingredient is present, means for notifying the user of the determination result, means for storing the determination result in a database, means for extracting text from the image using optical character recognition technology, means for temporarily saving image data photographed by the user, means for preparing data for the user to transmit their allergy information to the server, means for saving the user's allergy information, and means for displaying the determination result within the app. This allows users to efficiently and accurately confirm the safety of foods and purchase or consume them with peace of mind.

[0547] "Users" are consumers who have food allergies and want to make safe food choices.

[0548] "Allergy information" is a list of ingredients that cause allergic reactions in specific foods, and is entered by the user.

[0549] "Food packaging" refers to the packaging or label that lists the product information and ingredients of a food product.

[0550] "Server" means a central computer system that receives and processes data submitted by users.

[0551] "Image" refers to photographic data of food packaging taken with a smartphone or other device.

[0552] "Optical character recognition (OCR) technology" is a technology for recognizing and extracting text data from images.

[0553] A "database" is a system for structuring and storing data such as user allergy information and food ingredient lists.

[0554] An "ingredient list" is a list of the ingredients and materials contained in a food product, and is text data extracted from the packaging.

[0555] "Notification" refers to a message or warning that the server sends to the user to inform them of the judgment result.

[0556] An "alert" is a warning message that is displayed to prompt the user to take a specific action.

[0557] A "pop-up message" is a message that appears temporarily on the screen to attract the user's attention.

[0558] The "determination result" refers to the result of determining whether or not an allergy ingredient is present based on a comparison between the extracted ingredient list and the user's allergy database.

[0559] The "user profile setting screen" is the screen within the app where users can enter, edit, and save their personal information and allergy information.

[0560] The present invention provides a system that enables users with food allergies to safely purchase food and enjoy meals. This system executes a series of processes using the following hardware and software:

[0561] Hardware

[0562] Device: A mobile information device such as a smartphone or tablet.

[0563] Server: A centralized computer system.

[0564] software

[0565] Local database: Manage user allergy information stored on the device using a lightweight database such as SQLite.

[0566] Optical Character Recognition (OCR) technology: Using libraries such as OpenCV and Tesseract, the server extracts text from images.

[0567] Translation API: Using the Google Cloud Translation API or similar, translate the ingredient list into a user-understandable language as needed.

[0568] Notification system: The result of the judgment is notified to the user using GCM (Google Cloud Messaging) or APNs (Apple Push Notification service).

[0569] Operating Procedures and Processes

[0570] 1. Allergy Information Registration:

[0571] User: Launch the app and open the user profile settings screen. Enter your allergy information (e.g., "wheat," "peanuts," "dairy products," etc.).

[0572] Terminal: Stores the entered allergy information in a local database and prepares the data for transmission to the server.

[0573] Server: Receives allergy information sent from the device and stores it in an allergy database for each user.

[0574] 2. Food packaging photography:

[0575] User: Take a photo of the packaging of a food item they are considering purchasing with their smartphone camera, adjusting the camera so that the entire package is clearly visible.

[0576] Terminal: Temporarily stores the captured image data and prepares a request to send it to the server.

[0577] 3. Parse package information:

[0578] Server: Analyzes the received image data and extracts the text within the image using optical character recognition (OCR) technology.

[0579] Specific examples

[0580] A case where a user checks a package of "chocolate" will be described.

[0581] 1. User: Registers in the app that they are allergic to wheat, peanuts, and dairy products.

[0582] 2. Device: Stores allergy information locally and sends it to the server.

[0583] 3. Server: Stores allergy information in a database.

[0584] 4. User: Takes a photo of the chocolate packaging with their smartphone.

[0585] 5. Terminal: Sends image data to the server.

[0586] 6. Server: Performs image analysis and extracts text information. For example, "cocoa, sugar, milk components, lecithin (derived from soybeans)" is extracted.

[0587] 7. Server: Checks the ingredients list and identifies that it contains the user's allergic ingredient, "dairy."

[0588] 8. Server: Sends the judgment result to the terminal and generates a warning message.

[0589] 9. Device: Display a warning to the user that "This product contains milk ingredients."

[0590] Examples of prompt statements

[0591] "Open your user profile settings screen and enter your allergy information."

[0592] "Take a photo of the packaging of the food you're considering purchasing."

[0593] "The captured image data is being sent to the server. Please wait."

[0594] "The ingredients list has been analyzed. We are currently reviewing the results."

[0595] "Avoid this food because it contains dairy."

[0596] This system provides support to users with food allergies by efficiently and accurately checking food safety and enabling them to select foods with peace of mind.

[0597] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0598] Step 1:

[0599] User: Launches the app and opens the user profile setting screen. The user enters their allergy information (e.g., "wheat," "peanuts," "dairy products," etc.). This is the input data.

[0600] Terminal: The entered allergy information is saved in a local database (e.g., SQLite). Data calculations at this stage involve converting the information entered by the user into a database format and saving it. Data is also prepared for sending to the server. Specific operations include writing to the database and converting the data format for sending to the server. The output is the allergy information saved locally and the data to be sent to the server.

[0601] Server: Receives allergy information sent from the device. Stores the received data in an allergy database for each user. Specifically, the data is written to the allergy database (e.g., MySQL or PostgreSQL). This is the output data.

[0602] Step 2:

[0603] User: Take a photo of the packaging of a food item they are considering purchasing with their smartphone camera. When taking the photo, they need to adjust the camera so that the entire package is clearly visible. This is the input data.

[0604] Terminal: Temporarily stores the captured image data. Data calculation at this stage involves converting the image data into an appropriate format (e.g., JPEG or PNG) and saving it. After the user presses the capture button, a request is immediately prepared to upload the image to the server. The specific operation is to convert the image data for network transmission. The output is image data ready for transmission to the server.

[0605] Step 3:

[0606] Server: The server receives the image data sent from the terminal. It analyzes the received image data and extracts the text within the image using optical character recognition (OCR) technology (e.g., Tesseract). This is the input data. Data processing at this stage involves converting the image data into text data, including noise removal and image preprocessing. The output is the extracted text data of the ingredient list.

[0607] Step 4:

[0608] Server: The extracted text data is normalized and translated into a specific language using a translation API (e.g., Google Cloud Translation API) if necessary to conform to a standard format. This is the input data. Data operations at this stage involve cleaning and normalizing the text data, and translating it if necessary. The output is the normalized and translated text data of the ingredient list.

[0609] Step 5:

[0610] Server: Compares the normalized ingredient list with the user's allergy database. The specific operation is to compare the allergy information in the database with the ingredient list. This is the input data. The data operation at this stage is to identify matching ingredients using a comparison algorithm. The output is a judgment result on the presence or absence of allergens.

[0611] Step 6:

[0612] Server: If an allergen is identified, a warning message is generated. This is the input data. Data calculation at this stage generates a warning message based on the judgment result. The output is a warning message and the judgment result.

[0613] Server: Sends the result of the judgment to the user's device. Specifically, it sends a message in real time via a notification system such as GCM or APNs. This is the input data. The output is the sent notification message.

[0614] Device: Receives the sent judgment result and displays it to the user as an alert or pop-up message within the app. The specific behavior is to display a notification on the device's user interface. This is the input data, and the output is the displayed warning message.

[0615] Step 7:

[0616] Server: Each decision result is stored in a database. This is the input data. Data calculations at this stage are to save the decision results in a database (e.g. MySQL or PostgreSQL). The output is the updated database.

[0617] The above are the specific processing steps in the system of the present invention, which allow users with food allergies to select foods efficiently and safely.

[0618] (Application example 1)

[0619] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0620] When consumers with food allergies purchase food at physical stores, they need a method to quickly and accurately identify allergens and safely select foods. Conventional methods often require complicated and time-consuming food safety checks, making it difficult to ensure consumer peace of mind.

[0621] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0622] In this invention, the server includes means for a user to input their own allergy information, means for photographing the food package, means for transmitting the image to the server, means for the server to analyze the image and extract a list of the food's ingredients, means for comparing the extracted ingredient list with an allergy database to determine whether or not an allergen is present, means for notifying the user of the determination result, means for storing the determination result in the database, and means for displaying the allergen determination result in real time using a smart device at a physical store. This allows users to instantly check whether or not a food product contains an allergen by scanning the food package at the physical store, enabling them to select safe foods.

[0623] "User" refers to an individual who uses the system and is the entity that inputs allergy information and photographs food packaging.

[0624] "Allergy information" refers to information that indicates the specific ingredient names and food names for which the user has a food allergy.

[0625] "Food packaging" refers to the packaging in which food is packaged, and is the target from which ingredient information written inside is extracted.

[0626] "Smart devices" refer to electronic devices that can be carried by users, such as smartphones, smart glasses, and head-mounted displays.

[0627] "Server" refers to a computer system that processes information for the entire system, and performs image analysis and database matching.

[0628] "Image analysis" is the process of processing images of food packaging to extract useful textual information from them.

[0629] "Optical Character Recognition (OCR) technology" is a technology that recognizes characters from images or printed text as electronic data.

[0630] "Allergy database" refers to a database that stores the user's allergy information and is used for matching purposes.

[0631] The "determination result" is a conclusion as to whether or not an allergen is present, obtained by comparing the extracted ingredient list with the allergy database.

[0632] "Notification" refers to the process of communicating the results of a determination to the user, which is done through a warning message or a pop-up message.

[0633] MODE FOR CARRYING OUT THE INVENTION

[0634] This invention is a system that allows users with food allergies to safely select food in physical stores. How this system is implemented will be described below in detail.

[0635] System Program Overview

[0636] The server executes a series of processes, including setting up a user profile, photographing the food packaging, analyzing the image, extracting ingredients, comparing it with a database, notifying the user of the results, and storing the results, allowing users to quickly determine the safety of food in physical stores.

[0637] Hardware / Software used

[0638] Hardware:

[0639] Smartphone (Android or iOS)

[0640] server

[0641] software:

[0642] For server-side processing, the Flask framework is used.

[0643] For image analysis, pytesseract is used as an optical character recognition (OCR) technology.

[0644] For image processing, we use PIL, a Python image processing library.

[0645] SQLite is used as the database.

[0646] Data processing and calculation

[0647] User:

[0648] 1. The user launches the app on their smartphone, enters their allergy information (e.g., "wheat," "peanuts," "dairy products," etc.), and registers it.

[0649] 2. The registered allergy information is stored in the smartphone's local database and also sent to the server.

[0650] server:

[0651] 1. The server receives allergy information sent by the user and stores it in a dedicated allergy database.

[0652] 2. When a user takes a photo of a food package in a physical store with their smartphone camera, the image data is sent to the server.

[0653] 3. The server analyzes the received image data using optical character recognition (OCR) technology to extract the ingredient list from the package. This analysis involves preprocessing to remove noise and improve character recognition accuracy.

[0654] 4. The extracted ingredient list is normalized to a standard format and compared with the user's allergy database to determine whether or not an allergen is present.

[0655] Notification of decision:

[0656] 1. The server generates a warning message if an allergen is found. The result includes a list of identified allergens.

[0657] 2. The judgment result is sent to the user's smartphone, and a notification message is displayed so that the user can view it immediately.

[0658] Data accumulation:

[0659] 1. The results of the assessment are stored in a server database. This data is used to speed up future analysis processes.

[0660] Specific examples

[0661] A case will be described where the user wishes to check the packaging of "Chocolate".

[0662] 1. The user registers in a smartphone app that they are allergic to wheat, peanuts, and dairy products.

[0663] 2. The smartphone stores the allergy information locally and sends it to the server.

[0664] 3. The server stores this information in a dedicated database.

[0665] 4. The user takes a photo of the chocolate packaging with their smartphone in a physical store.

[0666] 5. The captured image data is sent to the server.

[0667] 6. The server analyzes the image and extracts text information, such as "cocoa, sugar, milk components, lecithin (derived from soybeans)."

[0668] 7. The extracted ingredient list is compared and it is determined that the product contains the user's allergic ingredient, "dairy products."

[0669] 8. The server sends the result of the judgment to the smartphone and generates a warning message.

[0670] 9. Display a "This product contains dairy" warning on your smartphone.

[0671] Prompt Sentence Examples

[0672] "Create an application that, when a user scans a food package in a physical store with their smartphone camera, analyzes the ingredients contained in the package in real time and compares them with the allergy information registered by the user to detect allergens."

[0673] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0674] Step 1:

[0675] The user launches the app on their smartphone and enters their allergy information (e.g., "wheat," "peanuts," "dairy products," etc.). The entered allergy information is stored in a local database and is also prepared as data to be sent to the server.

[0676] Input: Allergy information entered by the user

[0677] Output: Allergy information stored in the local database and sent to the server

[0678] Step 2:

[0679] The terminal receives the allergy information entered by the user, stores it in a local database, and then transmits it to the server.

[0680] Input: Allergy information entered by the user

[0681] Output: Allergy information sent to the server

[0682] Step 3:

[0683] The server receives the allergy information sent from the terminal and stores the information in an allergy database.

[0684] Input: Allergy information sent from the device

[0685] Output: Allergy information stored in the allergy database

[0686] Step 4:

[0687] The user takes a photo of the packaging of a food item they are considering purchasing at a physical store using their smartphone camera, adjusting the camera so that the entire package is clearly visible.

[0688] Input: Food packaging you're considering buying

[0689] Output: Package image saved on your smartphone

[0690] Step 5:

[0691] The device temporarily stores the captured image data and prepares a request to send it to the server. When the user presses the capture button, the image is immediately uploaded to the server.

[0692] Input: Photographed package image

[0693] Output: Image data sent to the server

[0694] Step 6:

[0695] The server analyzes the received image data, extracts the text within the image using optical character recognition (OCR) technology, and processes it as an ingredient list.

[0696] Input: Received image data

[0697] Output: Extracted ingredient list

[0698] Step 7:

[0699] The server normalizes the extracted ingredient list, matches it with a standard format in an allergy database, and may translate it into specific languages ​​if necessary.

[0700] Input: Extracted ingredient list

[0701] Output: Normalized ingredient list

[0702] Step 8:

[0703] The server checks the normalized ingredient list against the user's allergy database to see if the product contains any allergens.

[0704] Input: Normalized ingredient list, allergy database

[0705] Output: Judgment result regarding the presence or absence of allergens

[0706] Step 9:

[0707] If an allergy ingredient is identified, the server generates a warning message and creates a message to notify the user based on the generated judgment result.

[0708] Input: Judgment result regarding the presence or absence of allergens

[0709] Output: Warning message

[0710] Step 10:

[0711] The device receives the judgment results sent from the server and displays them to the user as an alert within the app, allowing the user to select safe foods.

[0712] Input: Verification result sent from the server

[0713] Output: The alert message that is displayed to the user

[0714] Step 11:

[0715] The server stores each judgment result in a database, which allows it to quickly provide results based on existing data the next time the same food is analyzed.

[0716] Input: Judgment result

[0717] Output: Judgment results stored in the database

[0718] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0719] The present invention provides a new approach to a system that allows users with food allergies to safely purchase food and enjoy meals, taking into account the emotional state of the user. Specific embodiments of this system are described in detail below.

[0720] Registering allergy information

[0721] User: Launch the app and open the "Register Allergy Information" screen from the settings menu. The user selects and enters the ingredients they are allergic to (e.g., "wheat," "peanuts," "dairy products") from a list.

[0722] Terminal: Stores the entered allergy information in a local database, prepares the data for synchronization with the server, and creates a transmission request.

[0723] Server: Receives allergy information sent from the device and stores it in a user-specific allergy database. This data is later used to match food ingredients against a food ingredient list.

[0724] Food packaging photography

[0725] User: Using their smartphone camera, they take a picture of the packaging of a food item they are considering purchasing. They frame the photo so that the entire label is visible.

[0726] Terminal: Temporarily stores the captured image data and prepares a request to send it to the server. When the user presses the send button, the image data is uploaded to the server.

[0727] Parsing package information

[0728] Server: Receives image data sent from the device. The received data is input into an optical character recognition (OCR) engine for image analysis. During this process, text information is extracted from the image.

[0729] Server: Preprocesses the text data extracted by the OCR engine, including noise removal and text normalization. For example, ingredients such as "sugar" and "dairy" are listed.

[0730] Database Matching

[0731] Server: Compares the preprocessed text data with the allergy information stored in the database, and verifies that the allergens registered by the user in advance match the analyzed ingredient list.

[0732] Notification of judgment results and emotion engine

[0733] Server: Makes a decision based on the match. If a matching allergen is found, identifies the allergen and adds it to the list. Generates a warning message and prepares a notification to the user.

[0734] Device: Notify the user of the received test result by displaying a message such as "This product contains milk ingredients" in a pop-up or notification bar within the app.

[0735] Emotion engine: Collects user reactions through cameras and sensors and analyzes the user's emotional state. For example, it can read emotions from facial expressions and tone of voice after a user sees a warning message.

[0736] On the device: If the emotion engine analyzes the user's emotional state and determines they are anxious or stressed, the display of the warning message can be adjusted, for example, by making the message more friendly or by providing additional support information.

[0737] Data accumulation

[0738] Server: Each judgment result and the emotion engine's analysis results are stored in a database. This allows for a quick response the next time the same product is analyzed. Emotion data is also stored, and the server learns how to display the optimal warning message for each user.

[0739] Specific examples

[0740] Let us consider the case where a user wants to purchase "chocolate." The system operates as follows:

[0741] 1. User: Registers in the app that they are allergic to wheat, peanuts, and dairy products.

[0742] 2. Device: Stores allergy information locally and sends it to the server.

[0743] 3. Server: Stores allergy information in a database.

[0744] 4. User: Takes a photo of the chocolate packaging with their smartphone.

[0745] 5. Terminal: Sends image data to the server.

[0746] 6. Server: Performs image analysis and extracts text information, such as "cocoa, sugar, milk components, lecithin (derived from soybeans)."

[0747] 7. Server: Checks the ingredients list and identifies that it contains the user's allergic ingredient, "dairy."

[0748] 8. Server: Sends the judgment result to the terminal and generates a warning message.

[0749] 9. Device: Display a warning to the user that "This product contains milk ingredients."

[0750] 10. Emotion Engine: Detects the user's reaction to receiving a warning message and analyzes their emotional state.

[0751] 11. Device: Based on the results of the emotion engine, adjust the message display as needed (e.g., add a reassuring message).

[0752] 12. Server: Stores the judgment results and emotion data in a database.

[0753] In this way, the present invention provides fast and efficient assistance to users with food allergies in making safe food choices, while also providing more appropriate warnings that take into account the user's emotional state.

[0754] The processing flow will be explained below.

[0755] Step 1:

[0756] User: Start the app and open the "Register Allergy Information" screen from the settings menu. Select and enter the ingredients to which you are allergic (e.g., "wheat," "peanuts," "dairy products") from the list.

[0757] Step 2:

[0758] Terminal: Stores the entered allergy information in a local database, prepares the data for synchronization with the server, and creates a transmission request.

[0759] Step 3:

[0760] Server: Receives allergy information sent from the device and stores it in a user-specific allergy database. This data is later used to match food ingredients against a food ingredient list.

[0761] Step 4:

[0762] User: Uses the smartphone camera to take a picture of the packaging of a food item they are considering purchasing, adjusting the camera so that the entire package is visible within the frame.

[0763] Step 5:

[0764] Terminal: Temporarily stores the captured image data and prepares a request to send it to the server. When the user presses the send button, the image data is uploaded to the server.

[0765] Step 6:

[0766] Server: Receives image data sent from the device. The received image data is input into the OCR engine and text information within the image is extracted. During this process, preprocessing is performed to remove noise and improve character recognition accuracy.

[0767] Step 7:

[0768] Server: Preprocesses the extracted text data and normalizes it into an ingredient list, for example, listing ingredients such as "sugar" and "dairy."

[0769] Step 8:

[0770] Server: Compares the normalized ingredient list with the user's allergy database. Checks for a match against the user's allergy ingredients.

[0771] Step 9:

[0772] Server: Based on the matching result, if there is a matching allergy ingredient, identify the ingredient and add it to the list. Generate a warning message and prepare the content to notify the user.

[0773] Step 10:

[0774] Server: Sends the judgment result and warning message to the user's device. The judgment result includes the identified allergen and the warning message.

[0775] Step 11:

[0776] Device: Notify the user of the received test result by displaying a message such as "This product contains milk ingredients" in a pop-up or notification bar within the app.

[0777] Step 12:

[0778] User: Checks the warning message. Their facial expressions, tone of voice, and other reactions are collected by the emotion engine.

[0779] Step 13:

[0780] On the device: The emotion engine analyzes the user's reactions and determines their emotional state (e.g., anxiety, surprise, relief, etc.), for example by analyzing the user's facial expressions and tone of voice via a camera or microphone.

[0781] Step 14:

[0782] On the device: If the emotion engine determines that the user's emotional state indicates anxiety or stress, the display of the warning message will be adjusted to be more friendly and provide additional reassurance or support information.

[0783] Step 15:

[0784] Server: The judgment results and the emotion engine analysis results are stored in a database for future reference and system improvement.

[0785] Step 16:

[0786] Server: Based on the information accumulated in the database, the server learns to provide appropriate warning messages that take the user's emotional state into account from the next time onwards.

[0787] These are the specific processing steps and operations of the system that combines the emotion engine. This not only allows users to select foods with peace of mind, but also provides appropriate support that takes into account their emotional state.

[0788] Example 2

[0789] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0790] Existing food allergy prevention systems provide basic functions to reduce the risk of users purchasing foods containing allergens, but they do not provide warning messages that take into account the user's emotional state. This leaves users with a problem: the anxiety and stress they feel when receiving a warning message is not alleviated, making it difficult for them to make safe and secure food choices.

[0791] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for using optical character recognition technology to extract text from an image, means for comparing the extracted ingredient list with an allergy database to determine whether or not an allergic ingredient is present, means for analyzing the user's emotional state, and means for adjusting the display method of the warning message based on the user's emotional state. This not only enables the user to safely select foods to which they have allergies, but also gives them a sense of security when receiving a warning message, thereby reducing stress.

[0792] "User" refers to an individual who uses the system, specifically someone who inputs food allergy information and makes safe food choices.

[0793] "Allergy information" is data about allergens that the user knows about, and the system uses this information to determine the safety of food.

[0794] "Food packaging" refers to packaging material on which food product information is printed, and is the object that a user photographs using a camera.

[0795] "Server" refers to a computer device that receives and processes data sent from user terminals and performs database management and analysis.

[0796] "Image analysis" refers to the process of extracting information from photographed images of food packaging and recognizing it as text data.

[0797] "Optical character recognition technology" refers to technology that digitizes and recognizes characters in an image and extracts them as text data.

[0798] An "ingredient list" is a list of ingredients contained in a food product, and is used to check whether it contains specific allergens.

[0799] "Allergy database" refers to a database that stores and manages allergy information registered by users.

[0800] "Determination" refers to the process of checking the ingredient list against an allergy database to determine whether it contains any ingredients that may be dangerous to the user.

[0801] "Emotional state" refers to the psychological reaction (e.g., anxiety, relief, stress) of the user when they receive a warning message, and is analyzed by the emotion engine.

[0802] "Emotion engine" refers to software or hardware for detecting and analyzing a user's emotional state.

[0803] A "warning message" is a notification that informs the user that a particular food contains an allergic ingredient, and is used to ensure the user's safety.

[0804] "Database" refers to an electronic information storage medium that organizes large amounts of data processed by a system and efficiently stores, searches, and manages them.

[0805] "Adjustment" refers to the act of providing a more appropriate user experience by changing the content and display method of a warning message based on the user's emotional state.

[0806] This invention provides a system for safely purchasing and consuming food for users with food allergies, and offers a new approach that takes into account the emotional state of the user. The system has specific roles for the user, the terminal, and the server.

[0807] Registering allergy information

[0808] User: Launch the app and open the "Register Allergy Information" screen from the settings menu. The user selects and enters the ingredients to which they are allergic (e.g., "wheat," "peanuts," "dairy products") from a list.

[0809] Terminal: Save the allergy information entered by the user to a local database. After saving, generate JSON data to be synchronized with the server and create a send request.

[0810] Server: Receives allergy information sent from the device and stores it in a personal allergy database. This data is used to match food ingredients with the food's ingredient list in a later step.

[0811] Food packaging photography

[0812] User: Use your smartphone camera to take a photo of the entire package of a food item you are considering purchasing, using the frame guidelines to guide you in the best position to take the photo.

[0813] Device: The captured image data is temporarily stored in local storage, and a thumbnail is displayed for the user to confirm. When the user confirms and presses the send button, a request to send the image data to the server is prepared.

[0814] Parsing package information

[0815] Server: Receives image data sent from the device. Inputs the received data into an OCR (Optical Character Recognition) engine. The OCR engine extracts text information from the image.

[0816] Server: Preprocesses the text data extracted by the OCR engine, including noise reduction and text normalization, such as removing symbols and unnecessary spaces, and listing important ingredient information (e.g., "sugar," "dairy").

[0817] Database Matching

[0818] Server: Matches the preprocessed text data with the allergy information stored in the database. It uses SQL queries to match the allergens registered by the user with the extracted ingredient list.

[0819] Server: If a matching allergen exists, identify and list the ingredient. For example, if "dairy" is included, add this information to the warning list.

[0820] Notification of judgment results and emotion engine

[0821] Server: Generates a warning message based on the matching result. For example, if the ingredient to which the user is allergic is "dairy products," prepare a warning message stating "This product contains dairy ingredients."

[0822] Device: Notify the user of received alert messages. Display alert messages within the app using a pop-up window or notification bar.

[0823] Emotion engine: The system uses camera and microphone sensors to collect data on how users view warning messages. For example, it analyzes changes in facial expressions and speech in real time the moment the user sees the message.

[0824] On the device: Based on the results of the emotion engine, the message display can be adjusted depending on the user's emotional state. For example, if the user expresses anxiety, more friendly language or additional support information can be displayed.

[0825] Data accumulation

[0826] Server: Stores each judgment result and the emotion engine's analysis results in a database. This allows for quick response when the same product is scanned in the future. It also continuously learns how to display the most appropriate warning message to the user based on emotion data.

[0827] Specific examples

[0828] As an example, a case where a user wants to purchase "chocolate" will be described.

[0829] 1. User: Registers in the app that they have allergies to "wheat," "peanuts," and "dairy products." The user opens the "Register Allergy Information" screen from the settings menu, selects each item, and confirms.

[0830] 2. Terminal: Stores this information in a local database, packages the allergy information in JSON format, and generates a request to send to the server.

[0831] 3. Server: Receives the submitted allergy information and stores it in a personal allergy database for each user. This information is stored in an SQL database and saved in a format that can be easily verified later.

[0832] 4. User: Use the smartphone camera to take a photo of the entire chocolate package, following the photography guidelines and centering the food label.

[0833] 5. Terminal: Displays thumbnails of the captured images for the user to check. When the user checks the image and presses the send button, a request to send the image data to the server is prepared.

[0834] 6. Server: Passes the received image data to the OCR engine, which extracts text information such as "cocoa, sugar, milk components, lecithin (soybean-derived)" from the image.

[0835] 7. Server: Preprocesses the text data, removing noise and normalizing the necessary raw material information into a list.

[0836] 8. Server: Checks the normalized ingredient list against the user's allergy database, using an SQL query to ensure "dairy" is included.

[0837] 9. Server: Based on the matching results, a warning message such as "This product contains milk ingredients" is generated and sent to the terminal.

[0838] 10. Terminal: Displays a popup to alert the user of the received message.

[0839] 11. Emotion Engine: User reactions are collected using cameras and sensors and emotion analysis is performed. If the user shows surprise or anxiety in response to a displayed message, the emotional data is analyzed.

[0840] 12. On the device: Based on the results of the emotion engine, an additional message is displayed to reassure the user. For example, a friendly message such as "Please avoid purchasing this product as it contains dairy ingredients" is displayed.

[0841] 13. Server: The final judgment result and the user's emotional data are stored in a database, which will enable faster and more appropriate responses when the same situation occurs in the future.

[0842] In this way, the present invention helps users with food allergies make safe food choices and further enhances the user experience by adjusting the display of warning messages taking into account the user's emotional state in real time.

[0843] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0844] Step 1: The user launches the app and opens the "Register Allergy Information" screen from the settings menu. The user selects and enters the ingredients to which they are allergic (e.g., "wheat," "peanuts," "dairy products") from a list. The entered allergy information is saved in JSON format on the device. This becomes the input data.

[0845] Step 2: The device generates a request to send the saved JSON data to the server. Specifically, the created JSON data is sent to the server as an HTTP POST request. The input is the allergy information data in JSON format, and the output is the request sent to the server.

[0846] Step 3: The server parses the received allergy information and stores it in a per-user allergy database. Specifically, it executes a SQL query to parse the received data and insert it into the database. The input is the JSON-formatted allergy information sent to the server, and the output is the updated database state.

[0847] Step 4: The user uses the smartphone camera to take a picture of the entire food package they are considering purchasing. Guidelines are displayed to guide the user in taking the picture so that the entire label fits in the picture. This results in a captured image. The input is a physical image of the food package, and the output is a captured digital image.

[0848] Step 5: The device temporarily saves the captured image in local storage, generates a thumbnail, and asks the user to confirm it. When the user confirms and presses the send button, a request is generated to send the image data to the server. The input is the captured digital image, and the output is an image data request sent to the server.

[0849] Step 6: The server inputs the received image data into the OCR engine, which extracts the text information in the image and stores it as text data. Specifically, it performs binary analysis on the image data and applies a character recognition algorithm. The input is the digital image sent to the server, and the output is the extracted text data.

[0850] Step 7: The server preprocesses the text data extracted by the OCR engine. This includes noise removal and normalization. Specifically, it removes symbols and unnecessary spaces, extracts important keywords, and compiles them into a list. The input is the extracted text data, and the output is a preprocessed raw material list.

[0851] Step 8: The server compares the preprocessed text data with the allergy information stored in the database. Specifically, it executes an SQL query to retrieve the user's allergy ingredients from the database and compare them with the ingredient list. The input is the preprocessed ingredient list and the allergy information in the database, and the output is the comparison result.

[0852] Step 9: The server generates a warning message based on the matching result. For example, if the ingredient "dairy" is matched, it prepares a warning message saying "This product contains dairy ingredients." The input is the matching result, and the output is the generated warning message.

[0853] Step 10: The device notifies the user of the received warning message. Specifically, it displays the message in the app using a popup or notification bar. The input is the warning message, and the output is the notification displayed to the user.

[0854] Step 11: The emotion engine collects and analyzes the user's reaction to the warning message through the camera and microphone sensors. It analyzes the user's facial expressions and tone of voice to evaluate their emotional state in real time. The input is the user's reaction data, and the output is the analyzed emotional state.

[0855] Step 12: The device adjusts the display of the warning message based on the results of the emotion engine. Specifically, it provides appropriate information to the user by adding reassuring and friendly language. The input is the analyzed emotional state, and the output is the adjusted warning message.

[0856] Step 13: The server saves the final judgment result and the user's emotion data in the database. This allows for quick response when the same product is scanned in the future and enables learning based on the emotion data. The input is the final judgment result and emotion data, and the output is the updated database.

[0857] (Application example 2)

[0858] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0859] Previously, there were systems that allowed users to register allergy information and check food ingredients. However, these systems did not take into account the user's emotional state and lacked measures to address the anxiety and stress users felt when choosing foods containing allergens. This limited the effectiveness of these systems in increasing users' sense of security and created a psychological burden when choosing food.

[0860] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting and analyzing the user's emotional state, means for adjusting the display method of the warning message based on the analyzed emotional state, and means for storing the determination result and the emotional state in a database. This makes it possible to provide flexible warning messages that take the user's emotional state into consideration.

[0861] "User" refers to an individual who uses the system to register their own allergy information and confirm food safety.

[0862] "Allergy information" is data registered by users regarding their allergens. This information is used to confirm food safety when selecting foods.

[0863] "Food packaging" refers to materials such as paper, plastic, and metal used to pack or wrap food, and which bear the name of the food, as well as information about its ingredients and components.

[0864] "Means for taking photographs" refers to a function that allows a user to take a photograph of a food package using a smartphone or camera.

[0865] A "server" is a computer system that receives, analyzes, and stores data sent by users.

[0866] The "means for transmitting images" is a function that allows a user to upload an image of a food package taken by the user to a server.

[0867] The "means for the server to analyze the image" refers to the function of analyzing the received image data and extracting the list of ingredients for the food. Optical character recognition (OCR) technology is generally used.

[0868] A "food ingredient list" is a list of ingredient names listed on a food package.

[0869] The "allergy database" is a collection of numerical information that stores allergy ingredient information registered by the user.

[0870] The "means for determining" is a function that checks the extracted ingredient list against an allergy database to determine whether or not an allergic ingredient is present.

[0871] "Means for notifying" refers to a function for displaying or sending a message to inform the user of the analysis results.

[0872] The "user's emotional state" refers to the user's psychological reaction or emotions when seeing food containing an allergic ingredient.

[0873] "Means of collection and analysis" refers to the function of collecting the user's emotional state through cameras and sensors and analyzing it using dedicated software and algorithms.

[0874] The "means for adjusting the display method of the warning message" is a function for changing the content and display method of the warning message depending on the emotional state of the user.

[0875] "Means for storing the judgment results and emotional state in a database" is a function that stores the analysis results and the user's emotional data in a database and makes them useful for future analysis and responses.

[0876] This system allows users to register allergy information, photograph food packages, and analyze their ingredients to help them choose foods safely. It also collects and analyzes the user's emotional state and provides appropriate messages to enhance the user's sense of security.

[0877] 1. Registering allergy information

[0878] The user starts the application and opens the "Register Allergy Information" screen from the settings menu. Here, the user selects and enters the ingredients to which they are allergic (e.g., "wheat," "peanuts," "dairy products") from a list. The device saves this allergy information locally and then sends the data to the cloud server.

[0879] 2. Take a photo of the food package and send it

[0880] A user takes a photo of the packaging of a food item they are considering purchasing in a physical store with their smartphone camera. The device temporarily stores the captured image data and prepares a request to send it to the server. When the user presses the send button, the image is uploaded to the cloud server.

[0881] 3. Package information analysis

[0882] The server uses optical character recognition (OCR) technology to extract text information from the received image data. This is done using Google Cloud Vision API, among other tools. The server then preprocesses the extracted text data and lists the ingredients. For example, ingredients such as "cocoa, sugar, milk components, lecithin (soybean-derived)" are identified here.

[0883] 4. Database Matching

[0884] The server compares the preprocessed text data with the allergy information stored in the database, checks whether the allergens registered by the user beforehand match the analyzed ingredient list, and generates a warning message if there is a match.

[0885] 5. Notification of judgment results and emotion analysis

[0886] The device will then notify the user of a warning message, such as "This product contains milk ingredients." The device will then collect the user's reactions through its camera and sensors and analyze their emotional state. This analysis utilizes Azure Emotion API and other tools. Based on the analysis results, if the user's emotional state indicates anxiety or stress, the server will adjust the message display and provide additional support information.

[0887] 6. Data accumulation

[0888] The server stores the results of each judgment and the emotion engine's analysis in a database. This allows for a quick response the next time the same product is analyzed. Emotion data is also stored, and the server learns how to display the optimal warning message for each user.

[0889] Specific examples

[0890] Let's consider the case where a user is about to purchase "chocolate." First, the user registers in the app that they are allergic to "wheat," "peanuts," and "dairy products." The device saves the allergy information locally and sends it to the server. The server saves the allergy information in a database. The user takes a photo of the "chocolate" packaging with their smartphone, and the device sends the image data to the server. The server analyzes the image and extracts text information. Ingredients such as "cocoa, sugar, milk ingredients, and lecithin (soybean-derived)" are extracted, and the information is then compared with the ingredient list to determine that the product contains "dairy products," the user's allergic ingredient. The server sends the result of the assessment to the device and generates a warning message. The device displays a warning to the user stating "This product contains milk ingredients," and the emotion engine detects the user's reaction and analyzes their emotional state. The device adjusts the message display as necessary, and the server saves the assessment result and emotion data in a database.

[0891] Prompt Sentence Examples

[0892] "Just enter your allergy information (e.g., wheat, peanuts, dairy) and take a photo of a peanut butter package. The system will detect the allergen and notify you. It will also provide additional support information as needed based on your emotional state."

[0893] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0894] Step 1: Register your allergy information

[0895] The user starts the application and opens the "Register Allergy Information" screen. As input, the user selects and enters the ingredients to which they are allergic (e.g., "wheat," "peanuts," "dairy products") from a list. The device saves this allergy information in local storage and generates a request to send to the server. The input data is the user's allergy information, and the output data is the data request sent to the server. Specifically, the device sends the data to the server using a REST API.

[0896] Step 2: Photograph and send the food package

[0897] A user takes a photo of the packaging of a food item they are considering purchasing with their smartphone camera. The input is the food packaging to be photographed, and the output is the captured image data. The device temporarily stores this image data and prepares a request to send to the server. Specifically, when the user presses the send button, the device uploads the image data to the server. An HTTP POST request is used to upload the image data.

[0898] Step 3: Parse package information

[0899] The server extracts text information from the received image data using optical character recognition (OCR) technology. The input is image data and the output is extracted text data. The OCR engine uses services such as Google Cloud Vision API. Specifically, the server sends data to the OCR engine and receives text data as the analysis result.

[0900] Step 4: Check against the database

[0901] The server compares the preprocessed text data with the allergy database. The input is the extracted text data and the user's allergy information, and the output is the comparison result. The comparison process identifies ingredients that match the allergens in the database. Specifically, the text data is used as a search query to query the database to see if there are any matching ingredients.

[0902] Step 5: Notification of decision

[0903] The server generates a warning message based on the matching result and sends it to the terminal. The input is the matching result and the output is the warning message. Specifically, the server analyzes the matching result and generates a warning message if the product contains an ingredient to which the user is allergic. The terminal displays this message to the user via a pop-up or notification bar.

[0904] Step 6: Sentiment Analysis

[0905] The device collects user reactions through cameras and sensors and analyzes their emotional state. The input is user reaction data, and the output is emotional state data. The emotion engine uses the Azure Emotion API, etc. Specifically, it sends reaction data to the emotion engine and receives the emotional state as the analysis result.

[0906] Step 7: Adjust message display

[0907] The server adjusts the display of warning messages based on the emotional state. The input is the emotional state data and the current warning message, and the output is the adjusted warning message. Specifically, it analyzes the emotional state data and provides friendly messages or additional support information as needed.

[0908] Step 8: Accumulate data

[0909] The server accumulates the judgment results and emotion analysis results in a database. The input is the judgment results and emotional state data, and the output is an updated database. Specifically, the server saves this data in the database and makes it available for future analysis.

[0910] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0911] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0912] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0913] [Third embodiment]

[0914] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0915] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0916] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0917] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0918] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0919] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0920] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0921] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0922] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0923] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0924] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0925] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0926] The present invention is a system that enables users with food allergies to safely purchase food and enjoy their meals, and is realized through a series of processes: registering allergy information, photographing food packaging, analyzing the images, extracting and comparing ingredient lists, notifying the user of the results, and storing the data. Specific embodiments of the system are described below.

[0927] Registering allergy information

[0928] User: Launches the app and opens the user profile settings screen. The user enters their allergy information (e.g., "wheat," "peanuts," "dairy," etc.). This information can be added, deleted, or edited at any time from the application's settings menu.

[0929] Terminal: The entered allergy information is saved in a local database and the data is prepared for transmission to the server. The following is the process for registering the user's allergy information.

[0930] Server: Receives allergy information sent from the device and stores it in a user-specific allergy database. This data is later used to match food ingredients against a food ingredient list.

[0931] Food packaging photography

[0932] User: Take a photo of the packaging of the food item you are considering purchasing using your smartphone camera. When taking the photo, make sure to adjust it so that the entire package is clearly visible.

[0933] Terminal: Temporarily stores the captured image data and prepares a request to send it to the server. After the user presses the capture button, the image is immediately uploaded to the server.

[0934] Parsing package information

[0935] Server: The server analyzes the received image data. As a first step in the analysis, it uses optical character recognition (OCR) technology to extract text from the image. The extracted text is then processed into a list of raw materials. During this process, preprocessing is performed to remove noise and improve character recognition accuracy.

[0936] Server: Normalizes the extracted ingredient list to match a standard format in the database, and translates it if necessary based on specific language codes.

[0937] Database Matching

[0938] Server: Compares the normalized ingredient list with the user's allergy database. Checks whether the ingredients match the allergy information in the database. If a match is found, identifies the ingredient and passes the result on to further processes.

[0939] Notification of the results

[0940] Server: Generates a warning message if allergens are identified. The result includes a list of identified allergens.

[0941] Server: Sends the result of the judgment to the user's device. A notification message is generated so that the user can view it.

[0942] Device: Receives the sent judgment result and displays it to the user as an alert or pop-up message within the app, allowing the user to make safe food choices.

[0943] Data accumulation

[0944] Server: Each judgment result is stored in a database. This allows the system to provide quick results based on existing data the next time the same food is analyzed. The stored data is used to improve the efficiency of the entire system.

[0945] Specific examples

[0946] A case where a user checks a package of "chocolate" will be described.

[0947] 1. User: Registers in the app that they are allergic to wheat, peanuts, and dairy products.

[0948] 2. Device: Stores allergy information locally and sends it to the server.

[0949] 3. Server: Stores allergy information in a database.

[0950] 4. User: Takes a photo of the chocolate packaging with their smartphone.

[0951] 5. Terminal: Sends image data to the server.

[0952] 6. Server: Performs image analysis and extracts text information, such as "cocoa, sugar, milk components, lecithin (derived from soybeans)."

[0953] 7. Server: Checks the ingredients list and identifies that it contains the user's allergic ingredient, "dairy."

[0954] 8. Server: Sends the judgment result to the terminal and generates a warning message.

[0955] 9. Device: Display a warning to the user that "This product contains milk ingredients."

[0956] 10. Server: Stores the judgment results in a database.

[0957] Thus, the present invention provides fast and efficient assistance to users with food allergies in making safe food choices.

[0958] The processing flow will be explained below.

[0959] Step 1:

[0960] User: Launch the app and open the "Register Allergy Information" screen from the settings menu. The user selects and enters the ingredients they are allergic to (e.g., "wheat," "peanuts," "dairy products") from a list.

[0961] Step 2:

[0962] Terminal: Stores the entered allergy information in a local database, prepares the data for synchronization with the server, and creates a transmission request.

[0963] Step 3:

[0964] Server: Receives allergy information sent from the device. Stores the received data in a user-specific allergy database. At this stage, the data is formatted for matching.

[0965] Step 4:

[0966] User: Using their smartphone camera, they take a picture of the packaging of a food item they are considering purchasing. They frame the photo so that the entire label is visible.

[0967] Step 5:

[0968] Terminal: Temporarily stores the captured image and prepares a request to send it to the server. When the user presses the send button, the image data is uploaded to the server.

[0969] Step 6:

[0970] Server: Receives image data sent from the device. The received data is input into an optical character recognition (OCR) engine for image analysis. During this process, text information is extracted from the image.

[0971] Step 7:

[0972] Server: Preprocesses the text data extracted by the OCR engine, including noise removal and text normalization. For example, ingredients such as "sugar" and "dairy" are listed.

[0973] Step 8:

[0974] Server: Compares the preprocessed text data with the allergy information stored in the database, and verifies that the allergens registered by the user in advance match the analyzed ingredient list.

[0975] Step 9:

[0976] Server: Makes a decision based on the match. If a matching allergen is found, identifies the allergen and adds it to the list. Generates a warning message and prepares a notification to the user.

[0977] Step 10:

[0978] Server: Sends the judgment result to the user's device, which includes the identified allergens and a warning message.

[0979] Step 11:

[0980] Device: Notify the user of the received test result by displaying a message such as "This product contains milk ingredients" in a pop-up or notification bar within the app.

[0981] Step 12:

[0982] Server: Each judgment result is stored in a database. This allows for a quick response the next time the same product is analyzed. The data is saved for future reference and analysis, contributing to the efficiency of the entire system.

[0983] The above are the specific processing steps and operations of the proposed system.

[0984] Example 1

[0985] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0986] For consumers with food allergies, the process of checking food safety when making everyday food purchases is extremely cumbersome. Manually checking ingredient lists is time-consuming and laborious, and there is a risk of overlooking or misreading information. Furthermore, in many markets, product information is displayed in different languages, which can be difficult to understand. Therefore, there is a need for a system that allows users with food allergies to efficiently and accurately check food safety and purchase and consume with peace of mind.

[0987] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0988] In this invention, the server includes means for a user to input their own allergy information, means for photographing a food package, means for transmitting the image to the server, means for the server to analyze the image and extract a list of ingredients for the food, means for comparing the extracted ingredient list with an allergy database to determine whether or not an allergic ingredient is present, means for notifying the user of the determination result, means for storing the determination result in a database, means for extracting text from the image using optical character recognition technology, means for temporarily saving image data photographed by the user, means for preparing data for the user to transmit their allergy information to the server, means for saving the user's allergy information, and means for displaying the determination result within the app. This allows users to efficiently and accurately confirm the safety of foods and purchase or consume them with peace of mind.

[0989] "Users" are consumers who have food allergies and want to make safe food choices.

[0990] "Allergy information" is a list of ingredients that cause allergic reactions in specific foods, and is entered by the user.

[0991] "Food packaging" refers to the packaging or label that lists the product information and ingredients of a food product.

[0992] "Server" means a central computer system that receives and processes data submitted by users.

[0993] "Image" refers to photographic data of food packaging taken with a smartphone or other device.

[0994] "Optical character recognition (OCR) technology" is a technology for recognizing and extracting text data from images.

[0995] A "database" is a system for structuring and storing data such as user allergy information and food ingredient lists.

[0996] An "ingredient list" is a list of the ingredients and materials contained in a food product, and is text data extracted from the packaging.

[0997] "Notification" refers to a message or warning that the server sends to the user to inform them of the judgment result.

[0998] An "alert" is a warning message that is displayed to prompt the user to take a specific action.

[0999] A "pop-up message" is a message that appears temporarily on the screen to attract the user's attention.

[1000] The "determination result" refers to the result of determining whether or not an allergy ingredient is present based on a comparison between the extracted ingredient list and the user's allergy database.

[1001] The "user profile setting screen" is the screen within the app where users can enter, edit, and save their personal information and allergy information.

[1002] The present invention provides a system that enables users with food allergies to safely purchase food and enjoy meals. This system executes a series of processes using the following hardware and software:

[1003] Hardware

[1004] Device: A mobile information device such as a smartphone or tablet.

[1005] Server: A centralized computer system.

[1006] software

[1007] Local database: Manage user allergy information stored on the device using a lightweight database such as SQLite.

[1008] Optical Character Recognition (OCR) technology: Using libraries such as OpenCV and Tesseract, the server extracts text from images.

[1009] Translation API: Using the Google Cloud Translation API or similar, translate the ingredient list into a user-understandable language as needed.

[1010] Notification system: The result of the judgment is notified to the user using GCM (Google Cloud Messaging) or APNs (Apple Push Notification service).

[1011] Operating Procedures and Processes

[1012] 1. Allergy Information Registration:

[1013] User: Launch the app and open the user profile settings screen. Enter your allergy information (e.g., "wheat," "peanuts," "dairy products," etc.).

[1014] Terminal: Stores the entered allergy information in a local database and prepares the data for transmission to the server.

[1015] Server: Receives allergy information sent from the device and stores it in an allergy database for each user.

[1016] 2. Food packaging photography:

[1017] User: Take a photo of the packaging of a food item they are considering purchasing with their smartphone camera, adjusting the camera so that the entire package is clearly visible.

[1018] Terminal: Temporarily stores the captured image data and prepares a request to send it to the server.

[1019] 3. Parse package information:

[1020] Server: Analyzes the received image data and extracts the text within the image using optical character recognition (OCR) technology.

[1021] Specific examples

[1022] A case where a user checks a package of "chocolate" will be described.

[1023] 1. User: Registers in the app that they are allergic to wheat, peanuts, and dairy products.

[1024] 2. Device: Stores allergy information locally and sends it to the server.

[1025] 3. Server: Stores allergy information in a database.

[1026] 4. User: Takes a photo of the chocolate packaging with their smartphone.

[1027] 5. Terminal: Sends image data to the server.

[1028] 6. Server: Performs image analysis and extracts text information. For example, "cocoa, sugar, milk components, lecithin (derived from soybeans)" is extracted.

[1029] 7. Server: Checks the ingredients list and identifies that it contains the user's allergic ingredient, "dairy."

[1030] 8. Server: Sends the judgment result to the terminal and generates a warning message.

[1031] 9. Device: Display a warning to the user that "This product contains milk ingredients."

[1032] Examples of prompt statements

[1033] "Open your user profile settings screen and enter your allergy information."

[1034] "Take a photo of the packaging of the food you're considering purchasing."

[1035] "The captured image data is being sent to the server. Please wait."

[1036] "The ingredients list has been analyzed. We are currently reviewing the results."

[1037] "Avoid this food because it contains dairy."

[1038] This system provides support to users with food allergies by efficiently and accurately checking food safety and enabling them to select foods with peace of mind.

[1039] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1040] Step 1:

[1041] User: Launches the app and opens the user profile setting screen. The user enters their allergy information (e.g., "wheat," "peanuts," "dairy products," etc.). This is the input data.

[1042] Terminal: The entered allergy information is saved in a local database (e.g., SQLite). Data calculations at this stage involve converting the information entered by the user into a database format and saving it. Data is also prepared for sending to the server. Specific operations include writing to the database and converting the data format for sending to the server. The output is the allergy information saved locally and the data to be sent to the server.

[1043] Server: Receives allergy information sent from the device. Stores the received data in an allergy database for each user. Specifically, the data is written to the allergy database (e.g., MySQL or PostgreSQL). This is the output data.

[1044] Step 2:

[1045] User: Take a photo of the packaging of a food item they are considering purchasing with their smartphone camera. When taking the photo, they need to adjust the camera so that the entire package is clearly visible. This is the input data.

[1046] Terminal: Temporarily stores the captured image data. Data calculation at this stage involves converting the image data into an appropriate format (e.g., JPEG or PNG) and saving it. After the user presses the capture button, a request is immediately prepared to upload the image to the server. The specific operation is to convert the image data for network transmission. The output is image data ready for transmission to the server.

[1047] Step 3:

[1048] Server: The server receives the image data sent from the terminal. It analyzes the received image data and extracts the text within the image using optical character recognition (OCR) technology (e.g., Tesseract). This is the input data. Data processing at this stage involves converting the image data into text data, including noise removal and image preprocessing. The output is the extracted text data of the ingredient list.

[1049] Step 4:

[1050] Server: The extracted text data is normalized and translated into a specific language using a translation API (e.g., Google Cloud Translation API) if necessary to conform to a standard format. This is the input data. Data operations at this stage involve cleaning and normalizing the text data, and translating it if necessary. The output is the normalized and translated text data of the ingredient list.

[1051] Step 5:

[1052] Server: Compares the normalized ingredient list with the user's allergy database. The specific operation is to compare the allergy information in the database with the ingredient list. This is the input data. The data operation at this stage is to identify matching ingredients using a comparison algorithm. The output is a judgment result on the presence or absence of allergens.

[1053] Step 6:

[1054] Server: If an allergen is identified, a warning message is generated. This is the input data. Data calculation at this stage generates a warning message based on the judgment result. The output is a warning message and the judgment result.

[1055] Server: Sends the result of the judgment to the user's device. Specifically, it sends a message in real time via a notification system such as GCM or APNs. This is the input data. The output is the sent notification message.

[1056] Device: Receives the sent judgment result and displays it to the user as an alert or pop-up message within the app. The specific behavior is to display a notification on the device's user interface. This is the input data, and the output is the displayed warning message.

[1057] Step 7:

[1058] Server: Each decision result is stored in a database. This is the input data. Data calculations at this stage are to save the decision results in a database (e.g. MySQL or PostgreSQL). The output is the updated database.

[1059] The above are the specific processing steps in the system of the present invention, which allow users with food allergies to select foods efficiently and safely.

[1060] (Application example 1)

[1061] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1062] When consumers with food allergies purchase food at physical stores, they need a method to quickly and accurately identify allergens and safely select foods. Conventional methods often require complicated and time-consuming food safety checks, making it difficult to ensure consumer peace of mind.

[1063] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1064] In this invention, the server includes means for a user to input their own allergy information, means for photographing the food package, means for transmitting the image to the server, means for the server to analyze the image and extract a list of the food's ingredients, means for comparing the extracted ingredient list with an allergy database to determine whether or not an allergen is present, means for notifying the user of the determination result, means for storing the determination result in the database, and means for displaying the allergen determination result in real time using a smart device at a physical store. This allows users to instantly check whether or not a food product contains an allergen by scanning the food package at the physical store, enabling them to select safe foods.

[1065] "User" refers to an individual who uses the system and is the entity that inputs allergy information and photographs food packaging.

[1066] "Allergy information" refers to information that indicates the specific ingredient names and food names for which the user has a food allergy.

[1067] "Food packaging" refers to the packaging in which food is packaged, and is the target from which ingredient information written inside is extracted.

[1068] "Smart devices" refer to electronic devices that can be carried by users, such as smartphones, smart glasses, and head-mounted displays.

[1069] "Server" refers to a computer system that processes information for the entire system, and performs image analysis and database matching.

[1070] "Image analysis" is the process of processing images of food packaging to extract useful textual information from them.

[1071] "Optical Character Recognition (OCR) technology" is a technology that recognizes characters from images or printed text as electronic data.

[1072] "Allergy database" refers to a database that stores the user's allergy information and is used for matching purposes.

[1073] The "determination result" is a conclusion as to whether or not an allergen is present, obtained by comparing the extracted ingredient list with the allergy database.

[1074] "Notification" refers to the process of communicating the results of a determination to the user, which is done through a warning message or a pop-up message.

[1075] MODE FOR CARRYING OUT THE INVENTION

[1076] This invention is a system that allows users with food allergies to safely select food in physical stores. How this system is implemented will be described below in detail.

[1077] System Program Overview

[1078] The server executes a series of processes, including setting up a user profile, photographing the food packaging, analyzing the image, extracting ingredients, comparing it with a database, notifying the user of the results, and storing the results, allowing users to quickly determine the safety of food in physical stores.

[1079] Hardware / Software used

[1080] Hardware:

[1081] Smartphone (Android or iOS)

[1082] server

[1083] software:

[1084] For server-side processing, the Flask framework is used.

[1085] For image analysis, pytesseract is used as an optical character recognition (OCR) technology.

[1086] For image processing, we use PIL, a Python image processing library.

[1087] SQLite is used as the database.

[1088] Data processing and calculation

[1089] User:

[1090] 1. The user launches the app on their smartphone, enters their allergy information (e.g., "wheat," "peanuts," "dairy products," etc.), and registers it.

[1091] 2. The registered allergy information is stored in the smartphone's local database and also sent to the server.

[1092] server:

[1093] 1. The server receives allergy information sent by the user and stores it in a dedicated allergy database.

[1094] 2. When a user takes a photo of a food package in a physical store with their smartphone camera, the image data is sent to the server.

[1095] 3. The server analyzes the received image data using optical character recognition (OCR) technology to extract the ingredient list from the package. This analysis involves preprocessing to remove noise and improve character recognition accuracy.

[1096] 4. The extracted ingredient list is normalized to a standard format and compared with the user's allergy database to determine whether or not an allergen is present.

[1097] Notification of decision:

[1098] 1. The server generates a warning message if an allergen is found. The result includes a list of identified allergens.

[1099] 2. The judgment result is sent to the user's smartphone, and a notification message is displayed so that the user can view it immediately.

[1100] Data accumulation:

[1101] 1. The results of the assessment are stored in a server database. This data is used to speed up future analysis processes.

[1102] Specific examples

[1103] A case will be described where the user wishes to check the packaging of "Chocolate".

[1104] 1. The user registers in a smartphone app that they are allergic to wheat, peanuts, and dairy products.

[1105] 2. The smartphone stores the allergy information locally and sends it to the server.

[1106] 3. The server stores this information in a dedicated database.

[1107] 4. The user takes a photo of the chocolate packaging with their smartphone in a physical store.

[1108] 5. The captured image data is sent to the server.

[1109] 6. The server analyzes the image and extracts text information, such as "cocoa, sugar, milk components, lecithin (derived from soybeans)."

[1110] 7. The extracted ingredient list is compared and it is determined that the product contains the user's allergic ingredient, "dairy products."

[1111] 8. The server sends the result of the judgment to the smartphone and generates a warning message.

[1112] 9. Display a "This product contains dairy" warning on your smartphone.

[1113] Prompt Sentence Examples

[1114] "Create an application that, when a user scans a food package in a physical store with their smartphone camera, analyzes the ingredients contained in the package in real time and compares them with the allergy information registered by the user to detect allergens."

[1115] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1116] Step 1:

[1117] The user launches the app on their smartphone and enters their allergy information (e.g., "wheat," "peanuts," "dairy products," etc.). The entered allergy information is stored in a local database and is also prepared as data to be sent to the server.

[1118] Input: Allergy information entered by the user

[1119] Output: Allergy information stored in the local database and sent to the server

[1120] Step 2:

[1121] The terminal receives the allergy information entered by the user, stores it in a local database, and then transmits it to the server.

[1122] Input: Allergy information entered by the user

[1123] Output: Allergy information sent to the server

[1124] Step 3:

[1125] The server receives the allergy information sent from the terminal and stores the information in an allergy database.

[1126] Input: Allergy information sent from the device

[1127] Output: Allergy information stored in the allergy database

[1128] Step 4:

[1129] The user takes a photo of the packaging of a food item they are considering purchasing at a physical store using their smartphone camera, adjusting the camera so that the entire package is clearly visible.

[1130] Input: Food packaging you're considering buying

[1131] Output: Package image saved on your smartphone

[1132] Step 5:

[1133] The device temporarily stores the captured image data and prepares a request to send it to the server. When the user presses the capture button, the image is immediately uploaded to the server.

[1134] Input: Photographed package image

[1135] Output: Image data sent to the server

[1136] Step 6:

[1137] The server analyzes the received image data, extracts the text within the image using optical character recognition (OCR) technology, and processes it as an ingredient list.

[1138] Input: Received image data

[1139] Output: Extracted ingredient list

[1140] Step 7:

[1141] The server normalizes the extracted ingredient list, matches it with a standard format in an allergy database, and may translate it into specific languages ​​if necessary.

[1142] Input: Extracted ingredient list

[1143] Output: Normalized ingredient list

[1144] Step 8:

[1145] The server checks the normalized ingredient list against the user's allergy database to see if the product contains any allergens.

[1146] Input: Normalized ingredient list, allergy database

[1147] Output: Judgment result regarding the presence or absence of allergens

[1148] Step 9:

[1149] If an allergy ingredient is identified, the server generates a warning message and creates a message to notify the user based on the generated judgment result.

[1150] Input: Judgment result regarding the presence or absence of allergens

[1151] Output: Warning message

[1152] Step 10:

[1153] The device receives the judgment results sent from the server and displays them to the user as an alert within the app, allowing the user to select safe foods.

[1154] Input: Verification result sent from the server

[1155] Output: The alert message that is displayed to the user

[1156] Step 11:

[1157] The server stores each judgment result in a database, which allows it to quickly provide results based on existing data the next time the same food is analyzed.

[1158] Input: Judgment result

[1159] Output: Judgment results stored in the database

[1160] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1161] The present invention provides a new approach to a system that allows users with food allergies to safely purchase food and enjoy meals, taking into account the emotional state of the user. Specific embodiments of this system are described in detail below.

[1162] Registering allergy information

[1163] User: Launch the app and open the "Register Allergy Information" screen from the settings menu. The user selects and enters the ingredients they are allergic to (e.g., "wheat," "peanuts," "dairy products") from a list.

[1164] Terminal: Stores the entered allergy information in a local database, prepares the data for synchronization with the server, and creates a transmission request.

[1165] Server: Receives allergy information sent from the device and stores it in a user-specific allergy database. This data is later used to match food ingredients against a food ingredient list.

[1166] Food packaging photography

[1167] User: Using their smartphone camera, they take a picture of the packaging of a food item they are considering purchasing. They frame the photo so that the entire label is visible.

[1168] Terminal: Temporarily stores the captured image data and prepares a request to send it to the server. When the user presses the send button, the image data is uploaded to the server.

[1169] Parsing package information

[1170] Server: Receives image data sent from the device. The received data is input into an optical character recognition (OCR) engine for image analysis. During this process, text information is extracted from the image.

[1171] Server: Preprocesses the text data extracted by the OCR engine, including noise removal and text normalization. For example, ingredients such as "sugar" and "dairy" are listed.

[1172] Database Matching

[1173] Server: Compares the preprocessed text data with the allergy information stored in the database, and verifies that the allergens registered by the user in advance match the analyzed ingredient list.

[1174] Notification of judgment results and emotion engine

[1175] Server: Makes a decision based on the match. If a matching allergen is found, identifies the allergen and adds it to the list. Generates a warning message and prepares a notification to the user.

[1176] Device: Notify the user of the received test result by displaying a message such as "This product contains milk ingredients" in a pop-up or notification bar within the app.

[1177] Emotion engine: Collects user reactions through cameras and sensors and analyzes the user's emotional state. For example, it can read emotions from facial expressions and tone of voice after a user sees a warning message.

[1178] On the device: If the emotion engine analyzes the user's emotional state and determines they are anxious or stressed, the display of the warning message can be adjusted, for example, by making the message more friendly or by providing additional support information.

[1179] Data accumulation

[1180] Server: Each judgment result and the emotion engine's analysis results are stored in a database. This allows for a quick response the next time the same product is analyzed. Emotion data is also stored, and the server learns how to display the optimal warning message for each user.

[1181] Specific examples

[1182] Let us consider the case where a user wants to purchase "chocolate." The system operates as follows:

[1183] 1. User: Registers in the app that they are allergic to wheat, peanuts, and dairy products.

[1184] 2. Device: Stores allergy information locally and sends it to the server.

[1185] 3. Server: Stores allergy information in a database.

[1186] 4. User: Takes a photo of the chocolate packaging with their smartphone.

[1187] 5. Terminal: Sends image data to the server.

[1188] 6. Server: Performs image analysis and extracts text information, such as "cocoa, sugar, milk components, lecithin (derived from soybeans)."

[1189] 7. Server: Checks the ingredients list and identifies that it contains the user's allergic ingredient, "dairy."

[1190] 8. Server: Sends the judgment result to the terminal and generates a warning message.

[1191] 9. Device: Display a warning to the user that "This product contains milk ingredients."

[1192] 10. Emotion Engine: Detects the user's reaction to receiving a warning message and analyzes their emotional state.

[1193] 11. Device: Based on the results of the emotion engine, adjust the message display as needed (e.g., add a reassuring message).

[1194] 12. Server: Stores the judgment results and emotion data in a database.

[1195] In this way, the present invention provides fast and efficient assistance to users with food allergies in making safe food choices, while also providing more appropriate warnings that take into account the user's emotional state.

[1196] The processing flow will be explained below.

[1197] Step 1:

[1198] User: Start the app and open the "Register Allergy Information" screen from the settings menu. Select and enter the ingredients to which you are allergic (e.g., "wheat," "peanuts," "dairy products") from the list.

[1199] Step 2:

[1200] Terminal: Stores the entered allergy information in a local database, prepares the data for synchronization with the server, and creates a transmission request.

[1201] Step 3:

[1202] Server: Receives allergy information sent from the device and stores it in a user-specific allergy database. This data is later used to match food ingredients against a food ingredient list.

[1203] Step 4:

[1204] User: Uses the smartphone camera to take a picture of the packaging of a food item they are considering purchasing, adjusting the camera so that the entire package is visible within the frame.

[1205] Step 5:

[1206] Terminal: Temporarily stores the captured image data and prepares a request to send it to the server. When the user presses the send button, the image data is uploaded to the server.

[1207] Step 6:

[1208] Server: Receives image data sent from the device. The received image data is input into the OCR engine and text information within the image is extracted. During this process, preprocessing is performed to remove noise and improve character recognition accuracy.

[1209] Step 7:

[1210] Server: Preprocesses the extracted text data and normalizes it into an ingredient list, for example, listing ingredients such as "sugar" and "dairy."

[1211] Step 8:

[1212] Server: Compares the normalized ingredient list with the user's allergy database. Checks for a match against the user's allergy ingredients.

[1213] Step 9:

[1214] Server: Based on the matching result, if there is a matching allergy ingredient, identify the ingredient and add it to the list. Generate a warning message and prepare the content to notify the user.

[1215] Step 10:

[1216] Server: Sends the judgment result and warning message to the user's device. The judgment result includes the identified allergen and the warning message.

[1217] Step 11:

[1218] Device: Notify the user of the received test result by displaying a message such as "This product contains milk ingredients" in a pop-up or notification bar within the app.

[1219] Step 12:

[1220] User: Checks the warning message. Their facial expressions, tone of voice, and other reactions are collected by the emotion engine.

[1221] Step 13:

[1222] On the device: The emotion engine analyzes the user's reactions and determines their emotional state (e.g., anxiety, surprise, relief, etc.), for example by analyzing the user's facial expressions and tone of voice via a camera or microphone.

[1223] Step 14:

[1224] On the device: If the emotion engine determines that the user's emotional state indicates anxiety or stress, the display of the warning message will be adjusted to be more friendly and provide additional reassurance or support information.

[1225] Step 15:

[1226] Server: The judgment results and the emotion engine analysis results are stored in a database for future reference and system improvement.

[1227] Step 16:

[1228] Server: Based on the information accumulated in the database, the server learns to provide appropriate warning messages that take the user's emotional state into account from the next time onwards.

[1229] These are the specific processing steps and operations of the system that combines the emotion engine. This not only allows users to select foods with peace of mind, but also provides appropriate support that takes into account their emotional state.

[1230] Example 2

[1231] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1232] Existing food allergy prevention systems provide basic functions to reduce the risk of users purchasing foods containing allergens, but they do not provide warning messages that take into account the user's emotional state. This leaves users with a problem: the anxiety and stress they feel when receiving a warning message is not alleviated, making it difficult for them to make safe and secure food choices.

[1233] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for using optical character recognition technology to extract text from an image, means for comparing the extracted ingredient list with an allergy database to determine whether or not an allergic ingredient is present, means for analyzing the user's emotional state, and means for adjusting the display method of the warning message based on the user's emotional state. This not only enables the user to safely select foods to which they have allergies, but also gives them a sense of security when receiving a warning message, thereby reducing stress.

[1234] "User" refers to an individual who uses the system, specifically someone who inputs food allergy information and makes safe food choices.

[1235] "Allergy information" is data about allergens that the user knows about, and the system uses this information to determine the safety of food.

[1236] "Food packaging" refers to packaging material on which food product information is printed, and is the object that a user photographs using a camera.

[1237] "Server" refers to a computer device that receives and processes data sent from user terminals and performs database management and analysis.

[1238] "Image analysis" refers to the process of extracting information from photographed images of food packaging and recognizing it as text data.

[1239] "Optical character recognition technology" refers to technology that digitizes and recognizes characters in an image and extracts them as text data.

[1240] An "ingredient list" is a list of ingredients contained in a food product, and is used to check whether it contains specific allergens.

[1241] "Allergy database" refers to a database that stores and manages allergy information registered by users.

[1242] "Determination" refers to the process of checking the ingredient list against an allergy database to determine whether it contains any ingredients that may be dangerous to the user.

[1243] "Emotional state" refers to the psychological reaction (e.g., anxiety, relief, stress) of the user when they receive a warning message, and is analyzed by the emotion engine.

[1244] "Emotion engine" refers to software or hardware for detecting and analyzing a user's emotional state.

[1245] A "warning message" is a notification that informs the user that a particular food contains an allergic ingredient, and is used to ensure the user's safety.

[1246] "Database" refers to an electronic information storage medium that organizes large amounts of data processed by a system and efficiently stores, searches, and manages them.

[1247] "Adjustment" refers to the act of providing a more appropriate user experience by changing the content and display method of a warning message based on the user's emotional state.

[1248] This invention provides a system for safely purchasing and consuming food for users with food allergies, and offers a new approach that takes into account the emotional state of the user. The system has specific roles for the user, the terminal, and the server.

[1249] Registering allergy information

[1250] User: Launch the app and open the "Register Allergy Information" screen from the settings menu. The user selects and enters the ingredients to which they are allergic (e.g., "wheat," "peanuts," "dairy products") from a list.

[1251] Terminal: Save the allergy information entered by the user to a local database. After saving, generate JSON data to be synchronized with the server and create a send request.

[1252] Server: Receives allergy information sent from the device and stores it in a personal allergy database. This data is used to match food ingredients with the food's ingredient list in a later step.

[1253] Food packaging photography

[1254] User: Use your smartphone camera to take a photo of the entire package of a food item you are considering purchasing, using the frame guidelines to guide you in the best position to take the photo.

[1255] Device: The captured image data is temporarily stored in local storage, and a thumbnail is displayed for the user to confirm. When the user confirms and presses the send button, a request to send the image data to the server is prepared.

[1256] Parsing package information

[1257] Server: Receives image data sent from the device. Inputs the received data into an OCR (Optical Character Recognition) engine. The OCR engine extracts text information from the image.

[1258] Server: Preprocesses the text data extracted by the OCR engine, including noise reduction and text normalization, such as removing symbols and unnecessary spaces, and listing important ingredient information (e.g., "sugar," "dairy").

[1259] Database Matching

[1260] Server: Matches the preprocessed text data with the allergy information stored in the database. It uses SQL queries to match the allergens registered by the user with the extracted ingredient list.

[1261] Server: If a matching allergen exists, identify and list the ingredient. For example, if "dairy" is included, add this information to the warning list.

[1262] Notification of judgment results and emotion engine

[1263] Server: Generates a warning message based on the matching result. For example, if the ingredient to which the user is allergic is "dairy products," prepare a warning message stating "This product contains dairy ingredients."

[1264] Device: Notify the user of received alert messages. Display alert messages within the app using a pop-up window or notification bar.

[1265] Emotion engine: The system uses camera and microphone sensors to collect data on how users view warning messages. For example, it analyzes changes in facial expressions and speech in real time the moment the user sees the message.

[1266] On the device: Based on the results of the emotion engine, the message display can be adjusted depending on the user's emotional state. For example, if the user expresses anxiety, more friendly language or additional support information can be displayed.

[1267] Data accumulation

[1268] Server: Stores each judgment result and the emotion engine's analysis results in a database. This allows for quick response when the same product is scanned in the future. It also continuously learns how to display the most appropriate warning message to the user based on emotion data.

[1269] Specific examples

[1270] As an example, a case where a user wants to purchase "chocolate" will be described.

[1271] 1. User: Registers in the app that they have allergies to "wheat," "peanuts," and "dairy products." The user opens the "Register Allergy Information" screen from the settings menu, selects each item, and confirms.

[1272] 2. Terminal: Stores this information in a local database, packages the allergy information in JSON format, and generates a request to send to the server.

[1273] 3. Server: Receives the submitted allergy information and stores it in a personal allergy database for each user. This information is stored in an SQL database and saved in a format that can be easily verified later.

[1274] 4. User: Use the smartphone camera to take a photo of the entire chocolate package, following the photography guidelines and centering the food label.

[1275] 5. Terminal: Displays thumbnails of the captured images for the user to check. When the user checks the image and presses the send button, a request to send the image data to the server is prepared.

[1276] 6. Server: Passes the received image data to the OCR engine, which extracts text information such as "cocoa, sugar, milk components, lecithin (soybean-derived)" from the image.

[1277] 7. Server: Preprocesses the text data, removing noise and normalizing the necessary raw material information into a list.

[1278] 8. Server: Checks the normalized ingredient list against the user's allergy database, using an SQL query to ensure "dairy" is included.

[1279] 9. Server: Based on the matching results, a warning message such as "This product contains milk ingredients" is generated and sent to the terminal.

[1280] 10. Terminal: Displays a popup to alert the user of the received message.

[1281] 11. Emotion Engine: User reactions are collected using cameras and sensors and emotion analysis is performed. If the user shows surprise or anxiety in response to a displayed message, the emotional data is analyzed.

[1282] 12. On the device: Based on the results of the emotion engine, an additional message is displayed to reassure the user. For example, a friendly message such as "Please avoid purchasing this product as it contains dairy ingredients" is displayed.

[1283] 13. Server: The final judgment result and the user's emotional data are stored in a database, which will enable faster and more appropriate responses when the same situation occurs in the future.

[1284] In this way, the present invention helps users with food allergies make safe food choices and further enhances the user experience by adjusting the display of warning messages taking into account the user's emotional state in real time.

[1285] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1286] Step 1: The user launches the app and opens the "Register Allergy Information" screen from the settings menu. The user selects and enters the ingredients to which they are allergic (e.g., "wheat," "peanuts," "dairy products") from a list. The entered allergy information is saved in JSON format on the device. This becomes the input data.

[1287] Step 2: The device generates a request to send the saved JSON data to the server. Specifically, the created JSON data is sent to the server as an HTTP POST request. The input is the allergy information data in JSON format, and the output is the request sent to the server.

[1288] Step 3: The server parses the received allergy information and stores it in a per-user allergy database. Specifically, it executes a SQL query to parse the received data and insert it into the database. The input is the JSON-formatted allergy information sent to the server, and the output is the updated database state.

[1289] Step 4: The user uses the smartphone camera to take a picture of the entire food package they are considering purchasing. Guidelines are displayed to guide the user in taking the picture so that the entire label fits in the picture. This results in a captured image. The input is a physical image of the food package, and the output is a captured digital image.

[1290] Step 5: The device temporarily saves the captured image in local storage, generates a thumbnail, and asks the user to confirm it. When the user confirms and presses the send button, a request is generated to send the image data to the server. The input is the captured digital image, and the output is an image data request sent to the server.

[1291] Step 6: The server inputs the received image data into the OCR engine, which extracts the text information in the image and stores it as text data. Specifically, it performs binary analysis on the image data and applies a character recognition algorithm. The input is the digital image sent to the server, and the output is the extracted text data.

[1292] Step 7: The server preprocesses the text data extracted by the OCR engine. This includes noise removal and normalization. Specifically, it removes symbols and unnecessary spaces, extracts important keywords, and compiles them into a list. The input is the extracted text data, and the output is a preprocessed raw material list.

[1293] Step 8: The server compares the preprocessed text data with the allergy information stored in the database. Specifically, it executes an SQL query to retrieve the user's allergy ingredients from the database and compare them with the ingredient list. The input is the preprocessed ingredient list and the allergy information in the database, and the output is the comparison result.

[1294] Step 9: The server generates a warning message based on the matching result. For example, if the ingredient "dairy" is matched, it prepares a warning message saying "This product contains dairy ingredients." The input is the matching result, and the output is the generated warning message.

[1295] Step 10: The device notifies the user of the received warning message. Specifically, it displays the message in the app using a popup or notification bar. The input is the warning message, and the output is the notification displayed to the user.

[1296] Step 11: The emotion engine collects and analyzes the user's reaction to the warning message through the camera and microphone sensors. It analyzes the user's facial expressions and tone of voice to evaluate their emotional state in real time. The input is the user's reaction data, and the output is the analyzed emotional state.

[1297] Step 12: The device adjusts the display of the warning message based on the results of the emotion engine. Specifically, it provides appropriate information to the user by adding reassuring and friendly language. The input is the analyzed emotional state, and the output is the adjusted warning message.

[1298] Step 13: The server saves the final judgment result and the user's emotion data in the database. This allows for quick response when the same product is scanned in the future and enables learning based on the emotion data. The input is the final judgment result and emotion data, and the output is the updated database.

[1299] (Application example 2)

[1300] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1301] Previously, there were systems that allowed users to register allergy information and check food ingredients. However, these systems did not take into account the user's emotional state and lacked measures to address the anxiety and stress users felt when choosing foods containing allergens. This limited the effectiveness of these systems in increasing users' sense of security and created a psychological burden when choosing food.

[1302] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting and analyzing the user's emotional state, means for adjusting the display method of the warning message based on the analyzed emotional state, and means for storing the determination result and the emotional state in a database. This makes it possible to provide flexible warning messages that take the user's emotional state into consideration.

[1303] "User" refers to an individual who uses the system to register their own allergy information and confirm food safety.

[1304] "Allergy information" is data registered by users regarding their allergens. This information is used to confirm food safety when selecting foods.

[1305] "Food packaging" refers to materials such as paper, plastic, and metal used to pack or wrap food, and which bear the name of the food, as well as information about its ingredients and components.

[1306] "Means for taking photographs" refers to a function that allows a user to take a photograph of a food package using a smartphone or camera.

[1307] A "server" is a computer system that receives, analyzes, and stores data sent by users.

[1308] The "means for transmitting images" is a function that allows a user to upload an image of a food package taken by the user to a server.

[1309] The "means for the server to analyze the image" refers to the function of analyzing the received image data and extracting the list of ingredients for the food. Optical character recognition (OCR) technology is generally used.

[1310] A "food ingredient list" is a list of ingredient names listed on a food package.

[1311] The "allergy database" is a collection of numerical information that stores allergy ingredient information registered by the user.

[1312] The "means for determining" is a function that checks the extracted ingredient list against an allergy database to determine whether or not an allergic ingredient is present.

[1313] "Means for notifying" refers to a function for displaying or sending a message to inform the user of the analysis results.

[1314] The "user's emotional state" refers to the user's psychological reaction or emotions when seeing food containing an allergic ingredient.

[1315] "Means of collection and analysis" refers to the function of collecting the user's emotional state through cameras and sensors and analyzing it using dedicated software and algorithms.

[1316] The "means for adjusting the display method of the warning message" is a function for changing the content and display method of the warning message depending on the emotional state of the user.

[1317] "Means for storing the judgment results and emotional state in a database" is a function that stores the analysis results and the user's emotional data in a database and makes them useful for future analysis and responses.

[1318] This system allows users to register allergy information, photograph food packages, and analyze their ingredients to help them choose foods safely. It also collects and analyzes the user's emotional state and provides appropriate messages to enhance the user's sense of security.

[1319] 1. Registering allergy information

[1320] The user starts the application and opens the "Register Allergy Information" screen from the settings menu. Here, the user selects and enters the ingredients to which they are allergic (e.g., "wheat," "peanuts," "dairy products") from a list. The device saves this allergy information locally and then sends the data to the cloud server.

[1321] 2. Take a photo of the food package and send it

[1322] A user takes a photo of the packaging of a food item they are considering purchasing in a physical store with their smartphone camera. The device temporarily stores the captured image data and prepares a request to send it to the server. When the user presses the send button, the image is uploaded to the cloud server.

[1323] 3. Package information analysis

[1324] The server uses optical character recognition (OCR) technology to extract text information from the received image data. This is done using Google Cloud Vision API, among other tools. The server then preprocesses the extracted text data and lists the ingredients. For example, ingredients such as "cocoa, sugar, milk components, lecithin (soybean-derived)" are identified here.

[1325] 4. Database Matching

[1326] The server compares the preprocessed text data with the allergy information stored in the database, checks whether the allergens registered by the user beforehand match the analyzed ingredient list, and generates a warning message if there is a match.

[1327] 5. Notification of judgment results and emotion analysis

[1328] The device will then notify the user of a warning message, such as "This product contains milk ingredients." The device will then collect the user's reactions through its camera and sensors and analyze their emotional state. This analysis utilizes Azure Emotion API and other tools. Based on the analysis results, if the user's emotional state indicates anxiety or stress, the server will adjust the message display and provide additional support information.

[1329] 6. Data accumulation

[1330] The server stores the results of each judgment and the emotion engine's analysis in a database. This allows for a quick response the next time the same product is analyzed. Emotion data is also stored, and the server learns how to display the optimal warning message for each user.

[1331] Specific examples

[1332] Let's consider the case where a user is about to purchase "chocolate." First, the user registers in the app that they are allergic to "wheat," "peanuts," and "dairy products." The device saves the allergy information locally and sends it to the server. The server saves the allergy information in a database. The user takes a photo of the "chocolate" packaging with their smartphone, and the device sends the image data to the server. The server analyzes the image and extracts text information. Ingredients such as "cocoa, sugar, milk ingredients, and lecithin (soybean-derived)" are extracted, and the information is then compared with the ingredient list to determine that the product contains "dairy products," the user's allergic ingredient. The server sends the result of the assessment to the device and generates a warning message. The device displays a warning to the user stating "This product contains milk ingredients," and the emotion engine detects the user's reaction and analyzes their emotional state. The device adjusts the message display as necessary, and the server saves the assessment result and emotion data in a database.

[1333] Prompt Sentence Examples

[1334] "Just enter your allergy information (e.g., wheat, peanuts, dairy) and take a photo of a peanut butter package. The system will detect the allergen and notify you. It will also provide additional support information as needed based on your emotional state."

[1335] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1336] Step 1: Register your allergy information

[1337] The user starts the application and opens the "Register Allergy Information" screen. As input, the user selects and enters the ingredients to which they are allergic (e.g., "wheat," "peanuts," "dairy products") from a list. The device saves this allergy information in local storage and generates a request to send to the server. The input data is the user's allergy information, and the output data is the data request sent to the server. Specifically, the device sends the data to the server using a REST API.

[1338] Step 2: Photograph and send the food package

[1339] A user takes a photo of the packaging of a food item they are considering purchasing with their smartphone camera. The input is the food packaging to be photographed, and the output is the captured image data. The device temporarily stores this image data and prepares a request to send to the server. Specifically, when the user presses the send button, the device uploads the image data to the server. An HTTP POST request is used to upload the image data.

[1340] Step 3: Parse package information

[1341] The server extracts text information from the received image data using optical character recognition (OCR) technology. The input is image data and the output is extracted text data. The OCR engine uses services such as Google Cloud Vision API. Specifically, the server sends data to the OCR engine and receives text data as the analysis result.

[1342] Step 4: Check against the database

[1343] The server compares the preprocessed text data with the allergy database. The input is the extracted text data and the user's allergy information, and the output is the comparison result. The comparison process identifies ingredients that match the allergens in the database. Specifically, the text data is used as a search query to query the database to see if there are any matching ingredients.

[1344] Step 5: Notification of decision

[1345] The server generates a warning message based on the matching result and sends it to the terminal. The input is the matching result and the output is the warning message. Specifically, the server analyzes the matching result and generates a warning message if the product contains an ingredient to which the user is allergic. The terminal displays this message to the user via a pop-up or notification bar.

[1346] Step 6: Sentiment Analysis

[1347] The device collects user reactions through cameras and sensors and analyzes their emotional state. The input is user reaction data, and the output is emotional state data. The emotion engine uses the Azure Emotion API, etc. Specifically, it sends reaction data to the emotion engine and receives the emotional state as the analysis result.

[1348] Step 7: Adjust message display

[1349] The server adjusts the display of warning messages based on the emotional state. The input is the emotional state data and the current warning message, and the output is the adjusted warning message. Specifically, it analyzes the emotional state data and provides friendly messages or additional support information as needed.

[1350] Step 8: Accumulate data

[1351] The server accumulates the judgment results and emotion analysis results in a database. The input is the judgment results and emotional state data, and the output is an updated database. Specifically, the server saves this data in the database and makes it available for future analysis.

[1352] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1353] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1354] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1355] [Fourth embodiment]

[1356] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1357] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1358] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1359] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1360] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1361] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1362] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1363] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1364] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1365] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1366] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1367] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1368] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1369] The present invention is a system that enables users with food allergies to safely purchase food and enjoy their meals, and is realized through a series of processes: registering allergy information, photographing food packaging, analyzing the images, extracting and comparing ingredient lists, notifying the user of the results, and storing the data. Specific embodiments of the system are described below.

[1370] Registering allergy information

[1371] User: Launches the app and opens the user profile settings screen. The user enters their allergy information (e.g., "wheat," "peanuts," "dairy," etc.). This information can be added, deleted, or edited at any time from the application's settings menu.

[1372] Terminal: The entered allergy information is saved in a local database and the data is prepared for transmission to the server. The following is the process for registering the user's allergy information.

[1373] Server: Receives allergy information sent from the device and stores it in a user-specific allergy database. This data is later used to match food ingredients against a food ingredient list.

[1374] Food packaging photography

[1375] User: Take a photo of the packaging of the food item you are considering purchasing using your smartphone camera. When taking the photo, make sure to adjust it so that the entire package is clearly visible.

[1376] Terminal: Temporarily stores the captured image data and prepares a request to send it to the server. After the user presses the capture button, the image is immediately uploaded to the server.

[1377] Parsing package information

[1378] Server: The server analyzes the received image data. As a first step in the analysis, it uses optical character recognition (OCR) technology to extract text from the image. The extracted text is then processed into a list of raw materials. During this process, preprocessing is performed to remove noise and improve character recognition accuracy.

[1379] Server: Normalizes the extracted ingredient list to match a standard format in the database, and translates it if necessary based on specific language codes.

[1380] Database Matching

[1381] Server: Compares the normalized ingredient list with the user's allergy database. Checks whether the ingredients match the allergy information in the database. If a match is found, identifies the ingredient and passes the result on to further processes.

[1382] Notification of the results

[1383] Server: Generates a warning message if allergens are identified. The result includes a list of identified allergens.

[1384] Server: Sends the result of the judgment to the user's device. A notification message is generated so that the user can view it.

[1385] Device: Receives the sent judgment result and displays it to the user as an alert or pop-up message within the app, allowing the user to make safe food choices.

[1386] Data accumulation

[1387] Server: Each judgment result is stored in a database. This allows the system to provide quick results based on existing data the next time the same food is analyzed. The stored data is used to improve the efficiency of the entire system.

[1388] Specific examples

[1389] A case where a user checks a package of "chocolate" will be described.

[1390] 1. User: Registers in the app that they are allergic to wheat, peanuts, and dairy products.

[1391] 2. Device: Stores allergy information locally and sends it to the server.

[1392] 3. Server: Stores allergy information in a database.

[1393] 4. User: Takes a photo of the chocolate packaging with their smartphone.

[1394] 5. Terminal: Sends image data to the server.

[1395] 6. Server: Performs image analysis and extracts text information, such as "cocoa, sugar, milk components, lecithin (derived from soybeans)."

[1396] 7. Server: Checks the ingredients list and identifies that it contains the user's allergic ingredient, "dairy."

[1397] 8. Server: Sends the judgment result to the terminal and generates a warning message.

[1398] 9. Device: Display a warning to the user that "This product contains milk ingredients."

[1399] 10. Server: Stores the judgment results in a database.

[1400] Thus, the present invention provides fast and efficient assistance to users with food allergies in making safe food choices.

[1401] The processing flow will be explained below.

[1402] Step 1:

[1403] User: Launch the app and open the "Register Allergy Information" screen from the settings menu. The user selects and enters the ingredients they are allergic to (e.g., "wheat," "peanuts," "dairy products") from a list.

[1404] Step 2:

[1405] Terminal: Stores the entered allergy information in a local database, prepares the data for synchronization with the server, and creates a transmission request.

[1406] Step 3:

[1407] Server: Receives allergy information sent from the device. Stores the received data in a user-specific allergy database. At this stage, the data is formatted for matching.

[1408] Step 4:

[1409] User: Using their smartphone camera, they take a picture of the packaging of a food item they are considering purchasing. They frame the photo so that the entire label is visible.

[1410] Step 5:

[1411] Terminal: Temporarily stores the captured image and prepares a request to send it to the server. When the user presses the send button, the image data is uploaded to the server.

[1412] Step 6:

[1413] Server: Receives image data sent from the device. The received data is input into an optical character recognition (OCR) engine for image analysis. During this process, text information is extracted from the image.

[1414] Step 7:

[1415] Server: Preprocesses the text data extracted by the OCR engine, including noise removal and text normalization. For example, ingredients such as "sugar" and "dairy" are listed.

[1416] Step 8:

[1417] Server: Compares the preprocessed text data with the allergy information stored in the database, and verifies that the allergens registered by the user in advance match the analyzed ingredient list.

[1418] Step 9:

[1419] Server: Makes a decision based on the match. If a matching allergen is found, identifies the allergen and adds it to the list. Generates a warning message and prepares a notification to the user.

[1420] Step 10:

[1421] Server: Sends the judgment result to the user's device, which includes the identified allergens and a warning message.

[1422] Step 11:

[1423] Device: Notify the user of the received test result by displaying a message such as "This product contains milk ingredients" in a pop-up or notification bar within the app.

[1424] Step 12:

[1425] Server: Each judgment result is stored in a database. This allows for a quick response the next time the same product is analyzed. The data is saved for future reference and analysis, contributing to the efficiency of the entire system.

[1426] The above are the specific processing steps and operations of the proposed system.

[1427] Example 1

[1428] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1429] For consumers with food allergies, the process of checking food safety when making everyday food purchases is extremely cumbersome. Manually checking ingredient lists is time-consuming and laborious, and there is a risk of overlooking or misreading information. Furthermore, in many markets, product information is displayed in different languages, which can be difficult to understand. Therefore, there is a need for a system that allows users with food allergies to efficiently and accurately check food safety and purchase and consume with peace of mind.

[1430] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1431] In this invention, the server includes means for a user to input their own allergy information, means for photographing a food package, means for transmitting the image to the server, means for the server to analyze the image and extract a list of ingredients for the food, means for comparing the extracted ingredient list with an allergy database to determine whether or not an allergic ingredient is present, means for notifying the user of the determination result, means for storing the determination result in a database, means for extracting text from the image using optical character recognition technology, means for temporarily saving image data photographed by the user, means for preparing data for the user to transmit their allergy information to the server, means for saving the user's allergy information, and means for displaying the determination result within the app. This allows users to efficiently and accurately confirm the safety of foods and purchase or consume them with peace of mind.

[1432] "Users" are consumers who have food allergies and want to make safe food choices.

[1433] "Allergy information" is a list of ingredients that cause allergic reactions in specific foods, and is entered by the user.

[1434] "Food packaging" refers to the packaging or label that lists the product information and ingredients of a food product.

[1435] "Server" means a central computer system that receives and processes data submitted by users.

[1436] "Image" refers to photographic data of food packaging taken with a smartphone or other device.

[1437] "Optical character recognition (OCR) technology" is a technology for recognizing and extracting text data from images.

[1438] A "database" is a system for structuring and storing data such as user allergy information and food ingredient lists.

[1439] An "ingredient list" is a list of the ingredients and materials contained in a food product, and is text data extracted from the packaging.

[1440] "Notification" refers to a message or warning that the server sends to the user to inform them of the judgment result.

[1441] An "alert" is a warning message that is displayed to prompt the user to take a specific action.

[1442] A "pop-up message" is a message that appears temporarily on the screen to attract the user's attention.

[1443] The "determination result" refers to the result of determining whether or not an allergy ingredient is present based on a comparison between the extracted ingredient list and the user's allergy database.

[1444] The "user profile setting screen" is the screen within the app where users can enter, edit, and save their personal information and allergy information.

[1445] The present invention provides a system that enables users with food allergies to safely purchase food and enjoy meals. This system executes a series of processes using the following hardware and software:

[1446] Hardware

[1447] Device: A mobile information device such as a smartphone or tablet.

[1448] Server: A centralized computer system.

[1449] software

[1450] Local database: Manage user allergy information stored on the device using a lightweight database such as SQLite.

[1451] Optical Character Recognition (OCR) technology: Using libraries such as OpenCV and Tesseract, the server extracts text from images.

[1452] Translation API: Using the Google Cloud Translation API or similar, translate the ingredient list into a user-understandable language as needed.

[1453] Notification system: The result of the judgment is notified to the user using GCM (Google Cloud Messaging) or APNs (Apple Push Notification service).

[1454] Operating Procedures and Processes

[1455] 1. Allergy Information Registration:

[1456] User: Launch the app and open the user profile settings screen. Enter your allergy information (e.g., "wheat," "peanuts," "dairy products," etc.).

[1457] Terminal: Stores the entered allergy information in a local database and prepares the data for transmission to the server.

[1458] Server: Receives allergy information sent from the device and stores it in an allergy database for each user.

[1459] 2. Food packaging photography:

[1460] User: Take a photo of the packaging of a food item they are considering purchasing with their smartphone camera, adjusting the camera so that the entire package is clearly visible.

[1461] Terminal: Temporarily stores the captured image data and prepares a request to send it to the server.

[1462] 3. Parse package information:

[1463] Server: Analyzes the received image data and extracts the text within the image using optical character recognition (OCR) technology.

[1464] Specific examples

[1465] A case where a user checks a package of "chocolate" will be described.

[1466] 1. User: Registers in the app that they are allergic to wheat, peanuts, and dairy products.

[1467] 2. Device: Stores allergy information locally and sends it to the server.

[1468] 3. Server: Stores allergy information in a database.

[1469] 4. User: Takes a photo of the chocolate packaging with their smartphone.

[1470] 5. Terminal: Sends image data to the server.

[1471] 6. Server: Performs image analysis and extracts text information. For example, "cocoa, sugar, milk components, lecithin (derived from soybeans)" is extracted.

[1472] 7. Server: Checks the ingredients list and identifies that it contains the user's allergic ingredient, "dairy."

[1473] 8. Server: Sends the judgment result to the terminal and generates a warning message.

[1474] 9. Device: Display a warning to the user that "This product contains milk ingredients."

[1475] Examples of prompt statements

[1476] "Open your user profile settings screen and enter your allergy information."

[1477] "Take a photo of the packaging of the food you're considering purchasing."

[1478] "The captured image data is being sent to the server. Please wait."

[1479] "The ingredients list has been analyzed. We are currently reviewing the results."

[1480] "Avoid this food because it contains dairy."

[1481] This system provides support to users with food allergies by efficiently and accurately checking food safety and enabling them to select foods with peace of mind.

[1482] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1483] Step 1:

[1484] User: Launches the app and opens the user profile setting screen. The user enters their allergy information (e.g., "wheat," "peanuts," "dairy products," etc.). This is the input data.

[1485] Terminal: The entered allergy information is saved in a local database (e.g., SQLite). Data calculations at this stage involve converting the information entered by the user into a database format and saving it. Data is also prepared for sending to the server. Specific operations include writing to the database and converting the data format for sending to the server. The output is the allergy information saved locally and the data to be sent to the server.

[1486] Server: Receives allergy information sent from the device. Stores the received data in an allergy database for each user. Specifically, the data is written to the allergy database (e.g., MySQL or PostgreSQL). This is the output data.

[1487] Step 2:

[1488] User: Take a photo of the packaging of a food item they are considering purchasing with their smartphone camera. When taking the photo, they need to adjust the camera so that the entire package is clearly visible. This is the input data.

[1489] Terminal: Temporarily stores the captured image data. Data calculation at this stage involves converting the image data into an appropriate format (e.g., JPEG or PNG) and saving it. After the user presses the capture button, a request is immediately prepared to upload the image to the server. The specific operation is to convert the image data for network transmission. The output is image data ready for transmission to the server.

[1490] Step 3:

[1491] Server: The server receives the image data sent from the terminal. It analyzes the received image data and extracts the text within the image using optical character recognition (OCR) technology (e.g., Tesseract). This is the input data. Data processing at this stage involves converting the image data into text data, including noise removal and image preprocessing. The output is the extracted text data of the ingredient list.

[1492] Step 4:

[1493] Server: The extracted text data is normalized and translated into a specific language using a translation API (e.g., Google Cloud Translation API) if necessary to conform to a standard format. This is the input data. Data operations at this stage involve cleaning and normalizing the text data, and translating it if necessary. The output is the normalized and translated text data of the ingredient list.

[1494] Step 5:

[1495] Server: Compares the normalized ingredient list with the user's allergy database. The specific operation is to compare the allergy information in the database with the ingredient list. This is the input data. The data operation at this stage is to identify matching ingredients using a comparison algorithm. The output is a judgment result on the presence or absence of allergens.

[1496] Step 6:

[1497] Server: If an allergen is identified, a warning message is generated. This is the input data. Data calculation at this stage generates a warning message based on the judgment result. The output is a warning message and the judgment result.

[1498] Server: Sends the result of the judgment to the user's device. Specifically, it sends a message in real time via a notification system such as GCM or APNs. This is the input data. The output is the sent notification message.

[1499] Device: Receives the sent judgment result and displays it to the user as an alert or pop-up message within the app. The specific behavior is to display a notification on the device's user interface. This is the input data, and the output is the displayed warning message.

[1500] Step 7:

[1501] Server: Each decision result is stored in a database. This is the input data. Data calculations at this stage are to save the decision results in a database (e.g. MySQL or PostgreSQL). The output is the updated database.

[1502] The above are the specific processing steps in the system of the present invention, which allow users with food allergies to select foods efficiently and safely.

[1503] (Application example 1)

[1504] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1505] When consumers with food allergies purchase food at physical stores, they need a method to quickly and accurately identify allergens and safely select foods. Conventional methods often require complicated and time-consuming food safety checks, making it difficult to ensure consumer peace of mind.

[1506] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1507] In this invention, the server includes means for a user to input their own allergy information, means for photographing the food package, means for transmitting the image to the server, means for the server to analyze the image and extract a list of the food's ingredients, means for comparing the extracted ingredient list with an allergy database to determine whether or not an allergen is present, means for notifying the user of the determination result, means for storing the determination result in the database, and means for displaying the allergen determination result in real time using a smart device at a physical store. This allows users to instantly check whether or not a food product contains an allergen by scanning the food package at the physical store, enabling them to select safe foods.

[1508] "User" refers to an individual who uses the system and is the entity that inputs allergy information and photographs food packaging.

[1509] "Allergy information" refers to information that indicates the specific ingredient names and food names for which the user has a food allergy.

[1510] "Food packaging" refers to the packaging in which food is packaged, and is the target from which ingredient information written inside is extracted.

[1511] "Smart devices" refer to electronic devices that can be carried by users, such as smartphones, smart glasses, and head-mounted displays.

[1512] "Server" refers to a computer system that processes information for the entire system, and performs image analysis and database matching.

[1513] "Image analysis" is the process of processing images of food packaging to extract useful textual information from them.

[1514] "Optical Character Recognition (OCR) technology" is a technology that recognizes characters from images or printed text as electronic data.

[1515] "Allergy database" refers to a database that stores the user's allergy information and is used for matching purposes.

[1516] The "determination result" is a conclusion as to whether or not an allergen is present, obtained by comparing the extracted ingredient list with the allergy database.

[1517] "Notification" refers to the process of communicating the results of a determination to the user, which is done through a warning message or a pop-up message.

[1518] MODE FOR CARRYING OUT THE INVENTION

[1519] This invention is a system that allows users with food allergies to safely select food in physical stores. How this system is implemented will be described below in detail.

[1520] System Program Overview

[1521] The server executes a series of processes, including setting up a user profile, photographing the food packaging, analyzing the image, extracting ingredients, comparing it with a database, notifying the user of the results, and storing the results, allowing users to quickly determine the safety of food in physical stores.

[1522] Hardware / Software used

[1523] Hardware:

[1524] Smartphone (Android or iOS)

[1525] server

[1526] software:

[1527] For server-side processing, the Flask framework is used.

[1528] For image analysis, pytesseract is used as an optical character recognition (OCR) technology.

[1529] For image processing, we use PIL, a Python image processing library.

[1530] SQLite is used as the database.

[1531] Data processing and calculation

[1532] User:

[1533] 1. The user launches the app on their smartphone, enters their allergy information (e.g., "wheat," "peanuts," "dairy products," etc.), and registers it.

[1534] 2. The registered allergy information is stored in the smartphone's local database and also sent to the server.

[1535] server:

[1536] 1. The server receives allergy information sent by the user and stores it in a dedicated allergy database.

[1537] 2. When a user takes a photo of a food package in a physical store with their smartphone camera, the image data is sent to the server.

[1538] 3. The server analyzes the received image data using optical character recognition (OCR) technology to extract the ingredient list from the package. This analysis involves preprocessing to remove noise and improve character recognition accuracy.

[1539] 4. The extracted ingredient list is normalized to a standard format and compared with the user's allergy database to determine whether or not an allergen is present.

[1540] Notification of decision:

[1541] 1. The server generates a warning message if an allergen is found. The result includes a list of identified allergens.

[1542] 2. The judgment result is sent to the user's smartphone, and a notification message is displayed so that the user can view it immediately.

[1543] Data accumulation:

[1544] 1. The results of the assessment are stored in a server database. This data is used to speed up future analysis processes.

[1545] Specific examples

[1546] A case will be described where the user wishes to check the packaging of "Chocolate".

[1547] 1. The user registers in a smartphone app that they are allergic to wheat, peanuts, and dairy products.

[1548] 2. The smartphone stores the allergy information locally and sends it to the server.

[1549] 3. The server stores this information in a dedicated database.

[1550] 4. The user takes a photo of the chocolate packaging with their smartphone in a physical store.

[1551] 5. The captured image data is sent to the server.

[1552] 6. The server analyzes the image and extracts text information, such as "cocoa, sugar, milk components, lecithin (derived from soybeans)."

[1553] 7. The extracted ingredient list is compared and it is determined that the product contains the user's allergic ingredient, "dairy products."

[1554] 8. The server sends the result of the judgment to the smartphone and generates a warning message.

[1555] 9. Display a "This product contains dairy" warning on your smartphone.

[1556] Prompt Sentence Examples

[1557] "Create an application that, when a user scans a food package in a physical store with their smartphone camera, analyzes the ingredients contained in the package in real time and compares them with the allergy information registered by the user to detect allergens."

[1558] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1559] Step 1:

[1560] The user launches the app on their smartphone and enters their allergy information (e.g., "wheat," "peanuts," "dairy products," etc.). The entered allergy information is stored in a local database and is also prepared as data to be sent to the server.

[1561] Input: Allergy information entered by the user

[1562] Output: Allergy information stored in the local database and sent to the server

[1563] Step 2:

[1564] The terminal receives the allergy information entered by the user, stores it in a local database, and then transmits it to the server.

[1565] Input: Allergy information entered by the user

[1566] Output: Allergy information sent to the server

[1567] Step 3:

[1568] The server receives the allergy information sent from the terminal and stores the information in an allergy database.

[1569] Input: Allergy information sent from the device

[1570] Output: Allergy information stored in the allergy database

[1571] Step 4:

[1572] The user takes a photo of the packaging of a food item they are considering purchasing at a physical store using their smartphone camera, adjusting the camera so that the entire package is clearly visible.

[1573] Input: Food packaging you're considering buying

[1574] Output: Package image saved on your smartphone

[1575] Step 5:

[1576] The device temporarily stores the captured image data and prepares a request to send it to the server. When the user presses the capture button, the image is immediately uploaded to the server.

[1577] Input: Photographed package image

[1578] Output: Image data sent to the server

[1579] Step 6:

[1580] The server analyzes the received image data, extracts the text within the image using optical character recognition (OCR) technology, and processes it as an ingredient list.

[1581] Input: Received image data

[1582] Output: Extracted ingredient list

[1583] Step 7:

[1584] The server normalizes the extracted ingredient list, matches it with a standard format in an allergy database, and may translate it into specific languages ​​if necessary.

[1585] Input: Extracted ingredient list

[1586] Output: Normalized ingredient list

[1587] Step 8:

[1588] The server checks the normalized ingredient list against the user's allergy database to see if the product contains any allergens.

[1589] Input: Normalized ingredient list, allergy database

[1590] Output: Judgment result regarding the presence or absence of allergens

[1591] Step 9:

[1592] If an allergy ingredient is identified, the server generates a warning message and creates a message to notify the user based on the generated judgment result.

[1593] Input: Judgment result regarding the presence or absence of allergens

[1594] Output: Warning message

[1595] Step 10:

[1596] The device receives the judgment results sent from the server and displays them to the user as an alert within the app, allowing the user to select safe foods.

[1597] Input: Verification result sent from the server

[1598] Output: The alert message that is displayed to the user

[1599] Step 11:

[1600] The server stores each judgment result in a database, which allows it to quickly provide results based on existing data the next time the same food is analyzed.

[1601] Input: Judgment result

[1602] Output: Judgment results stored in the database

[1603] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1604] The present invention provides a new approach to a system that allows users with food allergies to safely purchase food and enjoy meals, taking into account the emotional state of the user. Specific embodiments of this system are described in detail below.

[1605] Registering allergy information

[1606] User: Launch the app and open the "Register Allergy Information" screen from the settings menu. The user selects and enters the ingredients they are allergic to (e.g., "wheat," "peanuts," "dairy products") from a list.

[1607] Terminal: Stores the entered allergy information in a local database, prepares the data for synchronization with the server, and creates a transmission request.

[1608] Server: Receives allergy information sent from the device and stores it in a user-specific allergy database. This data is later used to match food ingredients against a food ingredient list.

[1609] Food packaging photography

[1610] User: Using their smartphone camera, they take a picture of the packaging of a food item they are considering purchasing. They frame the photo so that the entire label is visible.

[1611] Terminal: Temporarily stores the captured image data and prepares a request to send it to the server. When the user presses the send button, the image data is uploaded to the server.

[1612] Parsing package information

[1613] Server: Receives image data sent from the device. The received data is input into an optical character recognition (OCR) engine for image analysis. During this process, text information is extracted from the image.

[1614] Server: Preprocesses the text data extracted by the OCR engine, including noise removal and text normalization. For example, ingredients such as "sugar" and "dairy" are listed.

[1615] Database Matching

[1616] Server: Compares the preprocessed text data with the allergy information stored in the database, and verifies that the allergens registered by the user in advance match the analyzed ingredient list.

[1617] Notification of judgment results and emotion engine

[1618] Server: Makes a decision based on the match. If a matching allergen is found, identifies the allergen and adds it to the list. Generates a warning message and prepares a notification to the user.

[1619] Device: Notify the user of the received test result by displaying a message such as "This product contains milk ingredients" in a pop-up or notification bar within the app.

[1620] Emotion engine: Collects user reactions through cameras and sensors and analyzes the user's emotional state. For example, it can read emotions from facial expressions and tone of voice after a user sees a warning message.

[1621] On the device: If the emotion engine analyzes the user's emotional state and determines they are anxious or stressed, the display of the warning message can be adjusted, for example, by making the message more friendly or by providing additional support information.

[1622] Data accumulation

[1623] Server: Each judgment result and the emotion engine's analysis results are stored in a database. This allows for a quick response the next time the same product is analyzed. Emotion data is also stored, and the server learns how to display the optimal warning message for each user.

[1624] Specific examples

[1625] Let us consider the case where a user wants to purchase "chocolate." The system operates as follows:

[1626] 1. User: Registers in the app that they are allergic to wheat, peanuts, and dairy products.

[1627] 2. Device: Stores allergy information locally and sends it to the server.

[1628] 3. Server: Stores allergy information in a database.

[1629] 4. User: Takes a photo of the chocolate packaging with their smartphone.

[1630] 5. Terminal: Sends image data to the server.

[1631] 6. Server: Performs image analysis and extracts text information, such as "cocoa, sugar, milk components, lecithin (derived from soybeans)."

[1632] 7. Server: Checks the ingredients list and identifies that it contains the user's allergic ingredient, "dairy."

[1633] 8. Server: Sends the judgment result to the terminal and generates a warning message.

[1634] 9. Device: Display a warning to the user that "This product contains milk ingredients."

[1635] 10. Emotion Engine: Detects the user's reaction to receiving a warning message and analyzes their emotional state.

[1636] 11. Device: Based on the results of the emotion engine, adjust the message display as needed (e.g., add a reassuring message).

[1637] 12. Server: Stores the judgment results and emotion data in a database.

[1638] In this way, the present invention provides fast and efficient assistance to users with food allergies in making safe food choices, while also providing more appropriate warnings that take into account the user's emotional state.

[1639] The processing flow will be explained below.

[1640] Step 1:

[1641] User: Start the app and open the "Register Allergy Information" screen from the settings menu. Select and enter the ingredients to which you are allergic (e.g., "wheat," "peanuts," "dairy products") from the list.

[1642] Step 2:

[1643] Terminal: Stores the entered allergy information in a local database, prepares the data for synchronization with the server, and creates a transmission request.

[1644] Step 3:

[1645] Server: Receives allergy information sent from the device and stores it in a user-specific allergy database. This data is later used to match food ingredients against a food ingredient list.

[1646] Step 4:

[1647] User: Uses the smartphone camera to take a picture of the packaging of a food item they are considering purchasing, adjusting the camera so that the entire package is visible within the frame.

[1648] Step 5:

[1649] Terminal: Temporarily stores the captured image data and prepares a request to send it to the server. When the user presses the send button, the image data is uploaded to the server.

[1650] Step 6:

[1651] Server: Receives image data sent from the device. The received image data is input into the OCR engine and text information within the image is extracted. During this process, preprocessing is performed to remove noise and improve character recognition accuracy.

[1652] Step 7:

[1653] Server: Preprocesses the extracted text data and normalizes it into an ingredient list, for example, listing ingredients such as "sugar" and "dairy."

[1654] Step 8:

[1655] Server: Compares the normalized ingredient list with the user's allergy database. Checks for a match against the user's allergy ingredients.

[1656] Step 9:

[1657] Server: Based on the matching result, if there is a matching allergy ingredient, identify the ingredient and add it to the list. Generate a warning message and prepare the content to notify the user.

[1658] Step 10:

[1659] Server: Sends the judgment result and warning message to the user's device. The judgment result includes the identified allergen and the warning message.

[1660] Step 11:

[1661] Device: Notify the user of the received test result by displaying a message such as "This product contains milk ingredients" in a pop-up or notification bar within the app.

[1662] Step 12:

[1663] User: Checks the warning message. Their facial expressions, tone of voice, and other reactions are collected by the emotion engine.

[1664] Step 13:

[1665] On the device: The emotion engine analyzes the user's reactions and determines their emotional state (e.g., anxiety, surprise, relief, etc.), for example by analyzing the user's facial expressions and tone of voice via a camera or microphone.

[1666] Step 14:

[1667] On the device: If the emotion engine determines that the user's emotional state indicates anxiety or stress, the display of the warning message will be adjusted to be more friendly and provide additional reassurance or support information.

[1668] Step 15:

[1669] Server: The judgment results and the emotion engine analysis results are stored in a database for future reference and system improvement.

[1670] Step 16:

[1671] Server: Based on the information accumulated in the database, the server learns to provide appropriate warning messages that take the user's emotional state into account from the next time onwards.

[1672] These are the specific processing steps and operations of the system that combines the emotion engine. This not only allows users to select foods with peace of mind, but also provides appropriate support that takes into account their emotional state.

[1673] Example 2

[1674] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1675] Existing food allergy prevention systems provide basic functions to reduce the risk of users purchasing foods containing allergens, but they do not provide warning messages that take into account the user's emotional state. This leaves users with a problem: the anxiety and stress they feel when receiving a warning message is not alleviated, making it difficult for them to make safe and secure food choices.

[1676] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for using optical character recognition technology to extract text from an image, means for comparing the extracted ingredient list with an allergy database to determine whether or not an allergic ingredient is present, means for analyzing the user's emotional state, and means for adjusting the display method of the warning message based on the user's emotional state. This not only enables the user to safely select foods to which they have allergies, but also gives them a sense of security when receiving a warning message, thereby reducing stress.

[1677] "User" refers to an individual who uses the system, specifically someone who inputs food allergy information and makes safe food choices.

[1678] "Allergy information" is data about allergens that the user knows about, and the system uses this information to determine the safety of food.

[1679] "Food packaging" refers to packaging material on which food product information is printed, and is the object that a user photographs using a camera.

[1680] "Server" refers to a computer device that receives and processes data sent from user terminals and performs database management and analysis.

[1681] "Image analysis" refers to the process of extracting information from photographed images of food packaging and recognizing it as text data.

[1682] "Optical character recognition technology" refers to technology that digitizes and recognizes characters in an image and extracts them as text data.

[1683] An "ingredient list" is a list of ingredients contained in a food product, and is used to check whether it contains specific allergens.

[1684] "Allergy database" refers to a database that stores and manages allergy information registered by users.

[1685] "Determination" refers to the process of checking the ingredient list against an allergy database to determine whether it contains any ingredients that may be dangerous to the user.

[1686] "Emotional state" refers to the psychological reaction (e.g., anxiety, relief, stress) of the user when they receive a warning message, and is analyzed by the emotion engine.

[1687] "Emotion engine" refers to software or hardware for detecting and analyzing a user's emotional state.

[1688] A "warning message" is a notification that informs the user that a particular food contains an allergic ingredient, and is used to ensure the user's safety.

[1689] "Database" refers to an electronic information storage medium that organizes large amounts of data processed by a system and efficiently stores, searches, and manages them.

[1690] "Adjustment" refers to the act of providing a more appropriate user experience by changing the content and display method of a warning message based on the user's emotional state.

[1691] This invention provides a system for safely purchasing and consuming food for users with food allergies, and offers a new approach that takes into account the emotional state of the user. The system has specific roles for the user, the terminal, and the server.

[1692] Registering allergy information

[1693] User: Launch the app and open the "Register Allergy Information" screen from the settings menu. The user selects and enters the ingredients to which they are allergic (e.g., "wheat," "peanuts," "dairy products") from a list.

[1694] Terminal: Save the allergy information entered by the user to a local database. After saving, generate JSON data to be synchronized with the server and create a send request.

[1695] Server: Receives allergy information sent from the device and stores it in a personal allergy database. This data is used to match food ingredients with the food's ingredient list in a later step.

[1696] Food packaging photography

[1697] User: Use your smartphone camera to take a photo of the entire package of a food item you are considering purchasing, using the frame guidelines to guide you in the best position to take the photo.

[1698] Device: The captured image data is temporarily stored in local storage, and a thumbnail is displayed for the user to confirm. When the user confirms and presses the send button, a request to send the image data to the server is prepared.

[1699] Parsing package information

[1700] Server: Receives image data sent from the device. Inputs the received data into an OCR (Optical Character Recognition) engine. The OCR engine extracts text information from the image.

[1701] Server: Preprocesses the text data extracted by the OCR engine, including noise reduction and text normalization, such as removing symbols and unnecessary spaces, and listing important ingredient information (e.g., "sugar," "dairy").

[1702] Database Matching

[1703] Server: Matches the preprocessed text data with the allergy information stored in the database. It uses SQL queries to match the allergens registered by the user with the extracted ingredient list.

[1704] Server: If a matching allergen exists, identify and list the ingredient. For example, if "dairy" is included, add this information to the warning list.

[1705] Notification of judgment results and emotion engine

[1706] Server: Generates a warning message based on the matching result. For example, if the ingredient to which the user is allergic is "dairy products," prepare a warning message stating "This product contains dairy ingredients."

[1707] Device: Notify the user of received alert messages. Display alert messages within the app using a pop-up window or notification bar.

[1708] Emotion engine: The system uses camera and microphone sensors to collect data on how users view warning messages. For example, it analyzes changes in facial expressions and speech in real time the moment the user sees the message.

[1709] On the device: Based on the results of the emotion engine, the message display can be adjusted depending on the user's emotional state. For example, if the user expresses anxiety, more friendly language or additional support information can be displayed.

[1710] Data accumulation

[1711] Server: Stores each judgment result and the emotion engine's analysis results in a database. This allows for quick response when the same product is scanned in the future. It also continuously learns how to display the most appropriate warning message to the user based on emotion data.

[1712] Specific examples

[1713] As an example, a case where a user wants to purchase "chocolate" will be described.

[1714] 1. User: Registers in the app that they have allergies to "wheat," "peanuts," and "dairy products." The user opens the "Register Allergy Information" screen from the settings menu, selects each item, and confirms.

[1715] 2. Terminal: Stores this information in a local database, packages the allergy information in JSON format, and generates a request to send to the server.

[1716] 3. Server: Receives the submitted allergy information and stores it in a personal allergy database for each user. This information is stored in an SQL database and saved in a format that can be easily verified later.

[1717] 4. User: Use the smartphone camera to take a photo of the entire chocolate package, following the photography guidelines and centering the food label.

[1718] 5. Terminal: Displays thumbnails of the captured images for the user to check. When the user checks the image and presses the send button, a request to send the image data to the server is prepared.

[1719] 6. Server: Passes the received image data to the OCR engine, which extracts text information such as "cocoa, sugar, milk components, lecithin (soybean-derived)" from the image.

[1720] 7. Server: Preprocesses the text data, removing noise and normalizing the necessary raw material information into a list.

[1721] 8. Server: Checks the normalized ingredient list against the user's allergy database, using an SQL query to ensure "dairy" is included.

[1722] 9. Server: Based on the matching results, a warning message such as "This product contains milk ingredients" is generated and sent to the terminal.

[1723] 10. Terminal: Displays a popup to alert the user of the received message.

[1724] 11. Emotion Engine: User reactions are collected using cameras and sensors and emotion analysis is performed. If the user shows surprise or anxiety in response to a displayed message, the emotional data is analyzed.

[1725] 12. On the device: Based on the results of the emotion engine, an additional message is displayed to reassure the user. For example, a friendly message such as "Please avoid purchasing this product as it contains dairy ingredients" is displayed.

[1726] 13. Server: The final judgment result and the user's emotional data are stored in a database, which will enable faster and more appropriate responses when the same situation occurs in the future.

[1727] In this way, the present invention helps users with food allergies make safe food choices and further enhances the user experience by adjusting the display of warning messages taking into account the user's emotional state in real time.

[1728] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1729] Step 1: The user launches the app and opens the "Register Allergy Information" screen from the settings menu. The user selects and enters the ingredients to which they are allergic (e.g., "wheat," "peanuts," "dairy products") from a list. The entered allergy information is saved in JSON format on the device. This becomes the input data.

[1730] Step 2: The device generates a request to send the saved JSON data to the server. Specifically, the created JSON data is sent to the server as an HTTP POST request. The input is the allergy information data in JSON format, and the output is the request sent to the server.

[1731] Step 3: The server parses the received allergy information and stores it in a per-user allergy database. Specifically, it executes a SQL query to parse the received data and insert it into the database. The input is the JSON-formatted allergy information sent to the server, and the output is the updated database state.

[1732] Step 4: The user uses the smartphone camera to take a picture of the entire food package they are considering purchasing. Guidelines are displayed to guide the user in taking the picture so that the entire label fits in the picture. This results in a captured image. The input is a physical image of the food package, and the output is a captured digital image.

[1733] Step 5: The device temporarily saves the captured image in local storage, generates a thumbnail, and asks the user to confirm it. When the user confirms and presses the send button, a request is generated to send the image data to the server. The input is the captured digital image, and the output is an image data request sent to the server.

[1734] Step 6: The server inputs the received image data into the OCR engine, which extracts the text information in the image and stores it as text data. Specifically, it performs binary analysis on the image data and applies a character recognition algorithm. The input is the digital image sent to the server, and the output is the extracted text data.

[1735] Step 7: The server preprocesses the text data extracted by the OCR engine. This includes noise removal and normalization. Specifically, it removes symbols and unnecessary spaces, extracts important keywords, and compiles them into a list. The input is the extracted text data, and the output is a preprocessed raw material list.

[1736] Step 8: The server compares the preprocessed text data with the allergy information stored in the database. Specifically, it executes an SQL query to retrieve the user's allergy ingredients from the database and compare them with the ingredient list. The input is the preprocessed ingredient list and the allergy information in the database, and the output is the comparison result.

[1737] Step 9: The server generates a warning message based on the matching result. For example, if the ingredient "dairy" is matched, it prepares a warning message saying "This product contains dairy ingredients." The input is the matching result, and the output is the generated warning message.

[1738] Step 10: The device notifies the user of the received warning message. Specifically, it displays the message in the app using a popup or notification bar. The input is the warning message, and the output is the notification displayed to the user.

[1739] Step 11: The emotion engine collects and analyzes the user's reaction to the warning message through the camera and microphone sensors. It analyzes the user's facial expressions and tone of voice to evaluate their emotional state in real time. The input is the user's reaction data, and the output is the analyzed emotional state.

[1740] Step 12: The device adjusts the display of the warning message based on the results of the emotion engine. Specifically, it provides appropriate information to the user by adding reassuring and friendly language. The input is the analyzed emotional state, and the output is the adjusted warning message.

[1741] Step 13: The server saves the final judgment result and the user's emotion data in the database. This allows for quick response when the same product is scanned in the future and enables learning based on the emotion data. The input is the final judgment result and emotion data, and the output is the updated database.

[1742] (Application example 2)

[1743] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1744] Previously, there were systems that allowed users to register allergy information and check food ingredients. However, these systems did not take into account the user's emotional state and lacked measures to address the anxiety and stress users felt when choosing foods containing allergens. This limited the effectiveness of these systems in increasing users' sense of security and created a psychological burden when choosing food.

[1745] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting and analyzing the user's emotional state, means for adjusting the display method of the warning message based on the analyzed emotional state, and means for storing the determination result and the emotional state in a database. This makes it possible to provide flexible warning messages that take the user's emotional state into consideration.

[1746] "User" refers to an individual who uses the system to register their own allergy information and confirm food safety.

[1747] "Allergy information" is data registered by users regarding their allergens. This information is used to confirm food safety when selecting foods.

[1748] "Food packaging" refers to materials such as paper, plastic, and metal used to pack or wrap food, and which bear the name of the food, as well as information about its ingredients and components.

[1749] "Means for taking photographs" refers to a function that allows a user to take a photograph of a food package using a smartphone or camera.

[1750] A "server" is a computer system that receives, analyzes, and stores data sent by users.

[1751] The "means for transmitting images" is a function that allows a user to upload an image of a food package taken by the user to a server.

[1752] The "means for the server to analyze the image" refers to the function of analyzing the received image data and extracting the list of ingredients for the food. Optical character recognition (OCR) technology is generally used.

[1753] A "food ingredient list" is a list of ingredient names listed on a food package.

[1754] The "allergy database" is a collection of numerical information that stores allergy ingredient information registered by the user.

[1755] The "means for determining" is a function that checks the extracted ingredient list against an allergy database to determine whether or not an allergic ingredient is present.

[1756] "Means for notifying" refers to a function for displaying or sending a message to inform the user of the analysis results.

[1757] The "user's emotional state" refers to the user's psychological reaction or emotions when seeing food containing an allergic ingredient.

[1758] "Means of collection and analysis" refers to the function of collecting the user's emotional state through cameras and sensors and analyzing it using dedicated software and algorithms.

[1759] The "means for adjusting the display method of the warning message" is a function for changing the content and display method of the warning message depending on the emotional state of the user.

[1760] "Means for storing the judgment results and emotional state in a database" is a function that stores the analysis results and the user's emotional data in a database and makes them useful for future analysis and responses.

[1761] This system allows users to register allergy information, photograph food packages, and analyze their ingredients to help them choose foods safely. It also collects and analyzes the user's emotional state and provides appropriate messages to enhance the user's sense of security.

[1762] 1. Registering allergy information

[1763] The user starts the application and opens the "Register Allergy Information" screen from the settings menu. Here, the user selects and enters the ingredients to which they are allergic (e.g., "wheat," "peanuts," "dairy products") from a list. The device saves this allergy information locally and then sends the data to the cloud server.

[1764] 2. Take a photo of the food package and send it

[1765] A user takes a photo of the packaging of a food item they are considering purchasing in a physical store with their smartphone camera. The device temporarily stores the captured image data and prepares a request to send it to the server. When the user presses the send button, the image is uploaded to the cloud server.

[1766] 3. Package information analysis

[1767] The server uses optical character recognition (OCR) technology to extract text information from the received image data. This is done using Google Cloud Vision API, among other tools. The server then preprocesses the extracted text data and lists the ingredients. For example, ingredients such as "cocoa, sugar, milk components, lecithin (soybean-derived)" are identified here.

[1768] 4. Database Matching

[1769] The server compares the preprocessed text data with the allergy information stored in the database, checks whether the allergens registered by the user beforehand match the analyzed ingredient list, and generates a warning message if there is a match.

[1770] 5. Notification of judgment results and emotion analysis

[1771] The device will then notify the user of a warning message, such as "This product contains milk ingredients." The device will then collect the user's reactions through its camera and sensors and analyze their emotional state. This analysis utilizes Azure Emotion API and other tools. Based on the analysis results, if the user's emotional state indicates anxiety or stress, the server will adjust the message display and provide additional support information.

[1772] 6. Data accumulation

[1773] The server stores the results of each judgment and the emotion engine's analysis in a database. This allows for a quick response the next time the same product is analyzed. Emotion data is also stored, and the server learns how to display the optimal warning message for each user.

[1774] Specific examples

[1775] Let's consider the case where a user is about to purchase "chocolate." First, the user registers in the app that they are allergic to "wheat," "peanuts," and "dairy products." The device saves the allergy information locally and sends it to the server. The server saves the allergy information in a database. The user takes a photo of the "chocolate" packaging with their smartphone, and the device sends the image data to the server. The server analyzes the image and extracts text information. Ingredients such as "cocoa, sugar, milk ingredients, and lecithin (soybean-derived)" are extracted, and the information is then compared with the ingredient list to determine that the product contains "dairy products," the user's allergic ingredient. The server sends the result of the assessment to the device and generates a warning message. The device displays a warning to the user stating "This product contains milk ingredients," and the emotion engine detects the user's reaction and analyzes their emotional state. The device adjusts the message display as necessary, and the server saves the assessment result and emotion data in a database.

[1776] Prompt Sentence Examples

[1777] "Just enter your allergy information (e.g., wheat, peanuts, dairy) and take a photo of a peanut butter package. The system will detect the allergen and notify you. It will also provide additional support information as needed based on your emotional state."

[1778] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1779] Step 1: Register your allergy information

[1780] The user starts the application and opens the "Register Allergy Information" screen. As input, the user selects and enters the ingredients to which they are allergic (e.g., "wheat," "peanuts," "dairy products") from a list. The device saves this allergy information in local storage and generates a request to send to the server. The input data is the user's allergy information, and the output data is the data request sent to the server. Specifically, the device sends the data to the server using a REST API.

[1781] Step 2: Photograph and send the food package

[1782] A user takes a photo of the packaging of a food item they are considering purchasing with their smartphone camera. The input is the food packaging to be photographed, and the output is the captured image data. The device temporarily stores this image data and prepares a request to send to the server. Specifically, when the user presses the send button, the device uploads the image data to the server. An HTTP POST request is used to upload the image data.

[1783] Step 3: Parse package information

[1784] The server extracts text information from the received image data using optical character recognition (OCR) technology. The input is image data and the output is extracted text data. The OCR engine uses services such as Google Cloud Vision API. Specifically, the server sends data to the OCR engine and receives text data as the analysis result.

[1785] Step 4: Check against the database

[1786] The server compares the preprocessed text data with the allergy database. The input is the extracted text data and the user's allergy information, and the output is the comparison result. The comparison process identifies ingredients that match the allergens in the database. Specifically, the text data is used as a search query to query the database to see if there are any matching ingredients.

[1787] Step 5: Notification of decision

[1788] The server generates a warning message based on the matching result and sends it to the terminal. The input is the matching result and the output is the warning message. Specifically, the server analyzes the matching result and generates a warning message if the product contains an ingredient to which the user is allergic. The terminal displays this message to the user via a pop-up or notification bar.

[1789] Step 6: Sentiment Analysis

[1790] The device collects user reactions through cameras and sensors and analyzes their emotional state. The input is user reaction data, and the output is emotional state data. The emotion engine uses the Azure Emotion API, etc. Specifically, it sends reaction data to the emotion engine and receives the emotional state as the analysis result.

[1791] Step 7: Adjust message display

[1792] The server adjusts the display of warning messages based on the emotional state. The input is the emotional state data and the current warning message, and the output is the adjusted warning message. Specifically, it analyzes the emotional state data and provides friendly messages or additional support information as needed.

[1793] Step 8: Accumulate data

[1794] The server accumulates the judgment results and emotion analysis results in a database. The input is the judgment results and emotional state data, and the output is an updated database. Specifically, the server saves this data in the database and makes it available for future analysis.

[1795] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1796] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1797] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1798] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1799] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1800] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1801] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1802] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1803] The emotion map defines two emotions that prom...

Claims

1. a means for the user to input their allergy information; A means of photographing food packaging; means for transmitting the image to a server; a means for the server to analyze the image and extract a list of ingredients of the food; A means for comparing the extracted ingredient list with an allergy database to determine whether or not an allergen is present; means for notifying a user of the determination result; A means of storing the judgment results in a database A system including:

2. The system of claim 1, further comprising means for displaying a warning message to a user if an allergenic ingredient is contained.

3. 10. The system of claim 1, wherein the server includes means for using optical character recognition (OCR) techniques to extract text from the image.

Images