System

Abstract

A system is provided.SOLUTION: A system including camera means for capturing images of food, image processing means for pre-processing the captured images, communication means for transmitting the pre-processed images to a cloud server, generative model analysis means for analyzing the images at the cloud server and identifying food components, matching means for matching the food components to an allergy database, and visual and audio warning means for displaying a warning to a user if the matching results in the detection of the presence of an allergen.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, the number of individuals with food allergies is increasing, and the risk is particularly severe for children. Anaphylactic shock caused by food allergies can be life-threatening if allergens cannot be detected quickly and accurately. Conventional methods require manual label reading to identify food ingredients, which is time-consuming and labor-intensive. Furthermore, in the event of a disaster, label information may not even be available. Therefore, there is a need for a system that can quickly and accurately detect allergens and provide immediate visual and audio warnings. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides a system with the following features: It includes a camera means and image processing means that preprocess captured food images and send them to a cloud server. It also includes a generative model analysis means in the cloud server that analyzes the images based on a generative model to identify and extract food ingredients, and a comparison means that compares the identified ingredient information with an allergy database. It also includes a visual warning means and an audio warning means that display a warning to the user if an allergen is detected as a result of the comparison. It also includes a history management function that provides warnings based on the past intake history of allergens, and a function that can detect edible objects, poisons, and harmful substances in the event of a disaster, thereby realizing an integrated system that reduces the risk of allergies.

[0006] "Camera means" refers to a device for capturing an image of food, and has the ability to obtain an image of food within the user's field of view.

[0007] "Image processing means" refers to devices or algorithms for pre-processing captured images, such as noise reduction and resolution adjustment.

[0008] "Communication means" refers to the network interface and protocol for transmitting captured and pre-processed images to the cloud server, and has the ability to transmit data via HTTP requests.

[0009] "Generative model analysis means" refers to the AI ​​model or machine learning algorithm used to analyze the received images and identify food ingredients in the cloud server.

[0010] The "comparison means" has the function of comparing the food ingredient information identified by the generative model with an allergy database, and plays a role in confirming the presence of allergens.

[0011] "Visual warning means" means a device or software that displays a warning message on a screen to the user if an allergen is detected.

[0012] "Audio warning means" means a device or software that provides an audio warning to a user if an allergen is detected.

[0013] "History management function" refers to software or a database that has the function of recording and managing the past intake history of allergens and providing warnings based on that history.

[0014] "Disaster detection function" refers to a device or software that has the function to detect edible items, poisons, and harmful substances in the event of a disaster, and assists users in making safe food selections. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention relates to a system for quickly and accurately detecting allergens and providing a warning to a user. Specific embodiments are described below.

[0037] Overall system configuration

[0038] This system consists of the following main components:

[0039] 1. Camera means (terminal):

[0040] It is a camera built into glasses worn by the user that captures images of food within the user's field of view.

[0041] 2. Image processing means (terminal):

[0042] A device or algorithm that preprocesses a captured image to remove noise or adjust resolution.

[0043] 3. Communication means (terminal):

[0044] A network interface and protocol for sending preprocessed images to a cloud server.

[0045] 4. Generative model analysis means (server):

[0046] An AI model or machine learning algorithm that analyzes images on a cloud server and identifies food ingredients.

[0047] 5. Verification method (server):

[0048] A function that compares identified ingredient information with an allergy database to confirm the presence of allergens.

[0049] 6. Visual and audio warning means (terminals):

[0050] A device or software that provides visual and audio warnings to the user if an allergen is detected.

[0051] 7. History management function (terminal or server):

[0052] A function that manages past intake history of allergens and provides warnings based on that information.

[0053] 8. Disaster detection function (terminal or server):

[0054] A function that identifies edible, poisonous, and harmful substances during a disaster and provides information to users.

[0055] Basic operation flow

[0056] 1. Image Capture and Preprocessing (Device)

[0057] When a user looks at a food item, the device's camera captures an image of it. The captured image undergoes pre-processing, such as adjusting resolution and brightness and removing noise, and is then temporarily saved to local storage.

[0058] 2. Sending images to the server (terminal)

[0059] The device sends the preprocessed image to the cloud server by uploading the data via an HTTP request (POST request).

[0060] 3. Food ingredient analysis (server)

[0061] The server analyzes the received images using an AI model. The generative model analysis means processes the image data to identify and extract food ingredients.

[0062] 4. Verification (Server)

[0063] The identified ingredient information is checked against an allergy database. A checking means on the server compares the ingredient information with the database to confirm the presence of allergens.

[0064] 5. Alert notification (terminal)

[0065] If an allergen is detected, the server sends the results to the device, which then displays a visual warning message on the screen and provides an audio warning to the user based on the results received.

[0066] Specific examples

[0067] For example, imagine a user picking up a chocolate bar at a supermarket. When the user looks at the product, the camera in the glasses captures an image, which is preprocessed on the device and then sent to the server. The server analyzes the image using an AI model to identify its ingredients: sugar, milk, and peanuts. This ingredient information is then compared with an allergy database, confirming that the bar contains peanuts. The server then sends this information to the device, which displays "Dangerous Food 💀" on the screen and warns the user audibly that "Contains peanuts."

[0068] In this way, this system reduces the risk of food allergies by quickly and accurately detecting allergens in foods that come into the user's field of vision and providing immediate visual and audio warnings. Furthermore, by incorporating history management and disaster detection functions, it protects the user's safety in a variety of situations.

[0069] The processing flow will be explained below.

[0070] Step 1:

[0071] A user picks up food at a supermarket, etc. A camera built into the glasses captures an image of the food in the user's field of view.

[0072] Step 2:

[0073] It takes the image captured by the device's camera and performs pre-processing on it, specifically adjusting the image resolution and applying a noise reduction filter.

[0074] Step 3:

[0075] The device temporarily stores the preprocessed image in local storage.

[0076] Step 4:

[0077] The device sends the saved images to the cloud server using an HTTP request (POST request), uploading the data to the server via the Internet.

[0078] Step 5:

[0079] The server receives the image data and stores it in a database.

[0080] Step 6:

[0081] The server analyzes the received image using a generative model analysis method. Specifically, the image is input into an AI model to identify and extract food ingredients.

[0082] Step 7:

[0083] The server compares the identified ingredient information with an allergy database to see if the product contains any allergens.

[0084] Step 8:

[0085] The server returns the matching results to the device in a format such as JSON.

[0086] Step 9:

[0087] The device receives the results from the server, analyzes the received data, and extracts the necessary information.

[0088] Step 10:

[0089] If your device contains an allergen, it will display a visual warning message on the screen, such as "Dangerous Food 💀."

[0090] Step 11:

[0091] The device will provide audible warnings to the user, using text-to-speech to say things like "Contains peanuts."

[0092] These steps allow users to quickly and accurately find out the allergens in food, allowing them to make safe food choices.

[0093] Example 1

[0094] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0095] In modern society, there is a demand for the rapid and accurate detection of allergens contained in food and the provision of warnings to users. However, conventional methods have problems such as delays in allergen detection and a lack of accuracy. Furthermore, in times of disaster, it is difficult to determine which foods are safe to consume. Furthermore, there is a lack of systems that can manage the past intake history of allergens and provide appropriate warnings based on that information.

[0096] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0097] In this invention, the server includes an imaging means for capturing an image of the food, an image processing means for preprocessing the captured image, a communication means for transmitting the preprocessed image to the data server, a generative AI model analysis means for analyzing the image in the data server and identifying the food components, a comparison means for comparing the food components with an allergy database, and a visual warning means and an audio warning means for displaying a warning to the user if the presence of an allergen is detected as a result of the comparison. This makes it possible to quickly and accurately detect allergens contained in food and provide an immediate warning to the user.

[0098] "Imaging means" refers to a device used to capture an image of a food item, and includes, for example, a camera or an image sensor.

[0099] "Image processing means" refers to devices or algorithms for pre-processing captured images, such as adjusting resolution, correcting brightness, and removing noise.

[0100] "Communication means" refers to a device or protocol for transmitting preprocessed images to a data server, including a network interface and communication software.

[0101] "Generative AI model analysis means" includes artificial intelligence models and machine learning algorithms for analyzing images received by the data server and identifying food ingredients.

[0102] A "verification means" is a device or software that compares identified food ingredients with an allergy database to confirm the presence of allergens.

[0103] A "visual alert means" is a device or interface for displaying a visual alert to a user if an allergen is detected.

[0104] An "audio warning means" is a device or system for providing an audio warning to a user when an allergen is detected.

[0105] The "history management function" is a function for managing the past intake history of allergens and providing warnings to the user based on that history.

[0106] The "disaster detection function" is a function that identifies edible items, poisons, and harmful substances during a disaster and provides information to the user.

[0107] MODE FOR CARRYING OUT THE INVENTION

[0108] The present invention relates to a system for quickly and accurately detecting allergens and providing a warning to a user. DETAILED DESCRIPTION OF THE INVENTION ...

[0109] System configuration

[0110] This system consists of the following main components:

[0111] 1. Imaging means (terminal):

[0112] A camera built into the glasses worn by the user captures images of food within the user's field of view. Specifically, a small camera or image sensor is used.

[0113] 2. Image processing means (terminal):

[0114] The captured image is pre-processed by the device's built-in image processor or algorithms, which may include adjusting the resolution, correcting brightness, and removing noise, making the image easier to analyze.

[0115] 3. Communication means (terminal):

[0116] The preprocessed image is sent to the cloud server via the device's communication interface. Specifically, the data is uploaded using the POST method of the HTTP request.

[0117] 4. Generative AI model analysis means (server):

[0118] The cloud server analyzes the submitted images using generative AI models, which are used to identify food ingredients. Specifically, these include algorithms using machine learning and deep learning techniques.

[0119] 5. Verification method (server):

[0120] The server's verification means compares the identified ingredient information with an allergy database to confirm the presence of allergens. By comparing the information with the database, the presence of allergens is detected.

[0121] 6. Visual and audio warning means (terminals):

[0122] If an allergen is detected, the server sends the result to the device, which then provides a visual and audio warning to the user based on the received information. Specifically, a warning message is displayed on the device's display and the warning content is read aloud.

[0123] 7. History management function (terminal or server):

[0124] It has a function to manage the past intake history of allergens and provide appropriate warnings based on that, which allows for customized warnings based on the user's allergy history.

[0125] 8. Disaster detection function (terminal or server):

[0126] The system has the ability to identify edible foods, poisons, and harmful substances and provide information to users, enabling them to make safe food choices even during disasters.

[0127] Specific operation example

[0128] For example, consider a user picking up a chocolate bar at a supermarket. When the user looks at the product, the camera built into the glasses captures an image, which is then sent to a cloud server after the device adjusts the resolution and removes noise. A generative AI model on the server analyzes the image and identifies ingredients such as "sugar, milk, and peanuts." The server's matching means compares the ingredient information with an allergy database and confirms that it contains peanuts. The results are sent to the device, which displays "Dangerous Food 💀" and provides a voice warning to the user that "Contains peanuts."

[0129] Also, a specific example of a prompt sentence to input to the generative AI model is as follows:

[0130] "I have provided an image. This image shows a food product. Please identify the ingredients in the food product and detect allergens. Please return the ingredients in the full document. Please check if the ingredients contain the following allergens: milk, peanuts, eggs."

[0131] Using this prompt, the AI ​​model analyzes food ingredient information and detects the presence of allergens.

[0132] Other Features

[0133] This system has the ability to manage the user's allergen intake history, allowing it to provide more detailed and personalized warnings. It also has the ability to identify foods and harmful substances that can be ingested in the event of a disaster, ensuring safety in emergencies.

[0134] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0135] The flow of this system's program processing

[0136] Step 1:

[0137] (User sees food)

[0138] When a user looks at a food item, the device's imaging means (built-in camera) automatically captures an image of the food item. For example, when a user picks up a chocolate bar in a supermarket, an image of the chocolate bar's packaging is captured. Input: User's visual behavior Output: Captured image data

[0139] Step 2:

[0140] (Device preprocesses the image)

[0141] The captured image is pre-processed by the device's image processing means. Specifically, the resolution is adjusted, brightness is corrected, and noise is removed. This prepares the image in a format that is easy to analyze. Input: Captured image data Output: Pre-processed image data

[0142] Step 3:

[0143] (The device sends the image to the cloud server)

[0144] The preprocessed image is sent to the cloud server via the device's communication means. Specifically, the image data is uploaded using an HTTP POST request. Input: Preprocessed image data Output: Image data sent to the server

[0145] Step 4:

[0146] (The server analyzes the image using a generative AI model)

[0147] The cloud server analyzes the received image data using the AI ​​model analysis means. The AI ​​model performs data calculations to identify food ingredients based on the input image data. For example, the AI ​​model extracts ingredient information such as "sugar, milk, peanuts" from an image of a chocolate bar package. Input: Image data sent to the server Output: Identified food ingredient information

[0148] Step 5:

[0149] (The server checks the ingredient information against the allergy database)

[0150] The server's verification means compares the identified ingredient information with the allergy database. This verification confirms the presence of allergens in the food. For example, the verification result confirms that the food contains peanuts. Input: Identified food ingredient information Output: Verification result (presence or absence of allergens)

[0151] Step 6:

[0152] (The server sends the matching results to the device.)

[0153] Once the matching result is obtained, the server sends it to the device. Input: Matching result Output: Alert information sent to the device

[0154] Step 7:

[0155] (Device provides warning to user)

[0156] The device will provide a visual and audio warning to the user based on the received alert information. Specifically, it will display "Dangerous Food 💀" on the device's display and warn the user by voice, "Contains peanuts." Input: Alert information sent to the device Output: Visual and audio warning to the user

[0157] Step 8:

[0158] (The device or server manages the history.)

[0159] The history management function on the device or server records information about the ingredients of the food the user is about to consume and the history of allergen detection. This improves the accuracy of future warnings. Input: User's intake history and detection history. Output: Updated history data.

[0160] Step 9:

[0161] (The terminal or server will detect food in the event of a disaster.)

[0162] In the event of a disaster, the terminal or server provides information on foods that can be consumed by the user and on harmful substances. This allows users to safely select foods even in an emergency. Input: Environmental information and food information during a disaster. Output: Safe food information provided to the user.

[0163] The above is the flow of specific processing steps for this system.

[0164] (Application example 1)

[0165] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0166] Conventional allergy detection systems lack the speed and accuracy to quickly identify food allergens and provide appropriate warnings to users. Furthermore, methods for accurately identifying food allergens in moving environments, such as autonomous vehicles, have not been fully established. Therefore, an efficient and reliable system for reducing food allergy risks is needed.

[0167] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0168] In this invention, the server includes an image capturing means, an image processing means, a communication means, a generative model analysis means, a matching means, a visual warning means, an audio warning means, and a means for capturing images obtained from an internal camera of the autonomous vehicle and identifying food allergens. This makes it possible to quickly and accurately identify food allergens in a moving autonomous vehicle and provide appropriate visual and audio warnings to the user.

[0169] - "Photography means" refers to any device or technology used to capture an image of a food item.

[0170] "Image processing means" refers to devices or techniques for pre-processing captured images, such as noise reduction and resolution adjustment.

[0171] "Communication means" refers to the network interface and protocols used to transmit preprocessed images to a cloud data center.

[0172] "Generative model analysis means" refers to AI models and machine learning algorithms that analyze images in a cloud data center and identify food elements.

[0173] "Matching means" refers to the technology or method used to match food elements with an allergy database to confirm the presence of allergens.

[0174] "Visual warning means" means any device or software that provides a visual warning to the user upon detecting the presence of an allergen.

[0175] "Audio warning means" means a device or software that provides an audio warning to a user when it detects the presence of an allergen.

[0176] "Internal camera in an autonomous vehicle" refers to a camera installed inside an autonomous vehicle that captures images of food brought in by passengers.

[0177] The present invention relates to a system for quickly and accurately detecting food allergens in autonomous vehicles and providing warnings to users. The system consists of the following main components:

[0178] Overall system configuration

[0179] 1. Photography Method:

[0180] This is a camera installed inside the autonomous vehicle that captures images of food brought in by passengers, specifically when a passenger in the vehicle picks up the food.

[0181] 2. Image processing means:

[0182] Software for pre-processing the captured images to remove noise and adjust the resolution. For example, the OpenCV library can be used for image processing.

[0183] 3. Means of communication:

[0184] A network interface for sending preprocessed images to a cloud data center, specifically by sending data via HTTP POST requests.

[0185] 4. Generative model analysis methods:

[0186] AI models for analyzing images and identifying food elements in cloud data centers, for example, using the TensorFlow library to identify food ingredients.

[0187] 5. Verification method:

[0188] The identified food elements are checked against an allergy database to confirm the presence of allergens. Database checks are performed using an SQL server.

[0189] 6. Visual and audio warning means:

[0190] If an allergen is detected, the device will provide users with visual and audio alerts: visual alerts will be displayed on the smartphone screen and audio alerts will be played through the speaker.

[0191] 7. Means for capturing images obtained from an internal camera of an autonomous vehicle:

[0192] An interface for utilizing images acquired by the internal camera throughout the system. This interface allows real-time images acquired by the camera to be imported into the system.

[0193] Specific operation flow

[0194] 1. Cameras inside the self-driving vehicle capture images of food brought in by passengers.

[0195] 2. This image is pre-processed by image processing means to remove noise and adjust the resolution.

[0196] 3. The pre-processed images are sent to the cloud data center via communication means.

[0197] 4. The cloud data center uses generative model analysis tools to analyze the image and identify food elements.

[0198] 5. The identified food elements are matched against an allergy database via a matching means to confirm the presence of allergens.

[0199] 6. If an allergen is detected, a warning is provided to the user through visual and audio warning means.

[0200] Prompt Sentence Examples

[0201] Based on the image of food captured by the user with the camera, identify the ingredients contained in it, match it with an allergy database, and return the results.

[0202] Image data:<image_data>

[0203] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0204] Step 1:

[0205] Input: Food images obtained from a camera mounted inside an autonomous vehicle.

[0206] How it works: The device activates a camera installed inside the autonomous vehicle and captures an image of the food brought in by the passenger.

[0207] Output: Image data of the captured food.

[0208] Step 2:

[0209] Input: Captured food image data.

[0210] What it does: The device pre-processes the captured image by removing noise and adjusting the resolution, specifically by resizing and color correcting the image using the OpenCV library.

[0211] Output: Preprocessed image data.

[0212] Step 3:

[0213] Input: Preprocessed image data.

[0214] How it works: The device sends the preprocessed image data to the cloud data center, where it uploads the image data to the specified URL using an HTTP POST request.

[0215] Output: Notification of completion of image data transmission to the cloud data center.

[0216] Step 4:

[0217] Input: Image data sent to the cloud data center.

[0218] How it works: The server receives the image data and performs generative model analysis using the AI ​​model. It then uses the TensorFlow library to perform data calculations to identify food elements from the image.

[0219] Output: Parsed food element data.

[0220] Step 5:

[0221] Input: Food element data analyzed by the AI ​​model.

[0222] How it works: The server checks the food element data it retrieves against the allergy database. Using an SQL server, it compares it with existing data in the database to determine whether or not any allergens are present.

[0223] Output: Matching result (whether or not allergens were detected and details).

[0224] Step 6:

[0225] Input: Whether or not allergens are detected and detailed data.

[0226] How it works: The server sends the matching results to the device, and if an allergen is detected, the device provides visual and audio warnings to the user. A warning message is displayed on the smartphone screen and an audio warning is played through the speaker.

[0227] Output: Visual and audio warnings to the user.

[0228] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0229] The present invention relates to a system that quickly and accurately detects allergens and provides a warning to the user, and further combines it with an emotion engine that recognizes the user's emotions, allowing the content and method of the warning to be adapted to the user's emotional state. Specific embodiments are described below.

[0230] Overall system configuration

[0231] This system consists of the following main components:

[0232] 1. Camera means (terminal):

[0233] A camera built into glasses worn by the user that captures images of food in the user's field of view and images of the user's face.

[0234] 2. Image processing means (terminal):

[0235] A device or algorithm that preprocesses a captured image to remove noise or adjust resolution.

[0236] 3. Communication means (terminal):

[0237] A network interface and protocol for sending preprocessed images to a cloud server.

[0238] 4. Generative model analysis means (server):

[0239] An AI model or machine learning algorithm that analyzes images on a cloud server and identifies food ingredients.

[0240] 5. Verification method (server):

[0241] A function that compares identified ingredient information with an allergy database to confirm the presence of allergens.

[0242] 6. Emotion engine (terminal or server):

[0243] An algorithm that recognizes emotions from a user's facial expressions and assesses their stress level and emotional state.

[0244] 7. Visual and audio warning means (terminals):

[0245] A device or software that provides visual and audio warnings to the user when an allergen is detected, based on the results of the emotion engine.

[0246] 8. History management function (terminal or server):

[0247] A function that manages past intake history of allergens and provides warnings based on that information.

[0248] 9. Disaster detection function (terminal or server):

[0249] A function that identifies edible, poisonous, and harmful substances during a disaster and provides information to users.

[0250] Basic operation flow

[0251] 1. Image Capture and Preprocessing (Device)

[0252] When a user looks at a food item, the device's camera captures an image of it, along with an image of the user's face. The captured image is then pre-processed to adjust resolution, brightness, and noise reduction.

[0253] 2. Sending images to the server (terminal)

[0254] The device sends the preprocessed food and face images to the cloud server by uploading the data via an HTTP request (POST request).

[0255] 3. Food ingredient analysis and emotion recognition (server)

[0256] The server analyzes the received food image data using an AI model to identify and extract food ingredients. At the same time, it inputs facial image data into an emotion engine to recognize the user's emotions. The emotion engine evaluates the user's stress level and emotional state.

[0257] 4. Verification (Server)

[0258] The identified ingredient information is checked against an allergy database to determine whether it contains any allergens, and the emotion engine results are also sent to the device.

[0259] 5. Alert notification (terminal)

[0260] If an allergen is detected, the device will display a warning message based on the results of the emotion engine. For example, if the user is relaxed, it will display a gentle message such as "Be careful," while if the user is stressed, it will display a more emphatic message such as "Avoid immediately." Audio warnings are also adjusted in the same way.

[0261] Specific examples

[0262] For example, consider a user choosing food in a cafeteria. The user looks at the food and the camera captures an image of it. At the same time, the user's facial expression is also captured. This data is sent from the device to the server, which analyzes the food's ingredients and identifies them as "egg yolk, soy, and wheat flour." Meanwhile, the emotion engine analyzes the user's facial expression and recognizes that the user is nervous. The analysis confirms that the food contains soy, an allergen. The device receives this information, displays an emergency message on the screen saying "Dangerous food 💀," and issues a voice warning saying "Contains soy, be careful." The user can avoid the food based on this warning, thereby reducing the risk of allergies.

[0263] In this way, this system quickly and accurately detects allergens in food and provides warnings according to the user's emotional state, thereby achieving safe and effective allergy management.By combining it with history management and disaster detection functions, it is possible to protect user safety in a variety of situations.

[0264] The processing flow will be explained below.

[0265] Step 1:

[0266] The user looks at food, and the camera built into the glasses captures an image of the food in the user's field of view and an image of the user's face.

[0267] Step 2:

[0268] The device captures food images and facial images, which are then pre-processed to remove noise and adjust resolution.

[0269] Step 3:

[0270] The device temporarily stores the preprocessed image data in local storage.

[0271] Step 4:

[0272] The device sends the stored image data to the cloud server using an HTTP POST request, sending the image data over the network.

[0273] Step 5:

[0274] The server analyzes the received image data. The server's generative model analysis means processes the food image using an AI model to identify and extract food ingredients.

[0275] Step 6:

[0276] The server inputs facial images into the emotion engine, which recognizes emotions from the user's facial expressions and evaluates their stress level and emotional state.

[0277] Step 7:

[0278] The server checks the identified food ingredient information against an allergy database. A checking means compares the ingredient information with the database to confirm the presence of allergens.

[0279] Step 8:

[0280] The server returns the food ingredient matching results and the emotion engine evaluation results to the device. The results are formatted in JSON and sent as an HTTP response.

[0281] Step 9:

[0282] The terminal analyzes the results received from the server, and if an allergen is found, a visual warning means and an audio warning means are activated.

[0283] Step 10:

[0284] Based on the results of the emotion engine, the device displays the most appropriate visual warning to the user. For example, if the user is relaxed, it displays "Caution required," and if the user is tense, it displays "Avoid immediately."

[0285] Step 11:

[0286] The device provides the user with a voice alert based on the results of the emotion engine. The voice alert is also adjusted according to the user's emotions. For example, if the user is relaxed, the alert will be softer in tone, and if the user is tense, the alert will be more emphatic.

[0287] This allows users to quickly and accurately learn about food allergens, and also to receive appropriate warnings according to their current emotions.

[0288] Example 2

[0289] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0290] In modern society, the presence of allergens in food poses a serious risk to the user's health. Furthermore, conventional allergy detection systems simply detect allergens and are unable to adapt the content and method of warnings to take the user's emotional state into account. This can make it difficult for users to properly accept the warnings, and can result in insufficient reduction of allergy risk. Furthermore, the system does not take into account the identification of edibles, poisons, or harmful substances in the event of a disaster. Thus, providing a safe environment that takes the user's emotional state into account is a challenge for allergy risk management systems.

[0291] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an imaging means for capturing an image of the food, an image processing means for preprocessing the captured image, a communication means for transmitting the preprocessed image to a remote server, a generative model analysis means for analyzing the image in the remote server and identifying food components, a comparison means for comparing the food components with an allergy database, a visual warning means and an audio warning means for displaying a warning to the user if the presence of an allergen is detected as a result of the comparison, an emotion recognition means for capturing an image of the user's face and evaluating their emotional state, and an adaptive control means for adapting the content and method of the warning based on the emotional state. This enables safe and effective allergy management by quickly and accurately detecting allergens contained in food and providing a warning according to the user's emotional state.

[0292] "Photographing means for capturing images of food" is a general term for devices and techniques that record images of food within the user's field of view.

[0293] "Image processing means for pre-processing captured images" is a general term for devices or software that perform noise reduction and resolution adjustment to improve the quality of captured images.

[0294] "Communication means for transmitting preprocessed images to a remote server" is a general term for interfaces and protocols for uploading processed image data to a remote server over a network.

[0295] "Generative model analysis means that analyzes images on a remote server and identifies food components" is a general term for technologies and algorithms that process received image data using an AI model to identify and extract food components.

[0296] "Means for matching food ingredients with an allergy database" is a general term for technologies and functions for comparing identified food ingredients with an existing allergy database to confirm the presence of allergens.

[0297] "Visual and audio warning means that display a warning to the user when the presence of an allergen is detected as a result of the comparison" is a general term for devices and functions that provide a warning by screen display or audio to inform the user that an allergen is present.

[0298] "Emotion recognition means for capturing a user's facial image and evaluating the user's emotional state" is a general term for devices and algorithms for analyzing a user's facial expression and evaluating the user's emotional state.

[0299] "Adaptive control means for adapting the content and method of warning based on the emotional state" is a general term for techniques and functions for changing the content and method of warning depending on the emotional state of the user.

[0300] The present invention relates to a system that quickly and accurately detects allergens and provides a warning to the user. This system is combined with an emotion engine that recognizes the user's emotions, allowing the content and method of the warning to be adapted to the user's emotional state. Specific embodiments are described below.

[0301] Overall system configuration

[0302] This system consists of the following main components:

[0303] 1. Shooting method (device):

[0304] A camera built into glasses worn by the user captures images of food in the user's field of view as well as images of the user's face. For example, if a user is choosing a salad in a cafeteria, the camera in the glasses automatically captures the salad and the user's facial expression.

[0305] 2. Image processing means (terminal):

[0306] The captured images are pre-processed to improve their quality. This pre-processing includes adjusting the resolution and removing noise. For example, the resolution is standardized to 1080p and a Gaussian filter is applied to make the image clearer.

[0307] 3. Communication means (terminal):

[0308] A network interface and protocol for transmitting preprocessed images to a remote server using HTTP POST requests to upload image data over Wi-Fi or cellular networks.

[0309] 4. Generative model analysis means (server):

[0310] The remote server inputs the received image data into an AI model to identify and extract the food's ingredients. This analysis uses AI models such as SVM and ResNet. For example, the food's ingredients are identified as "egg yolk, soybeans, and wheat flour."

[0311] 5. Emotion recognition means (server):

[0312] The remote server inputs facial image data into the emotion engine, which analyzes the user's facial expressions and evaluates their emotional state. The emotion engine uses deep learning algorithms to detect the user's stress level and emotional state. For example, it can recognize that the user is "tense."

[0313] 6. Verification method (server):

[0314] The remote server checks the identified ingredient information against an allergy database to confirm the presence of allergens, for example, soy is found to be an allergen.

[0315] 7. Adaptive control means (terminal):

[0316] Based on the analysis results, the device will adapt the content and method of warnings to suit the user's emotional state. For example, if the user is nervous, an emergency message will be displayed on the screen saying "Dangerous Food 💀" and a voice warning will be given saying "Contains soy, be careful."

[0317] Specific examples

[0318] Usage scene in the cafeteria

[0319] Consider a scenario in which a user is choosing food in a restaurant. At this time, a camera captures images of the food the user is looking at and their facial expression. The device preprocesses these images and sends them to a remote server. The server analyzes the food's ingredients and identifies "egg yolk, soybeans, and wheat flour." At the same time, the emotion engine analyzes the user's facial expression and recognizes that the user is "nervous."

[0320] If the analysis results confirm that the food contains soy, the device will display a warning message according to the user's "tension" state. "Dangerous Food 💀" will be displayed on the screen, and a voice message will be played saying, "Contains soy. Be careful." Users can reduce their risk of allergies by avoiding the food based on this warning.

[0321] Prompt Sentence Examples

[0322] "Imagine a scene where a user is choosing food in a restaurant. The camera captures images of the food the user is looking at and their facial expressions. The server then analyzes the food's ingredients from the received data and identifies allergens. At the same time, it recognizes the user's emotional state and displays appropriate warning messages."

[0323] As described above, the present invention achieves safe and effective allergy management by quickly and accurately detecting allergens in food and providing warnings tailored to the user's emotional state.

[0324] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0325] Step 1:

[0326] User views food:

[0327] When a user looks at a food item, a camera built into the device captures an image of the food item and an image of the user's face.

[0328] Input: Food and its facial expression in the user's field of view through the glasses.

[0329] Output: Captured food images and user's face image data.

[0330] Step 2:

[0331] The device performs image preprocessing:

[0332] The device performs pre-processing on the captured image, such as adjusting the resolution and removing noise.

[0333] Input: A captured food image and a user's face image.

[0334] Data processing: Unification of resolution (e.g. 1080p), noise removal (e.g. applying Gaussian filter).

[0335] Output: Preprocessed food images and face images.

[0336] Step 3:

[0337] The device sends data to the server:

[0338] The device sends the preprocessed image data to the remote server using an HTTP POST request.

[0339] Input: Preprocessed food images and face images.

[0340] Data processing: Packaging data into HTTP requests.

[0341] Output: Image data sent to the remote server.

[0342] Step 4:

[0343] The server analyzes the food ingredients:

[0344] The server inputs the received food image data into an AI model to identify and extract the food's ingredients.

[0345] Input: Food image data sent to the server.

[0346] Data processing: Image analysis using AI models (e.g., SVM and ResNet).

[0347] Output: Identified food ingredients (e.g., "egg yolk, soy, wheat flour").

[0348] Step 5:

[0349] The server analyzes the user's emotions:

[0350] The server inputs the received facial image into an emotion engine, analyzes the user's facial expression, and evaluates their emotional state.

[0351] Input: User's facial image data sent to the server.

[0352] Data processing: Facial expression analysis using emotion engine (deep learning algorithm).

[0353] Output: The user's assessed emotional state (e.g., "tense").

[0354] Step 6:

[0355] The server verifies the ingredients:

[0356] The server compares the identified food ingredient information with an allergy database to determine whether the food contains any allergens.

[0357] Input: Identified food ingredient information.

[0358] Data processing: Comparison with allergy database.

[0359] Output: Matching result (e.g. "Soy is an allergen").

[0360] Step 7:

[0361] The server sends the analysis results to the device:

[0362] The server sends the results of the user's emotion analysis and food ingredient analysis to the terminal.

[0363] Input: Matching result and emotional state.

[0364] Data processing: Generate JSON data containing the information.

[0365] Output: Analysis result data sent to the device.

[0366] Step 8:

[0367] The device provides a warning to the user:

[0368] Based on the analysis results, the device provides the user with visual and audio warnings, adapting the content and method of the warning depending on the user's emotional state.

[0369] Input: Analysis result data (matching results and emotional state).

[0370] Data processing: Generate warning messages and configure display and sound settings.

[0371] Output: Visual and audio warning messages (e.g. "Dangerous Food 💀" and "Contains soy, be careful").

[0372] The above are the specific processing steps of the program of this system.

[0373] (Application example 2)

[0374] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0375] Rapid and accurate detection of allergens in food is very important to many people. However, systems that provide uniform warnings without considering the user's emotional state may cause excessive stress to the user. In particular, in situations such as food delivery, appropriate warnings based on the user's emotional state are required. Therefore, it is necessary to provide a system that can detect allergens in food and provide warnings based on the user's emotional state.

[0376] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0377] In this invention, the server includes a camera means for capturing images of food, an image processing means for preprocessing the captured images, a communication means for transmitting the preprocessed images to a cloud server, a generative model analysis means for analyzing the images in the cloud server and identifying food ingredients, a comparison means for comparing the food ingredients with an allergy database, an emotion engine for recognizing emotions from the user's facial expressions and evaluating their stress level and emotional state, and a visual warning means and an audio warning means for displaying a warning to the user based on the results of the emotion engine if the presence of an allergen is detected as a result of the comparison. This makes it possible to quickly and accurately detect allergens in food and provide an appropriate warning according to the user's emotional state.

[0378] "Camera means" refers to a device that is built into a user-worn device and has the ability to capture images of food and the user's face.

[0379] "Image processing means" refers to devices or algorithms that pre-process captured images to remove noise or adjust resolution.

[0380] "Communication means" refers to the network interface and protocol used to send preprocessed images to the cloud server.

[0381] The "generative model analysis means" is an AI model or machine learning algorithm that analyzes images received by the cloud server and identifies food ingredients.

[0382] The "comparison means" is a function that compares the identified food ingredient information with an allergy database to confirm the presence of allergens.

[0383] The "emotion engine" is an algorithm that recognizes emotions from the user's facial expressions and evaluates their stress level and emotional state.

[0384] A "visual warning means" is a device or software that displays a visual warning to the user based on the results of the emotion engine when an allergen is detected.

[0385] The "audio warning means" is a device or software that provides an audio warning to the user based on the results of the emotion engine when an allergen is detected.

[0386] The "history management function" is a function that manages the past intake history of allergens and provides warnings based on that history.

[0387] The "disaster detection function" is a function that identifies edible items, poisons, and harmful substances during a disaster and provides information to the user.

[0388] The present invention is a system that quickly and accurately detects food allergens and provides appropriate warnings according to the user's emotional state. The overall system configuration consists of the following main components:

[0389] 1. Camera Means

[0390] The camera is built into a device worn by the user and captures images of the food and the user's face. This image capture collects data on the food the user is viewing and their facial expression.

[0391] 2. Image Processing Methods

[0392] The captured images are pre-processed, including noise removal, resolution adjustment, normalization, etc. Specifically, OpenCV is used to perform these image processing.

[0393] 3. Means of communication

[0394] The preprocessed image is sent to the cloud server using a network interface and a communication protocol such as an HTTP request (POST request).

[0395] 4. Generative Model Analysis Methods

[0396] The cloud server analyzes the received image data using an AI model to identify and extract food ingredients. This analysis uses machine learning algorithms such as TensorFlow and Keras. An example prompt is, "Analyze the food image to identify its ingredients and determine whether it contains any of the following substances to which you are allergic: egg yolk, soy, or wheat flour."

[0397] 5. Verification Method

[0398] The identified food ingredient information is compared with an allergy database to confirm the presence of allergens, thereby determining whether the ingredients extracted by the generative model analysis means are pre-registered allergens.

[0399] 6. Emotion Engine

[0400] This algorithm recognizes emotions based on images of a user's face and evaluates their stress level and emotional state. Emotion recognition is performed using an emotion analysis engine such as EmotionRecognitionEngine. This engine analyzes the user's facial expression data and classifies their emotional state as "tension" or "stress," etc.

[0401] 7. Visual and audio warning means

[0402] If an allergen is detected, a warning will be provided to the user based on the results of the emotion engine. For example, if the user is nervous, it will display "Dangerous Food 💀 - Contains {Detected Allergen}, Be Careful," and if the user is relaxed, it will display "Contains {Detected Allergen}, Be Careful." Audio warnings will also be adjusted accordingly.

[0403] Specific examples

[0404] For example, when a user receives a food delivery, they view the food through the smart glasses and capture an image of it using the camera means. The image is then preprocessed by the image processing means and sent to a cloud server via the communication means. The server identifies the food ingredients using the generative model analysis means and checks for allergens using the matching means. At the same time, the emotion engine recognizes the user's emotional state from their facial expressions. Finally, if an allergen is detected, the emotion engine provides the user with an appropriate visual and audio warning based on the evaluation results of the emotion engine.

[0405] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0406] Step 1:

[0407] When a user receives a food delivery, they view the food through the smart glasses. A camera means (terminal) captures an image of the food and an image of the user's face. The input is the viewed food and the user's facial expression, and the output is the captured image data. This captured image data is used in the next step.

[0408] Step 2:

[0409] The device preprocesses the captured image data using image processing means. Specifically, it uses OpenCV to perform noise removal, resolution adjustment, and normalization. The input is the captured image data, and the output is the preprocessed image data. This preprocessed image data is used in the next step.

[0410] Step 3:

[0411] The preprocessed image data is sent to the cloud server via the communication means (terminal). This communication is performed using an HTTP request (POST request) via a network interface. The input is the preprocessed image data, and the output is the upload completion status to the cloud server.

[0412] Step 4:

[0413] The cloud server analyzes the received image data using a generative model analysis method. Specifically, it uses TensorFlow and Keras to identify and extract food ingredients. Here, the prompt is "Analyze the food image, identify the ingredients contained, and determine whether the following allergens are present: egg yolk, soy, wheat flour." The input is the image data uploaded to the cloud server, and the output is the identified food ingredient information.

[0414] Step 5:

[0415] The verification means (server) verifies the identified food ingredient information against the allergy database to confirm the presence of allergens. The input is the identified food ingredient information, and the output is the allergen detection result. This result is used in the next step.

[0416] Step 6:

[0417] At the same time, the server uses an emotion engine to analyze the user's facial image and recognize their emotional state. Specifically, it uses the EmotionRecognitionEngine to analyze the user's facial expression data and classify the user's emotions as "tension" or "stress." The input is the facial image data uploaded to the cloud server, and the output is the classification result of the user's emotional state. This classification result of the emotional state is used in the next step.

[0418] Step 7:

[0419] If an allergen is detected, the device will provide a visual and audio warning to the user based on the emotion engine's results. Specifically, if the user is nervous, the device will display "Dangerous Food 💀 - Contains {Detected Allergen}, Be Careful," and if the user is relaxed, the device will display "Contains {Detected Allergen}, Be Careful." The audio warning is also adjusted in a similar way. The input is the allergen detection result and the classification result of the user's emotional state, and the output is a warning message to the user.

[0420] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0421] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0422] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0423] [Second embodiment]

[0424] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0425] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0426] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0427] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0428] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0429] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0430] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0431] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0432] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0433] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0434] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0435] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0436] The present invention relates to a system for quickly and accurately detecting allergens and providing a warning to a user. Specific embodiments are described below.

[0437] Overall system configuration

[0438] This system consists of the following main components:

[0439] 1. Camera means (terminal):

[0440] It is a camera built into glasses worn by the user that captures images of food within the user's field of view.

[0441] 2. Image processing means (terminal):

[0442] A device or algorithm that preprocesses a captured image to remove noise or adjust resolution.

[0443] 3. Communication means (terminal):

[0444] A network interface and protocol for sending preprocessed images to a cloud server.

[0445] 4. Generative model analysis means (server):

[0446] An AI model or machine learning algorithm that analyzes images on a cloud server and identifies food ingredients.

[0447] 5. Verification method (server):

[0448] A function that compares identified ingredient information with an allergy database to confirm the presence of allergens.

[0449] 6. Visual and audio warning means (terminals):

[0450] A device or software that provides visual and audio warnings to the user if an allergen is detected.

[0451] 7. History management function (terminal or server):

[0452] A function that manages past intake history of allergens and provides warnings based on that information.

[0453] 8. Disaster detection function (terminal or server):

[0454] A function that identifies edible, poisonous, and harmful substances during a disaster and provides information to users.

[0455] Basic operation flow

[0456] 1. Image Capture and Preprocessing (Device)

[0457] When a user looks at a food item, the device's camera captures an image of it. The captured image undergoes pre-processing, such as adjusting resolution and brightness and removing noise, and is then temporarily saved to local storage.

[0458] 2. Sending images to the server (terminal)

[0459] The device sends the preprocessed image to the cloud server by uploading the data via an HTTP request (POST request).

[0460] 3. Food ingredient analysis (server)

[0461] The server analyzes the received images using an AI model. The generative model analysis means processes the image data to identify and extract food ingredients.

[0462] 4. Verification (Server)

[0463] The identified ingredient information is checked against an allergy database. A checking means on the server compares the ingredient information with the database to confirm the presence of allergens.

[0464] 5. Alert notification (terminal)

[0465] If an allergen is detected, the server sends the results to the device, which then displays a visual warning message on the screen and provides an audio warning to the user based on the results received.

[0466] Specific examples

[0467] For example, imagine a user picking up a chocolate bar at a supermarket. When the user looks at the product, the camera in the glasses captures an image, which is preprocessed on the device and then sent to the server. The server analyzes the image using an AI model to identify its ingredients: sugar, milk, and peanuts. This ingredient information is then compared with an allergy database, confirming that the bar contains peanuts. The server then sends this information to the device, which displays "Dangerous Food 💀" on the screen and warns the user audibly that "Contains peanuts."

[0468] In this way, this system reduces the risk of food allergies by quickly and accurately detecting allergens in foods that come into the user's field of vision and providing immediate visual and audio warnings. Furthermore, by incorporating history management and disaster detection functions, it protects the user's safety in a variety of situations.

[0469] The processing flow will be explained below.

[0470] Step 1:

[0471] A user picks up food at a supermarket, etc. A camera built into the glasses captures an image of the food in the user's field of view.

[0472] Step 2:

[0473] It takes the image captured by the device's camera and performs pre-processing on it, specifically adjusting the image resolution and applying a noise reduction filter.

[0474] Step 3:

[0475] The device temporarily stores the preprocessed image in local storage.

[0476] Step 4:

[0477] The device sends the saved images to the cloud server using an HTTP request (POST request), uploading the data to the server via the Internet.

[0478] Step 5:

[0479] The server receives the image data and stores it in a database.

[0480] Step 6:

[0481] The server analyzes the received image using a generative model analysis method. Specifically, the image is input into an AI model to identify and extract food ingredients.

[0482] Step 7:

[0483] The server compares the identified ingredient information with an allergy database to see if the product contains any allergens.

[0484] Step 8:

[0485] The server returns the matching results to the device in a format such as JSON.

[0486] Step 9:

[0487] The device receives the results from the server, analyzes the received data, and extracts the necessary information.

[0488] Step 10:

[0489] If your device contains an allergen, it will display a visual warning message on the screen, such as "Dangerous Food 💀."

[0490] Step 11:

[0491] The device will provide audible warnings to the user, using text-to-speech to say things like "Contains peanuts."

[0492] These steps allow users to quickly and accurately find out the allergens in food, allowing them to make safe food choices.

[0493] Example 1

[0494] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0495] In modern society, there is a demand for the rapid and accurate detection of allergens contained in food and the provision of warnings to users. However, conventional methods have problems such as delays in allergen detection and a lack of accuracy. Furthermore, in times of disaster, it is difficult to determine which foods are safe to consume. Furthermore, there is a lack of systems that can manage the past intake history of allergens and provide appropriate warnings based on that information.

[0496] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0497] In this invention, the server includes an imaging means for capturing an image of the food, an image processing means for preprocessing the captured image, a communication means for transmitting the preprocessed image to the data server, a generative AI model analysis means for analyzing the image in the data server and identifying the food components, a comparison means for comparing the food components with an allergy database, and a visual warning means and an audio warning means for displaying a warning to the user if the presence of an allergen is detected as a result of the comparison. This makes it possible to quickly and accurately detect allergens contained in food and provide an immediate warning to the user.

[0498] "Imaging means" refers to a device used to capture an image of a food item, and includes, for example, a camera or an image sensor.

[0499] "Image processing means" refers to devices or algorithms for pre-processing captured images, such as adjusting resolution, correcting brightness, and removing noise.

[0500] "Communication means" refers to a device or protocol for transmitting preprocessed images to a data server, including a network interface and communication software.

[0501] "Generative AI model analysis means" includes artificial intelligence models and machine learning algorithms for analyzing images received by the data server and identifying food ingredients.

[0502] A "verification means" is a device or software that compares identified food ingredients with an allergy database to confirm the presence of allergens.

[0503] A "visual alert means" is a device or interface for displaying a visual alert to a user if an allergen is detected.

[0504] An "audio warning means" is a device or system for providing an audio warning to a user when an allergen is detected.

[0505] The "history management function" is a function for managing the past intake history of allergens and providing warnings to the user based on that history.

[0506] The "disaster detection function" is a function that identifies edible items, poisons, and harmful substances during a disaster and provides information to the user.

[0507] MODE FOR CARRYING OUT THE INVENTION

[0508] The present invention relates to a system for quickly and accurately detecting allergens and providing a warning to a user. DETAILED DESCRIPTION OF THE INVENTION ...

[0509] System configuration

[0510] This system consists of the following main components:

[0511] 1. Imaging means (terminal):

[0512] A camera built into the glasses worn by the user captures images of food within the user's field of view. Specifically, a small camera or image sensor is used.

[0513] 2. Image processing means (terminal):

[0514] The captured image is pre-processed by the device's built-in image processor or algorithms, which may include adjusting the resolution, correcting brightness, and removing noise, making the image easier to analyze.

[0515] 3. Communication means (terminal):

[0516] The preprocessed image is sent to the cloud server via the device's communication interface. Specifically, the data is uploaded using the POST method of the HTTP request.

[0517] 4. Generative AI model analysis means (server):

[0518] The cloud server analyzes the submitted images using generative AI models, which are used to identify food ingredients. Specifically, these include algorithms using machine learning and deep learning techniques.

[0519] 5. Verification method (server):

[0520] The server's verification means compares the identified ingredient information with an allergy database to confirm the presence of allergens. By comparing the information with the database, the presence of allergens is detected.

[0521] 6. Visual and audio warning means (terminals):

[0522] If an allergen is detected, the server sends the result to the device, which then provides a visual and audio warning to the user based on the received information. Specifically, a warning message is displayed on the device's display and the warning content is read aloud.

[0523] 7. History management function (terminal or server):

[0524] It has a function to manage the past intake history of allergens and provide appropriate warnings based on that, which allows for customized warnings based on the user's allergy history.

[0525] 8. Disaster detection function (terminal or server):

[0526] The system has the ability to identify edible foods, poisons, and harmful substances and provide information to users, enabling them to make safe food choices even during disasters.

[0527] Specific operation example

[0528] For example, consider a user picking up a chocolate bar at a supermarket. When the user looks at the product, the camera built into the glasses captures an image, which is then sent to a cloud server after the device adjusts the resolution and removes noise. A generative AI model on the server analyzes the image and identifies ingredients such as "sugar, milk, and peanuts." The server's matching means compares the ingredient information with an allergy database and confirms that it contains peanuts. The results are sent to the device, which displays "Dangerous Food 💀" and provides a voice warning to the user that "Contains peanuts."

[0529] Also, a specific example of a prompt sentence to input to the generative AI model is as follows:

[0530] "I have provided an image. This image shows a food product. Please identify the ingredients in the food product and detect allergens. Please return the ingredients in the full document. Please check if the ingredients contain the following allergens: milk, peanuts, eggs."

[0531] Using this prompt, the AI ​​model analyzes food ingredient information and detects the presence of allergens.

[0532] Other Features

[0533] This system has the ability to manage the user's allergen intake history, allowing it to provide more detailed and personalized warnings. It also has the ability to identify foods and harmful substances that can be ingested in the event of a disaster, ensuring safety in emergencies.

[0534] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0535] The flow of this system's program processing

[0536] Step 1:

[0537] (User sees food)

[0538] When a user looks at a food item, the device's imaging means (built-in camera) automatically captures an image of the food item. For example, when a user picks up a chocolate bar in a supermarket, an image of the chocolate bar's packaging is captured. Input: User's visual behavior Output: Captured image data

[0539] Step 2:

[0540] (Device preprocesses the image)

[0541] The captured image is pre-processed by the device's image processing means. Specifically, the resolution is adjusted, brightness is corrected, and noise is removed. This prepares the image in a format that is easy to analyze. Input: Captured image data Output: Pre-processed image data

[0542] Step 3:

[0543] (The device sends the image to the cloud server)

[0544] The preprocessed image is sent to the cloud server via the device's communication means. Specifically, the image data is uploaded using an HTTP POST request. Input: Preprocessed image data Output: Image data sent to the server

[0545] Step 4:

[0546] (The server analyzes the image using a generative AI model)

[0547] The cloud server analyzes the received image data using the AI ​​model analysis means. The AI ​​model performs data calculations to identify food ingredients based on the input image data. For example, the AI ​​model extracts ingredient information such as "sugar, milk, peanuts" from an image of a chocolate bar package. Input: Image data sent to the server Output: Identified food ingredient information

[0548] Step 5:

[0549] (The server checks the ingredient information against the allergy database)

[0550] The server's verification means compares the identified ingredient information with the allergy database. This verification confirms the presence of allergens in the food. For example, the verification result confirms that the food contains peanuts. Input: Identified food ingredient information Output: Verification result (presence or absence of allergens)

[0551] Step 6:

[0552] (The server sends the matching results to the device.)

[0553] Once the matching result is obtained, the server sends it to the device. Input: Matching result Output: Alert information sent to the device

[0554] Step 7:

[0555] (Device provides warning to user)

[0556] The device will provide a visual and audio warning to the user based on the received alert information. Specifically, it will display "Dangerous Food 💀" on the device's display and warn the user by voice, "Contains peanuts." Input: Alert information sent to the device Output: Visual and audio warning to the user

[0557] Step 8:

[0558] (The device or server manages the history.)

[0559] The history management function on the device or server records information about the ingredients of the food the user is about to consume and the history of allergen detection. This improves the accuracy of future warnings. Input: User's intake history and detection history. Output: Updated history data.

[0560] Step 9:

[0561] (The terminal or server will detect food in the event of a disaster.)

[0562] In the event of a disaster, the terminal or server provides information on foods that can be consumed by the user and on harmful substances. This allows users to safely select foods even in an emergency. Input: Environmental information and food information during a disaster. Output: Safe food information provided to the user.

[0563] The above is the flow of specific processing steps for this system.

[0564] (Application example 1)

[0565] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0566] Conventional allergy detection systems lack the speed and accuracy to quickly identify food allergens and provide appropriate warnings to users. Furthermore, methods for accurately identifying food allergens in moving environments, such as autonomous vehicles, have not been fully established. Therefore, an efficient and reliable system for reducing food allergy risks is needed.

[0567] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0568] In this invention, the server includes an image capturing means, an image processing means, a communication means, a generative model analysis means, a matching means, a visual warning means, an audio warning means, and a means for capturing images obtained from an internal camera of the autonomous vehicle and identifying food allergens. This makes it possible to quickly and accurately identify food allergens in a moving autonomous vehicle and provide appropriate visual and audio warnings to the user.

[0569] - "Photography means" refers to any device or technology used to capture an image of a food item.

[0570] "Image processing means" refers to devices or techniques for pre-processing captured images, such as noise reduction and resolution adjustment.

[0571] "Communication means" refers to the network interface and protocols used to transmit preprocessed images to a cloud data center.

[0572] "Generative model analysis means" refers to AI models and machine learning algorithms that analyze images in a cloud data center and identify food elements.

[0573] "Matching means" refers to the technology or method used to match food elements with an allergy database to confirm the presence of allergens.

[0574] "Visual warning means" means any device or software that provides a visual warning to the user upon detecting the presence of an allergen.

[0575] "Audio warning means" means a device or software that provides an audio warning to a user when it detects the presence of an allergen.

[0576] "Internal camera in an autonomous vehicle" refers to a camera installed inside an autonomous vehicle that captures images of food brought in by passengers.

[0577] The present invention relates to a system for quickly and accurately detecting food allergens in autonomous vehicles and providing warnings to users. The system consists of the following main components:

[0578] Overall system configuration

[0579] 1. Photography Method:

[0580] This is a camera installed inside the autonomous vehicle that captures images of food brought in by passengers, specifically when a passenger in the vehicle picks up the food.

[0581] 2. Image processing means:

[0582] Software for pre-processing the captured images to remove noise and adjust the resolution. For example, the OpenCV library can be used for image processing.

[0583] 3. Means of communication:

[0584] A network interface for sending preprocessed images to a cloud data center, specifically by sending data via HTTP POST requests.

[0585] 4. Generative model analysis methods:

[0586] AI models for analyzing images and identifying food elements in cloud data centers, for example, using the TensorFlow library to identify food ingredients.

[0587] 5. Verification method:

[0588] The identified food elements are checked against an allergy database to confirm the presence of allergens. Database checks are performed using an SQL server.

[0589] 6. Visual and audio warning means:

[0590] If an allergen is detected, the device will provide users with visual and audio alerts: visual alerts will be displayed on the smartphone screen and audio alerts will be played through the speaker.

[0591] 7. Means for capturing images obtained from an internal camera of an autonomous vehicle:

[0592] An interface for utilizing images acquired by the internal camera throughout the system. This interface allows real-time images acquired by the camera to be imported into the system.

[0593] Specific operation flow

[0594] 1. Cameras inside the self-driving vehicle capture images of food brought in by passengers.

[0595] 2. This image is pre-processed by image processing means to remove noise and adjust the resolution.

[0596] 3. The pre-processed images are sent to the cloud data center via communication means.

[0597] 4. The cloud data center uses generative model analysis tools to analyze the image and identify food elements.

[0598] 5. The identified food elements are matched against an allergy database via a matching means to confirm the presence of allergens.

[0599] 6. If an allergen is detected, a warning is provided to the user through visual and audio warning means.

[0600] Prompt Sentence Examples

[0601] Based on the image of food captured by the user with the camera, identify the ingredients contained in it, match it with an allergy database, and return the results.

[0602] Image data:<image_data>

[0603] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0604] Step 1:

[0605] Input: Food images obtained from a camera mounted inside an autonomous vehicle.

[0606] How it works: The device activates a camera installed inside the autonomous vehicle and captures an image of the food brought in by the passenger.

[0607] Output: Image data of the captured food.

[0608] Step 2:

[0609] Input: Captured food image data.

[0610] What it does: The device pre-processes the captured image by removing noise and adjusting the resolution, specifically by resizing and color correcting the image using the OpenCV library.

[0611] Output: Preprocessed image data.

[0612] Step 3:

[0613] Input: Preprocessed image data.

[0614] How it works: The device sends the preprocessed image data to the cloud data center, where it uploads the image data to the specified URL using an HTTP POST request.

[0615] Output: Notification of completion of image data transmission to the cloud data center.

[0616] Step 4:

[0617] Input: Image data sent to the cloud data center.

[0618] How it works: The server receives the image data and performs generative model analysis using the AI ​​model. It then uses the TensorFlow library to perform data calculations to identify food elements from the image.

[0619] Output: Parsed food element data.

[0620] Step 5:

[0621] Input: Food element data analyzed by the AI ​​model.

[0622] How it works: The server checks the food element data it retrieves against the allergy database. Using an SQL server, it compares it with existing data in the database to determine whether or not any allergens are present.

[0623] Output: Matching result (whether or not allergens were detected and details).

[0624] Step 6:

[0625] Input: Whether or not allergens are detected and detailed data.

[0626] How it works: The server sends the matching results to the device, and if an allergen is detected, the device provides visual and audio warnings to the user. A warning message is displayed on the smartphone screen and an audio warning is played through the speaker.

[0627] Output: Visual and audio warnings to the user.

[0628] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0629] The present invention relates to a system that quickly and accurately detects allergens and provides a warning to the user, and further combines it with an emotion engine that recognizes the user's emotions, allowing the content and method of the warning to be adapted to the user's emotional state. Specific embodiments are described below.

[0630] Overall system configuration

[0631] This system consists of the following main components:

[0632] 1. Camera means (terminal):

[0633] A camera built into glasses worn by the user that captures images of food in the user's field of view and images of the user's face.

[0634] 2. Image processing means (terminal):

[0635] A device or algorithm that preprocesses a captured image to remove noise or adjust resolution.

[0636] 3. Communication means (terminal):

[0637] A network interface and protocol for sending preprocessed images to a cloud server.

[0638] 4. Generative model analysis means (server):

[0639] An AI model or machine learning algorithm that analyzes images on a cloud server and identifies food ingredients.

[0640] 5. Verification method (server):

[0641] A function that compares identified ingredient information with an allergy database to confirm the presence of allergens.

[0642] 6. Emotion engine (terminal or server):

[0643] An algorithm that recognizes emotions from a user's facial expressions and assesses their stress level and emotional state.

[0644] 7. Visual and audio warning means (terminals):

[0645] A device or software that provides visual and audio warnings to the user when an allergen is detected, based on the results of the emotion engine.

[0646] 8. History management function (terminal or server):

[0647] A function that manages past intake history of allergens and provides warnings based on that information.

[0648] 9. Disaster detection function (terminal or server):

[0649] A function that identifies edible, poisonous, and harmful substances during a disaster and provides information to users.

[0650] Basic operation flow

[0651] 1. Image Capture and Preprocessing (Device)

[0652] When a user looks at a food item, the device's camera captures an image of it, along with an image of the user's face. The captured image is then pre-processed to adjust resolution, brightness, and noise reduction.

[0653] 2. Sending images to the server (terminal)

[0654] The device sends the preprocessed food and face images to the cloud server by uploading the data via an HTTP request (POST request).

[0655] 3. Food ingredient analysis and emotion recognition (server)

[0656] The server analyzes the received food image data using an AI model to identify and extract food ingredients. At the same time, it inputs facial image data into an emotion engine to recognize the user's emotions. The emotion engine evaluates the user's stress level and emotional state.

[0657] 4. Verification (Server)

[0658] The identified ingredient information is checked against an allergy database to determine whether it contains any allergens, and the emotion engine results are also sent to the device.

[0659] 5. Alert notification (terminal)

[0660] If an allergen is detected, the device will display a warning message based on the results of the emotion engine. For example, if the user is relaxed, it will display a gentle message such as "Be careful," while if the user is stressed, it will display a more emphatic message such as "Avoid immediately." Audio warnings are also adjusted in the same way.

[0661] Specific examples

[0662] For example, consider a user choosing food in a cafeteria. The user looks at the food and the camera captures an image of it. At the same time, the user's facial expression is also captured. This data is sent from the device to the server, which analyzes the food's ingredients and identifies them as "egg yolk, soy, and wheat flour." Meanwhile, the emotion engine analyzes the user's facial expression and recognizes that the user is nervous. The analysis confirms that the food contains soy, an allergen. The device receives this information, displays an emergency message on the screen saying "Dangerous food 💀," and issues a voice warning saying "Contains soy, be careful." The user can avoid the food based on this warning, thereby reducing the risk of allergies.

[0663] In this way, this system quickly and accurately detects allergens in food and provides warnings according to the user's emotional state, thereby achieving safe and effective allergy management.By combining it with history management and disaster detection functions, it is possible to protect user safety in a variety of situations.

[0664] The processing flow will be explained below.

[0665] Step 1:

[0666] The user looks at food, and the camera built into the glasses captures an image of the food in the user's field of view and an image of the user's face.

[0667] Step 2:

[0668] The device captures food images and facial images, which are then pre-processed to remove noise and adjust resolution.

[0669] Step 3:

[0670] The device temporarily stores the preprocessed image data in local storage.

[0671] Step 4:

[0672] The device sends the stored image data to the cloud server using an HTTP POST request, sending the image data over the network.

[0673] Step 5:

[0674] The server analyzes the received image data. The server's generative model analysis means processes the food image using an AI model to identify and extract food ingredients.

[0675] Step 6:

[0676] The server inputs facial images into the emotion engine, which recognizes emotions from the user's facial expressions and evaluates their stress level and emotional state.

[0677] Step 7:

[0678] The server checks the identified food ingredient information against an allergy database. A checking means compares the ingredient information with the database to confirm the presence of allergens.

[0679] Step 8:

[0680] The server returns the food ingredient matching results and the emotion engine evaluation results to the device. The results are formatted in JSON and sent as an HTTP response.

[0681] Step 9:

[0682] The terminal analyzes the results received from the server, and if an allergen is found, a visual warning means and an audio warning means are activated.

[0683] Step 10:

[0684] Based on the results of the emotion engine, the device displays the most appropriate visual warning to the user. For example, if the user is relaxed, it displays "Caution required," and if the user is tense, it displays "Avoid immediately."

[0685] Step 11:

[0686] The device provides the user with a voice alert based on the results of the emotion engine. The voice alert is also adjusted according to the user's emotions. For example, if the user is relaxed, the alert will be softer in tone, and if the user is tense, the alert will be more emphatic.

[0687] This allows users to quickly and accurately learn about food allergens, and also to receive appropriate warnings according to their current emotions.

[0688] Example 2

[0689] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0690] In modern society, the presence of allergens in food poses a serious risk to the user's health. Furthermore, conventional allergy detection systems simply detect allergens and are unable to adapt the content and method of warnings to take the user's emotional state into account. This can make it difficult for users to properly accept the warnings, and can result in insufficient reduction of allergy risk. Furthermore, the system does not take into account the identification of edibles, poisons, or harmful substances in the event of a disaster. Thus, providing a safe environment that takes the user's emotional state into account is a challenge for allergy risk management systems.

[0691] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an imaging means for capturing an image of the food, an image processing means for preprocessing the captured image, a communication means for transmitting the preprocessed image to a remote server, a generative model analysis means for analyzing the image in the remote server and identifying food components, a comparison means for comparing the food components with an allergy database, a visual warning means and an audio warning means for displaying a warning to the user if the presence of an allergen is detected as a result of the comparison, an emotion recognition means for capturing an image of the user's face and evaluating their emotional state, and an adaptive control means for adapting the content and method of the warning based on the emotional state. This enables safe and effective allergy management by quickly and accurately detecting allergens contained in food and providing a warning according to the user's emotional state.

[0692] "Photographing means for capturing images of food" is a general term for devices and techniques that record images of food within the user's field of view.

[0693] "Image processing means for pre-processing captured images" is a general term for devices or software that perform noise reduction and resolution adjustment to improve the quality of captured images.

[0694] "Communication means for transmitting preprocessed images to a remote server" is a general term for interfaces and protocols for uploading processed image data to a remote server over a network.

[0695] "Generative model analysis means that analyzes images on a remote server and identifies food components" is a general term for technologies and algorithms that process received image data using an AI model to identify and extract food components.

[0696] "Means for matching food ingredients with an allergy database" is a general term for technologies and functions for comparing identified food ingredients with an existing allergy database to confirm the presence of allergens.

[0697] "Visual and audio warning means that display a warning to the user when the presence of an allergen is detected as a result of the comparison" is a general term for devices and functions that provide a warning by screen display or audio to inform the user that an allergen is present.

[0698] "Emotion recognition means for capturing a user's facial image and evaluating the user's emotional state" is a general term for devices and algorithms for analyzing a user's facial expression and evaluating the user's emotional state.

[0699] "Adaptive control means for adapting the content and method of warning based on the emotional state" is a general term for techniques and functions for changing the content and method of warning depending on the emotional state of the user.

[0700] The present invention relates to a system that quickly and accurately detects allergens and provides a warning to the user. This system is combined with an emotion engine that recognizes the user's emotions, allowing the content and method of the warning to be adapted to the user's emotional state. Specific embodiments are described below.

[0701] Overall system configuration

[0702] This system consists of the following main components:

[0703] 1. Shooting method (device):

[0704] A camera built into glasses worn by the user captures images of food in the user's field of view as well as images of the user's face. For example, if a user is choosing a salad in a cafeteria, the camera in the glasses automatically captures the salad and the user's facial expression.

[0705] 2. Image processing means (terminal):

[0706] The captured images are pre-processed to improve their quality. This pre-processing includes adjusting the resolution and removing noise. For example, the resolution is standardized to 1080p and a Gaussian filter is applied to make the image clearer.

[0707] 3. Communication means (terminal):

[0708] A network interface and protocol for transmitting preprocessed images to a remote server using HTTP POST requests to upload image data over Wi-Fi or cellular networks.

[0709] 4. Generative model analysis means (server):

[0710] The remote server inputs the received image data into an AI model to identify and extract the food's ingredients. This analysis uses AI models such as SVM and ResNet. For example, the food's ingredients are identified as "egg yolk, soybeans, and wheat flour."

[0711] 5. Emotion recognition means (server):

[0712] The remote server inputs facial image data into the emotion engine, which analyzes the user's facial expressions and evaluates their emotional state. The emotion engine uses deep learning algorithms to detect the user's stress level and emotional state. For example, it can recognize that the user is "tense."

[0713] 6. Verification method (server):

[0714] The remote server checks the identified ingredient information against an allergy database to confirm the presence of allergens, for example, soy is found to be an allergen.

[0715] 7. Adaptive control means (terminal):

[0716] Based on the analysis results, the device will adapt the content and method of warnings to suit the user's emotional state. For example, if the user is nervous, an emergency message will be displayed on the screen saying "Dangerous Food 💀" and a voice warning will be given saying "Contains soy, be careful."

[0717] Specific examples

[0718] Usage scene in the cafeteria

[0719] Consider a scenario in which a user is choosing food in a restaurant. At this time, a camera captures images of the food the user is looking at and their facial expression. The device preprocesses these images and sends them to a remote server. The server analyzes the food's ingredients and identifies "egg yolk, soybeans, and wheat flour." At the same time, the emotion engine analyzes the user's facial expression and recognizes that the user is "nervous."

[0720] If the analysis results confirm that the food contains soy, the device will display a warning message according to the user's "tension" state. "Dangerous Food 💀" will be displayed on the screen, and a voice message will be played saying, "Contains soy. Be careful." Users can reduce their risk of allergies by avoiding the food based on this warning.

[0721] Prompt Sentence Examples

[0722] "Imagine a scene where a user is choosing food in a restaurant. The camera captures images of the food the user is looking at and their facial expressions. The server then analyzes the food's ingredients from the received data and identifies allergens. At the same time, it recognizes the user's emotional state and displays appropriate warning messages."

[0723] As described above, the present invention achieves safe and effective allergy management by quickly and accurately detecting allergens in food and providing warnings tailored to the user's emotional state.

[0724] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0725] Step 1:

[0726] User views food:

[0727] When a user looks at a food item, a camera built into the device captures an image of the food item and an image of the user's face.

[0728] Input: Food and its facial expression in the user's field of view through the glasses.

[0729] Output: Captured food images and user's face image data.

[0730] Step 2:

[0731] The device performs image preprocessing:

[0732] The device performs pre-processing on the captured image, such as adjusting the resolution and removing noise.

[0733] Input: A captured food image and a user's face image.

[0734] Data processing: Unification of resolution (e.g. 1080p), noise removal (e.g. applying Gaussian filter).

[0735] Output: Preprocessed food images and face images.

[0736] Step 3:

[0737] The device sends data to the server:

[0738] The device sends the preprocessed image data to the remote server using an HTTP POST request.

[0739] Input: Preprocessed food images and face images.

[0740] Data processing: Packaging data into HTTP requests.

[0741] Output: Image data sent to the remote server.

[0742] Step 4:

[0743] The server analyzes the food ingredients:

[0744] The server inputs the received food image data into an AI model to identify and extract the food's ingredients.

[0745] Input: Food image data sent to the server.

[0746] Data processing: Image analysis using AI models (e.g., SVM and ResNet).

[0747] Output: Identified food ingredients (e.g., "egg yolk, soy, wheat flour").

[0748] Step 5:

[0749] The server analyzes the user's emotions:

[0750] The server inputs the received facial image into an emotion engine, analyzes the user's facial expression, and evaluates their emotional state.

[0751] Input: User's facial image data sent to the server.

[0752] Data processing: Facial expression analysis using emotion engine (deep learning algorithm).

[0753] Output: The user's assessed emotional state (e.g., "tense").

[0754] Step 6:

[0755] The server verifies the ingredients:

[0756] The server compares the identified food ingredient information with an allergy database to determine whether the food contains any allergens.

[0757] Input: Identified food ingredient information.

[0758] Data processing: Comparison with allergy database.

[0759] Output: Matching result (e.g. "Soy is an allergen").

[0760] Step 7:

[0761] The server sends the analysis results to the device:

[0762] The server sends the results of the user's emotion analysis and food ingredient analysis to the terminal.

[0763] Input: Matching result and emotional state.

[0764] Data processing: Generate JSON data containing the information.

[0765] Output: Analysis result data sent to the device.

[0766] Step 8:

[0767] The device provides a warning to the user:

[0768] Based on the analysis results, the device provides the user with visual and audio warnings, adapting the content and method of the warning depending on the user's emotional state.

[0769] Input: Analysis result data (matching results and emotional state).

[0770] Data processing: Generate warning messages and configure display and sound settings.

[0771] Output: Visual and audio warning messages (e.g. "Dangerous Food 💀" and "Contains soy, be careful").

[0772] The above are the specific processing steps of the program of this system.

[0773] (Application example 2)

[0774] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0775] Rapid and accurate detection of allergens in food is very important to many people. However, systems that provide uniform warnings without considering the user's emotional state may cause excessive stress to the user. In particular, in situations such as food delivery, appropriate warnings based on the user's emotional state are required. Therefore, it is necessary to provide a system that can detect allergens in food and provide warnings based on the user's emotional state.

[0776] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0777] In this invention, the server includes a camera means for capturing images of food, an image processing means for preprocessing the captured images, a communication means for transmitting the preprocessed images to a cloud server, a generative model analysis means for analyzing the images in the cloud server and identifying food ingredients, a comparison means for comparing the food ingredients with an allergy database, an emotion engine for recognizing emotions from the user's facial expressions and evaluating their stress level and emotional state, and a visual warning means and an audio warning means for displaying a warning to the user based on the results of the emotion engine if the presence of an allergen is detected as a result of the comparison. This makes it possible to quickly and accurately detect allergens in food and provide an appropriate warning according to the user's emotional state.

[0778] "Camera means" refers to a device that is built into a user-worn device and has the ability to capture images of food and the user's face.

[0779] "Image processing means" refers to devices or algorithms that pre-process captured images to remove noise or adjust resolution.

[0780] "Communication means" refers to the network interface and protocol used to send preprocessed images to the cloud server.

[0781] The "generative model analysis means" is an AI model or machine learning algorithm that analyzes images received by the cloud server and identifies food ingredients.

[0782] The "comparison means" is a function that compares the identified food ingredient information with an allergy database to confirm the presence of allergens.

[0783] The "emotion engine" is an algorithm that recognizes emotions from the user's facial expressions and evaluates their stress level and emotional state.

[0784] A "visual warning means" is a device or software that displays a visual warning to the user based on the results of the emotion engine when an allergen is detected.

[0785] The "audio warning means" is a device or software that provides an audio warning to the user based on the results of the emotion engine when an allergen is detected.

[0786] The "history management function" is a function that manages the past intake history of allergens and provides warnings based on that history.

[0787] The "disaster detection function" is a function that identifies edible items, poisons, and harmful substances during a disaster and provides information to the user.

[0788] The present invention is a system that quickly and accurately detects food allergens and provides appropriate warnings according to the user's emotional state. The overall system configuration consists of the following main components:

[0789] 1. Camera Means

[0790] The camera is built into a device worn by the user and captures images of the food and the user's face. This image capture collects data on the food the user is viewing and their facial expression.

[0791] 2. Image Processing Methods

[0792] The captured images are pre-processed, including noise removal, resolution adjustment, normalization, etc. Specifically, OpenCV is used to perform these image processing.

[0793] 3. Means of communication

[0794] The preprocessed image is sent to the cloud server using a network interface and a communication protocol such as an HTTP request (POST request).

[0795] 4. Generative Model Analysis Methods

[0796] The cloud server analyzes the received image data using an AI model to identify and extract food ingredients. This analysis uses machine learning algorithms such as TensorFlow and Keras. An example prompt is, "Analyze the food image to identify its ingredients and determine whether it contains any of the following substances to which you are allergic: egg yolk, soy, or wheat flour."

[0797] 5. Verification Method

[0798] The identified food ingredient information is compared with an allergy database to confirm the presence of allergens, thereby determining whether the ingredients extracted by the generative model analysis means are pre-registered allergens.

[0799] 6. Emotion Engine

[0800] This algorithm recognizes emotions based on images of a user's face and evaluates their stress level and emotional state. Emotion recognition is performed using an emotion analysis engine such as EmotionRecognitionEngine. This engine analyzes the user's facial expression data and classifies their emotional state as "tension" or "stress," etc.

[0801] 7. Visual and audio warning means

[0802] If an allergen is detected, a warning will be provided to the user based on the results of the emotion engine. For example, if the user is nervous, it will display "Dangerous Food 💀 - Contains {Detected Allergen}, Be Careful," and if the user is relaxed, it will display "Contains {Detected Allergen}, Be Careful." Audio warnings will also be adjusted accordingly.

[0803] Specific examples

[0804] For example, when a user receives a food delivery, they view the food through the smart glasses and capture an image of it using the camera means. The image is then preprocessed by the image processing means and sent to a cloud server via the communication means. The server identifies the food ingredients using the generative model analysis means and checks for allergens using the matching means. At the same time, the emotion engine recognizes the user's emotional state from their facial expressions. Finally, if an allergen is detected, the emotion engine provides the user with an appropriate visual and audio warning based on the evaluation results of the emotion engine.

[0805] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0806] Step 1:

[0807] When a user receives a food delivery, they view the food through the smart glasses. A camera means (terminal) captures an image of the food and an image of the user's face. The input is the viewed food and the user's facial expression, and the output is the captured image data. This captured image data is used in the next step.

[0808] Step 2:

[0809] The device preprocesses the captured image data using image processing means. Specifically, it uses OpenCV to perform noise removal, resolution adjustment, and normalization. The input is the captured image data, and the output is the preprocessed image data. This preprocessed image data is used in the next step.

[0810] Step 3:

[0811] The preprocessed image data is sent to the cloud server via the communication means (terminal). This communication is performed using an HTTP request (POST request) via a network interface. The input is the preprocessed image data, and the output is the upload completion status to the cloud server.

[0812] Step 4:

[0813] The cloud server analyzes the received image data using a generative model analysis method. Specifically, it uses TensorFlow and Keras to identify and extract food ingredients. Here, the prompt is "Analyze the food image, identify the ingredients contained, and determine whether the following allergens are present: egg yolk, soy, wheat flour." The input is the image data uploaded to the cloud server, and the output is the identified food ingredient information.

[0814] Step 5:

[0815] The verification means (server) verifies the identified food ingredient information against the allergy database to confirm the presence of allergens. The input is the identified food ingredient information, and the output is the allergen detection result. This result is used in the next step.

[0816] Step 6:

[0817] At the same time, the server uses an emotion engine to analyze the user's facial image and recognize their emotional state. Specifically, it uses the EmotionRecognitionEngine to analyze the user's facial expression data and classify the user's emotions as "tension" or "stress." The input is the facial image data uploaded to the cloud server, and the output is the classification result of the user's emotional state. This classification result of the emotional state is used in the next step.

[0818] Step 7:

[0819] If an allergen is detected, the device will provide a visual and audio warning to the user based on the emotion engine's results. Specifically, if the user is nervous, the device will display "Dangerous Food 💀 - Contains {Detected Allergen}, Be Careful," and if the user is relaxed, the device will display "Contains {Detected Allergen}, Be Careful." The audio warning is also adjusted in a similar way. The input is the allergen detection result and the classification result of the user's emotional state, and the output is a warning message to the user.

[0820] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0821] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0822] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0823] [Third embodiment]

[0824] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0825] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0826] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0827] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0828] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0829] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0830] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0831] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0832] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0833] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0834] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0835] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0836] The present invention relates to a system for quickly and accurately detecting allergens and providing a warning to a user. Specific embodiments are described below.

[0837] Overall system configuration

[0838] This system consists of the following main components:

[0839] 1. Camera means (terminal):

[0840] It is a camera built into glasses worn by the user that captures images of food within the user's field of view.

[0841] 2. Image processing means (terminal):

[0842] A device or algorithm that preprocesses a captured image to remove noise or adjust resolution.

[0843] 3. Communication means (terminal):

[0844] A network interface and protocol for sending preprocessed images to a cloud server.

[0845] 4. Generative model analysis means (server):

[0846] An AI model or machine learning algorithm that analyzes images on a cloud server and identifies food ingredients.

[0847] 5. Verification method (server):

[0848] A function that compares identified ingredient information with an allergy database to confirm the presence of allergens.

[0849] 6. Visual and audio warning means (terminals):

[0850] A device or software that provides visual and audio warnings to the user if an allergen is detected.

[0851] 7. History management function (terminal or server):

[0852] A function that manages past intake history of allergens and provides warnings based on that information.

[0853] 8. Disaster detection function (terminal or server):

[0854] A function that identifies edible, poisonous, and harmful substances during a disaster and provides information to users.

[0855] Basic operation flow

[0856] 1. Image Capture and Preprocessing (Device)

[0857] When a user looks at a food item, the device's camera captures an image of it. The captured image undergoes pre-processing, such as adjusting resolution and brightness and removing noise, and is then temporarily saved to local storage.

[0858] 2. Sending images to the server (terminal)

[0859] The device sends the preprocessed image to the cloud server by uploading the data via an HTTP request (POST request).

[0860] 3. Food ingredient analysis (server)

[0861] The server analyzes the received images using an AI model. The generative model analysis means processes the image data to identify and extract food ingredients.

[0862] 4. Verification (Server)

[0863] The identified ingredient information is checked against an allergy database. A checking means on the server compares the ingredient information with the database to confirm the presence of allergens.

[0864] 5. Alert notification (terminal)

[0865] If an allergen is detected, the server sends the results to the device, which then displays a visual warning message on the screen and provides an audio warning to the user based on the results received.

[0866] Specific examples

[0867] For example, imagine a user picking up a chocolate bar at a supermarket. When the user looks at the product, the camera in the glasses captures an image, which is preprocessed on the device and then sent to the server. The server analyzes the image using an AI model to identify its ingredients: sugar, milk, and peanuts. This ingredient information is then compared with an allergy database, confirming that the bar contains peanuts. The server then sends this information to the device, which displays "Dangerous Food 💀" on the screen and warns the user audibly that "Contains peanuts."

[0868] In this way, this system reduces the risk of food allergies by quickly and accurately detecting allergens in foods that come into the user's field of vision and providing immediate visual and audio warnings. Furthermore, by incorporating history management and disaster detection functions, it protects the user's safety in a variety of situations.

[0869] The processing flow will be explained below.

[0870] Step 1:

[0871] A user picks up food at a supermarket, etc. A camera built into the glasses captures an image of the food in the user's field of view.

[0872] Step 2:

[0873] It takes the image captured by the device's camera and performs pre-processing on it, specifically adjusting the image resolution and applying a noise reduction filter.

[0874] Step 3:

[0875] The device temporarily stores the preprocessed image in local storage.

[0876] Step 4:

[0877] The device sends the saved images to the cloud server using an HTTP request (POST request), uploading the data to the server via the Internet.

[0878] Step 5:

[0879] The server receives the image data and stores it in a database.

[0880] Step 6:

[0881] The server analyzes the received image using a generative model analysis method. Specifically, the image is input into an AI model to identify and extract food ingredients.

[0882] Step 7:

[0883] The server compares the identified ingredient information with an allergy database to see if the product contains any allergens.

[0884] Step 8:

[0885] The server returns the matching results to the device in a format such as JSON.

[0886] Step 9:

[0887] The device receives the results from the server, analyzes the received data, and extracts the necessary information.

[0888] Step 10:

[0889] If your device contains an allergen, it will display a visual warning message on the screen, such as "Dangerous Food 💀."

[0890] Step 11:

[0891] The device will provide audible warnings to the user, using text-to-speech to say things like "Contains peanuts."

[0892] These steps allow users to quickly and accurately find out the allergens in food, allowing them to make safe food choices.

[0893] Example 1

[0894] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0895] In modern society, there is a demand for the rapid and accurate detection of allergens contained in food and the provision of warnings to users. However, conventional methods have problems such as delays in allergen detection and a lack of accuracy. Furthermore, in times of disaster, it is difficult to determine which foods are safe to consume. Furthermore, there is a lack of systems that can manage the past intake history of allergens and provide appropriate warnings based on that information.

[0896] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0897] In this invention, the server includes an imaging means for capturing an image of the food, an image processing means for preprocessing the captured image, a communication means for transmitting the preprocessed image to the data server, a generative AI model analysis means for analyzing the image in the data server and identifying the food components, a comparison means for comparing the food components with an allergy database, and a visual warning means and an audio warning means for displaying a warning to the user if the presence of an allergen is detected as a result of the comparison. This makes it possible to quickly and accurately detect allergens contained in food and provide an immediate warning to the user.

[0898] "Imaging means" refers to a device used to capture an image of a food item, and includes, for example, a camera or an image sensor.

[0899] "Image processing means" refers to devices or algorithms for pre-processing captured images, such as adjusting resolution, correcting brightness, and removing noise.

[0900] "Communication means" refers to a device or protocol for transmitting preprocessed images to a data server, including a network interface and communication software.

[0901] "Generative AI model analysis means" includes artificial intelligence models and machine learning algorithms for analyzing images received by the data server and identifying food ingredients.

[0902] A "verification means" is a device or software that compares identified food ingredients with an allergy database to confirm the presence of allergens.

[0903] A "visual alert means" is a device or interface for displaying a visual alert to a user if an allergen is detected.

[0904] An "audio warning means" is a device or system for providing an audio warning to a user when an allergen is detected.

[0905] The "history management function" is a function for managing the past intake history of allergens and providing warnings to the user based on that history.

[0906] The "disaster detection function" is a function that identifies edible items, poisons, and harmful substances during a disaster and provides information to the user.

[0907] MODE FOR CARRYING OUT THE INVENTION

[0908] The present invention relates to a system for quickly and accurately detecting allergens and providing a warning to a user. DETAILED DESCRIPTION OF THE INVENTION ...

[0909] System configuration

[0910] This system consists of the following main components:

[0911] 1. Imaging means (terminal):

[0912] A camera built into the glasses worn by the user captures images of food within the user's field of view. Specifically, a small camera or image sensor is used.

[0913] 2. Image processing means (terminal):

[0914] The captured image is pre-processed by the device's built-in image processor or algorithms, which may include adjusting the resolution, correcting brightness, and removing noise, making the image easier to analyze.

[0915] 3. Communication means (terminal):

[0916] The preprocessed image is sent to the cloud server via the device's communication interface. Specifically, the data is uploaded using the POST method of the HTTP request.

[0917] 4. Generative AI model analysis means (server):

[0918] The cloud server analyzes the submitted images using generative AI models, which are used to identify food ingredients. Specifically, these include algorithms using machine learning and deep learning techniques.

[0919] 5. Verification method (server):

[0920] The server's verification means compares the identified ingredient information with an allergy database to confirm the presence of allergens. By comparing the information with the database, the presence of allergens is detected.

[0921] 6. Visual and audio warning means (terminals):

[0922] If an allergen is detected, the server sends the result to the device, which then provides a visual and audio warning to the user based on the received information. Specifically, a warning message is displayed on the device's display and the warning content is read aloud.

[0923] 7. History management function (terminal or server):

[0924] It has a function to manage the past intake history of allergens and provide appropriate warnings based on that, which allows for customized warnings based on the user's allergy history.

[0925] 8. Disaster detection function (terminal or server):

[0926] The system has the ability to identify edible foods, poisons, and harmful substances and provide information to users, enabling them to make safe food choices even during disasters.

[0927] Specific operation example

[0928] For example, consider a user picking up a chocolate bar at a supermarket. When the user looks at the product, the camera built into the glasses captures an image, which is then sent to a cloud server after the device adjusts the resolution and removes noise. A generative AI model on the server analyzes the image and identifies ingredients such as "sugar, milk, and peanuts." The server's matching means compares the ingredient information with an allergy database and confirms that it contains peanuts. The results are sent to the device, which displays "Dangerous Food 💀" and provides a voice warning to the user that "Contains peanuts."

[0929] Also, a specific example of a prompt sentence to input to the generative AI model is as follows:

[0930] "I have provided an image. This image shows a food product. Please identify the ingredients in the food product and detect allergens. Please return the ingredients in the full document. Please check if the ingredients contain the following allergens: milk, peanuts, eggs."

[0931] Using this prompt, the AI ​​model analyzes food ingredient information and detects the presence of allergens.

[0932] Other Features

[0933] This system has the ability to manage the user's allergen intake history, allowing it to provide more detailed and personalized warnings. It also has the ability to identify foods and harmful substances that can be ingested in the event of a disaster, ensuring safety in emergencies.

[0934] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0935] The flow of this system's program processing

[0936] Step 1:

[0937] (User sees food)

[0938] When a user looks at a food item, the device's imaging means (built-in camera) automatically captures an image of the food item. For example, when a user picks up a chocolate bar in a supermarket, an image of the chocolate bar's packaging is captured. Input: User's visual behavior Output: Captured image data

[0939] Step 2:

[0940] (Device preprocesses the image)

[0941] The captured image is pre-processed by the device's image processing means. Specifically, the resolution is adjusted, brightness is corrected, and noise is removed. This prepares the image in a format that is easy to analyze. Input: Captured image data Output: Pre-processed image data

[0942] Step 3:

[0943] (The device sends the image to the cloud server)

[0944] The preprocessed image is sent to the cloud server via the device's communication means. Specifically, the image data is uploaded using an HTTP POST request. Input: Preprocessed image data Output: Image data sent to the server

[0945] Step 4:

[0946] (The server analyzes the image using a generative AI model)

[0947] The cloud server analyzes the received image data using the AI ​​model analysis means. The AI ​​model performs data calculations to identify food ingredients based on the input image data. For example, the AI ​​model extracts ingredient information such as "sugar, milk, peanuts" from an image of a chocolate bar package. Input: Image data sent to the server Output: Identified food ingredient information

[0948] Step 5:

[0949] (The server checks the ingredient information against the allergy database)

[0950] The server's verification means compares the identified ingredient information with the allergy database. This verification confirms the presence of allergens in the food. For example, the verification result confirms that the food contains peanuts. Input: Identified food ingredient information Output: Verification result (presence or absence of allergens)

[0951] Step 6:

[0952] (The server sends the matching results to the device.)

[0953] Once the matching result is obtained, the server sends it to the device. Input: Matching result Output: Alert information sent to the device

[0954] Step 7:

[0955] (Device provides warning to user)

[0956] The device will provide a visual and audio warning to the user based on the received alert information. Specifically, it will display "Dangerous Food 💀" on the device's display and warn the user by voice, "Contains peanuts." Input: Alert information sent to the device Output: Visual and audio warning to the user

[0957] Step 8:

[0958] (The device or server manages the history.)

[0959] The history management function on the device or server records information about the ingredients of the food the user is about to consume and the history of allergen detection. This improves the accuracy of future warnings. Input: User's intake history and detection history. Output: Updated history data.

[0960] Step 9:

[0961] (The terminal or server will detect food in the event of a disaster.)

[0962] In the event of a disaster, the terminal or server provides information on foods that can be consumed by the user and on harmful substances. This allows users to safely select foods even in an emergency. Input: Environmental information and food information during a disaster. Output: Safe food information provided to the user.

[0963] The above is the flow of specific processing steps for this system.

[0964] (Application example 1)

[0965] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0966] Conventional allergy detection systems lack the speed and accuracy to quickly identify food allergens and provide appropriate warnings to users. Furthermore, methods for accurately identifying food allergens in moving environments, such as autonomous vehicles, have not been fully established. Therefore, an efficient and reliable system for reducing food allergy risks is needed.

[0967] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0968] In this invention, the server includes an image capturing means, an image processing means, a communication means, a generative model analysis means, a matching means, a visual warning means, an audio warning means, and a means for capturing images obtained from an internal camera of the autonomous vehicle and identifying food allergens. This makes it possible to quickly and accurately identify food allergens in a moving autonomous vehicle and provide appropriate visual and audio warnings to the user.

[0969] - "Photography means" refers to any device or technology used to capture an image of a food item.

[0970] "Image processing means" refers to devices or techniques for pre-processing captured images, such as noise reduction and resolution adjustment.

[0971] "Communication means" refers to the network interface and protocols used to transmit preprocessed images to a cloud data center.

[0972] "Generative model analysis means" refers to AI models and machine learning algorithms that analyze images in a cloud data center and identify food elements.

[0973] "Matching means" refers to the technology or method used to match food elements with an allergy database to confirm the presence of allergens.

[0974] "Visual warning means" means any device or software that provides a visual warning to the user upon detecting the presence of an allergen.

[0975] "Audio warning means" means a device or software that provides an audio warning to a user when it detects the presence of an allergen.

[0976] "Internal camera in an autonomous vehicle" refers to a camera installed inside an autonomous vehicle that captures images of food brought in by passengers.

[0977] The present invention relates to a system for quickly and accurately detecting food allergens in autonomous vehicles and providing warnings to users. The system consists of the following main components:

[0978] Overall system configuration

[0979] 1. Photography Method:

[0980] This is a camera installed inside the autonomous vehicle that captures images of food brought in by passengers, specifically when a passenger in the vehicle picks up the food.

[0981] 2. Image processing means:

[0982] Software for pre-processing the captured images to remove noise and adjust the resolution. For example, the OpenCV library can be used for image processing.

[0983] 3. Means of communication:

[0984] A network interface for sending preprocessed images to a cloud data center, specifically by sending data via HTTP POST requests.

[0985] 4. Generative model analysis methods:

[0986] AI models for analyzing images and identifying food elements in cloud data centers, for example, using the TensorFlow library to identify food ingredients.

[0987] 5. Verification method:

[0988] The identified food elements are checked against an allergy database to confirm the presence of allergens. Database checks are performed using an SQL server.

[0989] 6. Visual and audio warning means:

[0990] If an allergen is detected, the device will provide users with visual and audio alerts: visual alerts will be displayed on the smartphone screen and audio alerts will be played through the speaker.

[0991] 7. Means for capturing images obtained from an internal camera of an autonomous vehicle:

[0992] An interface for utilizing images acquired by the internal camera throughout the system. This interface allows real-time images acquired by the camera to be imported into the system.

[0993] Specific operation flow

[0994] 1. Cameras inside the self-driving vehicle capture images of food brought in by passengers.

[0995] 2. This image is pre-processed by image processing means to remove noise and adjust the resolution.

[0996] 3. The pre-processed images are sent to the cloud data center via communication means.

[0997] 4. The cloud data center uses generative model analysis tools to analyze the image and identify food elements.

[0998] 5. The identified food elements are matched against an allergy database via a matching means to confirm the presence of allergens.

[0999] 6. If an allergen is detected, a warning is provided to the user through visual and audio warning means.

[1000] Prompt Sentence Examples

[1001] Based on the image of food captured by the user with the camera, identify the ingredients contained in it, match it with an allergy database, and return the results.

[1002] Image data:<image_data>

[1003] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1004] Step 1:

[1005] Input: Food images obtained from a camera mounted inside an autonomous vehicle.

[1006] How it works: The device activates a camera installed inside the autonomous vehicle and captures an image of the food brought in by the passenger.

[1007] Output: Image data of the captured food.

[1008] Step 2:

[1009] Input: Captured food image data.

[1010] What it does: The device pre-processes the captured image by removing noise and adjusting the resolution, specifically by resizing and color correcting the image using the OpenCV library.

[1011] Output: Preprocessed image data.

[1012] Step 3:

[1013] Input: Preprocessed image data.

[1014] How it works: The device sends the preprocessed image data to the cloud data center, where it uploads the image data to the specified URL using an HTTP POST request.

[1015] Output: Notification of completion of image data transmission to the cloud data center.

[1016] Step 4:

[1017] Input: Image data sent to the cloud data center.

[1018] How it works: The server receives the image data and performs generative model analysis using the AI ​​model. It then uses the TensorFlow library to perform data calculations to identify food elements from the image.

[1019] Output: Parsed food element data.

[1020] Step 5:

[1021] Input: Food element data analyzed by the AI ​​model.

[1022] How it works: The server checks the food element data it retrieves against the allergy database. Using an SQL server, it compares it with existing data in the database to determine whether or not any allergens are present.

[1023] Output: Matching result (whether or not allergens were detected and details).

[1024] Step 6:

[1025] Input: Whether or not allergens are detected and detailed data.

[1026] How it works: The server sends the matching results to the device, and if an allergen is detected, the device provides visual and audio warnings to the user. A warning message is displayed on the smartphone screen and an audio warning is played through the speaker.

[1027] Output: Visual and audio warnings to the user.

[1028] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1029] The present invention relates to a system that quickly and accurately detects allergens and provides a warning to the user, and further combines it with an emotion engine that recognizes the user's emotions, allowing the content and method of the warning to be adapted to the user's emotional state. Specific embodiments are described below.

[1030] Overall system configuration

[1031] This system consists of the following main components:

[1032] 1. Camera means (terminal):

[1033] A camera built into glasses worn by the user that captures images of food in the user's field of view and images of the user's face.

[1034] 2. Image processing means (terminal):

[1035] A device or algorithm that preprocesses a captured image to remove noise or adjust resolution.

[1036] 3. Communication means (terminal):

[1037] A network interface and protocol for sending preprocessed images to a cloud server.

[1038] 4. Generative model analysis means (server):

[1039] An AI model or machine learning algorithm that analyzes images on a cloud server and identifies food ingredients.

[1040] 5. Verification method (server):

[1041] A function that compares identified ingredient information with an allergy database to confirm the presence of allergens.

[1042] 6. Emotion engine (terminal or server):

[1043] An algorithm that recognizes emotions from a user's facial expressions and assesses their stress level and emotional state.

[1044] 7. Visual and audio warning means (terminals):

[1045] A device or software that provides visual and audio warnings to the user when an allergen is detected, based on the results of the emotion engine.

[1046] 8. History management function (terminal or server):

[1047] A function that manages past intake history of allergens and provides warnings based on that information.

[1048] 9. Disaster detection function (terminal or server):

[1049] A function that identifies edible, poisonous, and harmful substances during a disaster and provides information to users.

[1050] Basic operation flow

[1051] 1. Image Capture and Preprocessing (Device)

[1052] When a user looks at a food item, the device's camera captures an image of it, along with an image of the user's face. The captured image is then pre-processed to adjust resolution, brightness, and noise reduction.

[1053] 2. Sending images to the server (terminal)

[1054] The device sends the preprocessed food and face images to the cloud server by uploading the data via an HTTP request (POST request).

[1055] 3. Food ingredient analysis and emotion recognition (server)

[1056] The server analyzes the received food image data using an AI model to identify and extract food ingredients. At the same time, it inputs facial image data into an emotion engine to recognize the user's emotions. The emotion engine evaluates the user's stress level and emotional state.

[1057] 4. Verification (Server)

[1058] The identified ingredient information is checked against an allergy database to determine whether it contains any allergens, and the emotion engine results are also sent to the device.

[1059] 5. Alert notification (terminal)

[1060] If an allergen is detected, the device will display a warning message based on the results of the emotion engine. For example, if the user is relaxed, it will display a gentle message such as "Be careful," while if the user is stressed, it will display a more emphatic message such as "Avoid immediately." Audio warnings are also adjusted in the same way.

[1061] Specific examples

[1062] For example, consider a user choosing food in a cafeteria. The user looks at the food and the camera captures an image of it. At the same time, the user's facial expression is also captured. This data is sent from the device to the server, which analyzes the food's ingredients and identifies them as "egg yolk, soy, and wheat flour." Meanwhile, the emotion engine analyzes the user's facial expression and recognizes that the user is nervous. The analysis confirms that the food contains soy, an allergen. The device receives this information, displays an emergency message on the screen saying "Dangerous food 💀," and issues a voice warning saying "Contains soy, be careful." The user can avoid the food based on this warning, thereby reducing the risk of allergies.

[1063] In this way, this system quickly and accurately detects allergens in food and provides warnings according to the user's emotional state, thereby achieving safe and effective allergy management.By combining it with history management and disaster detection functions, it is possible to protect user safety in a variety of situations.

[1064] The processing flow will be explained below.

[1065] Step 1:

[1066] The user looks at food, and the camera built into the glasses captures an image of the food in the user's field of view and an image of the user's face.

[1067] Step 2:

[1068] The device captures food images and facial images, which are then pre-processed to remove noise and adjust resolution.

[1069] Step 3:

[1070] The device temporarily stores the preprocessed image data in local storage.

[1071] Step 4:

[1072] The device sends the stored image data to the cloud server using an HTTP POST request, sending the image data over the network.

[1073] Step 5:

[1074] The server analyzes the received image data. The server's generative model analysis means processes the food image using an AI model to identify and extract food ingredients.

[1075] Step 6:

[1076] The server inputs facial images into the emotion engine, which recognizes emotions from the user's facial expressions and evaluates their stress level and emotional state.

[1077] Step 7:

[1078] The server checks the identified food ingredient information against an allergy database. A checking means compares the ingredient information with the database to confirm the presence of allergens.

[1079] Step 8:

[1080] The server returns the food ingredient matching results and the emotion engine evaluation results to the device. The results are formatted in JSON and sent as an HTTP response.

[1081] Step 9:

[1082] The terminal analyzes the results received from the server, and if an allergen is found, a visual warning means and an audio warning means are activated.

[1083] Step 10:

[1084] Based on the results of the emotion engine, the device displays the most appropriate visual warning to the user. For example, if the user is relaxed, it displays "Caution required," and if the user is tense, it displays "Avoid immediately."

[1085] Step 11:

[1086] The device provides the user with a voice alert based on the results of the emotion engine. The voice alert is also adjusted according to the user's emotions. For example, if the user is relaxed, the alert will be softer in tone, and if the user is tense, the alert will be more emphatic.

[1087] This allows users to quickly and accurately learn about food allergens, and also to receive appropriate warnings according to their current emotions.

[1088] Example 2

[1089] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1090] In modern society, the presence of allergens in food poses a serious risk to the user's health. Furthermore, conventional allergy detection systems simply detect allergens and are unable to adapt the content and method of warnings to take the user's emotional state into account. This can make it difficult for users to properly accept the warnings, and can result in insufficient reduction of allergy risk. Furthermore, the system does not take into account the identification of edibles, poisons, or harmful substances in the event of a disaster. Thus, providing a safe environment that takes the user's emotional state into account is a challenge for allergy risk management systems.

[1091] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an imaging means for capturing an image of the food, an image processing means for preprocessing the captured image, a communication means for transmitting the preprocessed image to a remote server, a generative model analysis means for analyzing the image in the remote server and identifying food components, a comparison means for comparing the food components with an allergy database, a visual warning means and an audio warning means for displaying a warning to the user if the presence of an allergen is detected as a result of the comparison, an emotion recognition means for capturing an image of the user's face and evaluating their emotional state, and an adaptive control means for adapting the content and method of the warning based on the emotional state. This enables safe and effective allergy management by quickly and accurately detecting allergens contained in food and providing a warning according to the user's emotional state.

[1092] "Photographing means for capturing images of food" is a general term for devices and techniques that record images of food within the user's field of view.

[1093] "Image processing means for pre-processing captured images" is a general term for devices or software that perform noise reduction and resolution adjustment to improve the quality of captured images.

[1094] "Communication means for transmitting preprocessed images to a remote server" is a general term for interfaces and protocols for uploading processed image data to a remote server over a network.

[1095] "Generative model analysis means that analyzes images on a remote server and identifies food components" is a general term for technologies and algorithms that process received image data using an AI model to identify and extract food components.

[1096] "Means for matching food ingredients with an allergy database" is a general term for technologies and functions for comparing identified food ingredients with an existing allergy database to confirm the presence of allergens.

[1097] "Visual and audio warning means that display a warning to the user when the presence of an allergen is detected as a result of the comparison" is a general term for devices and functions that provide a warning by screen display or audio to inform the user that an allergen is present.

[1098] "Emotion recognition means for capturing a user's facial image and evaluating the user's emotional state" is a general term for devices and algorithms for analyzing a user's facial expression and evaluating the user's emotional state.

[1099] "Adaptive control means for adapting the content and method of warning based on the emotional state" is a general term for techniques and functions for changing the content and method of warning depending on the emotional state of the user.

[1100] The present invention relates to a system that quickly and accurately detects allergens and provides a warning to the user. This system is combined with an emotion engine that recognizes the user's emotions, allowing the content and method of the warning to be adapted to the user's emotional state. Specific embodiments are described below.

[1101] Overall system configuration

[1102] This system consists of the following main components:

[1103] 1. Shooting method (device):

[1104] A camera built into glasses worn by the user captures images of food in the user's field of view as well as images of the user's face. For example, if a user is choosing a salad in a cafeteria, the camera in the glasses automatically captures the salad and the user's facial expression.

[1105] 2. Image processing means (terminal):

[1106] The captured images are pre-processed to improve their quality. This pre-processing includes adjusting the resolution and removing noise. For example, the resolution is standardized to 1080p and a Gaussian filter is applied to make the image clearer.

[1107] 3. Communication means (terminal):

[1108] A network interface and protocol for transmitting preprocessed images to a remote server using HTTP POST requests to upload image data over Wi-Fi or cellular networks.

[1109] 4. Generative model analysis means (server):

[1110] The remote server inputs the received image data into an AI model to identify and extract the food's ingredients. This analysis uses AI models such as SVM and ResNet. For example, the food's ingredients are identified as "egg yolk, soybeans, and wheat flour."

[1111] 5. Emotion recognition means (server):

[1112] The remote server inputs facial image data into the emotion engine, which analyzes the user's facial expressions and evaluates their emotional state. The emotion engine uses deep learning algorithms to detect the user's stress level and emotional state. For example, it can recognize that the user is "tense."

[1113] 6. Verification method (server):

[1114] The remote server checks the identified ingredient information against an allergy database to confirm the presence of allergens, for example, soy is found to be an allergen.

[1115] 7. Adaptive control means (terminal):

[1116] Based on the analysis results, the device will adapt the content and method of warnings to suit the user's emotional state. For example, if the user is nervous, an emergency message will be displayed on the screen saying "Dangerous Food 💀" and a voice warning will be given saying "Contains soy, be careful."

[1117] Specific examples

[1118] Usage scene in the cafeteria

[1119] Consider a scenario in which a user is choosing food in a restaurant. At this time, a camera captures images of the food the user is looking at and their facial expression. The device preprocesses these images and sends them to a remote server. The server analyzes the food's ingredients and identifies "egg yolk, soybeans, and wheat flour." At the same time, the emotion engine analyzes the user's facial expression and recognizes that the user is "nervous."

[1120] If the analysis results confirm that the food contains soy, the device will display a warning message according to the user's "tension" state. "Dangerous Food 💀" will be displayed on the screen, and a voice message will be played saying, "Contains soy. Be careful." Users can reduce their risk of allergies by avoiding the food based on this warning.

[1121] Prompt Sentence Examples

[1122] "Imagine a scene where a user is choosing food in a restaurant. The camera captures images of the food the user is looking at and their facial expressions. The server then analyzes the food's ingredients from the received data and identifies allergens. At the same time, it recognizes the user's emotional state and displays appropriate warning messages."

[1123] As described above, the present invention achieves safe and effective allergy management by quickly and accurately detecting allergens in food and providing warnings tailored to the user's emotional state.

[1124] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1125] Step 1:

[1126] User views food:

[1127] When a user looks at a food item, a camera built into the device captures an image of the food item and an image of the user's face.

[1128] Input: Food and its facial expression in the user's field of view through the glasses.

[1129] Output: Captured food images and user's face image data.

[1130] Step 2:

[1131] The device performs image preprocessing:

[1132] The device performs pre-processing on the captured image, such as adjusting the resolution and removing noise.

[1133] Input: A captured food image and a user's face image.

[1134] Data processing: Unification of resolution (e.g. 1080p), noise removal (e.g. applying Gaussian filter).

[1135] Output: Preprocessed food images and face images.

[1136] Step 3:

[1137] The device sends data to the server:

[1138] The device sends the preprocessed image data to the remote server using an HTTP POST request.

[1139] Input: Preprocessed food images and face images.

[1140] Data processing: Packaging data into HTTP requests.

[1141] Output: Image data sent to the remote server.

[1142] Step 4:

[1143] The server analyzes the food ingredients:

[1144] The server inputs the received food image data into an AI model to identify and extract the food's ingredients.

[1145] Input: Food image data sent to the server.

[1146] Data processing: Image analysis using AI models (e.g., SVM and ResNet).

[1147] Output: Identified food ingredients (e.g., "egg yolk, soy, wheat flour").

[1148] Step 5:

[1149] The server analyzes the user's emotions:

[1150] The server inputs the received facial image into an emotion engine, analyzes the user's facial expression, and evaluates their emotional state.

[1151] Input: User's facial image data sent to the server.

[1152] Data processing: Facial expression analysis using emotion engine (deep learning algorithm).

[1153] Output: The user's assessed emotional state (e.g., "tense").

[1154] Step 6:

[1155] The server verifies the ingredients:

[1156] The server compares the identified food ingredient information with an allergy database to determine whether the food contains any allergens.

[1157] Input: Identified food ingredient information.

[1158] Data processing: Comparison with allergy database.

[1159] Output: Matching result (e.g. "Soy is an allergen").

[1160] Step 7:

[1161] The server sends the analysis results to the device:

[1162] The server sends the results of the user's emotion analysis and food ingredient analysis to the terminal.

[1163] Input: Matching result and emotional state.

[1164] Data processing: Generate JSON data containing the information.

[1165] Output: Analysis result data sent to the device.

[1166] Step 8:

[1167] The device provides a warning to the user:

[1168] Based on the analysis results, the device provides the user with visual and audio warnings, adapting the content and method of the warning depending on the user's emotional state.

[1169] Input: Analysis result data (matching results and emotional state).

[1170] Data processing: Generate warning messages and configure display and sound settings.

[1171] Output: Visual and audio warning messages (e.g. "Dangerous Food 💀" and "Contains soy, be careful").

[1172] The above are the specific processing steps of the program of this system.

[1173] (Application example 2)

[1174] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1175] Rapid and accurate detection of allergens in food is very important to many people. However, systems that provide uniform warnings without considering the user's emotional state may cause excessive stress to the user. In particular, in situations such as food delivery, appropriate warnings based on the user's emotional state are required. Therefore, it is necessary to provide a system that can detect allergens in food and provide warnings based on the user's emotional state.

[1176] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1177] In this invention, the server includes a camera means for capturing images of food, an image processing means for preprocessing the captured images, a communication means for transmitting the preprocessed images to a cloud server, a generative model analysis means for analyzing the images in the cloud server and identifying food ingredients, a comparison means for comparing the food ingredients with an allergy database, an emotion engine for recognizing emotions from the user's facial expressions and evaluating their stress level and emotional state, and a visual warning means and an audio warning means for displaying a warning to the user based on the results of the emotion engine if the presence of an allergen is detected as a result of the comparison. This makes it possible to quickly and accurately detect allergens in food and provide an appropriate warning according to the user's emotional state.

[1178] "Camera means" refers to a device that is built into a user-worn device and has the ability to capture images of food and the user's face.

[1179] "Image processing means" refers to devices or algorithms that pre-process captured images to remove noise or adjust resolution.

[1180] "Communication means" refers to the network interface and protocol used to send preprocessed images to the cloud server.

[1181] The "generative model analysis means" is an AI model or machine learning algorithm that analyzes images received by the cloud server and identifies food ingredients.

[1182] The "comparison means" is a function that compares the identified food ingredient information with an allergy database to confirm the presence of allergens.

[1183] The "emotion engine" is an algorithm that recognizes emotions from the user's facial expressions and evaluates their stress level and emotional state.

[1184] A "visual warning means" is a device or software that displays a visual warning to the user based on the results of the emotion engine when an allergen is detected.

[1185] The "audio warning means" is a device or software that provides an audio warning to the user based on the results of the emotion engine when an allergen is detected.

[1186] The "history management function" is a function that manages the past intake history of allergens and provides warnings based on that history.

[1187] The "disaster detection function" is a function that identifies edible items, poisons, and harmful substances during a disaster and provides information to the user.

[1188] The present invention is a system that quickly and accurately detects food allergens and provides appropriate warnings according to the user's emotional state. The overall system configuration consists of the following main components:

[1189] 1. Camera Means

[1190] The camera is built into a device worn by the user and captures images of the food and the user's face. This image capture collects data on the food the user is viewing and their facial expression.

[1191] 2. Image Processing Methods

[1192] The captured images are pre-processed, including noise removal, resolution adjustment, normalization, etc. Specifically, OpenCV is used to perform these image processing.

[1193] 3. Means of communication

[1194] The preprocessed image is sent to the cloud server using a network interface and a communication protocol such as an HTTP request (POST request).

[1195] 4. Generative Model Analysis Methods

[1196] The cloud server analyzes the received image data using an AI model to identify and extract food ingredients. This analysis uses machine learning algorithms such as TensorFlow and Keras. An example prompt is, "Analyze the food image to identify its ingredients and determine whether it contains any of the following substances to which you are allergic: egg yolk, soy, or wheat flour."

[1197] 5. Verification Method

[1198] The identified food ingredient information is compared with an allergy database to confirm the presence of allergens, thereby determining whether the ingredients extracted by the generative model analysis means are pre-registered allergens.

[1199] 6. Emotion Engine

[1200] This algorithm recognizes emotions based on images of a user's face and evaluates their stress level and emotional state. Emotion recognition is performed using an emotion analysis engine such as EmotionRecognitionEngine. This engine analyzes the user's facial expression data and classifies their emotional state as "tension" or "stress," etc.

[1201] 7. Visual and audio warning means

[1202] If an allergen is detected, a warning will be provided to the user based on the results of the emotion engine. For example, if the user is nervous, it will display "Dangerous Food 💀 - Contains {Detected Allergen}, Be Careful," and if the user is relaxed, it will display "Contains {Detected Allergen}, Be Careful." Audio warnings will also be adjusted accordingly.

[1203] Specific examples

[1204] For example, when a user receives a food delivery, they view the food through the smart glasses and capture an image of it using the camera means. The image is then preprocessed by the image processing means and sent to a cloud server via the communication means. The server identifies the food ingredients using the generative model analysis means and checks for allergens using the matching means. At the same time, the emotion engine recognizes the user's emotional state from their facial expressions. Finally, if an allergen is detected, the emotion engine provides the user with an appropriate visual and audio warning based on the evaluation results of the emotion engine.

[1205] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1206] Step 1:

[1207] When a user receives a food delivery, they view the food through the smart glasses. A camera means (terminal) captures an image of the food and an image of the user's face. The input is the viewed food and the user's facial expression, and the output is the captured image data. This captured image data is used in the next step.

[1208] Step 2:

[1209] The device preprocesses the captured image data using image processing means. Specifically, it uses OpenCV to perform noise removal, resolution adjustment, and normalization. The input is the captured image data, and the output is the preprocessed image data. This preprocessed image data is used in the next step.

[1210] Step 3:

[1211] The preprocessed image data is sent to the cloud server via the communication means (terminal). This communication is performed using an HTTP request (POST request) via a network interface. The input is the preprocessed image data, and the output is the upload completion status to the cloud server.

[1212] Step 4:

[1213] The cloud server analyzes the received image data using a generative model analysis method. Specifically, it uses TensorFlow and Keras to identify and extract food ingredients. Here, the prompt is "Analyze the food image, identify the ingredients contained, and determine whether the following allergens are present: egg yolk, soy, wheat flour." The input is the image data uploaded to the cloud server, and the output is the identified food ingredient information.

[1214] Step 5:

[1215] The verification means (server) verifies the identified food ingredient information against the allergy database to confirm the presence of allergens. The input is the identified food ingredient information, and the output is the allergen detection result. This result is used in the next step.

[1216] Step 6:

[1217] At the same time, the server uses an emotion engine to analyze the user's facial image and recognize their emotional state. Specifically, it uses the EmotionRecognitionEngine to analyze the user's facial expression data and classify the user's emotions as "tension" or "stress." The input is the facial image data uploaded to the cloud server, and the output is the classification result of the user's emotional state. This classification result of the emotional state is used in the next step.

[1218] Step 7:

[1219] If an allergen is detected, the device will provide a visual and audio warning to the user based on the emotion engine's results. Specifically, if the user is nervous, the device will display "Dangerous Food 💀 - Contains {Detected Allergen}, Be Careful," and if the user is relaxed, the device will display "Contains {Detected Allergen}, Be Careful." The audio warning is also adjusted in a similar way. The input is the allergen detection result and the classification result of the user's emotional state, and the output is a warning message to the user.

[1220] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1221] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1222] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1223] [Fourth embodiment]

[1224] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1225] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1226] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1227] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1228] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1229] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1230] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1231] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1232] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1233] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1234] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1235] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1236] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1237] The present invention relates to a system for quickly and accurately detecting allergens and providing a warning to a user. Specific embodiments are described below.

[1238] Overall system configuration

[1239] This system consists of the following main components:

[1240] 1. Camera means (terminal):

[1241] It is a camera built into glasses worn by the user that captures images of food within the user's field of view.

[1242] 2. Image processing means (terminal):

[1243] A device or algorithm that preprocesses a captured image to remove noise or adjust resolution.

[1244] 3. Communication means (terminal):

[1245] A network interface and protocol for sending preprocessed images to a cloud server.

[1246] 4. Generative model analysis means (server):

[1247] An AI model or machine learning algorithm that analyzes images on a cloud server and identifies food ingredients.

[1248] 5. Verification method (server):

[1249] A function that compares identified ingredient information with an allergy database to confirm the presence of allergens.

[1250] 6. Visual and audio warning means (terminals):

[1251] A device or software that provides visual and audio warnings to the user if an allergen is detected.

[1252] 7. History management function (terminal or server):

[1253] A function that manages past intake history of allergens and provides warnings based on that information.

[1254] 8. Disaster detection function (terminal or server):

[1255] A function that identifies edible, poisonous, and harmful substances during a disaster and provides information to users.

[1256] Basic operation flow

[1257] 1. Image Capture and Preprocessing (Device)

[1258] When a user looks at a food item, the device's camera captures an image of it. The captured image undergoes pre-processing, such as adjusting resolution and brightness and removing noise, and is then temporarily saved to local storage.

[1259] 2. Sending images to the server (terminal)

[1260] The device sends the preprocessed image to the cloud server by uploading the data via an HTTP request (POST request).

[1261] 3. Food ingredient analysis (server)

[1262] The server analyzes the received images using an AI model. The generative model analysis means processes the image data to identify and extract food ingredients.

[1263] 4. Verification (Server)

[1264] The identified ingredient information is checked against an allergy database. A checking means on the server compares the ingredient information with the database to confirm the presence of allergens.

[1265] 5. Alert notification (terminal)

[1266] If an allergen is detected, the server sends the results to the device, which then displays a visual warning message on the screen and provides an audio warning to the user based on the results received.

[1267] Specific examples

[1268] For example, imagine a user picking up a chocolate bar at a supermarket. When the user looks at the product, the camera in the glasses captures an image, which is preprocessed on the device and then sent to the server. The server analyzes the image using an AI model to identify its ingredients: sugar, milk, and peanuts. This ingredient information is then compared with an allergy database, confirming that the bar contains peanuts. The server then sends this information to the device, which displays "Dangerous Food 💀" on the screen and warns the user audibly that "Contains peanuts."

[1269] In this way, this system reduces the risk of food allergies by quickly and accurately detecting allergens in foods that come into the user's field of vision and providing immediate visual and audio warnings. Furthermore, by incorporating history management and disaster detection functions, it protects the user's safety in a variety of situations.

[1270] The processing flow will be explained below.

[1271] Step 1:

[1272] A user picks up food at a supermarket, etc. A camera built into the glasses captures an image of the food in the user's field of view.

[1273] Step 2:

[1274] It takes the image captured by the device's camera and performs pre-processing on it, specifically adjusting the image resolution and applying a noise reduction filter.

[1275] Step 3:

[1276] The device temporarily stores the preprocessed image in local storage.

[1277] Step 4:

[1278] The device sends the saved images to the cloud server using an HTTP request (POST request), uploading the data to the server via the Internet.

[1279] Step 5:

[1280] The server receives the image data and stores it in a database.

[1281] Step 6:

[1282] The server analyzes the received image using a generative model analysis method. Specifically, the image is input into an AI model to identify and extract food ingredients.

[1283] Step 7:

[1284] The server compares the identified ingredient information with an allergy database to see if the product contains any allergens.

[1285] Step 8:

[1286] The server returns the matching results to the device in a format such as JSON.

[1287] Step 9:

[1288] The device receives the results from the server, analyzes the received data, and extracts the necessary information.

[1289] Step 10:

[1290] If your device contains an allergen, it will display a visual warning message on the screen, such as "Dangerous Food 💀."

[1291] Step 11:

[1292] The device will provide audible warnings to the user, using text-to-speech to say things like "Contains peanuts."

[1293] These steps allow users to quickly and accurately find out the allergens in food, allowing them to make safe food choices.

[1294] Example 1

[1295] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1296] In modern society, there is a demand for the rapid and accurate detection of allergens contained in food and the provision of warnings to users. However, conventional methods have problems such as delays in allergen detection and a lack of accuracy. Furthermore, in times of disaster, it is difficult to determine which foods are safe to consume. Furthermore, there is a lack of systems that can manage the past intake history of allergens and provide appropriate warnings based on that information.

[1297] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1298] In this invention, the server includes an imaging means for capturing an image of the food, an image processing means for preprocessing the captured image, a communication means for transmitting the preprocessed image to the data server, a generative AI model analysis means for analyzing the image in the data server and identifying the food components, a comparison means for comparing the food components with an allergy database, and a visual warning means and an audio warning means for displaying a warning to the user if the presence of an allergen is detected as a result of the comparison. This makes it possible to quickly and accurately detect allergens contained in food and provide an immediate warning to the user.

[1299] "Imaging means" refers to a device used to capture an image of a food item, and includes, for example, a camera or an image sensor.

[1300] "Image processing means" refers to devices or algorithms for pre-processing captured images, such as adjusting resolution, correcting brightness, and removing noise.

[1301] "Communication means" refers to a device or protocol for transmitting preprocessed images to a data server, including a network interface and communication software.

[1302] "Generative AI model analysis means" includes artificial intelligence models and machine learning algorithms for analyzing images received by the data server and identifying food ingredients.

[1303] A "verification means" is a device or software that compares identified food ingredients with an allergy database to confirm the presence of allergens.

[1304] A "visual alert means" is a device or interface for displaying a visual alert to a user if an allergen is detected.

[1305] An "audio warning means" is a device or system for providing an audio warning to a user when an allergen is detected.

[1306] The "history management function" is a function for managing the past intake history of allergens and providing warnings to the user based on that history.

[1307] The "disaster detection function" is a function that identifies edible items, poisons, and harmful substances during a disaster and provides information to the user.

[1308] MODE FOR CARRYING OUT THE INVENTION

[1309] The present invention relates to a system for quickly and accurately detecting allergens and providing a warning to a user. DETAILED DESCRIPTION OF THE INVENTION ...

[1310] System configuration

[1311] This system consists of the following main components:

[1312] 1. Imaging means (terminal):

[1313] A camera built into the glasses worn by the user captures images of food within the user's field of view. Specifically, a small camera or image sensor is used.

[1314] 2. Image processing means (terminal):

[1315] The captured image is pre-processed by the device's built-in image processor or algorithms, which may include adjusting the resolution, correcting brightness, and removing noise, making the image easier to analyze.

[1316] 3. Communication means (terminal):

[1317] The preprocessed image is sent to the cloud server via the device's communication interface. Specifically, the data is uploaded using the POST method of the HTTP request.

[1318] 4. Generative AI model analysis means (server):

[1319] The cloud server analyzes the submitted images using generative AI models, which are used to identify food ingredients. Specifically, these include algorithms using machine learning and deep learning techniques.

[1320] 5. Verification method (server):

[1321] The server's verification means compares the identified ingredient information with an allergy database to confirm the presence of allergens. By comparing the information with the database, the presence of allergens is detected.

[1322] 6. Visual and audio warning means (terminals):

[1323] If an allergen is detected, the server sends the result to the device, which then provides a visual and audio warning to the user based on the received information. Specifically, a warning message is displayed on the device's display and the warning content is read aloud.

[1324] 7. History management function (terminal or server):

[1325] It has a function to manage the past intake history of allergens and provide appropriate warnings based on that, which allows for customized warnings based on the user's allergy history.

[1326] 8. Disaster detection function (terminal or server):

[1327] The system has the ability to identify edible foods, poisons, and harmful substances and provide information to users, enabling them to make safe food choices even during disasters.

[1328] Specific operation example

[1329] For example, consider a user picking up a chocolate bar at a supermarket. When the user looks at the product, the camera built into the glasses captures an image, which is then sent to a cloud server after the device adjusts the resolution and removes noise. A generative AI model on the server analyzes the image and identifies ingredients such as "sugar, milk, and peanuts." The server's matching means compares the ingredient information with an allergy database and confirms that it contains peanuts. The results are sent to the device, which displays "Dangerous Food 💀" and provides a voice warning to the user that "Contains peanuts."

[1330] Also, a specific example of a prompt sentence to input to the generative AI model is as follows:

[1331] "I have provided an image. This image shows a food product. Please identify the ingredients in the food product and detect allergens. Please return the ingredients in the full document. Please check if the ingredients contain the following allergens: milk, peanuts, eggs."

[1332] Using this prompt, the AI ​​model analyzes food ingredient information and detects the presence of allergens.

[1333] Other Features

[1334] This system has the ability to manage the user's allergen intake history, allowing it to provide more detailed and personalized warnings. It also has the ability to identify foods and harmful substances that can be ingested in the event of a disaster, ensuring safety in emergencies.

[1335] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1336] The flow of this system's program processing

[1337] Step 1:

[1338] (User sees food)

[1339] When a user looks at a food item, the device's imaging means (built-in camera) automatically captures an image of the food item. For example, when a user picks up a chocolate bar in a supermarket, an image of the chocolate bar's packaging is captured. Input: User's visual behavior Output: Captured image data

[1340] Step 2:

[1341] (Device preprocesses the image)

[1342] The captured image is pre-processed by the device's image processing means. Specifically, the resolution is adjusted, brightness is corrected, and noise is removed. This prepares the image in a format that is easy to analyze. Input: Captured image data Output: Pre-processed image data

[1343] Step 3:

[1344] (The device sends the image to the cloud server)

[1345] The preprocessed image is sent to the cloud server via the device's communication means. Specifically, the image data is uploaded using an HTTP POST request. Input: Preprocessed image data Output: Image data sent to the server

[1346] Step 4:

[1347] (The server analyzes the image using a generative AI model)

[1348] The cloud server analyzes the received image data using the AI ​​model analysis means. The AI ​​model performs data calculations to identify food ingredients based on the input image data. For example, the AI ​​model extracts ingredient information such as "sugar, milk, peanuts" from an image of a chocolate bar package. Input: Image data sent to the server Output: Identified food ingredient information

[1349] Step 5:

[1350] (The server checks the ingredient information against the allergy database)

[1351] The server's verification means compares the identified ingredient information with the allergy database. This verification confirms the presence of allergens in the food. For example, the verification result confirms that the food contains peanuts. Input: Identified food ingredient information Output: Verification result (presence or absence of allergens)

[1352] Step 6:

[1353] (The server sends the matching results to the device.)

[1354] Once the matching result is obtained, the server sends it to the device. Input: Matching result Output: Alert information sent to the device

[1355] Step 7:

[1356] (Device provides warning to user)

[1357] The device will provide a visual and audio warning to the user based on the received alert information. Specifically, it will display "Dangerous Food 💀" on the device's display and warn the user by voice, "Contains peanuts." Input: Alert information sent to the device Output: Visual and audio warning to the user

[1358] Step 8:

[1359] (The device or server manages the history.)

[1360] The history management function on the device or server records information about the ingredients of the food the user is about to consume and the history of allergen detection. This improves the accuracy of future warnings. Input: User's intake history and detection history. Output: Updated history data.

[1361] Step 9:

[1362] (The terminal or server will detect food in the event of a disaster.)

[1363] In the event of a disaster, the terminal or server provides information on foods that can be consumed by the user and on harmful substances. This allows users to safely select foods even in an emergency. Input: Environmental information and food information during a disaster. Output: Safe food information provided to the user.

[1364] The above is the flow of specific processing steps for this system.

[1365] (Application example 1)

[1366] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1367] Conventional allergy detection systems lack the speed and accuracy to quickly identify food allergens and provide appropriate warnings to users. Furthermore, methods for accurately identifying food allergens in moving environments, such as autonomous vehicles, have not been fully established. Therefore, an efficient and reliable system for reducing food allergy risks is needed.

[1368] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1369] In this invention, the server includes an image capturing means, an image processing means, a communication means, a generative model analysis means, a matching means, a visual warning means, an audio warning means, and a means for capturing images obtained from an internal camera of the autonomous vehicle and identifying food allergens. This makes it possible to quickly and accurately identify food allergens in a moving autonomous vehicle and provide appropriate visual and audio warnings to the user.

[1370] - "Photography means" refers to any device or technology used to capture an image of a food item.

[1371] "Image processing means" refers to devices or techniques for pre-processing captured images, such as noise reduction and resolution adjustment.

[1372] "Communication means" refers to the network interface and protocols used to transmit preprocessed images to a cloud data center.

[1373] "Generative model analysis means" refers to AI models and machine learning algorithms that analyze images in a cloud data center and identify food elements.

[1374] "Matching means" refers to the technology or method used to match food elements with an allergy database to confirm the presence of allergens.

[1375] "Visual warning means" means any device or software that provides a visual warning to the user upon detecting the presence of an allergen.

[1376] "Audio warning means" means a device or software that provides an audio warning to a user when it detects the presence of an allergen.

[1377] "Internal camera in an autonomous vehicle" refers to a camera installed inside an autonomous vehicle that captures images of food brought in by passengers.

[1378] The present invention relates to a system for quickly and accurately detecting food allergens in autonomous vehicles and providing warnings to users. The system consists of the following main components:

[1379] Overall system configuration

[1380] 1. Photography Method:

[1381] This is a camera installed inside the autonomous vehicle that captures images of food brought in by passengers, specifically when a passenger in the vehicle picks up the food.

[1382] 2. Image processing means:

[1383] Software for pre-processing the captured images to remove noise and adjust the resolution. For example, the OpenCV library can be used for image processing.

[1384] 3. Means of communication:

[1385] A network interface for sending preprocessed images to a cloud data center, specifically by sending data via HTTP POST requests.

[1386] 4. Generative model analysis methods:

[1387] AI models for analyzing images and identifying food elements in cloud data centers, for example, using the TensorFlow library to identify food ingredients.

[1388] 5. Verification method:

[1389] The identified food elements are checked against an allergy database to confirm the presence of allergens. Database checks are performed using an SQL server.

[1390] 6. Visual and audio warning means:

[1391] If an allergen is detected, the device will provide users with visual and audio alerts: visual alerts will be displayed on the smartphone screen and audio alerts will be played through the speaker.

[1392] 7. Means for capturing images obtained from an internal camera of an autonomous vehicle:

[1393] An interface for utilizing images acquired by the internal camera throughout the system. This interface allows real-time images acquired by the camera to be imported into the system.

[1394] Specific operation flow

[1395] 1. Cameras inside the self-driving vehicle capture images of food brought in by passengers.

[1396] 2. This image is pre-processed by image processing means to remove noise and adjust the resolution.

[1397] 3. The pre-processed images are sent to the cloud data center via communication means.

[1398] 4. The cloud data center uses generative model analysis tools to analyze the image and identify food elements.

[1399] 5. The identified food elements are matched against an allergy database via a matching means to confirm the presence of allergens.

[1400] 6. If an allergen is detected, a warning is provided to the user through visual and audio warning means.

[1401] Prompt Sentence Examples

[1402] Based on the image of food captured by the user with the camera, identify the ingredients contained in it, match it with an allergy database, and return the results.

[1403] Image data:<image_data>

[1404] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1405] Step 1:

[1406] Input: Food images obtained from a camera mounted inside an autonomous vehicle.

[1407] How it works: The device activates a camera installed inside the autonomous vehicle and captures an image of the food brought in by the passenger.

[1408] Output: Image data of the captured food.

[1409] Step 2:

[1410] Input: Captured food image data.

[1411] What it does: The device pre-processes the captured image by removing noise and adjusting the resolution, specifically by resizing and color correcting the image using the OpenCV library.

[1412] Output: Preprocessed image data.

[1413] Step 3:

[1414] Input: Preprocessed image data.

[1415] How it works: The device sends the preprocessed image data to the cloud data center, where it uploads the image data to the specified URL using an HTTP POST request.

[1416] Output: Notification of completion of image data transmission to the cloud data center.

[1417] Step 4:

[1418] Input: Image data sent to the cloud data center.

[1419] How it works: The server receives the image data and performs generative model analysis using the AI ​​model. It then uses the TensorFlow library to perform data calculations to identify food elements from the image.

[1420] Output: Parsed food element data.

[1421] Step 5:

[1422] Input: Food element data analyzed by the AI ​​model.

[1423] How it works: The server checks the food element data it retrieves against the allergy database. Using an SQL server, it compares it with existing data in the database to determine whether or not any allergens are present.

[1424] Output: Matching result (whether or not allergens were detected and details).

[1425] Step 6:

[1426] Input: Whether or not allergens are detected and detailed data.

[1427] How it works: The server sends the matching results to the device, and if an allergen is detected, the device provides visual and audio warnings to the user. A warning message is displayed on the smartphone screen and an audio warning is played through the speaker.

[1428] Output: Visual and audio warnings to the user.

[1429] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1430] The present invention relates to a system that quickly and accurately detects allergens and provides a warning to the user, and further combines it with an emotion engine that recognizes the user's emotions, allowing the content and method of the warning to be adapted to the user's emotional state. Specific embodiments are described below.

[1431] Overall system configuration

[1432] This system consists of the following main components:

[1433] 1. Camera means (terminal):

[1434] A camera built into glasses worn by the user that captures images of food in the user's field of view and images of the user's face.

[1435] 2. Image processing means (terminal):

[1436] A device or algorithm that preprocesses a captured image to remove noise or adjust resolution.

[1437] 3. Communication means (terminal):

[1438] A network interface and protocol for sending preprocessed images to a cloud server.

[1439] 4. Generative model analysis means (server):

[1440] An AI model or machine learning algorithm that analyzes images on a cloud server and identifies food ingredients.

[1441] 5. Verification method (server):

[1442] A function that compares identified ingredient information with an allergy database to confirm the presence of allergens.

[1443] 6. Emotion engine (terminal or server):

[1444] An algorithm that recognizes emotions from a user's facial expressions and assesses their stress level and emotional state.

[1445] 7. Visual and audio warning means (terminals):

[1446] A device or software that provides visual and audio warnings to the user when an allergen is detected, based on the results of the emotion engine.

[1447] 8. History management function (terminal or server):

[1448] A function that manages past intake history of allergens and provides warnings based on that information.

[1449] 9. Disaster detection function (terminal or server):

[1450] A function that identifies edible, poisonous, and harmful substances during a disaster and provides information to users.

[1451] Basic operation flow

[1452] 1. Image Capture and Preprocessing (Device)

[1453] When a user looks at a food item, the device's camera captures an image of it, along with an image of the user's face. The captured image is then pre-processed to adjust resolution, brightness, and noise reduction.

[1454] 2. Sending images to the server (terminal)

[1455] The device sends the preprocessed food and face images to the cloud server by uploading the data via an HTTP request (POST request).

[1456] 3. Food ingredient analysis and emotion recognition (server)

[1457] The server analyzes the received food image data using an AI model to identify and extract food ingredients. At the same time, it inputs facial image data into an emotion engine to recognize the user's emotions. The emotion engine evaluates the user's stress level and emotional state.

[1458] 4. Verification (Server)

[1459] The identified ingredient information is checked against an allergy database to determine whether it contains any allergens, and the emotion engine results are also sent to the device.

[1460] 5. Alert notification (terminal)

[1461] If an allergen is detected, the device will display a warning message based on the results of the emotion engine. For example, if the user is relaxed, it will display a gentle message such as "Be careful," while if the user is stressed, it will display a more emphatic message such as "Avoid immediately." Audio warnings are also adjusted in the same way.

[1462] Specific examples

[1463] For example, consider a user choosing food in a cafeteria. The user looks at the food and the camera captures an image of it. At the same time, the user's facial expression is also captured. This data is sent from the device to the server, which analyzes the food's ingredients and identifies them as "egg yolk, soy, and wheat flour." Meanwhile, the emotion engine analyzes the user's facial expression and recognizes that the user is nervous. The analysis confirms that the food contains soy, an allergen. The device receives this information, displays an emergency message on the screen saying "Dangerous food 💀," and issues a voice warning saying "Contains soy, be careful." The user can avoid the food based on this warning, thereby reducing the risk of allergies.

[1464] In this way, this system quickly and accurately detects allergens in food and provides warnings according to the user's emotional state, thereby achieving safe and effective allergy management.By combining it with history management and disaster detection functions, it is possible to protect user safety in a variety of situations.

[1465] The processing flow will be explained below.

[1466] Step 1:

[1467] The user looks at food, and the camera built into the glasses captures an image of the food in the user's field of view and an image of the user's face.

[1468] Step 2:

[1469] The device captures food images and facial images, which are then pre-processed to remove noise and adjust resolution.

[1470] Step 3:

[1471] The device temporarily stores the preprocessed image data in local storage.

[1472] Step 4:

[1473] The device sends the stored image data to the cloud server using an HTTP POST request, sending the image data over the network.

[1474] Step 5:

[1475] The server analyzes the received image data. The server's generative model analysis means processes the food image using an AI model to identify and extract food ingredients.

[1476] Step 6:

[1477] The server inputs facial images into the emotion engine, which recognizes emotions from the user's facial expressions and evaluates their stress level and emotional state.

[1478] Step 7:

[1479] The server checks the identified food ingredient information against an allergy database. A checking means compares the ingredient information with the database to confirm the presence of allergens.

[1480] Step 8:

[1481] The server returns the food ingredient matching results and the emotion engine evaluation results to the device. The results are formatted in JSON and sent as an HTTP response.

[1482] Step 9:

[1483] The terminal analyzes the results received from the server, and if an allergen is found, a visual warning means and an audio warning means are activated.

[1484] Step 10:

[1485] Based on the results of the emotion engine, the device displays the most appropriate visual warning to the user. For example, if the user is relaxed, it displays "Caution required," and if the user is tense, it displays "Avoid immediately."

[1486] Step 11:

[1487] The device provides the user with a voice alert based on the results of the emotion engine. The voice alert is also adjusted according to the user's emotions. For example, if the user is relaxed, the alert will be softer in tone, and if the user is tense, the alert will be more emphatic.

[1488] This allows users to quickly and accurately learn about food allergens, and also to receive appropriate warnings according to their current emotions.

[1489] Example 2

[1490] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1491] In modern society, the presence of allergens in food poses a serious risk to the user's health. Furthermore, conventional allergy detection systems simply detect allergens and are unable to adapt the content and method of warnings to take the user's emotional state into account. This can make it difficult for users to properly accept the warnings, and can result in insufficient reduction of allergy risk. Furthermore, the system does not take into account the identification of edibles, poisons, or harmful substances in the event of a disaster. Thus, providing a safe environment that takes the user's emotional state into account is a challenge for allergy risk management systems.

[1492] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an imaging means for capturing an image of the food, an image processing means for preprocessing the captured image, a communication means for transmitting the preprocessed image to a remote server, a generative model analysis means for analyzing the image in the remote server and identifying food components, a comparison means for comparing the food components with an allergy database, a visual warning means and an audio warning means for displaying a warning to the user if the presence of an allergen is detected as a result of the comparison, an emotion recognition means for capturing an image of the user's face and evaluating their emotional state, and an adaptive control means for adapting the content and method of the warning based on the emotional state. This enables safe and effective allergy management by quickly and accurately detecting allergens contained in food and providing a warning according to the user's emotional state.

[1493] "Photographing means for capturing images of food" is a general term for devices and techniques that record images of food within the user's field of view.

[1494] "Image processing means for pre-processing captured images" is a general term for devices or software that perform noise reduction and resolution adjustment to improve the quality of captured images.

[1495] "Communication means for transmitting preprocessed images to a remote server" is a general term for interfaces and protocols for uploading processed image data to a remote server over a network.

[1496] "Generative model analysis means that analyzes images on a remote server and identifies food components" is a general term for technologies and algorithms that process received image data using an AI model to identify and extract food components.

[1497] "Means for matching food ingredients with an allergy database" is a general term for technologies and functions for comparing identified food ingredients with an existing allergy database to confirm the presence of allergens.

[1498] "Visual and audio warning means that display a warning to the user when the presence of an allergen is detected as a result of the comparison" is a general term for devices and functions that provide a warning by screen display or audio to inform the user that an allergen is present.

[1499] "Emotion recognition means for capturing a user's facial image and evaluating the user's emotional state" is a general term for devices and algorithms for analyzing a user's facial expression and evaluating the user's emotional state.

[1500] "Adaptive control means for adapting the content and method of warning based on the emotional state" is a general term for techniques and functions for changing the content and method of warning depending on the emotional state of the user.

[1501] The present invention relates to a system that quickly and accurately detects allergens and provides a warning to the user. This system is combined with an emotion engine that recognizes the user's emotions, allowing the content and method of the warning to be adapted to the user's emotional state. Specific embodiments are described below.

[1502] Overall system configuration

[1503] This system consists of the following main components:

[1504] 1. Shooting method (device):

[1505] A camera built into glasses worn by the user captures images of food in the user's field of view as well as images of the user's face. For example, if a user is choosing a salad in a cafeteria, the camera in the glasses automatically captures the salad and the user's facial expression.

[1506] 2. Image processing means (terminal):

[1507] The captured images are pre-processed to improve their quality. This pre-processing includes adjusting the resolution and removing noise. For example, the resolution is standardized to 1080p and a Gaussian filter is applied to make the image clearer.

[1508] 3. Communication means (terminal):

[1509] A network interface and protocol for transmitting preprocessed images to a remote server using HTTP POST requests to upload image data over Wi-Fi or cellular networks.

[1510] 4. Generative model analysis means (server):

[1511] The remote server inputs the received image data into an AI model to identify and extract the food's ingredients. This analysis uses AI models such as SVM and ResNet. For example, the food's ingredients are identified as "egg yolk, soybeans, and wheat flour."

[1512] 5. Emotion recognition means (server):

[1513] The remote server inputs facial image data into the emotion engine, which analyzes the user's facial expressions and evaluates their emotional state. The emotion engine uses deep learning algorithms to detect the user's stress level and emotional state. For example, it can recognize that the user is "tense."

[1514] 6. Verification method (server):

[1515] The remote server checks the identified ingredient information against an allergy database to confirm the presence of allergens, for example, soy is found to be an allergen.

[1516] 7. Adaptive control means (terminal):

[1517] Based on the analysis results, the device will adapt the content and method of warnings to suit the user's emotional state. For example, if the user is nervous, an emergency message will be displayed on the screen saying "Dangerous Food 💀" and a voice warning will be given saying "Contains soy, be careful."

[1518] Specific examples

[1519] Usage scene in the cafeteria

[1520] Consider a scenario in which a user is choosing food in a restaurant. At this time, a camera captures images of the food the user is looking at and their facial expression. The device preprocesses these images and sends them to a remote server. The server analyzes the food's ingredients and identifies "egg yolk, soybeans, and wheat flour." At the same time, the emotion engine analyzes the user's facial expression and recognizes that the user is "nervous."

[1521] If the analysis results confirm that the food contains soy, the device will display a warning message according to the user's "tension" state. "Dangerous Food 💀" will be displayed on the screen, and a voice message will be played saying, "Contains soy. Be careful." Users can reduce their risk of allergies by avoiding the food based on this warning.

[1522] Prompt Sentence Examples

[1523] "Imagine a scene where a user is choosing food in a restaurant. The camera captures images of the food the user is looking at and their facial expressions. The server then analyzes the food's ingredients from the received data and identifies allergens. At the same time, it recognizes the user's emotional state and displays appropriate warning messages."

[1524] As described above, the present invention achieves safe and effective allergy management by quickly and accurately detecting allergens in food and providing warnings tailored to the user's emotional state.

[1525] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1526] Step 1:

[1527] User views food:

[1528] When a user looks at a food item, a camera built into the device captures an image of the food item and an image of the user's face.

[1529] Input: Food and its facial expression in the user's field of view through the glasses.

[1530] Output: Captured food images and user's face image data.

[1531] Step 2:

[1532] The device performs image preprocessing:

[1533] The device performs pre-processing on the captured image, such as adjusting the resolution and removing noise.

[1534] Input: A captured food image and a user's face image.

[1535] Data processing: Unification of resolution (e.g. 1080p), noise removal (e.g. applying Gaussian filter).

[1536] Output: Preprocessed food images and face images.

[1537] Step 3:

[1538] The device sends data to the server:

[1539] The device sends the preprocessed image data to the remote server using an HTTP POST request.

[1540] Input: Preprocessed food images and face images.

[1541] Data processing: Packaging data into HTTP requests.

[1542] Output: Image data sent to the remote server.

[1543] Step 4:

[1544] The server analyzes the food ingredients:

[1545] The server inputs the received food image data into an AI model to identify and extract the food's ingredients.

[1546] Input: Food image data sent to the server.

[1547] Data processing: Image analysis using AI models (e.g., SVM and ResNet).

[1548] Output: Identified food ingredients (e.g., "egg yolk, soy, wheat flour").

[1549] Step 5:

[1550] The server analyzes the user's emotions:

[1551] The server inputs the received facial image into an emotion engine, analyzes the user's facial expression, and evaluates their emotional state.

[1552] Input: User's facial image data sent to the server.

[1553] Data processing: Facial expression analysis using emotion engine (deep learning algorithm).

[1554] Output: The user's assessed emotional state (e.g., "tense").

[1555] Step 6:

[1556] The server verifies the ingredients:

[1557] The server compares the identified food ingredient information with an allergy database to determine whether the food contains any allergens.

[1558] Input: Identified food ingredient information.

[1559] Data processing: Comparison with allergy database.

[1560] Output: Matching result (e.g. "Soy is an allergen").

[1561] Step 7:

[1562] The server sends the analysis results to the device:

[1563] The server sends the results of the user's emotion analysis and food ingredient analysis to the terminal.

[1564] Input: Matching result and emotional state.

[1565] Data processing: Generate JSON data containing the information.

[1566] Output: Analysis result data sent to the device.

[1567] Step 8:

[1568] The device provides a warning to the user:

[1569] Based on the analysis results, the device provides the user with visual and audio warnings, adapting the content and method of the warning depending on the user's emotional state.

[1570] Input: Analysis result data (matching results and emotional state).

[1571] Data processing: Generate warning messages and configure display and sound settings.

[1572] Output: Visual and audio warning messages (e.g. "Dangerous Food 💀" and "Contains soy, be careful").

[1573] The above are the specific processing steps of the program of this system.

[1574] (Application example 2)

[1575] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1576] Rapid and accurate detection of allergens in food is very important to many people. However, systems that provide uniform warnings without considering the user's emotional state may cause excessive stress to the user. In particular, in situations such as food delivery, appropriate warnings based on the user's emotional state are required. Therefore, it is necessary to provide a system that can detect allergens in food and provide warnings based on the user's emotional state.

[1577] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1578] In this invention, the server includes a camera means for capturing images of food, an image processing means for preprocessing the captured images, a communication means for transmitting the preprocessed images to a cloud server, a generative model analysis means for analyzing the images in the cloud server and identifying food ingredients, a comparison means for comparing the food ingredients with an allergy database, an emotion engine for recognizing emotions from the user's facial expressions and evaluating their stress level and emotional state, and a visual warning means and an audio warning means for displaying a warning to the user based on the results of the emotion engine if the presence of an allergen is detected as a result of the comparison. This makes it possible to quickly and accurately detect allergens in food and provide an appropriate warning according to the user's emotional state.

[1579] "Camera means" refers to a device that is built into a user-worn device and has the ability to capture images of food and the user's face.

[1580] "Image processing means" refers to devices or algorithms that pre-process captured images to remove noise or adjust resolution.

[1581] "Communication means" refers to the network interface and protocol used to send preprocessed images to the cloud server.

[1582] The "generative model analysis means" is an AI model or machine learning algorithm that analyzes images received by the cloud server and identifies food ingredients.

[1583] The "comparison means" is a function that compares the identified food ingredient information with an allergy database to confirm the presence of allergens.

[1584] The "emotion engine" is an algorithm that recognizes emotions from the user's facial expressions and evaluates their stress level and emotional state.

[1585] A "visual warning means" is a device or software that displays a visual warning to the user based on the results of the emotion engine when an allergen is detected.

[1586] The "audio warning means" is a device or software that provides an audio warning to the user based on the results of the emotion engine when an allergen is detected.

[1587] The "history management function" is a function that manages the past intake history of allergens and provides warnings based on that history.

[1588] The "disaster detection function" is a function that identifies edible items, poisons, and harmful substances during a disaster and provides information to the user.

[1589] The present invention is a system that quickly and accurately detects food allergens and provides appropriate warnings according to the user's emotional state. The overall system configuration consists of the following main components:

[1590] 1. Camera Means

[1591] The camera is built into a device worn by the user and captures images of the food and the user's face. This image capture collects data on the food the user is viewing and their facial expression.

[1592] 2. Image Processing Methods

[1593] The captured images are pre-processed, including noise removal, resolution adjustment, normalization, etc. Specifically, OpenCV is used to perform these image processing.

[1594] 3. Means of communication

[1595] The preprocessed image is sent to the cloud server using a network interface and a communication protocol such as an HTTP request (POST request).

[1596] 4. Generative Model Analysis Methods

[1597] The cloud server analyzes the received image data using an AI model to identify and extract food ingredients. This analysis uses machine learning algorithms such as TensorFlow and Keras. An example prompt is, "Analyze the food image to identify its ingredients and determine whether it contains any of the following substances to which you are allergic: egg yolk, soy, or wheat flour."

[1598] 5. Verification Method

[1599] The identified food ingredient information is compared with an allergy database to confirm the presence of allergens, thereby determining whether the ingredients extracted by the generative model analysis means are pre-registered allergens.

[1600] 6. Emotion Engine

[1601] This algorithm recognizes emotions based on images of a user's face and evaluates their stress level and emotional state. Emotion recognition is performed using an emotion analysis engine such as EmotionRecognitionEngine. This engine analyzes the user's facial expression data and classifies their emotional state as "tension" or "stress," etc.

[1602] 7. Visual and audio warning means

[1603] If an allergen is detected, a warning will be provided to the user based on the results of the emotion engine. For example, if the user is nervous, it will display "Dangerous Food 💀 - Contains {Detected Allergen}, Be Careful," and if the user is relaxed, it will display "Contains {Detected Allergen}, Be Careful." Audio warnings will also be adjusted accordingly.

[1604] Specific examples

[1605] For example, when a user receives a food delivery, they view the food through the smart glasses and capture an image of it using the camera means. The image is then preprocessed by the image processing means and sent to a cloud server via the communication means. The server identifies the food ingredients using the generative model analysis means and checks for allergens using the matching means. At the same time, the emotion engine recognizes the user's emotional state from their facial expressions. Finally, if an allergen is detected, the emotion engine provides the user with an appropriate visual and audio warning based on the evaluation results of the emotion engine.

[1606] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1607] Step 1:

[1608] When a user receives a food delivery, they view the food through the smart glasses. A camera means (terminal) captures an image of the food and an image of the user's face. The input is the viewed food and the user's facial expression, and the output is the captured image data. This captured image data is used in the next step.

[1609] Step 2:

[1610] The device preprocesses the captured image data using image processing means. Specifically, it uses OpenCV to perform noise removal, resolution adjustment, and normalization. The input is the captured image data, and the output is the preprocessed image data. This preprocessed image data is used in the next step.

[1611] Step 3:

[1612] The preprocessed image data is sent to the cloud server via the communication means (terminal). This communication is performed using an HTTP request (POST request) via a network interface. The input is the preprocessed image data, and the output is the upload completion status to the cloud server.

[1613] Step 4:

[1614] The cloud server analyzes the received image data using a generative model analysis method. Specifically, it uses TensorFlow and Keras to identify and extract food ingredients. Here, the prompt is "Analyze the food image, identify the ingredients contained, and determine whether the following allergens are present: egg yolk, soy, wheat flour." The input is the image data uploaded to the cloud server, and the output is the identified food ingredient information.

[1615] Step 5:

[1616] The verification means (server) verifies the identified food ingredient information against the allergy database to confirm the presence of allergens. The input is the identified food ingredient information, and the output is the allergen detection result. This result is used in the next step.

[1617] Step 6:

[1618] At the same time, the server uses an emotion engine to analyze the user's facial image and recognize their emotional state. Specifically, it uses the EmotionRecognitionEngine to analyze the user's facial expression data and classify the user's emotions as "tension" or "stress." The input is the facial image data uploaded to the cloud server, and the output is the classification result of the user's emotional state. This classification result of the emotional state is used in the next step.

[1619] Step 7:

[1620] If an allergen is detected, the device will provide a visual and audio warning to the user based on the emotion engine's results. Specifically, if the user is nervous, the device will display "Dangerous Food 💀 - Contains {Detected Allergen}, Be Careful," and if the user is relaxed, the device will display "Contains {Detected Allergen}, Be Careful." The audio warning is also adjusted in a similar way. The input is the allergen detection result and the classification result of the user's emotional state, and the output is a warning message to the user.

[1621] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1622] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1623] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1624] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1625] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1626] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1627] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1628] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1629] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1630] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1631] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1632] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1633] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1634] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1635] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1636] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1637] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1638] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1639] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1640] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1641] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1642] The following is further disclosed regarding the above embodiment.

[1643] (Claim 1)

[1644] camera means for capturing an image of the food product;

[1645] image processing means for pre-processing the captured image;

[1646] a communication means for transmitting the preprocessed image to a cloud server;

[1647] a generative model analysis means for analyzing images on a cloud server and identifying food components;

[1648] a matching means for matching the food ingredient with an allergy database;

[1649] a visual warning means and an audio warning means for displaying a warning to the user when the presence of an allergen is detected as a result of the comparison;

[1650] A system including:

[1651] (Claim 2)

[1652] 10. The system of claim 1, further comprising a history management function that provides warnings based on a history of past intake of allergens.

[1653] (Claim 3)

[1654] 10. The system of claim 1, further comprising a function for detecting edible, poisonous, or harmful substances in the event of a disaster.

[1655] "Example 1"

[1656] (Claim 1)

[1657] imaging means for capturing an image of the food;

[1658] image processing means for pre-processing the captured image;

[1659] a communication means for transmitting the preprocessed image to a data server;

[1660] A generative AI model analysis means for analyzing images on a data server and identifying food components;

[1661] a matching means for matching the food ingredient with an allergy database;

[1662] a visual warning means and an audio warning means for displaying a warning to the user when the presence of an allergen is detected as a result of the comparison;

[1663] A system including:

[1664] (Claim 2)

[1665] 10. The system of claim 1, further comprising a history management function that provides warnings based on a history of past intake of allergens.

[1666] (Claim 3)

[1667] 10. The system of claim 1, further comprising a function for detecting edible, poisonous, or harmful substances in the event of a disaster.

[1668] "Application Example 1"

[1669] (Claim 1)

[1670] an imaging means for capturing an image of the food;

[1671] image processing means for pre-processing the captured image;

[1672] a communication means for transmitting the preprocessed image to a cloud data center;

[1673] a generative model analysis means for analyzing the image and identifying food elements in a cloud data center;

[1674] a matching means for matching the food element with an allergy database;

[1675] a visual warning means and an audio warning means for displaying a warning to the user when the presence of an allergen is detected as a result of the comparison;

[1676] a means for capturing images obtained from an internal camera of the autonomous vehicle and identifying food allergens;

[1677] A system including:

[1678] (Claim 2)

[1679] 10. The system of claim 1, further comprising a history management function that provides warnings based on a history of past intake of allergens.

[1680] (Claim 3)

[1681] The system of claim 1 further comprising a function for detecting edible items and harmful ingredients in the event of a disaster.

[1682] "Example 2: Combining Emotion Engines"

[1683] (Claim 1)

[1684] an imaging means for capturing an image of the food;

[1685] image processing means for pre-processing the captured image;

[1686] a communication means for transmitting the preprocessed image to a remote server;

[1687] a generative model analysis means for analyzing the image on a remote server and identifying food components;

[1688] a matching means for matching the food ingredient with an allergy database;

[1689] a visual warning means and an audio warning means for displaying a warning to a user when the presence of an allergen is detected as a result of the comparison;

[1690] emotion recognition means for capturing a facial image of a user and assessing the user's emotional state;

[1691] adaptive control means for adapting the content and manner of warnings based on the emotional state;

[1692] A system including:

[1693] (Claim 2)

[1694] 10. The system according to claim 1, further comprising a history management means for providing a warning based on a past intake history of allergens.

[1695] (Claim 3)

[1696] 10. The system of claim 1, further comprising a function for detecting edible, poisonous, or harmful substances in the event of a disaster.

[1697] "Application example 2 when combining emotion engines"

[1698] (Claim 1)

[1699] camera means for capturing an image of the food product;

[1700] image processing means for pre-processing the captured image;

[1701] a communication means for transmitting the preprocessed image to a cloud server;

[1702] a generative model analysis means for analyzing images on a cloud server and identifying food components;

[1703] a matching means for matching the food ingredient with an allergy database;

[1704] An emotion engine that recognizes emotions from the user's facial expressions and evaluates their stress level and emotional state;

[1705] a visual warning means and an audio warning means for displaying a warning to the user based on the result of the emotion engine when the presence of an allergen is detected as a result of the comparison;

[1706] A system including:

[1707] (Claim 2)

[1708] 10. The system of claim 1, further comprising a history management function that provides warnings based on a history of past intake of allergens.

[1709] (Claim 3)

[1710] 10. The system of claim 1, further comprising a function for detecting edible, poisonous, or harmful substances in the event of a disaster. [Explanation of symbols]

[1711] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. camera means for capturing an image of the food product; image processing means for pre-processing the captured image; a communication means for transmitting the preprocessed image to a cloud server; a generative model analysis means for analyzing images on a cloud server and identifying food components; a matching means for matching the food ingredient with an allergy database; a visual warning means and an audio warning means for displaying a warning to the user when the presence of an allergen is detected as a result of the comparison; A system including:

2. The system of claim 1 further comprising a history management function that provides warnings based on a past intake history of allergens.

3. The system according to claim 1, further comprising a function for detecting edible, poisonous, or harmful substances in the event of a disaster.

Images