System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring an image of a food product based on an operation of a user; means for transmitting the image to a server via the Internet; means for determining freshness of the food product using an image analysis technique in the server; and means for providing a result of the image analysis to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When consumers purchase fresh food at supermarkets or greengrocers, they often face the challenge of accurately determining the freshness of the food. It is particularly difficult to determine freshness by visual inspection alone for unpackaged fresh food, and the quality may not meet expectations after purchase. There is a need to address this issue and ensure that fresh food can be purchased. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a means for acquiring food images based on user operation and a means for transmitting the acquired images to a server via the Internet. Furthermore, a system is provided that includes a means for determining the freshness of the food using image analysis technology on the server and a means for providing the results of the image analysis to the user. This allows users to obtain freshness assessment results using AI from captured images, enabling more appropriate food selection. Furthermore, by using specific feature quantities to determine freshness, highly accurate analysis results can be obtained. A system is provided that can comprehensively evaluate the freshness of food by analyzing color, surface condition, and shape using a neural network.

[0006] "User" refers to any individual or entity that uses the system to determine the freshness of food.

[0007] "Operation" refers to the actions a user takes using the device to take a picture of the food and check the results.

[0008] "Food" refers to food in general, including fresh food in particular that consumers consider purchasing.

[0009] "Image" refers to still image data captured by a camera or other photographic device.

[0010] "Device" refers to an electronic device used by a user, such as a smartphone, tablet, or computer.

[0011] "Internet" refers to the global network infrastructure and means of communication that enables the sending and receiving of data.

[0012] "Server" refers to a computer system for processing data received from users and generating analytical results.

[0013] "Transmit" refers to the act of moving data from one device to another.

[0014] "Image analysis technology" refers to the technology of analyzing image data using artificial intelligence and machine learning to extract specific information.

[0015] "Freshness" refers to a quality indicator that indicates whether food is in good condition.

[0016] "Judgment" refers to the act of drawing a conclusion based on analytical data.

[0017] "Results" refers to the information and data obtained through image analysis technology.

[0018] "Specific features" refer to elements or indicators used to determine the freshness of food.

[0019] A "neural network" is a type of artificial intelligence that refers to a learning algorithm that mimics the neural network of the human brain.

[0020] "Color" refers to the color and tone of food.

[0021] "Surface condition" refers to the appearance of a food product, especially its surface texture and characteristics.

[0022] "Shape" refers to the shape and structure of the food.

[0023] "System" refers to a comprehensive organization consisting of a set of hardware, software, and network infrastructure to provide a specific function. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0025] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0026] First, the terms used in the following description will be explained.

[0027] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0028] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0029] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0030] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0031] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0032] [First embodiment]

[0033] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0034] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0035] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0036] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0037] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0038] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0039] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0040] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0041] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0042] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0043] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0044] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0045] The system according to the present invention provides a series of functions for users to determine the freshness of perishable foods, and specific embodiments thereof will be described below.

[0046] First, the user launches the app on a smartphone or other device and takes a picture of the fresh food. For example, if the user is looking to buy grapes at a supermarket, they can take a picture of the grapes using the app's camera function.

[0047] The captured image is encoded by the device and sent to a server via the Internet, along with the image data, including user identification information and food category information (e.g., fruit or vegetable).

[0048] The server receives the transmitted image data and performs an analysis process. This analysis process incorporates image analysis technology using neural networks. Specifically, the server inputs the image data into a neural network model and analyzes specific features (e.g., color, surface condition, shape, etc.) to determine the freshness of the fresh food.

[0049] The neural network analyzes the color, surface condition, and shape of food and calculates a freshness score based on each feature. For example, in the case of grapes, information such as the color of the skin, the presence or absence of bloom on the surface, the color and firmness of the stalk are analyzed.

[0050] The server sends the resulting freshness score and details to the user's device, where the analysis results are returned in JSON or other data formats.

[0051] The device then displays the analysis results to the user. Specifically, the app displays results such as "These grapes are fresh. Freshness score: 80 / 100," allowing the user to make a purchasing decision based on this information.

[0052] As described above, the system according to the present invention allows users to easily determine the freshness of perishable foods, and allows users to shop at supermarkets and greengrocers with greater peace of mind.

[0053] As a concrete example, consider the case where a user purchases commercially available tomatoes. The user launches a dedicated app and takes a picture of the tomato. The image data is then sent to a server, which uses a neural network to analyze the color and surface condition of the tomato (wrinkles, glossiness, etc.). The analysis result is displayed on the user's device as "This tomato is fresh. Freshness score: 85 / 100." The user can use this information to select fresh tomatoes.

[0054] Thus, the system of the present invention is a powerful tool that helps users select fresh foods in their daily shopping.

[0055] The processing flow will be explained below.

[0056] Step 1:

[0057] The user launches a dedicated app on their device and takes a picture of the fresh food they want to purchase (e.g., grapes).The user then uses the app's camera function to capture a clear image of the food under appropriate lighting conditions.

[0058] Step 2:

[0059] The device encodes the captured image data. Specifically, it compresses the image into JPEG or PNG format to reduce the data size. The encoded image data is temporarily saved.

[0060] Step 3:

[0061] The device packages the encoded image data with user identification information and food category information, and sends the packaged data to the server via the Internet. The image data and accompanying information are sent using an HTTP POST request.

[0062] Step 4:

[0063] The server receives the image data sent from the device. The server analyzes the received data and stores the image data, user identification information, food category information, etc. in the respective variables.

[0064] Step 5:

[0065] The server preprocesses the image data stored on the server, specifically by resizing the image, filtering noise, normalizing it, etc., to convert it into a form suitable for image analysis.

[0066] Step 6:

[0067] The server inputs the preprocessed image data into an AI analysis module, where a neural network analyzes the food's color, surface condition, shape, and other characteristics to extract specific features.

[0068] Step 7:

[0069] The AI ​​analysis module determines the freshness of perishable foods based on the extracted features. For example, in the case of grapes, the color of the skin, the presence or absence of bloom, and the color of the branch are indicators of freshness. These features are combined to calculate a freshness score.

[0070] Step 8:

[0071] The server packages the freshness score and detailed analysis results in JSON format or similar and sends them to the device. The data is returned using an HTTP response.

[0072] Step 9:

[0073] The device decodes the analysis results it receives and displays them on the user interface. Specifically, it displays a message such as "These grapes are fresh. Freshness score: 80 / 100" to provide the user with a visual result.

[0074] Step 10:

[0075] Users can then make a purchasing decision based on the displayed freshness score, or continue analyzing other fresh foods using other features within the app.

[0076] Example 1

[0077] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0078] When purchasing fresh food, determining its freshness is extremely important and presents a major challenge for consumers. However, it can be difficult to accurately judge freshness using visual information alone. This challenge is particularly severe for consumers who lack specialized knowledge or experience in assessing freshness. Currently, there is no reliable method for determining freshness, so there is a need to eliminate the uncertainty consumers have when selecting high-quality fresh food.

[0079] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0080] In this invention, the server includes a means for determining the freshness of the food using image analysis technology, a means for the server to analyze specific features of the food using a neural network model and calculate a freshness score, and a means for transmitting the freshness score to a user's device in JSON format, thereby enabling users to easily determine the freshness of fresh food and make purchasing decisions based on reliable information.

[0081] "User" refers to the consumer who uses the dedicated app to operate the system and determine the freshness of perishables.

[0082] "Terminal" refers to an electronic device used by a user, such as a smartphone or tablet, that acquires images and communicates with a server.

[0083] "Image acquisition means" refers to a function for taking images of fresh food using the camera function of the terminal.

[0084] "Transmission means" refers to a function for transmitting image data and additional information acquired from a terminal to a server via the Internet.

[0085] "Server" refers to a computer system that processes received image data using image analysis techniques to determine freshness.

[0086] "Image analysis technology" refers to technology that uses software algorithms running on a server to extract specific features from images of fresh food and determine its freshness.

[0087] A "neural network model" is a model that uses artificial intelligence technology that runs on a server and is used to analyze food characteristics such as color, surface condition, and shape.

[0088] "Freshness score" refers to a numerical indicator of the freshness of fresh food, calculated based on image analysis technology and neural network models.

[0089] "JSON format" is a type of data format used when sending analysis results and freshness scores to devices, and refers to a lightweight, structured data exchange format.

[0090] "Display means" refers to a function for visually showing the freshness score and analysis results received on the terminal to the user.

[0091] The present invention relates to a system for users to determine the freshness of fresh food. Implementing this system involves a series of processes: acquiring an image of the food based on user operations, transmitting the image to a server via the Internet, and the server using image analysis technology to determine the freshness and providing the analysis results to the user.

[0092] Hardware and software used

[0093] 1. The device used by the user is an electronic device such as a smartphone or tablet. Specific examples include an iPhone 12 or an Android-based device. A dedicated app (e.g., FreshCheck App) is installed on the device, and by launching this app, the user can use the camera function to take a picture of the food.

[0094] 2. The device encodes the captured image in JPEG format, adds user identification information and food category information (e.g., fruit, vegetable), and sends it to the server via HTTPS protocol.

[0095] 3. The server is a high-performance computer system equipped with software that performs image analysis techniques, specifically a neural network model using the TensorFlow library, which analyzes specific features of food, such as color, surface condition, and shape, to calculate a freshness score.

[0096] Example of operation

[0097] While selecting tomatoes to purchase at the supermarket, a user launches a dedicated app and takes a picture of the tomato. The device encodes this image into JPEG format, adds the information "User ID: 67890" and "Food category: Vegetables", and sends it to the server. When the server receives the image data, it inputs the image into a neural network model using TensorFlow, which analyzes the color, surface condition, etc. of the tomato. The freshness score calculated as a result of the analysis (e.g., 85 / 100) is formatted in JSON format and sent to the user's device. The device then displays the received result on the app interface as "This tomato is fresh. Freshness score: 85 / 100." The user can use this information to select fresh tomatoes.

[0098] Prompt Sentence Examples

[0099] An example of a prompt in text format is shown below.

[0100] The user launches the dedicated app "FreshCheck App" on their smartphone and takes a picture of the tomato they plan to purchase. The image data is sent to the server, which uses TensorFlow to analyze the color and surface condition. The analysis result is displayed on the user's device as "This tomato is fresh. Freshness score: 85 / 100." Based on this information, the user can select fresh tomatoes.

[0101] Thus, the system according to the present invention is a powerful tool that helps users to select fresh foods in their daily shopping.

[0102] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0103] Processing Steps

[0104] Step 1:

[0105] The user launches a dedicated app and takes a picture of the fresh food they plan to purchase.

[0106] Specific operation: The user launches the "FreshCheck App" on their smartphone and takes a picture of a tomato using the app's camera function.

[0107] Input: A camera activated by user action, an image of a tomato

[0108] Output: Image data of the photographed tomato

[0109] Step 2:

[0110] The device encodes the captured image data and sends it to a server via the Internet.

[0111] Specific operation: The device encodes the captured image into JPEG format, and then adds user identification information (e.g., UserID: 67890) and food category information (e.g., vegetables).

[0112] Input: Tomato image, user identification information, food category information

[0113] Output: Encoded JPEG image data and additional information

[0114] Step 3:

[0115] The terminal transmits the encoded image data and additional information to the server using the HTTPS protocol.

[0116] Specific operation: The terminal sends the encoded image data and additional information to the server as an HTTPS request.

[0117] Input: JPEG image data and additional information

[0118] Output: Image data and additional information are sent to the server as an HTTPS request.

[0119] Step 4:

[0120] The server performs image analysis using a neural network based on the image data and additional information received.

[0121] Specific operation: The server uses the TensorFlow library to input image data into a neural network model and analyzes features such as the color and surface condition of the tomato.

[0122] Input: Received JPEG image data and additional information

[0123] Output: Analyzed feature data (e.g., RGB values ​​of color, edge information of surface condition)

[0124] Step 5:

[0125] The server calculates a freshness score based on the analysis results and sends it to the user's device in JSON format.

[0126] Specific operation: The server calculates a freshness score (e.g., 85 / 100) based on the analyzed feature data and formats the result in JSON format.

[0127] Input: Analyzed feature data

[0128] Output: JSON format data containing the freshness score.

[0129] Step 6:

[0130] The server sends the parsed results in JSON format to the user's device.

[0131] Specific operation: The server sends the analysis result in JSON format, including the freshness score, to the user's device as an HTTPS response.

[0132] Input: JSON formatted data containing freshness scores

[0133] Output: Analysis results are sent to the device as an HTTPS response

[0134] Step 7:

[0135] The terminal displays the analysis results received to the user.

[0136] Specific operation: The device reads the JSON format analysis results received from the server and displays "This tomato is fresh. Freshness score: 85 / 100" on the app interface.

[0137] Input: Parsed result of received JSON format

[0138] Output: Freshness score and message displayed in the app

[0139] This flow of steps allows users to easily determine the freshness of perishable foods and make purchasing decisions based on reliable information.

[0140] (Application example 1)

[0141] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0142] Existing systems lack the means to not only determine the freshness of fresh food, but also to confirm its safety and quality. Furthermore, they do not meet the needs of users who want to detect counterfeit food and illegally distributed products. To solve this problem, a system is needed that can not only determine the freshness and quality of food, but also verify its authenticity using security tags.

[0143] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0144] In this invention, the server includes means for using image analysis technology to determine the freshness and quality of the item, means for verifying the security tag, and means for providing the results of the analysis and verification to the user, thereby enabling the user to verify not only the freshness and quality of the food they are purchasing, but also whether the food is authentic.

[0145] "Item" refers to a specific product or item that is the subject of analysis using a photograph or image.

[0146] "Freshness" is an index that indicates the degree of freshness that an item maintains and the quality of its preservation.

[0147] "Quality" refers to the characteristics and condition that make up the overall evaluation of a product, including its value and safety to the consumer.

[0148] "Image analysis technology" is a technology for analyzing the characteristics of an item based on a photograph or image to determine its freshness and quality.

[0149] A "security tag" is identification information, such as a QR code or barcode, attached to an item to prove its authenticity.

[0150] "User" means any individual or entity that uses the system to verify the freshness, quality, and authenticity of goods.

[0151] "Via the Internet" refers to a method of using public lines to send and receive data between a user's device and a server.

[0152] A "server" is a computer system that receives data sent from a user, analyzes it, and returns the results.

[0153] "Transmitting" refers to the act of sending images and related information from a user's device to a server via telecommunications means.

[0154] "Providing" refers to the act of presenting the results analyzed and confirmed by the server in the form of a display on the user's device.

[0155] System Program

[0156] The system program for this application example begins with a user taking an image of an item using a device such as a smartphone, smart glasses, or head-mounted display and sending it to a server via the Internet. The server analyzes the received image data, determines its freshness and quality, and then checks the security tag based on that information. The results of the analysis and check are then provided back to the user's device, and the user can use that information to determine the item's freshness and authenticity.

[0157] Natural language explanation of the process

[0158] This system is realized using the following hardware and software.

[0159] The hardware used includes devices such as smartphones, smart glasses, and head-mounted displays, and the software used is Python, OpenCV, Keras (a library for neural networks), and requests (a library for HTTP communication).

[0160] 1. Image capture: The user takes an image of an object using the device's camera, which can be done through an interface such as a smartphone, smart glasses, or a head-mounted display.

[0161] 2. Data transmission: The acquired image data is encoded by the device and sent to the server using the HTTP protocol. At this time, user identification information, item category information, and other information are sent along with the image data.

[0162] 3. Image analysis: The server inputs the received image data into a neural network model to analyze specific features (e.g., color, surface condition, shape, etc.) to determine freshness and quality. This uses a neural network built using Python and Keras.

[0163] 4. Security Verification: The server detects and analyzes the item's security tag (e.g., QR code, barcode, etc.) to verify that the item is authentic. In this step, an HTTP request is made to an external security tag verification service.

[0164] 5. Providing results: The analysis and verification results are returned to the user's device in a data format such as JSON. The analysis results are then visually displayed on the user's device. For example, "This item is fresh. Freshness score: 80 / 100. This item is authentic."

[0165] Specific examples

[0166] For example, if a user is looking to buy grapes at a supermarket, they launch a dedicated app and take a picture of the grapes. The captured image is sent to a server, where a neural network is used to analyze the grapes' color, surface condition, and branch condition to calculate a freshness score. At the same time, the QR code attached to the grapes' packaging is scanned to confirm their authenticity. The result is a message displayed on the smartphone saying, "These grapes are fresh. Freshness score: 85 / 100. These grapes are authentic."

[0167] Example prompts to be input to the generative AI model

[0168] "Analyze the features of fresh produce, such as color, surface condition, and shape, to calculate a freshness score. Also, use security tags to verify whether the food is authentic."

[0169] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0170] Step 1:

[0171] Image acquisition

[0172] input:

[0173] The user activates the camera on their smart device and takes a picture of an item (e.g., grapes).

[0174] Specific behavior:

[0175] The user activates the camera function on a smartphone or smart glasses interface and takes a picture of the item, which generates a high-resolution image of the item.

[0176] output:

[0177] Acquired item image data.

[0178] Step 2:

[0179] Sending data

[0180] input:

[0181] The item image data, user identification information, and item category information obtained in step 1.

[0182] Specific behavior:

[0183] The terminal transmits the acquired image data, user identification information (e.g., user ID, store ID), and item category information (e.g., fruit, vegetable) to the server as an HTTP request.

[0184] output:

[0185] Image data and related information sent to the server.

[0186] Step 3:

[0187] Image analysis

[0188] input:

[0189] The product image data and related information are sent to the server.

[0190] Specific behavior:

[0191] The server inputs the received image data into a neural network model built using Python and Keras to analyze features such as color, surface condition, and shape. For example, in the case of grapes, the skin color and surface gloss are analyzed. The neural network evaluates these features using a trained generative AI model and calculates a freshness score.

[0192] output:

[0193] Parsed freshness score and other feature data.

[0194] Step 4:

[0195] Check the security tag

[0196] input:

[0197] Item image data and related information obtained after analysis.

[0198] Specific behavior:

[0199] The server detects the security tag attached to the item (e.g., QR code or barcode) from the image and uses it to query an external security tag verification service. For example, it scans the QR code, extracts its content, and checks whether the item is authentic based on that content.

[0200] output:

[0201] Security tag verification results (genuine / non-genuine).

[0202] Step 5:

[0203] Providing analysis and verification results

[0204] input:

[0205] Freshness score and security tag validation results.

[0206] Specific behavior:

[0207] The server compiles the analysis results and security tag verification results in a data format such as JSON and returns them to the user's device. At that time, to make the results easier to understand, it generates a specific message such as "This item is fresh. Freshness score: 85 / 100. This item is authentic."

[0208] output:

[0209] Analysis and verification results are displayed on the user's device.

[0210] Step 6:

[0211] Displaying the results

[0212] input:

[0213] Analysis and verification result data returned from the server.

[0214] Specific behavior:

[0215] The device analyzes the received data and visually displays it on the user interface. For example, a message such as "These grapes are fresh. Freshness score: 85 / 100. These grapes are authentic" could be displayed on the smartphone screen, making the results easy for the user to understand.

[0216] output:

[0217] User perception of the results and purchasing decisions based on them.

[0218] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0219] The system according to the present invention has the function of not only determining the freshness of perishable foods but also recognizing the user's emotions and providing information based on those emotions. Specific embodiments of the system will be described below.

[0220] First, a user launches a dedicated smartphone app and takes a picture of the fresh food. For example, if a user is looking to buy tomatoes at a supermarket, they can use the app's camera to take a picture of the tomato. The captured image data is encoded by the device and sent to a server via the Internet.

[0221] The server receives the image data sent from the device and performs preprocessing, which includes image resizing, noise filtering, and normalization. The processed image is then input into a neural network to extract features such as the food's color, surface condition, and shape, and calculate a freshness score.

[0222] The newly added emotion engine feature uses facial recognition and voice analysis to determine a user's current emotional state. For example, if a user smiles into the smartphone camera or vocally expresses joy, that emotional data is collected.

[0223] The server analyzes the detected freshness score by combining it with the user's emotional data. If the emotional data is positive, it provides the user with a result that is more likely to be recommended. Conversely, if negative emotions are detected, it provides the user with appropriate feedback, such as detailed analysis information or other options.

[0224] For example, if a user is checking the freshness of a tomato and the emotion engine detects that the user is smiling, it will highlight the high freshness score. If the user looks dissatisfied, it will suggest, "Try looking for a fresher tomato."

[0225] The analysis results and emotional feedback are sent to the device and displayed visually within the app, allowing users to make informed purchasing decisions about fresh food.

[0226] Furthermore, the emotion engine can learn a user's emotional patterns based on their purchasing history and use them in future judgments. For example, if past data shows that a user tends to prefer foods of a certain color or shape, the engine can adjust the freshness judgment results based on that tendency and provide more personalized information.

[0227] In this way, the system of the present invention aims to provide users with a more appropriate and satisfying purchasing experience by integrating freshness determination and emotion recognition, thereby improving users' constant satisfaction and providing highly accurate freshness determination.

[0228] The processing flow will be explained below.

[0229] Step 1:

[0230] The user launches a dedicated app on their device and takes a picture of the fresh food (e.g., a tomato) they are considering purchasing. The user then uses the app's camera function to capture a clear image of the food under appropriate lighting.

[0231] Step 2:

[0232] The device encodes the captured image data. Specifically, it compresses the image into JPEG or PNG format to reduce the data size. The encoded image data is temporarily saved.

[0233] Step 3:

[0234] The device packages the encoded image data with user identification information and food category information, and sends the packaged data to the server via the Internet. The image data and accompanying information are sent using an HTTP POST request.

[0235] Step 4:

[0236] The server receives the image data sent from the device. The server analyzes the received data and stores the image data, user identification information, food category information, etc. in the respective variables.

[0237] Step 5:

[0238] The server preprocesses the image data stored on the server, specifically by resizing the image, filtering noise, normalizing it, etc., to convert it into a form suitable for image analysis.

[0239] Step 6:

[0240] The server inputs the preprocessed image data into an AI analysis module, where a neural network analyzes the food's color, surface condition, shape, and other characteristics to extract specific features.

[0241] Step 7:

[0242] The AI ​​analysis module determines the freshness of fresh food based on the extracted features. For example, in the case of tomatoes, the color of the skin and the presence or absence of blemishes or scratches on the surface are indicators of freshness. These features are combined to calculate a freshness score.

[0243] Step 8:

[0244] The device captures the user's face and voice in real time and sends them to the emotion engine, which then uses facial recognition and voice analysis to determine the user's emotional state. Emotional data is generated, such as whether the user is smiling, angry, or sad.

[0245] Step 9:

[0246] The server receives the emotion data from the emotion engine and makes a comprehensive judgment based on the freshness score. If the emotion data is positive, it returns a highly recommended result, and if it is negative, it makes adjustments such as presenting a detailed analysis.

[0247] Step 10:

[0248] The server packages the final analysis results in JSON format or similar and sends them to the terminal, returning the data using an HTTP response.

[0249] Step 11:

[0250] The device decodes the analysis results it receives and displays them on the user interface. Specifically, it displays a message such as "This tomato is fresh. Freshness score: 85 / 100" so that the user can visually confirm the results.

[0251] Step 12:

[0252] Users make purchasing decisions based on the displayed freshness score and their own emotional feedback. Users can also use other features within the app to continue analyzing other fresh foods.

[0253] Step 13:

[0254] The emotion engine learns the user's purchasing history and emotional tendencies, and reflects this in future analyses. Based on past data, a more personalized freshness assessment is made.

[0255] Example 2

[0256] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0257] In recent years, technology for evaluating the quality and freshness of fresh food has advanced, but systems that provide feedback that takes user emotions into account are still limited. Furthermore, conventional systems are limited to determining freshness through image analysis, making it difficult to make personalized recommendations based on the user's emotions and preferences. This makes it difficult to improve the user experience and support satisfying purchasing decisions.

[0258] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0259] In this invention, the server includes means for determining the freshness of the food using image analysis technology, means for performing face recognition and voice analysis to recognize the user's emotions, and means for performing analysis based on the determination results and emotion data and providing information to the user. This enables analysis that integrates freshness determination and user emotions, making it possible to provide a more appropriate and satisfying purchasing experience.

[0260] "User" refers to any individual or legal entity that uses the System.

[0261] An "operation" refers to an action taken by a user to achieve a specific goal using the system.

[0262] "Image" refers to visual data captured by a user using a device's camera or the like.

[0263] "Internet" refers to the technology that connects computers and databases around the world via information and communications networks.

[0264] "Server" refers to a set of hardware or software that receives requests and processes data.

[0265] "Image analysis technology" refers to software and algorithms for extracting specific information from image data.

[0266] "Freshness" refers to an indication of the appropriate quality and state of preservation of an item such as food.

[0267] "Facial recognition" refers to the technology that detects and recognizes the faces of people captured on camera.

[0268] "Voice analysis" refers to the technology of analyzing voice data and extracting information.

[0269] "Determination result" refers to a conclusion reached using image analysis techniques or other analytical methods.

[0270] "Emotional data" refers to the results of collecting and analyzing data that indicates the emotional state of a user.

[0271] "Analysis" refers to the process of deriving information and patterns from collected data.

[0272] "Providing information" refers to the act of communicating analysis results and recommendations to users.

[0273] "Visually displaying" refers to the act of graphically showing analytical results or information on a computer screen or device display.

[0274] "Goods" refers to a broad definition of the subject matter, including food and other subject items.

[0275] "Feedback" refers to the analysis results and recommendations provided to the user.

[0276] A "machine learning model" refers to an algorithm that learns patterns from data and makes predictions or classifications for new data.

[0277] The system of the present invention not only determines the freshness of perishable food, but also has the ability to recognize the user's emotions and provide appropriate feedback based on them.

[0278] Hardware and Software Configuration

[0279] The implementation of this system utilizes the following hardware and software:

[0280] Hardware:

[0281] Smartphone or tablet: Use the camera function to take a picture.

[0282] Server: Receives data, analyzes it, and generates feedback.

[0283] software:

[0284] Dedicated app: An application that provides an interface for users to take images and input emotions.

[0285] Image analysis software: Use libraries such as OpenCV to preprocess images.

[0286] Machine learning model: We use TensorFlow and PyTorch to build a neural network to extract food features and calculate a freshness score.

[0287] Emotion Recognition Software: Analyze user emotions using Emotion API or our own emotion recognition model.

[0288] Specific functions of the system

[0289] 1. Image capture and transmission:

[0290] Users launch a dedicated smartphone app and take a picture of the fresh food. The captured image data is encoded by the device and sent to a server via the Internet.

[0291] 2. Image preprocessing:

[0292] The server receives image data sent from the device and performs preprocessing such as resizing, noise filtering, and normalization, using an image processing library such as OpenCV.

[0293] 3. Feature extraction and freshness determination:

[0294] The preprocessed images are input into a neural network to extract features such as food color, surface condition, and shape, using a machine learning model (for example, TensorFlow or PyTorch).

[0295] 4. Emotion recognition:

[0296] When a user smiles at the camera or expresses emotion through voice, the device sends the data to the server, which then performs facial recognition and voice analysis to determine the user's emotional state, using the Emotion API or a proprietary emotion recognition model.

[0297] 5. Data Integration and Analysis:

[0298] The server integrates the freshness score and the user's emotional data and performs an analysis based on the results. If the emotional data is positive, the server provides positive feedback, and if it is negative, the server presents detailed analysis information or other options to the user.

[0299] 6. Providing Feedback:

[0300] The analysis results and emotional feedback are sent to the device and displayed visually within a dedicated app, allowing users to make purchasing decisions about fresh food based on this information.

[0301] 7. Learning Emotional Patterns:

[0302] The emotion engine learns the user's emotional patterns based on their purchasing behavior history and uses them to make decisions for future purchases. It also extracts the user's preferences from past data and provides personalized information.

[0303] Specific examples

[0304] For example, if a user is checking the freshness of tomatoes at a supermarket and the emotion engine detects that the user is smiling, it will highlight the high freshness score. If the user looks dissatisfied, it will suggest, "Try looking for a fresher tomato."

[0305] Example prompts for generative AI models

[0306] "I want to determine the freshness of a particular food item from an image. Can you please tell me some concrete ways to build a system that recognizes the user's emotions and provides feedback based on their reaction?"

[0307] By integrating freshness assessment and emotion recognition, this system aims to provide users with a more relevant and satisfying purchasing experience.

[0308] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0309] Step 1:

[0310] The user launches the app and takes a picture

[0311] Input: Fresh produce the user wants to buy (e.g. tomatoes)

[0312] Output: Image data taken with a smartphone

[0313] How it works: The user launches a dedicated app on their smartphone and takes a picture of the fresh food. The app uses the camera function to capture a high-resolution image.

[0314] Step 2:

[0315] The device encodes and transmits the image data.

[0316] Input: Photographed image data

[0317] Output: Encoded image data (e.g. JPEG, PNG format)

[0318] What it does: The device encodes the captured image and sends it over the Internet to a server, using the HTTPS protocol for secure data transfer.

[0319] Step 3:

[0320] The server receives the image data and performs preprocessing.

[0321] Input: Encoded image data sent from the device

[0322] Output: Preprocessed image data

[0323] Specific operation: The server performs preprocessing on the received image data, such as resizing (e.g., changing to 224x224 pixels), noise filtering, and normalization. This processing uses an image processing library such as OpenCV.

[0324] Step 4:

[0325] The server extracts features using a neural network

[0326] Input: Preprocessed image data

[0327] Output: Extracted features

[0328] Specific operation: The server inputs the preprocessed images into a neural network (e.g., using TensorFlow or PyTorch) to extract features such as the color, surface condition, and shape of the food.

[0329] Step 5:

[0330] The server calculates a freshness score based on the features.

[0331] Input: extracted features

[0332] Output: Freshness score

[0333] How it works: The server uses the extracted features to calculate the freshness score of the fresh produce, using a pre-trained model.

[0334] Step 6:

[0335] The user expresses their emotions through the camera and microphone

[0336] Input: User's facial expressions and voice

[0337] Output: Facial image data and audio data

[0338] Specific actions: The user smiles into the smartphone camera or expresses an emotion through voice.

[0339] Step 7:

[0340] The device sends the facial image data and voice data to the server.

[0341] Input: Facial image data and audio data

[0342] Output: Facial image data and audio data sent from the device

[0343] Specific operation: The device encodes the collected facial image data and voice data and sends them to the server. The data is transferred securely using the HTTPS protocol.

[0344] Step 8:

[0345] The server performs emotion recognition

[0346] Input: Facial image data and audio data

[0347] Output: Emotion data

[0348] How it works: The server performs facial recognition and voice analysis to determine the user's current emotional state, using an emotion recognition model (e.g., Emotion API or a proprietary emotion recognition model).

[0349] Step 9:

[0350] The server combines the freshness score and sentiment data for analysis.

[0351] Input: Freshness score and sentiment data

[0352] Output: Analysis results and feedback

[0353] How it works: The server combines the freshness score with the user's sentiment data and generates feedback based on the results. If the sentiment is positive, it provides positive feedback, and if it is negative, it provides detailed analysis information or other options.

[0354] Step 10:

[0355] The server sends the analysis results to the device.

[0356] Input: Analysis results and feedback

[0357] Output: Analysis results and feedback to the device

[0358] Specific operation: The server sends the analysis results and feedback to a dedicated app and then to the device for visual display.

[0359] Step 11:

[0360] The terminal displays the results visually.

[0361] Input: Analysis results and feedback sent from the server

[0362] Output: Analysis results and feedback displayed visually within a dedicated app

[0363] Specific operation: The device visually displays the analysis results and feedback within a dedicated app so that the user can check them.

[0364] Step 12:

[0365] The server learns emotional patterns

[0366] Input: User's past purchasing behavior history and emotional data

[0367] Output: Learned emotion patterns

[0368] Specific operation: The server uses an emotion engine to learn the user's emotional patterns based on their purchasing behavior history and utilizes them for future judgments. It also extracts the user's preferences from past data and optimizes freshness judgment and feedback based on them.

[0369] (Application example 2)

[0370] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0371] Conventional freshness assessment systems for fresh food simply assess freshness based on image data of the food, without providing feedback that takes into account the user's emotions or purchasing experience. As a result, even if a user confirms the freshness of the food, their satisfaction with the purchase may be low. Furthermore, because they are unable to provide detailed responses based on the user's emotions, it is difficult to influence the user's purchasing behavior.

[0372] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0373] In this invention, the server includes means for acquiring images of fresh food based on user operations, means for transmitting the images to the server via the Internet, means for determining the freshness of the fresh food using image analysis technology in the server, means for providing the results of the image analysis to the user, means for performing facial recognition and voice analysis to recognize the user's emotions, and means for adjusting the results of the image analysis based on the user's emotional state. This makes it possible to provide appropriate feedback that takes the user's emotions into consideration, thereby improving the user's purchasing experience and satisfaction.

[0374] "User operations" refers to a series of actions a user takes to use a system or application.

[0375] "Fresh food" refers to food for which freshness and quality are important and which has a limited shelf life.

[0376] "Means for acquiring an image" refers to a method for acquiring visual information as a digital image using a device such as a camera or scanner.

[0377] "Transmission via the Internet" refers to the technology that utilizes Internet protocols to transmit data to a remote server.

[0378] A "server" refers to a computer system that processes various data and provides services via a network.

[0379] "Image analysis technology" refers to methods and algorithms for extracting useful information from digital image data.

[0380] "Means for determining freshness" refers to methods for evaluating the freshness of fresh food based on various indicators and analytical results.

[0381] "Means of providing" refers to the method of notifying or displaying the analysis results or information to the user.

[0382] "Emotion recognition" refers to analyzing a person's facial expressions and voice data to determine their emotional state.

[0383] "Facial recognition" refers to the technology of identifying facial features in digital images or video streams to recognize and identify individuals.

[0384] "Voice analysis" refers to the technology of analyzing voice data and extracting information such as content, emotions, and speaker identification.

[0385] "Emotional state" refers to the user's mental and emotional state as indicated by the analyzed results.

[0386] "Adjustment means" refers to a method for changing or adjusting the analysis results or provided information according to the emotional state of the user.

[0387] "Feedback" refers to the reaction or information that a system provides to a user.

[0388] "Purchasing experience" refers to the series of experiences and emotions a user feels during the process of purchasing a product.

[0389] The system of the present invention aims to provide a more satisfying shopping experience by combining freshness assessment of fresh food in a physical store with user emotion recognition. This system is implemented using smart glasses.

[0390] First, a user wears smart glasses and makes a purchase in a physical store. The user picks up a fresh food item they are considering purchasing and takes a picture of it with the smart glasses' camera. The captured image is then captured as digital data by the glasses' built-in camera system. This captured image data is temporarily processed by a processor in the smart glasses and sent to a remote server via the Internet.

[0391] The server preprocesses the received image data and uses image analysis technology to determine the freshness of the fresh food. This preprocessing mainly includes image resizing, noise filtering, and normalization. The preprocessed image data is input into a neural network model to extract features such as the food's color, surface condition, and shape. This allows a freshness score to be calculated.

[0392] Next, a process to recognize the user's emotions is carried out. The camera and microphone attached to the smart glasses capture the user's facial expressions and voice in real time. The server receives this data and uses facial recognition and voice analysis technologies to determine the user's emotional state. Facial recognition reads emotions from the user's facial expressions, while voice analysis analyzes emotions from the user's tone and intonation of voice.

[0393] The server analyzes the freshness score by combining it with the user's emotional data. If the user is in a positive emotional state (e.g., smiling, happy voice), the system highlights the freshness score and provides feedback that increases the recommendation. Conversely, if a negative emotional state (e.g., dissatisfied face, flat voice) is detected, the system provides feedback such as detailed analysis information or other options.

[0394] The analysis results and emotional data feedback are then sent to the smart glasses via the internet and visually displayed on the screen, allowing users to make purchasing decisions about fresh food based on this information.

[0395] As a specific example, a user picks up a tomato in a supermarket and takes a picture of it with the smart glasses' camera. Image analysis determines that the captured image has a high freshness score, and the server feeds that score back to the user. At the same time, if the user smiles, the smart glasses' display will show "This product is fresh! Freshness score: 0.85." On the other hand, if the user looks dissatisfied, the display will show more detailed information, such as "Please try looking for a fresher product. Freshness score: 0.65."

[0396] Examples of prompts using generative AI models for emotion recognition and freshness determination include:

[0397] "A user picks up a tomato in a supermarket and uses smart glasses to check its freshness. If the user smiles, provide feedback highlighting a high freshness score. If the user looks dissatisfied, provide detailed analytics or alternative options."

[0398] As described above, the system of the present invention improves the user's purchasing experience and satisfaction by providing feedback to the user by integrating the freshness determination of perishable foods with the recognition of the user's emotions.

[0399] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0400] Step 1:

[0401] A user wears smart glasses and picks up fresh produce in a brick-and-mortar store. The smart glasses' camera is used to capture an image of the food, where the input is the physical appearance of the fresh produce and the output is digital image data. This digital image data is sent to the system for analysis.

[0402] Step 2:

[0403] The terminal (smart glasses) sends the captured image data to a server via the Internet. The input is the digital image data acquired in step 1, and the output is the image data transferred to the server. A network communication protocol is used for data transfer.

[0404] Step 3:

[0405] The server preprocesses the received image data by resizing, noise filtering, normalizing, and converting it into a format suitable for analysis. The input is the image data of the fresh food sent to the server, and the output is the preprocessed image data.

[0406] Step 4:

[0407] The server inputs the preprocessed image data into a neural network model to extract features such as the color, surface condition, and shape of the food, which then calculates a freshness score. The input is the preprocessed image data, and the output is the freshness score.

[0408] Step 5:

[0409] The device (smart glasses) captures the user's facial expressions and voice in real time. It uses the camera and microphone of the smart glasses to obtain facial image data and voice data. The input is the user's facial expressions and voice, and the output is facial image data and voice data.

[0410] Step 6:

[0411] The server analyzes the received facial image data and voice data to recognize the user's emotional state. Using facial recognition and voice analysis technologies, it determines whether the user's emotion is positive or negative. The input is facial image data and voice data, and the output is the user's emotional state.

[0412] Step 7:

[0413] The server performs an analysis by combining the calculated freshness score with the user's emotional state. If the emotional state is positive, it highlights the freshness score and generates feedback that increases the recommendation level. Conversely, if the emotional state is negative, it generates feedback that presents detailed analysis information and other options. The input is the freshness score and the emotional state, and the output is feedback information.

[0414] Step 8:

[0415] The server sends the generated feedback information to the terminal (smart glasses) via the Internet. The input is the feedback information, and the output is the feedback information displayed on the display of the smart glasses.

[0416] Step 9:

[0417] The user makes a purchasing decision on fresh food based on the feedback information displayed on the smart glasses display. The input is the feedback information, and the output is the purchase decision on fresh food.

[0418] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0419] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0420] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0421] [Second embodiment]

[0422] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0423] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0424] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0425] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0426] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0427] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0428] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0429] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0430] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0431] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0432] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0433] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0434] The system according to the present invention provides a series of functions for users to determine the freshness of perishable foods, and specific embodiments thereof will be described below.

[0435] First, the user launches the app on a smartphone or other device and takes a picture of the fresh food. For example, if the user is looking to buy grapes at a supermarket, they can take a picture of the grapes using the app's camera function.

[0436] The captured image is encoded by the device and sent to a server via the Internet, along with the image data, including user identification information and food category information (e.g., fruit or vegetable).

[0437] The server receives the transmitted image data and performs an analysis process. This analysis process incorporates image analysis technology using neural networks. Specifically, the server inputs the image data into a neural network model and analyzes specific features (e.g., color, surface condition, shape, etc.) to determine the freshness of the fresh food.

[0438] The neural network analyzes the color, surface condition, and shape of food and calculates a freshness score based on each feature. For example, in the case of grapes, information such as the color of the skin, the presence or absence of bloom on the surface, the color and firmness of the stalk are analyzed.

[0439] The server sends the resulting freshness score and details to the user's device, where the analysis results are returned in JSON or other data formats.

[0440] The device then displays the analysis results to the user. Specifically, the app displays results such as "These grapes are fresh. Freshness score: 80 / 100," allowing the user to make a purchasing decision based on this information.

[0441] As described above, the system according to the present invention allows users to easily determine the freshness of perishable foods, and allows users to shop at supermarkets and greengrocers with greater peace of mind.

[0442] As a concrete example, consider the case where a user purchases commercially available tomatoes. The user launches a dedicated app and takes a picture of the tomato. The image data is then sent to a server, which uses a neural network to analyze the color and surface condition of the tomato (wrinkles, glossiness, etc.). The analysis result is displayed on the user's device as "This tomato is fresh. Freshness score: 85 / 100." The user can use this information to select fresh tomatoes.

[0443] Thus, the system of the present invention is a powerful tool that helps users select fresh foods in their daily shopping.

[0444] The processing flow will be explained below.

[0445] Step 1:

[0446] The user launches a dedicated app on their device and takes a picture of the fresh food they want to purchase (e.g., grapes).The user then uses the app's camera function to capture a clear image of the food under appropriate lighting conditions.

[0447] Step 2:

[0448] The device encodes the captured image data. Specifically, it compresses the image into JPEG or PNG format to reduce the data size. The encoded image data is temporarily saved.

[0449] Step 3:

[0450] The device packages the encoded image data with user identification information and food category information, and sends the packaged data to the server via the Internet. The image data and accompanying information are sent using an HTTP POST request.

[0451] Step 4:

[0452] The server receives the image data sent from the device. The server analyzes the received data and stores the image data, user identification information, food category information, etc. in the respective variables.

[0453] Step 5:

[0454] The server preprocesses the image data stored on the server, specifically by resizing the image, filtering noise, normalizing it, etc., to convert it into a form suitable for image analysis.

[0455] Step 6:

[0456] The server inputs the preprocessed image data into an AI analysis module, where a neural network analyzes the food's color, surface condition, shape, and other characteristics to extract specific features.

[0457] Step 7:

[0458] The AI ​​analysis module determines the freshness of perishable foods based on the extracted features. For example, in the case of grapes, the color of the skin, the presence or absence of bloom, and the color of the branch are indicators of freshness. These features are combined to calculate a freshness score.

[0459] Step 8:

[0460] The server packages the freshness score and detailed analysis results in JSON format or similar and sends them to the device. The data is returned using an HTTP response.

[0461] Step 9:

[0462] The device decodes the analysis results it receives and displays them on the user interface. Specifically, it displays a message such as "These grapes are fresh. Freshness score: 80 / 100" to provide the user with a visual result.

[0463] Step 10:

[0464] Users can then make a purchasing decision based on the displayed freshness score, or continue analyzing other fresh foods using other features within the app.

[0465] Example 1

[0466] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0467] When purchasing fresh food, determining its freshness is extremely important and presents a major challenge for consumers. However, it can be difficult to accurately judge freshness using visual information alone. This challenge is particularly severe for consumers who lack specialized knowledge or experience in assessing freshness. Currently, there is no reliable method for determining freshness, so there is a need to eliminate the uncertainty consumers have when selecting high-quality fresh food.

[0468] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0469] In this invention, the server includes a means for determining the freshness of the food using image analysis technology, a means for the server to analyze specific features of the food using a neural network model and calculate a freshness score, and a means for transmitting the freshness score to a user's device in JSON format, thereby enabling users to easily determine the freshness of fresh food and make purchasing decisions based on reliable information.

[0470] "User" refers to the consumer who uses the dedicated app to operate the system and determine the freshness of perishables.

[0471] "Terminal" refers to an electronic device used by a user, such as a smartphone or tablet, that acquires images and communicates with a server.

[0472] "Image acquisition means" refers to a function for taking images of fresh food using the camera function of the terminal.

[0473] "Transmission means" refers to a function for transmitting image data and additional information acquired from a terminal to a server via the Internet.

[0474] "Server" refers to a computer system that processes received image data using image analysis techniques to determine freshness.

[0475] "Image analysis technology" refers to technology that uses software algorithms running on a server to extract specific features from images of fresh food and determine its freshness.

[0476] A "neural network model" is a model that uses artificial intelligence technology that runs on a server and is used to analyze food characteristics such as color, surface condition, and shape.

[0477] "Freshness score" refers to a numerical indicator of the freshness of fresh food, calculated based on image analysis technology and neural network models.

[0478] "JSON format" is a type of data format used when sending analysis results and freshness scores to devices, and refers to a lightweight, structured data exchange format.

[0479] "Display means" refers to a function for visually showing the freshness score and analysis results received on the terminal to the user.

[0480] The present invention relates to a system for users to determine the freshness of fresh food. Implementing this system involves a series of processes: acquiring an image of the food based on user operations, transmitting the image to a server via the Internet, and the server using image analysis technology to determine the freshness and providing the analysis results to the user.

[0481] Hardware and software used

[0482] 1. The device used by the user is an electronic device such as a smartphone or tablet. Specific examples include an iPhone 12 or an Android-based device. A dedicated app (e.g., FreshCheck App) is installed on the device, and by launching this app, the user can use the camera function to take a picture of the food.

[0483] 2. The device encodes the captured image in JPEG format, adds user identification information and food category information (e.g., fruit, vegetable), and sends it to the server via HTTPS protocol.

[0484] 3. The server is a high-performance computer system equipped with software that performs image analysis techniques, specifically a neural network model using the TensorFlow library, which analyzes specific features of food, such as color, surface condition, and shape, to calculate a freshness score.

[0485] Example of operation

[0486] While selecting tomatoes to purchase at the supermarket, a user launches a dedicated app and takes a picture of the tomato. The device encodes this image into JPEG format, adds the information "User ID: 67890" and "Food category: Vegetables", and sends it to the server. When the server receives the image data, it inputs the image into a neural network model using TensorFlow, which analyzes the color, surface condition, etc. of the tomato. The freshness score calculated as a result of the analysis (e.g., 85 / 100) is formatted in JSON format and sent to the user's device. The device then displays the received result on the app interface as "This tomato is fresh. Freshness score: 85 / 100." The user can use this information to select fresh tomatoes.

[0487] Prompt Sentence Examples

[0488] An example of a prompt in text format is shown below.

[0489] The user launches the dedicated app "FreshCheck App" on their smartphone and takes a picture of the tomato they plan to purchase. The image data is sent to the server, which uses TensorFlow to analyze the color and surface condition. The analysis result is displayed on the user's device as "This tomato is fresh. Freshness score: 85 / 100." Based on this information, the user can select fresh tomatoes.

[0490] Thus, the system according to the present invention is a powerful tool that helps users to select fresh foods in their daily shopping.

[0491] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0492] Processing Steps

[0493] Step 1:

[0494] The user launches a dedicated app and takes a picture of the fresh food they plan to purchase.

[0495] Specific operation: The user launches the "FreshCheck App" on their smartphone and takes a picture of a tomato using the app's camera function.

[0496] Input: A camera activated by user action, an image of a tomato

[0497] Output: Image data of the photographed tomato

[0498] Step 2:

[0499] The device encodes the captured image data and sends it to a server via the Internet.

[0500] Specific operation: The device encodes the captured image into JPEG format, and then adds user identification information (e.g., UserID: 67890) and food category information (e.g., vegetables).

[0501] Input: Tomato image, user identification information, food category information

[0502] Output: Encoded JPEG image data and additional information

[0503] Step 3:

[0504] The terminal transmits the encoded image data and additional information to the server using the HTTPS protocol.

[0505] Specific operation: The terminal sends the encoded image data and additional information to the server as an HTTPS request.

[0506] Input: JPEG image data and additional information

[0507] Output: Image data and additional information are sent to the server as an HTTPS request.

[0508] Step 4:

[0509] The server performs image analysis using a neural network based on the image data and additional information received.

[0510] Specific operation: The server uses the TensorFlow library to input image data into a neural network model and analyzes features such as the color and surface condition of the tomato.

[0511] Input: Received JPEG image data and additional information

[0512] Output: Analyzed feature data (e.g., RGB values ​​of color, edge information of surface condition)

[0513] Step 5:

[0514] The server calculates a freshness score based on the analysis results and sends it to the user's device in JSON format.

[0515] Specific operation: The server calculates a freshness score (e.g., 85 / 100) based on the analyzed feature data and formats the result in JSON format.

[0516] Input: Analyzed feature data

[0517] Output: JSON format data containing the freshness score.

[0518] Step 6:

[0519] The server sends the parsed results in JSON format to the user's device.

[0520] Specific operation: The server sends the analysis result in JSON format, including the freshness score, to the user's device as an HTTPS response.

[0521] Input: JSON formatted data containing freshness scores

[0522] Output: Analysis results are sent to the device as an HTTPS response

[0523] Step 7:

[0524] The terminal displays the analysis results received to the user.

[0525] Specific operation: The device reads the JSON format analysis results received from the server and displays "This tomato is fresh. Freshness score: 85 / 100" on the app interface.

[0526] Input: Parsed result of received JSON format

[0527] Output: Freshness score and message displayed in the app

[0528] This flow of steps allows users to easily determine the freshness of perishable foods and make purchasing decisions based on reliable information.

[0529] (Application example 1)

[0530] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0531] Existing systems lack the means to not only determine the freshness of fresh food, but also to confirm its safety and quality. Furthermore, they do not meet the needs of users who want to detect counterfeit food and illegally distributed products. To solve this problem, a system is needed that can not only determine the freshness and quality of food, but also verify its authenticity using security tags.

[0532] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0533] In this invention, the server includes means for using image analysis technology to determine the freshness and quality of the item, means for verifying the security tag, and means for providing the results of the analysis and verification to the user, thereby enabling the user to verify not only the freshness and quality of the food they are purchasing, but also whether the food is authentic.

[0534] "Item" refers to a specific product or item that is the subject of analysis using a photograph or image.

[0535] "Freshness" is an index that indicates the degree of freshness that an item maintains and the quality of its preservation.

[0536] "Quality" refers to the characteristics and condition that make up the overall evaluation of a product, including its value and safety to the consumer.

[0537] "Image analysis technology" is a technology for analyzing the characteristics of an item based on a photograph or image to determine its freshness and quality.

[0538] A "security tag" is identification information, such as a QR code or barcode, attached to an item to prove its authenticity.

[0539] "User" means any individual or entity that uses the system to verify the freshness, quality, and authenticity of goods.

[0540] "Via the Internet" refers to a method of using public lines to send and receive data between a user's device and a server.

[0541] A "server" is a computer system that receives data sent from a user, analyzes it, and returns the results.

[0542] "Transmitting" refers to the act of sending images and related information from a user's device to a server via telecommunications means.

[0543] "Providing" refers to the act of presenting the results analyzed and confirmed by the server in the form of a display on the user's device.

[0544] System Program

[0545] The system program for this application example begins with a user taking an image of an item using a device such as a smartphone, smart glasses, or head-mounted display and sending it to a server via the Internet. The server analyzes the received image data, determines its freshness and quality, and then checks the security tag based on that information. The results of the analysis and check are then provided back to the user's device, and the user can use that information to determine the item's freshness and authenticity.

[0546] Natural language explanation of the process

[0547] This system is realized using the following hardware and software.

[0548] The hardware used includes devices such as smartphones, smart glasses, and head-mounted displays, and the software used is Python, OpenCV, Keras (a library for neural networks), and requests (a library for HTTP communication).

[0549] 1. Image capture: The user takes an image of an object using the device's camera, which can be done through an interface such as a smartphone, smart glasses, or a head-mounted display.

[0550] 2. Data transmission: The acquired image data is encoded by the device and sent to the server using the HTTP protocol. At this time, user identification information, item category information, and other information are sent along with the image data.

[0551] 3. Image analysis: The server inputs the received image data into a neural network model to analyze specific features (e.g., color, surface condition, shape, etc.) to determine freshness and quality. This uses a neural network built using Python and Keras.

[0552] 4. Security Verification: The server detects and analyzes the item's security tag (e.g., QR code, barcode, etc.) to verify that the item is authentic. In this step, an HTTP request is made to an external security tag verification service.

[0553] 5. Providing results: The analysis and verification results are returned to the user's device in a data format such as JSON. The analysis results are then visually displayed on the user's device. For example, "This item is fresh. Freshness score: 80 / 100. This item is authentic."

[0554] Specific examples

[0555] For example, if a user is looking to buy grapes at a supermarket, they launch a dedicated app and take a picture of the grapes. The captured image is sent to a server, where a neural network is used to analyze the grapes' color, surface condition, and branch condition to calculate a freshness score. At the same time, the QR code attached to the grapes' packaging is scanned to confirm their authenticity. The result is a message displayed on the smartphone saying, "These grapes are fresh. Freshness score: 85 / 100. These grapes are authentic."

[0556] Example prompts to be input to the generative AI model

[0557] "Analyze the features of fresh produce, such as color, surface condition, and shape, to calculate a freshness score. Also, use security tags to verify whether the food is authentic."

[0558] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0559] Step 1:

[0560] Image acquisition

[0561] input:

[0562] The user activates the camera on their smart device and takes a picture of an item (e.g., grapes).

[0563] Specific behavior:

[0564] The user activates the camera function on a smartphone or smart glasses interface and takes a picture of the item, which generates a high-resolution image of the item.

[0565] output:

[0566] Acquired item image data.

[0567] Step 2:

[0568] Sending data

[0569] input:

[0570] The item image data, user identification information, and item category information obtained in step 1.

[0571] Specific behavior:

[0572] The terminal transmits the acquired image data, user identification information (e.g., user ID, store ID), and item category information (e.g., fruit, vegetable) to the server as an HTTP request.

[0573] output:

[0574] Image data and related information sent to the server.

[0575] Step 3:

[0576] Image analysis

[0577] input:

[0578] The product image data and related information are sent to the server.

[0579] Specific behavior:

[0580] The server inputs the received image data into a neural network model built using Python and Keras to analyze features such as color, surface condition, and shape. For example, in the case of grapes, the skin color and surface gloss are analyzed. The neural network evaluates these features using a trained generative AI model and calculates a freshness score.

[0581] output:

[0582] Parsed freshness score and other feature data.

[0583] Step 4:

[0584] Check the security tag

[0585] input:

[0586] Item image data and related information obtained after analysis.

[0587] Specific behavior:

[0588] The server detects the security tag attached to the item (e.g., QR code or barcode) from the image and uses it to query an external security tag verification service. For example, it scans the QR code, extracts its content, and checks whether the item is authentic based on that content.

[0589] output:

[0590] Security tag verification results (genuine / non-genuine).

[0591] Step 5:

[0592] Providing analysis and verification results

[0593] input:

[0594] Freshness score and security tag validation results.

[0595] Specific behavior:

[0596] The server compiles the analysis results and security tag verification results in a data format such as JSON and returns them to the user's device. At that time, to make the results easier to understand, it generates a specific message such as "This item is fresh. Freshness score: 85 / 100. This item is authentic."

[0597] output:

[0598] Analysis and verification results are displayed on the user's device.

[0599] Step 6:

[0600] Displaying the results

[0601] input:

[0602] Analysis and verification result data returned from the server.

[0603] Specific behavior:

[0604] The device analyzes the received data and visually displays it on the user interface. For example, a message such as "These grapes are fresh. Freshness score: 85 / 100. These grapes are authentic" could be displayed on the smartphone screen, making the results easy for the user to understand.

[0605] output:

[0606] User perception of the results and purchasing decisions based on them.

[0607] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0608] The system according to the present invention has the function of not only determining the freshness of perishable foods but also recognizing the user's emotions and providing information based on those emotions. Specific embodiments of the system will be described below.

[0609] First, a user launches a dedicated smartphone app and takes a picture of the fresh food. For example, if a user is looking to buy tomatoes at a supermarket, they can use the app's camera to take a picture of the tomato. The captured image data is encoded by the device and sent to a server via the Internet.

[0610] The server receives the image data sent from the device and performs preprocessing, which includes image resizing, noise filtering, and normalization. The processed image is then input into a neural network to extract features such as the food's color, surface condition, and shape, and calculate a freshness score.

[0611] The newly added emotion engine feature uses facial recognition and voice analysis to determine a user's current emotional state. For example, if a user smiles into the smartphone camera or vocally expresses joy, that emotional data is collected.

[0612] The server analyzes the detected freshness score by combining it with the user's emotional data. If the emotional data is positive, it provides the user with a result that is more likely to be recommended. Conversely, if negative emotions are detected, it provides the user with appropriate feedback, such as detailed analysis information or other options.

[0613] For example, if a user is checking the freshness of a tomato and the emotion engine detects that the user is smiling, it will highlight the high freshness score. If the user looks dissatisfied, it will suggest, "Try looking for a fresher tomato."

[0614] The analysis results and emotional feedback are sent to the device and displayed visually within the app, allowing users to make informed purchasing decisions about fresh food.

[0615] Furthermore, the emotion engine can learn a user's emotional patterns based on their purchasing history and use them in future judgments. For example, if past data shows that a user tends to prefer foods of a certain color or shape, the engine can adjust the freshness judgment results based on that tendency and provide more personalized information.

[0616] In this way, the system of the present invention aims to provide users with a more appropriate and satisfying purchasing experience by integrating freshness determination and emotion recognition, thereby improving users' constant satisfaction and providing highly accurate freshness determination.

[0617] The processing flow will be explained below.

[0618] Step 1:

[0619] The user launches a dedicated app on their device and takes a picture of the fresh food (e.g., a tomato) they are considering purchasing. The user then uses the app's camera function to capture a clear image of the food under appropriate lighting.

[0620] Step 2:

[0621] The device encodes the captured image data. Specifically, it compresses the image into JPEG or PNG format to reduce the data size. The encoded image data is temporarily saved.

[0622] Step 3:

[0623] The device packages the encoded image data with user identification information and food category information, and sends the packaged data to the server via the Internet. The image data and accompanying information are sent using an HTTP POST request.

[0624] Step 4:

[0625] The server receives the image data sent from the device. The server analyzes the received data and stores the image data, user identification information, food category information, etc. in the respective variables.

[0626] Step 5:

[0627] The server preprocesses the image data stored on the server, specifically by resizing the image, filtering noise, normalizing it, etc., to convert it into a form suitable for image analysis.

[0628] Step 6:

[0629] The server inputs the preprocessed image data into an AI analysis module, where a neural network analyzes the food's color, surface condition, shape, and other characteristics to extract specific features.

[0630] Step 7:

[0631] The AI ​​analysis module determines the freshness of fresh food based on the extracted features. For example, in the case of tomatoes, the color of the skin and the presence or absence of blemishes or scratches on the surface are indicators of freshness. These features are combined to calculate a freshness score.

[0632] Step 8:

[0633] The device captures the user's face and voice in real time and sends them to the emotion engine, which then uses facial recognition and voice analysis to determine the user's emotional state. Emotional data is generated, such as whether the user is smiling, angry, or sad.

[0634] Step 9:

[0635] The server receives the emotion data from the emotion engine and makes a comprehensive judgment based on the freshness score. If the emotion data is positive, it returns a highly recommended result, and if it is negative, it makes adjustments such as presenting a detailed analysis.

[0636] Step 10:

[0637] The server packages the final analysis results in JSON format or similar and sends them to the terminal, returning the data using an HTTP response.

[0638] Step 11:

[0639] The device decodes the analysis results it receives and displays them on the user interface. Specifically, it displays a message such as "This tomato is fresh. Freshness score: 85 / 100" so that the user can visually confirm the results.

[0640] Step 12:

[0641] Users make purchasing decisions based on the displayed freshness score and their own emotional feedback. Users can also use other features within the app to continue analyzing other fresh foods.

[0642] Step 13:

[0643] The emotion engine learns the user's purchasing history and emotional tendencies, and reflects this in future analyses. Based on past data, a more personalized freshness assessment is made.

[0644] Example 2

[0645] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0646] In recent years, technology for evaluating the quality and freshness of fresh food has advanced, but systems that provide feedback that takes user emotions into account are still limited. Furthermore, conventional systems are limited to determining freshness through image analysis, making it difficult to make personalized recommendations based on the user's emotions and preferences. This makes it difficult to improve the user experience and support satisfying purchasing decisions.

[0647] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0648] In this invention, the server includes means for determining the freshness of the food using image analysis technology, means for performing face recognition and voice analysis to recognize the user's emotions, and means for performing analysis based on the determination results and emotion data and providing information to the user. This enables analysis that integrates freshness determination and user emotions, making it possible to provide a more appropriate and satisfying purchasing experience.

[0649] "User" refers to any individual or legal entity that uses the System.

[0650] An "operation" refers to an action taken by a user to achieve a specific goal using the system.

[0651] "Image" refers to visual data captured by a user using a device's camera or the like.

[0652] "Internet" refers to the technology that connects computers and databases around the world via information and communications networks.

[0653] "Server" refers to a set of hardware or software that receives requests and processes data.

[0654] "Image analysis technology" refers to software and algorithms for extracting specific information from image data.

[0655] "Freshness" refers to an indication of the appropriate quality and state of preservation of an item such as food.

[0656] "Facial recognition" refers to the technology that detects and recognizes the faces of people captured on camera.

[0657] "Voice analysis" refers to the technology of analyzing voice data and extracting information.

[0658] "Determination result" refers to a conclusion reached using image analysis techniques or other analytical methods.

[0659] "Emotional data" refers to the results of collecting and analyzing data that indicates the emotional state of a user.

[0660] "Analysis" refers to the process of deriving information and patterns from collected data.

[0661] "Providing information" refers to the act of communicating analysis results and recommendations to users.

[0662] "Visually displaying" refers to the act of graphically showing analytical results or information on a computer screen or device display.

[0663] "Goods" refers to a broad definition of the subject matter, including food and other subject items.

[0664] "Feedback" refers to the analysis results and recommendations provided to the user.

[0665] A "machine learning model" refers to an algorithm that learns patterns from data and makes predictions or classifications for new data.

[0666] The system of the present invention not only determines the freshness of perishable food, but also has the ability to recognize the user's emotions and provide appropriate feedback based on them.

[0667] Hardware and Software Configuration

[0668] The implementation of this system utilizes the following hardware and software:

[0669] Hardware:

[0670] Smartphone or tablet: Use the camera function to take a picture.

[0671] Server: Receives data, analyzes it, and generates feedback.

[0672] software:

[0673] Dedicated app: An application that provides an interface for users to take images and input emotions.

[0674] Image analysis software: Use libraries such as OpenCV to preprocess images.

[0675] Machine learning model: We use TensorFlow and PyTorch to build a neural network to extract food features and calculate a freshness score.

[0676] Emotion Recognition Software: Analyze user emotions using Emotion API or our own emotion recognition model.

[0677] Specific functions of the system

[0678] 1. Image capture and transmission:

[0679] Users launch a dedicated smartphone app and take a picture of the fresh food. The captured image data is encoded by the device and sent to a server via the Internet.

[0680] 2. Image preprocessing:

[0681] The server receives image data sent from the device and performs preprocessing such as resizing, noise filtering, and normalization, using an image processing library such as OpenCV.

[0682] 3. Feature extraction and freshness determination:

[0683] The preprocessed images are input into a neural network to extract features such as food color, surface condition, and shape, using a machine learning model (for example, TensorFlow or PyTorch).

[0684] 4. Emotion recognition:

[0685] When a user smiles at the camera or expresses emotion through voice, the device sends the data to the server, which then performs facial recognition and voice analysis to determine the user's emotional state, using the Emotion API or a proprietary emotion recognition model.

[0686] 5. Data Integration and Analysis:

[0687] The server integrates the freshness score and the user's emotional data and performs an analysis based on the results. If the emotional data is positive, the server provides positive feedback, and if it is negative, the server presents detailed analysis information or other options to the user.

[0688] 6. Providing Feedback:

[0689] The analysis results and emotional feedback are sent to the device and displayed visually within a dedicated app, allowing users to make purchasing decisions about fresh food based on this information.

[0690] 7. Learning Emotional Patterns:

[0691] The emotion engine learns the user's emotional patterns based on their purchasing behavior history and uses them to make decisions for future purchases. It also extracts the user's preferences from past data and provides personalized information.

[0692] Specific examples

[0693] For example, if a user is checking the freshness of tomatoes at a supermarket and the emotion engine detects that the user is smiling, it will highlight the high freshness score. If the user looks dissatisfied, it will suggest, "Try looking for a fresher tomato."

[0694] Example prompts for generative AI models

[0695] "I want to determine the freshness of a particular food item from an image. Can you please tell me some concrete ways to build a system that recognizes the user's emotions and provides feedback based on their reaction?"

[0696] By integrating freshness assessment and emotion recognition, this system aims to provide users with a more relevant and satisfying purchasing experience.

[0697] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0698] Step 1:

[0699] The user launches the app and takes a picture

[0700] Input: Fresh produce the user wants to buy (e.g. tomatoes)

[0701] Output: Image data taken with a smartphone

[0702] How it works: The user launches a dedicated app on their smartphone and takes a picture of the fresh food. The app uses the camera function to capture a high-resolution image.

[0703] Step 2:

[0704] The device encodes and transmits the image data.

[0705] Input: Photographed image data

[0706] Output: Encoded image data (e.g. JPEG, PNG format)

[0707] What it does: The device encodes the captured image and sends it over the Internet to a server, using the HTTPS protocol for secure data transfer.

[0708] Step 3:

[0709] The server receives the image data and performs preprocessing.

[0710] Input: Encoded image data sent from the device

[0711] Output: Preprocessed image data

[0712] Specific operation: The server performs preprocessing on the received image data, such as resizing (e.g., changing to 224x224 pixels), noise filtering, and normalization. This processing uses an image processing library such as OpenCV.

[0713] Step 4:

[0714] The server extracts features using a neural network

[0715] Input: Preprocessed image data

[0716] Output: Extracted features

[0717] Specific operation: The server inputs the preprocessed images into a neural network (e.g., using TensorFlow or PyTorch) to extract features such as the color, surface condition, and shape of the food.

[0718] Step 5:

[0719] The server calculates a freshness score based on the features.

[0720] Input: extracted features

[0721] Output: Freshness score

[0722] How it works: The server uses the extracted features to calculate the freshness score of the fresh produce, using a pre-trained model.

[0723] Step 6:

[0724] The user expresses their emotions through the camera and microphone

[0725] Input: User's facial expressions and voice

[0726] Output: Facial image data and audio data

[0727] Specific actions: The user smiles into the smartphone camera or expresses an emotion through voice.

[0728] Step 7:

[0729] The device sends the facial image data and voice data to the server.

[0730] Input: Facial image data and audio data

[0731] Output: Facial image data and audio data sent from the device

[0732] Specific operation: The device encodes the collected facial image data and voice data and sends them to the server. The data is transferred securely using the HTTPS protocol.

[0733] Step 8:

[0734] The server performs emotion recognition

[0735] Input: Facial image data and audio data

[0736] Output: Emotion data

[0737] How it works: The server performs facial recognition and voice analysis to determine the user's current emotional state, using an emotion recognition model (e.g., Emotion API or a proprietary emotion recognition model).

[0738] Step 9:

[0739] The server combines the freshness score and sentiment data for analysis.

[0740] Input: Freshness score and sentiment data

[0741] Output: Analysis results and feedback

[0742] How it works: The server combines the freshness score with the user's sentiment data and generates feedback based on the results. If the sentiment is positive, it provides positive feedback, and if it is negative, it provides detailed analysis information or other options.

[0743] Step 10:

[0744] The server sends the analysis results to the device.

[0745] Input: Analysis results and feedback

[0746] Output: Analysis results and feedback to the device

[0747] Specific operation: The server sends the analysis results and feedback to a dedicated app and then to the device for visual display.

[0748] Step 11:

[0749] The terminal displays the results visually.

[0750] Input: Analysis results and feedback sent from the server

[0751] Output: Analysis results and feedback displayed visually within a dedicated app

[0752] Specific operation: The device visually displays the analysis results and feedback within a dedicated app so that the user can check them.

[0753] Step 12:

[0754] The server learns emotional patterns

[0755] Input: User's past purchasing behavior history and emotional data

[0756] Output: Learned emotion patterns

[0757] Specific operation: The server uses an emotion engine to learn the user's emotional patterns based on their purchasing behavior history and utilizes them for future judgments. It also extracts the user's preferences from past data and optimizes freshness judgment and feedback based on them.

[0758] (Application example 2)

[0759] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0760] Conventional freshness assessment systems for fresh food simply assess freshness based on image data of the food, without providing feedback that takes into account the user's emotions or purchasing experience. As a result, even if a user confirms the freshness of the food, their satisfaction with the purchase may be low. Furthermore, because they are unable to provide detailed responses based on the user's emotions, it is difficult to influence the user's purchasing behavior.

[0761] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0762] In this invention, the server includes means for acquiring images of fresh food based on user operations, means for transmitting the images to the server via the Internet, means for determining the freshness of the fresh food using image analysis technology in the server, means for providing the results of the image analysis to the user, means for performing facial recognition and voice analysis to recognize the user's emotions, and means for adjusting the results of the image analysis based on the user's emotional state. This makes it possible to provide appropriate feedback that takes the user's emotions into consideration, thereby improving the user's purchasing experience and satisfaction.

[0763] "User operations" refers to a series of actions a user takes to use a system or application.

[0764] "Fresh food" refers to food for which freshness and quality are important and which has a limited shelf life.

[0765] "Means for acquiring an image" refers to a method for acquiring visual information as a digital image using a device such as a camera or scanner.

[0766] "Transmission via the Internet" refers to the technology that utilizes Internet protocols to transmit data to a remote server.

[0767] A "server" refers to a computer system that processes various data and provides services via a network.

[0768] "Image analysis technology" refers to methods and algorithms for extracting useful information from digital image data.

[0769] "Means for determining freshness" refers to methods for evaluating the freshness of fresh food based on various indicators and analytical results.

[0770] "Means of providing" refers to the method of notifying or displaying the analysis results or information to the user.

[0771] "Emotion recognition" refers to analyzing a person's facial expressions and voice data to determine their emotional state.

[0772] "Facial recognition" refers to the technology of identifying facial features in digital images or video streams to recognize and identify individuals.

[0773] "Voice analysis" refers to the technology of analyzing voice data and extracting information such as content, emotions, and speaker identification.

[0774] "Emotional state" refers to the user's mental and emotional state as indicated by the analyzed results.

[0775] "Adjustment means" refers to a method for changing or adjusting the analysis results or provided information according to the emotional state of the user.

[0776] "Feedback" refers to the reaction or information that a system provides to a user.

[0777] "Purchasing experience" refers to the series of experiences and emotions a user feels during the process of purchasing a product.

[0778] The system of the present invention aims to provide a more satisfying shopping experience by combining freshness assessment of fresh food in a physical store with user emotion recognition. This system is implemented using smart glasses.

[0779] First, a user wears smart glasses and makes a purchase in a physical store. The user picks up a fresh food item they are considering purchasing and takes a picture of it with the smart glasses' camera. The captured image is then captured as digital data by the glasses' built-in camera system. This captured image data is temporarily processed by a processor in the smart glasses and sent to a remote server via the Internet.

[0780] The server preprocesses the received image data and uses image analysis technology to determine the freshness of the fresh food. This preprocessing mainly includes image resizing, noise filtering, and normalization. The preprocessed image data is input into a neural network model to extract features such as the food's color, surface condition, and shape. This allows a freshness score to be calculated.

[0781] Next, a process to recognize the user's emotions is carried out. The camera and microphone attached to the smart glasses capture the user's facial expressions and voice in real time. The server receives this data and uses facial recognition and voice analysis technologies to determine the user's emotional state. Facial recognition reads emotions from the user's facial expressions, while voice analysis analyzes emotions from the user's tone and intonation of voice.

[0782] The server analyzes the freshness score by combining it with the user's emotional data. If the user is in a positive emotional state (e.g., smiling, happy voice), the system highlights the freshness score and provides feedback that increases the recommendation. Conversely, if a negative emotional state (e.g., dissatisfied face, flat voice) is detected, the system provides feedback such as detailed analysis information or other options.

[0783] The analysis results and emotional data feedback are then sent to the smart glasses via the internet and visually displayed on the screen, allowing users to make purchasing decisions about fresh food based on this information.

[0784] As a specific example, a user picks up a tomato in a supermarket and takes a picture of it with the smart glasses' camera. Image analysis determines that the captured image has a high freshness score, and the server feeds that score back to the user. At the same time, if the user smiles, the smart glasses' display will show "This product is fresh! Freshness score: 0.85." On the other hand, if the user looks dissatisfied, the display will show more detailed information, such as "Please try looking for a fresher product. Freshness score: 0.65."

[0785] Examples of prompts using generative AI models for emotion recognition and freshness determination include:

[0786] "A user picks up a tomato in a supermarket and uses smart glasses to check its freshness. If the user smiles, provide feedback highlighting a high freshness score. If the user looks dissatisfied, provide detailed analytics or alternative options."

[0787] As described above, the system of the present invention improves the user's purchasing experience and satisfaction by providing feedback to the user by integrating the freshness determination of perishable foods with the recognition of the user's emotions.

[0788] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0789] Step 1:

[0790] A user wears smart glasses and picks up fresh produce in a brick-and-mortar store. The smart glasses' camera is used to capture an image of the food, where the input is the physical appearance of the fresh produce and the output is digital image data. This digital image data is sent to the system for analysis.

[0791] Step 2:

[0792] The terminal (smart glasses) sends the captured image data to a server via the Internet. The input is the digital image data acquired in step 1, and the output is the image data transferred to the server. A network communication protocol is used for data transfer.

[0793] Step 3:

[0794] The server preprocesses the received image data by resizing, noise filtering, normalizing, and converting it into a format suitable for analysis. The input is the image data of the fresh food sent to the server, and the output is the preprocessed image data.

[0795] Step 4:

[0796] The server inputs the preprocessed image data into a neural network model to extract features such as the color, surface condition, and shape of the food, which then calculates a freshness score. The input is the preprocessed image data, and the output is the freshness score.

[0797] Step 5:

[0798] The device (smart glasses) captures the user's facial expressions and voice in real time. It uses the camera and microphone of the smart glasses to obtain facial image data and voice data. The input is the user's facial expressions and voice, and the output is facial image data and voice data.

[0799] Step 6:

[0800] The server analyzes the received facial image data and voice data to recognize the user's emotional state. Using facial recognition and voice analysis technologies, it determines whether the user's emotion is positive or negative. The input is facial image data and voice data, and the output is the user's emotional state.

[0801] Step 7:

[0802] The server performs an analysis by combining the calculated freshness score with the user's emotional state. If the emotional state is positive, it highlights the freshness score and generates feedback that increases the recommendation level. Conversely, if the emotional state is negative, it generates feedback that presents detailed analysis information and other options. The input is the freshness score and the emotional state, and the output is feedback information.

[0803] Step 8:

[0804] The server sends the generated feedback information to the terminal (smart glasses) via the Internet. The input is the feedback information, and the output is the feedback information displayed on the display of the smart glasses.

[0805] Step 9:

[0806] The user makes a purchasing decision on fresh food based on the feedback information displayed on the smart glasses display. The input is the feedback information, and the output is the purchase decision on fresh food.

[0807] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0808] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0809] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0810] [Third embodiment]

[0811] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0812] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0813] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0814] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0815] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0816] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0817] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0818] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0819] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0820] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0821] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0822] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0823] The system according to the present invention provides a series of functions for users to determine the freshness of perishable foods, and specific embodiments thereof will be described below.

[0824] First, the user launches the app on a smartphone or other device and takes a picture of the fresh food. For example, if the user is looking to buy grapes at a supermarket, they can take a picture of the grapes using the app's camera function.

[0825] The captured image is encoded by the device and sent to a server via the Internet, along with the image data, including user identification information and food category information (e.g., fruit or vegetable).

[0826] The server receives the transmitted image data and performs an analysis process. This analysis process incorporates image analysis technology using neural networks. Specifically, the server inputs the image data into a neural network model and analyzes specific features (e.g., color, surface condition, shape, etc.) to determine the freshness of the fresh food.

[0827] The neural network analyzes the color, surface condition, and shape of food and calculates a freshness score based on each feature. For example, in the case of grapes, information such as the color of the skin, the presence or absence of bloom on the surface, the color and firmness of the stalk are analyzed.

[0828] The server sends the resulting freshness score and details to the user's device, where the analysis results are returned in JSON or other data formats.

[0829] The device then displays the analysis results to the user. Specifically, the app displays results such as "These grapes are fresh. Freshness score: 80 / 100," allowing the user to make a purchasing decision based on this information.

[0830] As described above, the system according to the present invention allows users to easily determine the freshness of perishable foods, and allows users to shop at supermarkets and greengrocers with greater peace of mind.

[0831] As a concrete example, consider the case where a user purchases commercially available tomatoes. The user launches a dedicated app and takes a picture of the tomato. The image data is then sent to a server, which uses a neural network to analyze the color and surface condition of the tomato (wrinkles, glossiness, etc.). The analysis result is displayed on the user's device as "This tomato is fresh. Freshness score: 85 / 100." The user can use this information to select fresh tomatoes.

[0832] Thus, the system of the present invention is a powerful tool that helps users select fresh foods in their daily shopping.

[0833] The processing flow will be explained below.

[0834] Step 1:

[0835] The user launches a dedicated app on their device and takes a picture of the fresh food they want to purchase (e.g., grapes).The user then uses the app's camera function to capture a clear image of the food under appropriate lighting conditions.

[0836] Step 2:

[0837] The device encodes the captured image data. Specifically, it compresses the image into JPEG or PNG format to reduce the data size. The encoded image data is temporarily saved.

[0838] Step 3:

[0839] The device packages the encoded image data with user identification information and food category information, and sends the packaged data to the server via the Internet. The image data and accompanying information are sent using an HTTP POST request.

[0840] Step 4:

[0841] The server receives the image data sent from the device. The server analyzes the received data and stores the image data, user identification information, food category information, etc. in the respective variables.

[0842] Step 5:

[0843] The server preprocesses the image data stored on the server, specifically by resizing the image, filtering noise, normalizing it, etc., to convert it into a form suitable for image analysis.

[0844] Step 6:

[0845] The server inputs the preprocessed image data into an AI analysis module, where a neural network analyzes the food's color, surface condition, shape, and other characteristics to extract specific features.

[0846] Step 7:

[0847] The AI ​​analysis module determines the freshness of perishable foods based on the extracted features. For example, in the case of grapes, the color of the skin, the presence or absence of bloom, and the color of the branch are indicators of freshness. These features are combined to calculate a freshness score.

[0848] Step 8:

[0849] The server packages the freshness score and detailed analysis results in JSON format or similar and sends them to the device. The data is returned using an HTTP response.

[0850] Step 9:

[0851] The device decodes the analysis results it receives and displays them on the user interface. Specifically, it displays a message such as "These grapes are fresh. Freshness score: 80 / 100" to provide the user with a visual result.

[0852] Step 10:

[0853] Users can then make a purchasing decision based on the displayed freshness score, or continue analyzing other fresh foods using other features within the app.

[0854] Example 1

[0855] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0856] When purchasing fresh food, determining its freshness is extremely important and presents a major challenge for consumers. However, it can be difficult to accurately judge freshness using visual information alone. This challenge is particularly severe for consumers who lack specialized knowledge or experience in assessing freshness. Currently, there is no reliable method for determining freshness, so there is a need to eliminate the uncertainty consumers have when selecting high-quality fresh food.

[0857] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0858] In this invention, the server includes a means for determining the freshness of the food using image analysis technology, a means for the server to analyze specific features of the food using a neural network model and calculate a freshness score, and a means for transmitting the freshness score to a user's device in JSON format, thereby enabling users to easily determine the freshness of fresh food and make purchasing decisions based on reliable information.

[0859] "User" refers to the consumer who uses the dedicated app to operate the system and determine the freshness of perishables.

[0860] "Terminal" refers to an electronic device used by a user, such as a smartphone or tablet, that acquires images and communicates with a server.

[0861] "Image acquisition means" refers to a function for taking images of fresh food using the camera function of the terminal.

[0862] "Transmission means" refers to a function for transmitting image data and additional information acquired from a terminal to a server via the Internet.

[0863] "Server" refers to a computer system that processes received image data using image analysis techniques to determine freshness.

[0864] "Image analysis technology" refers to technology that uses software algorithms running on a server to extract specific features from images of fresh food and determine its freshness.

[0865] A "neural network model" is a model that uses artificial intelligence technology that runs on a server and is used to analyze food characteristics such as color, surface condition, and shape.

[0866] "Freshness score" refers to a numerical index of the freshness of fresh food calculated based on image analysis technology and neural network models.

[0867] "JSON format" is a type of data format used when sending analysis results and freshness scores to devices, and refers to a lightweight, structured data exchange format.

[0868] "Display means" refers to a function for visually showing the freshness score and analysis results received on the terminal to the user.

[0869] The present invention relates to a system for users to determine the freshness of fresh food. Implementing this system involves a series of processes: acquiring an image of the food based on user operations, transmitting the image to a server via the Internet, and the server using image analysis technology to determine the freshness and providing the analysis results to the user.

[0870] Hardware and software used

[0871] 1. The device used by the user is an electronic device such as a smartphone or tablet. Specific examples include an iPhone 12 or an Android-based device. A dedicated app (e.g., FreshCheck App) is installed on the device, and by launching this app, the user can use the camera function to take a picture of the food.

[0872] 2. The device encodes the captured image in JPEG format, adds user identification information and food category information (e.g., fruit, vegetable), and sends it to the server via HTTPS protocol.

[0873] 3. The server is a high-performance computer system equipped with software that performs image analysis techniques, specifically a neural network model using the TensorFlow library, which analyzes specific features of food, such as color, surface condition, and shape, to calculate a freshness score.

[0874] Example of operation

[0875] While selecting tomatoes to purchase at the supermarket, a user launches a dedicated app and takes a picture of the tomato. The device encodes this image into JPEG format, adds the information "User ID: 67890" and "Food category: Vegetables", and sends it to the server. When the server receives the image data, it inputs the image into a neural network model using TensorFlow, which analyzes the color, surface condition, etc. of the tomato. The freshness score calculated as a result of the analysis (e.g., 85 / 100) is formatted in JSON format and sent to the user's device. The device then displays the received result on the app interface as "This tomato is fresh. Freshness score: 85 / 100." The user can use this information to select fresh tomatoes.

[0876] Prompt Sentence Examples

[0877] An example of a prompt in text format is shown below.

[0878] The user launches the dedicated app "FreshCheck App" on their smartphone and takes a picture of the tomato they plan to purchase. The image data is sent to the server, which uses TensorFlow to analyze the color and surface condition. The analysis result is displayed on the user's device as "This tomato is fresh. Freshness score: 85 / 100." Based on this information, the user can select fresh tomatoes.

[0879] Thus, the system according to the present invention is a powerful tool that helps users to select fresh foods in their daily shopping.

[0880] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0881] Processing Steps

[0882] Step 1:

[0883] The user launches a dedicated app and takes a picture of the fresh food they plan to purchase.

[0884] Specific operation: The user launches the "FreshCheck App" on their smartphone and takes a picture of a tomato using the app's camera function.

[0885] Input: A camera activated by user action, an image of a tomato

[0886] Output: Image data of the photographed tomato

[0887] Step 2:

[0888] The device encodes the captured image data and sends it to a server via the Internet.

[0889] Specific operation: The device encodes the captured image into JPEG format, and then adds user identification information (e.g., UserID: 67890) and food category information (e.g., vegetables).

[0890] Input: Tomato image, user identification information, food category information

[0891] Output: Encoded JPEG image data and additional information

[0892] Step 3:

[0893] The terminal transmits the encoded image data and additional information to the server using the HTTPS protocol.

[0894] Specific operation: The terminal sends the encoded image data and additional information to the server as an HTTPS request.

[0895] Input: JPEG image data and additional information

[0896] Output: Image data and additional information are sent to the server as an HTTPS request.

[0897] Step 4:

[0898] The server performs image analysis using a neural network based on the image data and additional information received.

[0899] Specific operation: The server uses the TensorFlow library to input image data into a neural network model and analyzes features such as the color and surface condition of the tomato.

[0900] Input: Received JPEG image data and additional information

[0901] Output: Analyzed feature data (e.g., RGB values ​​of color, edge information of surface condition)

[0902] Step 5:

[0903] The server calculates a freshness score based on the analysis results and sends it to the user's device in JSON format.

[0904] Specific operation: The server calculates a freshness score (e.g., 85 / 100) based on the analyzed feature data and formats the result in JSON format.

[0905] Input: Analyzed feature data

[0906] Output: JSON format data containing the freshness score.

[0907] Step 6:

[0908] The server sends the parsed results in JSON format to the user's device.

[0909] Specific operation: The server sends the analysis result in JSON format, including the freshness score, to the user's device as an HTTPS response.

[0910] Input: JSON formatted data containing freshness scores

[0911] Output: Analysis results are sent to the device as an HTTPS response

[0912] Step 7:

[0913] The terminal displays the analysis results received to the user.

[0914] Specific operation: The device reads the JSON format analysis results received from the server and displays "This tomato is fresh. Freshness score: 85 / 100" on the app interface.

[0915] Input: Parsed result of received JSON format

[0916] Output: Freshness score and message displayed in the app

[0917] This flow of steps allows users to easily determine the freshness of perishable foods and make purchasing decisions based on reliable information.

[0918] (Application example 1)

[0919] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0920] Existing systems lack the means to not only determine the freshness of fresh food, but also to confirm its safety and quality. Furthermore, they do not meet the needs of users who want to detect counterfeit food and illegally distributed products. To solve this problem, a system is needed that can not only determine the freshness and quality of food, but also verify its authenticity using security tags.

[0921] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0922] In this invention, the server includes means for using image analysis technology to determine the freshness and quality of the item, means for verifying the security tag, and means for providing the results of the analysis and verification to the user, thereby enabling the user to verify not only the freshness and quality of the food they are purchasing, but also whether the food is authentic.

[0923] "Item" refers to a specific product or item that is the subject of analysis using a photograph or image.

[0924] "Freshness" is an index that indicates the degree of freshness that an item maintains and the quality of its preservation.

[0925] "Quality" refers to the characteristics and condition that make up the overall evaluation of a product, including its value and safety to the consumer.

[0926] "Image analysis technology" is a technology for analyzing the characteristics of an item based on a photograph or image to determine its freshness and quality.

[0927] A "security tag" is identification information, such as a QR code or barcode, attached to an item to prove its authenticity.

[0928] "User" means any individual or entity that uses the system to verify the freshness, quality, and authenticity of goods.

[0929] "Via the Internet" refers to a method of using public lines to send and receive data between a user's device and a server.

[0930] A "server" is a computer system that receives data sent from a user, analyzes it, and returns the results.

[0931] "Transmitting" refers to the act of sending images and related information from a user's device to a server via telecommunications means.

[0932] "Providing" refers to the act of presenting the results analyzed and confirmed by the server in the form of a display on the user's device.

[0933] System Program

[0934] The system program for this application example begins with a user taking an image of an item using a device such as a smartphone, smart glasses, or head-mounted display and sending it to a server via the Internet. The server analyzes the received image data, determines its freshness and quality, and then checks the security tag based on that information. The results of the analysis and check are then provided back to the user's device, and the user can use that information to determine the item's freshness and authenticity.

[0935] Natural language explanation of the process

[0936] This system is realized using the following hardware and software.

[0937] The hardware used includes devices such as smartphones, smart glasses, and head-mounted displays, and the software used is Python, OpenCV, Keras (a library for neural networks), and requests (a library for HTTP communication).

[0938] 1. Image capture: The user takes an image of an object using the device's camera, which can be done through an interface such as a smartphone, smart glasses, or a head-mounted display.

[0939] 2. Data transmission: The acquired image data is encoded by the device and sent to the server using the HTTP protocol. At this time, user identification information, item category information, and other information are sent along with the image data.

[0940] 3. Image analysis: The server inputs the received image data into a neural network model to analyze specific features (e.g., color, surface condition, shape, etc.) to determine freshness and quality. This uses a neural network built using Python and Keras.

[0941] 4. Security Verification: The server detects and analyzes the item's security tag (e.g., QR code, barcode, etc.) to verify that the item is authentic. In this step, an HTTP request is made to an external security tag verification service.

[0942] 5. Providing results: The analysis and verification results are returned to the user's device in a data format such as JSON. The analysis results are then visually displayed on the user's device. For example, "This item is fresh. Freshness score: 80 / 100. This item is authentic."

[0943] Specific examples

[0944] For example, if a user is looking to buy grapes at a supermarket, they launch a dedicated app and take a picture of the grapes. The captured image is sent to a server, where a neural network is used to analyze the grapes' color, surface condition, and branch condition to calculate a freshness score. At the same time, the QR code attached to the grapes' packaging is scanned to confirm their authenticity. The result is a message displayed on the smartphone saying, "These grapes are fresh. Freshness score: 85 / 100. These grapes are authentic."

[0945] Example prompts to be input to the generative AI model

[0946] "Analyze the features of fresh produce, such as color, surface condition, and shape, to calculate a freshness score. Also, use security tags to verify whether the food is authentic."

[0947] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0948] Step 1:

[0949] Image acquisition

[0950] input:

[0951] The user activates the camera on their smart device and takes a picture of an item (e.g., grapes).

[0952] Specific behavior:

[0953] The user activates the camera function on a smartphone or smart glasses interface and takes a picture of the item, which generates a high-resolution image of the item.

[0954] output:

[0955] Acquired item image data.

[0956] Step 2:

[0957] Sending data

[0958] input:

[0959] The item image data, user identification information, and item category information obtained in step 1.

[0960] Specific behavior:

[0961] The terminal transmits the acquired image data, user identification information (e.g., user ID, store ID), and item category information (e.g., fruit, vegetable) to the server as an HTTP request.

[0962] output:

[0963] Image data and related information sent to the server.

[0964] Step 3:

[0965] Image analysis

[0966] input:

[0967] The product image data and related information are sent to the server.

[0968] Specific behavior:

[0969] The server inputs the received image data into a neural network model built using Python and Keras to analyze features such as color, surface condition, and shape. For example, in the case of grapes, the skin color and surface gloss are analyzed. The neural network evaluates these features using a trained generative AI model and calculates a freshness score.

[0970] output:

[0971] Parsed freshness score and other feature data.

[0972] Step 4:

[0973] Check the security tag

[0974] input:

[0975] Item image data and related information obtained after analysis.

[0976] Specific behavior:

[0977] The server detects the security tag attached to the item (e.g., QR code or barcode) from the image and uses it to query an external security tag verification service. For example, it scans the QR code, extracts its content, and checks whether the item is authentic based on that content.

[0978] output:

[0979] Security tag verification results (genuine / non-genuine).

[0980] Step 5:

[0981] Providing analysis and verification results

[0982] input:

[0983] Freshness score and security tag validation results.

[0984] Specific behavior:

[0985] The server compiles the analysis results and security tag verification results in a data format such as JSON and returns them to the user's device. At that time, to make the results easier to understand, it generates a specific message such as "This item is fresh. Freshness score: 85 / 100. This item is authentic."

[0986] output:

[0987] Analysis and verification results are displayed on the user's device.

[0988] Step 6:

[0989] Displaying the results

[0990] input:

[0991] Analysis and verification result data returned from the server.

[0992] Specific behavior:

[0993] The device analyzes the received data and visually displays it on the user interface. For example, a message such as "These grapes are fresh. Freshness score: 85 / 100. These grapes are authentic" could be displayed on the smartphone screen, making the results easy for the user to understand.

[0994] output:

[0995] User perception of the results and purchasing decisions based on them.

[0996] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0997] The system according to the present invention has the function of not only determining the freshness of perishable foods but also recognizing the user's emotions and providing information based on those emotions. Specific embodiments of the system will be described below.

[0998] First, a user launches a dedicated smartphone app and takes a picture of the fresh food. For example, if a user is looking to buy tomatoes at a supermarket, they can use the app's camera to take a picture of the tomato. The captured image data is encoded by the device and sent to a server via the Internet.

[0999] The server receives the image data sent from the device and performs preprocessing, which includes image resizing, noise filtering, and normalization. The processed image is then input into a neural network to extract features such as the food's color, surface condition, and shape, and calculate a freshness score.

[1000] The newly added emotion engine feature uses facial recognition and voice analysis to determine a user's current emotional state. For example, if a user smiles into the smartphone camera or vocally expresses joy, that emotional data is collected.

[1001] The server analyzes the detected freshness score by combining it with the user's emotional data. If the emotional data is positive, it provides the user with a result that is more likely to be recommended. Conversely, if negative emotions are detected, it provides the user with appropriate feedback, such as detailed analysis information or other options.

[1002] For example, if a user is checking the freshness of a tomato and the emotion engine detects that the user is smiling, it will highlight the high freshness score. If the user looks dissatisfied, it will suggest, "Try looking for a fresher tomato."

[1003] The analysis results and emotional feedback are sent to the device and displayed visually within the app, allowing users to make informed purchasing decisions about fresh food.

[1004] Furthermore, the emotion engine can learn a user's emotional patterns based on their purchasing history and use them in future judgments. For example, if past data shows that a user tends to prefer foods of a certain color or shape, the engine can adjust the freshness judgment results based on that tendency and provide more personalized information.

[1005] In this way, the system of the present invention aims to provide users with a more appropriate and satisfying purchasing experience by integrating freshness determination and emotion recognition, thereby improving users' constant satisfaction and providing highly accurate freshness determination.

[1006] The processing flow will be explained below.

[1007] Step 1:

[1008] The user launches a dedicated app on their device and takes a picture of the fresh food (e.g., a tomato) they are considering purchasing. The user then uses the app's camera function to capture a clear image of the food under appropriate lighting.

[1009] Step 2:

[1010] The device encodes the captured image data. Specifically, it compresses the image into JPEG or PNG format to reduce the data size. The encoded image data is temporarily saved.

[1011] Step 3:

[1012] The device packages the encoded image data with user identification information and food category information, and sends the packaged data to the server via the Internet. The image data and accompanying information are sent using an HTTP POST request.

[1013] Step 4:

[1014] The server receives the image data sent from the device. The server analyzes the received data and stores the image data, user identification information, food category information, etc. in the respective variables.

[1015] Step 5:

[1016] The server preprocesses the image data stored on the server, specifically by resizing the image, filtering noise, normalizing it, etc., to convert it into a form suitable for image analysis.

[1017] Step 6:

[1018] The server inputs the preprocessed image data into an AI analysis module, where a neural network analyzes the food's color, surface condition, shape, and other characteristics to extract specific features.

[1019] Step 7:

[1020] The AI ​​analysis module determines the freshness of fresh food based on the extracted features. For example, in the case of tomatoes, the color of the skin and the presence or absence of blemishes or scratches on the surface are indicators of freshness. These features are combined to calculate a freshness score.

[1021] Step 8:

[1022] The device captures the user's face and voice in real time and sends them to the emotion engine, which then uses facial recognition and voice analysis to determine the user's emotional state. Emotional data is generated, such as whether the user is smiling, angry, or sad.

[1023] Step 9:

[1024] The server receives the emotion data from the emotion engine and makes a comprehensive judgment based on the freshness score. If the emotion data is positive, it returns a highly recommended result, and if it is negative, it makes adjustments such as presenting a detailed analysis.

[1025] Step 10:

[1026] The server packages the final analysis results in JSON format or similar and sends them to the terminal, returning the data using an HTTP response.

[1027] Step 11:

[1028] The device decodes the analysis results it receives and displays them on the user interface. Specifically, it displays a message such as "This tomato is fresh. Freshness score: 85 / 100" so that the user can visually confirm the results.

[1029] Step 12:

[1030] Users make purchasing decisions based on the displayed freshness score and their own emotional feedback. Users can also use other features within the app to continue analyzing other fresh foods.

[1031] Step 13:

[1032] The emotion engine learns the user's purchasing history and emotional tendencies, and reflects this in future analyses. Based on past data, a more personalized freshness assessment is made.

[1033] Example 2

[1034] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1035] In recent years, technology for evaluating the quality and freshness of fresh food has advanced, but systems that provide feedback that takes user emotions into account are still limited. Furthermore, conventional systems are limited to determining freshness through image analysis, making it difficult to make personalized recommendations based on the user's emotions and preferences. This makes it difficult to improve the user experience and support satisfying purchasing decisions.

[1036] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1037] In this invention, the server includes means for determining the freshness of the food using image analysis technology, means for performing face recognition and voice analysis to recognize the user's emotions, and means for performing analysis based on the determination results and emotion data and providing information to the user. This enables analysis that integrates freshness determination and user emotions, making it possible to provide a more appropriate and satisfying purchasing experience.

[1038] "User" refers to any individual or legal entity that uses the System.

[1039] An "operation" refers to an action taken by a user to achieve a specific goal using the system.

[1040] "Image" refers to visual data captured by a user using a device's camera or the like.

[1041] "Internet" refers to the technology that connects computers and databases around the world via information and communications networks.

[1042] "Server" refers to a set of hardware or software that receives requests and processes data.

[1043] "Image analysis technology" refers to software and algorithms for extracting specific information from image data.

[1044] "Freshness" refers to an indication of the appropriate quality and state of preservation of an item such as food.

[1045] "Facial recognition" refers to the technology that detects and recognizes the faces of people captured on camera.

[1046] "Voice analysis" refers to the technology of analyzing voice data and extracting information.

[1047] "Determination result" refers to a conclusion reached using image analysis techniques or other analytical methods.

[1048] "Emotional data" refers to the results of collecting and analyzing data that indicates the emotional state of a user.

[1049] "Analysis" refers to the process of deriving information and patterns from collected data.

[1050] "Providing information" refers to the act of communicating analysis results and recommendations to users.

[1051] "Visually displaying" refers to the act of graphically showing analytical results or information on a computer screen or device display.

[1052] "Goods" refers to a broad definition of the subject matter, including food and other subject items.

[1053] "Feedback" refers to the analysis results and recommendations provided to the user.

[1054] A "machine learning model" refers to an algorithm that learns patterns from data and makes predictions or classifications for new data.

[1055] The system of the present invention not only determines the freshness of perishable food, but also has the ability to recognize the user's emotions and provide appropriate feedback based on them.

[1056] Hardware and Software Configuration

[1057] The implementation of this system utilizes the following hardware and software:

[1058] Hardware:

[1059] Smartphone or tablet: Use the camera function to take a picture.

[1060] Server: Receives data, analyzes it, and generates feedback.

[1061] software:

[1062] Dedicated app: An application that provides an interface for users to take images and input emotions.

[1063] Image analysis software: Use libraries such as OpenCV to preprocess images.

[1064] Machine learning model: We use TensorFlow and PyTorch to build a neural network to extract food features and calculate a freshness score.

[1065] Emotion Recognition Software: Analyze user emotions using Emotion API or our own emotion recognition model.

[1066] Specific functions of the system

[1067] 1. Image capture and transmission:

[1068] Users launch a dedicated smartphone app and take a picture of the fresh food. The captured image data is encoded by the device and sent to a server via the Internet.

[1069] 2. Image preprocessing:

[1070] The server receives image data sent from the device and performs preprocessing such as resizing, noise filtering, and normalization, using an image processing library such as OpenCV.

[1071] 3. Feature extraction and freshness determination:

[1072] The preprocessed images are input into a neural network to extract features such as food color, surface condition, and shape, using a machine learning model (for example, TensorFlow or PyTorch).

[1073] 4. Emotion recognition:

[1074] When a user smiles at the camera or expresses emotion through voice, the device sends the data to the server, which then performs facial recognition and voice analysis to determine the user's emotional state, using the Emotion API or a proprietary emotion recognition model.

[1075] 5. Data Integration and Analysis:

[1076] The server integrates the freshness score and the user's emotional data and performs an analysis based on the results. If the emotional data is positive, the server provides positive feedback, and if it is negative, the server presents detailed analysis information or other options to the user.

[1077] 6. Providing Feedback:

[1078] The analysis results and emotional feedback are sent to the device and displayed visually within a dedicated app, allowing users to make purchasing decisions about fresh food based on this information.

[1079] 7. Learning Emotional Patterns:

[1080] The emotion engine learns the user's emotional patterns based on their purchasing behavior history and uses them to make decisions for future purchases. It also extracts the user's preferences from past data and provides personalized information.

[1081] Specific examples

[1082] For example, if a user is checking the freshness of tomatoes at a supermarket and the emotion engine detects that the user is smiling, it will highlight the high freshness score. If the user looks dissatisfied, it will suggest, "Try looking for a fresher tomato."

[1083] Example prompts for generative AI models

[1084] "I want to determine the freshness of a particular food item from an image. Can you please tell me some concrete ways to build a system that recognizes the user's emotions and provides feedback based on their reaction?"

[1085] By integrating freshness assessment and emotion recognition, this system aims to provide users with a more relevant and satisfying purchasing experience.

[1086] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1087] Step 1:

[1088] The user launches the app and takes a picture

[1089] Input: Fresh produce the user wants to buy (e.g. tomatoes)

[1090] Output: Image data taken with a smartphone

[1091] How it works: The user launches a dedicated app on their smartphone and takes a picture of the fresh food. The app uses the camera function to capture a high-resolution image.

[1092] Step 2:

[1093] The device encodes and transmits the image data.

[1094] Input: Photographed image data

[1095] Output: Encoded image data (e.g. JPEG, PNG format)

[1096] What it does: The device encodes the captured image and sends it over the Internet to a server, using the HTTPS protocol for secure data transfer.

[1097] Step 3:

[1098] The server receives the image data and performs preprocessing.

[1099] Input: Encoded image data sent from the device

[1100] Output: Preprocessed image data

[1101] Specific operation: The server performs preprocessing on the received image data, such as resizing (e.g., changing to 224x224 pixels), noise filtering, and normalization. This processing uses an image processing library such as OpenCV.

[1102] Step 4:

[1103] The server extracts features using a neural network

[1104] Input: Preprocessed image data

[1105] Output: Extracted features

[1106] Specific operation: The server inputs the preprocessed images into a neural network (e.g., using TensorFlow or PyTorch) to extract features such as the color, surface condition, and shape of the food.

[1107] Step 5:

[1108] The server calculates a freshness score based on the features.

[1109] Input: extracted features

[1110] Output: Freshness score

[1111] How it works: The server uses the extracted features to calculate the freshness score of the fresh produce, using a pre-trained model.

[1112] Step 6:

[1113] The user expresses their emotions through the camera and microphone

[1114] Input: User's facial expressions and voice

[1115] Output: Facial image data and audio data

[1116] Specific actions: The user smiles into the smartphone camera or expresses an emotion through voice.

[1117] Step 7:

[1118] The device sends the facial image data and voice data to the server.

[1119] Input: Facial image data and audio data

[1120] Output: Facial image data and audio data sent from the device

[1121] Specific operation: The device encodes the collected facial image data and voice data and sends them to the server. The data is transferred securely using the HTTPS protocol.

[1122] Step 8:

[1123] The server performs emotion recognition

[1124] Input: Facial image data and audio data

[1125] Output: Emotion data

[1126] How it works: The server performs facial recognition and voice analysis to determine the user's current emotional state, using an emotion recognition model (e.g., Emotion API or a proprietary emotion recognition model).

[1127] Step 9:

[1128] The server combines the freshness score and sentiment data for analysis.

[1129] Input: Freshness score and sentiment data

[1130] Output: Analysis results and feedback

[1131] How it works: The server combines the freshness score with the user's sentiment data and generates feedback based on the results. If the sentiment is positive, it provides positive feedback, and if it is negative, it provides detailed analysis information or other options.

[1132] Step 10:

[1133] The server sends the analysis results to the device.

[1134] Input: Analysis results and feedback

[1135] Output: Analysis results and feedback to the device

[1136] Specific operation: The server sends the analysis results and feedback to a dedicated app and then to the device for visual display.

[1137] Step 11:

[1138] The terminal displays the results visually.

[1139] Input: Analysis results and feedback sent from the server

[1140] Output: Analysis results and feedback displayed visually within a dedicated app

[1141] Specific operation: The device visually displays the analysis results and feedback within a dedicated app so that the user can check them.

[1142] Step 12:

[1143] The server learns emotional patterns

[1144] Input: User's past purchasing behavior history and emotional data

[1145] Output: Learned emotion patterns

[1146] Specific operation: The server uses an emotion engine to learn the user's emotional patterns based on their purchasing behavior history and utilizes them for future judgments. It also extracts the user's preferences from past data and optimizes freshness judgment and feedback based on them.

[1147] (Application example 2)

[1148] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1149] Conventional freshness assessment systems for fresh food simply assess freshness based on image data of the food, without providing feedback that takes into account the user's emotions or purchasing experience. As a result, even if a user confirms the freshness of the food, their satisfaction with the purchase may be low. Furthermore, because they are unable to provide detailed responses based on the user's emotions, it is difficult to influence the user's purchasing behavior.

[1150] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1151] In this invention, the server includes means for acquiring images of fresh food based on user operations, means for transmitting the images to the server via the Internet, means for determining the freshness of the fresh food using image analysis technology in the server, means for providing the results of the image analysis to the user, means for performing facial recognition and voice analysis to recognize the user's emotions, and means for adjusting the results of the image analysis based on the user's emotional state. This makes it possible to provide appropriate feedback that takes the user's emotions into consideration, thereby improving the user's purchasing experience and satisfaction.

[1152] "User operations" refers to a series of actions a user takes to use a system or application.

[1153] "Fresh food" refers to food for which freshness and quality are important and which has a limited shelf life.

[1154] "Means for acquiring an image" refers to a method for acquiring visual information as a digital image using a device such as a camera or scanner.

[1155] "Transmission via the Internet" refers to the technology that utilizes Internet protocols to transmit data to a remote server.

[1156] A "server" refers to a computer system that processes various data and provides services via a network.

[1157] "Image analysis technology" refers to methods and algorithms for extracting useful information from digital image data.

[1158] "Means for determining freshness" refers to methods for evaluating the freshness of fresh food based on various indicators and analytical results.

[1159] "Means of providing" refers to the method of notifying or displaying the analysis results or information to the user.

[1160] "Emotion recognition" refers to analyzing a person's facial expressions and voice data to determine their emotional state.

[1161] "Facial recognition" refers to the technology of identifying facial features in digital images or video streams to recognize and identify individuals.

[1162] "Voice analysis" refers to the technology of analyzing voice data and extracting information such as content, emotions, and speaker identification.

[1163] "Emotional state" refers to the user's mental and emotional state as indicated by the analyzed results.

[1164] "Adjustment means" refers to a method for changing or adjusting the analysis results or provided information according to the emotional state of the user.

[1165] "Feedback" refers to the reaction or information that a system provides to a user.

[1166] "Purchasing experience" refers to the series of experiences and emotions a user feels during the process of purchasing a product.

[1167] The system of the present invention aims to provide a more satisfying shopping experience by combining freshness assessment of fresh food in a physical store with user emotion recognition. This system is implemented using smart glasses.

[1168] First, a user wears smart glasses and makes a purchase in a physical store. The user picks up a fresh food item they are considering purchasing and takes a picture of it with the smart glasses' camera. The captured image is then captured as digital data by the glasses' built-in camera system. This captured image data is temporarily processed by a processor in the smart glasses and sent to a remote server via the Internet.

[1169] The server preprocesses the received image data and uses image analysis technology to determine the freshness of the fresh food. This preprocessing mainly includes image resizing, noise filtering, and normalization. The preprocessed image data is input into a neural network model to extract features such as the food's color, surface condition, and shape. This allows a freshness score to be calculated.

[1170] Next, a process to recognize the user's emotions is carried out. The camera and microphone attached to the smart glasses capture the user's facial expressions and voice in real time. The server receives this data and uses facial recognition and voice analysis technologies to determine the user's emotional state. Facial recognition reads emotions from the user's facial expressions, while voice analysis analyzes emotions from the user's tone and intonation of voice.

[1171] The server analyzes the freshness score by combining it with the user's emotional data. If the user is in a positive emotional state (e.g., smiling, happy voice), the system highlights the freshness score and provides feedback that increases the recommendation. Conversely, if a negative emotional state (e.g., dissatisfied face, flat voice) is detected, the system provides feedback such as detailed analysis information or other options.

[1172] The analysis results and emotional data feedback are then sent to the smart glasses via the internet and visually displayed on the screen, allowing users to make purchasing decisions about fresh food based on this information.

[1173] As a specific example, a user picks up a tomato in a supermarket and takes a picture of it with the smart glasses' camera. Image analysis determines that the captured image has a high freshness score, and the server feeds that score back to the user. At the same time, if the user smiles, the smart glasses' display will show "This product is fresh! Freshness score: 0.85." On the other hand, if the user looks dissatisfied, the display will show more detailed information, such as "Please try looking for a fresher product. Freshness score: 0.65."

[1174] Examples of prompts using generative AI models for emotion recognition and freshness determination include:

[1175] "A user picks up a tomato in a supermarket and uses smart glasses to check its freshness. If the user smiles, provide feedback highlighting a high freshness score. If the user looks dissatisfied, provide detailed analytics or alternative options."

[1176] As described above, the system of the present invention improves the user's purchasing experience and satisfaction by providing feedback to the user by integrating the freshness determination of perishable foods with the recognition of the user's emotions.

[1177] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1178] Step 1:

[1179] A user wears smart glasses and picks up fresh produce in a brick-and-mortar store. The smart glasses' camera is used to capture an image of the food, where the input is the physical appearance of the fresh produce and the output is digital image data. This digital image data is sent to the system for analysis.

[1180] Step 2:

[1181] The terminal (smart glasses) sends the captured image data to a server via the Internet. The input is the digital image data acquired in step 1, and the output is the image data transferred to the server. A network communication protocol is used for data transfer.

[1182] Step 3:

[1183] The server preprocesses the received image data by resizing, noise filtering, normalizing, and converting it into a format suitable for analysis. The input is the image data of the fresh food sent to the server, and the output is the preprocessed image data.

[1184] Step 4:

[1185] The server inputs the preprocessed image data into a neural network model to extract features such as the color, surface condition, and shape of the food, which then calculates a freshness score. The input is the preprocessed image data, and the output is the freshness score.

[1186] Step 5:

[1187] The device (smart glasses) captures the user's facial expressions and voice in real time. It uses the camera and microphone of the smart glasses to obtain facial image data and voice data. The input is the user's facial expressions and voice, and the output is facial image data and voice data.

[1188] Step 6:

[1189] The server analyzes the received facial image data and voice data to recognize the user's emotional state. Using facial recognition and voice analysis technologies, it determines whether the user's emotion is positive or negative. The input is facial image data and voice data, and the output is the user's emotional state.

[1190] Step 7:

[1191] The server performs an analysis by combining the calculated freshness score with the user's emotional state. If the emotional state is positive, it highlights the freshness score and generates feedback that increases the recommendation level. Conversely, if the emotional state is negative, it generates feedback that presents detailed analysis information and other options. The input is the freshness score and the emotional state, and the output is feedback information.

[1192] Step 8:

[1193] The server sends the generated feedback information to the terminal (smart glasses) via the Internet. The input is the feedback information, and the output is the feedback information displayed on the display of the smart glasses.

[1194] Step 9:

[1195] The user makes a purchasing decision on fresh food based on the feedback information displayed on the smart glasses display. The input is the feedback information, and the output is the purchase decision on fresh food.

[1196] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1197] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1198] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1199] [Fourth embodiment]

[1200] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1201] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1202] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1203] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1204] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1205] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1206] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1207] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1208] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1209] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1210] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1211] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1212] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1213] The system according to the present invention provides a series of functions for users to determine the freshness of perishable foods, and specific embodiments thereof will be described below.

[1214] First, the user launches the app on a smartphone or other device and takes a picture of the fresh food. For example, if the user is looking to buy grapes at a supermarket, they can take a picture of the grapes using the app's camera function.

[1215] The captured image is encoded by the device and sent to a server via the Internet, along with the image data, including user identification information and food category information (e.g., fruit or vegetable).

[1216] The server receives the transmitted image data and performs an analysis process. This analysis process incorporates image analysis technology using neural networks. Specifically, the server inputs the image data into a neural network model and analyzes specific features (e.g., color, surface condition, shape, etc.) to determine the freshness of the fresh food.

[1217] The neural network analyzes the color, surface condition, and shape of food and calculates a freshness score based on each feature. For example, in the case of grapes, information such as the color of the skin, the presence or absence of bloom on the surface, the color and firmness of the stalk are analyzed.

[1218] The server sends the resulting freshness score and details to the user's device, where the analysis results are returned in JSON or other data formats.

[1219] The device then displays the analysis results to the user. Specifically, the app displays results such as "These grapes are fresh. Freshness score: 80 / 100," allowing the user to make a purchasing decision based on this information.

[1220] As described above, the system according to the present invention allows users to easily determine the freshness of perishable foods, and allows users to shop at supermarkets and greengrocers with greater peace of mind.

[1221] As a concrete example, consider the case where a user purchases commercially available tomatoes. The user launches a dedicated app and takes a picture of the tomato. The image data is then sent to a server, which uses a neural network to analyze the color and surface condition of the tomato (wrinkles, glossiness, etc.). The analysis result is displayed on the user's device as "This tomato is fresh. Freshness score: 85 / 100." The user can use this information to select fresh tomatoes.

[1222] Thus, the system of the present invention is a powerful tool that helps users select fresh foods in their daily shopping.

[1223] The processing flow will be explained below.

[1224] Step 1:

[1225] The user launches a dedicated app on their device and takes a picture of the fresh food they want to purchase (e.g., grapes).The user then uses the app's camera function to capture a clear image of the food under appropriate lighting conditions.

[1226] Step 2:

[1227] The device encodes the captured image data. Specifically, it compresses the image into JPEG or PNG format to reduce the data size. The encoded image data is temporarily saved.

[1228] Step 3:

[1229] The device packages the encoded image data with user identification information and food category information, and sends the packaged data to the server via the Internet. The image data and accompanying information are sent using an HTTP POST request.

[1230] Step 4:

[1231] The server receives the image data sent from the device. The server analyzes the received data and stores the image data, user identification information, food category information, etc. in the respective variables.

[1232] Step 5:

[1233] The server preprocesses the image data stored on the server, specifically by resizing the image, filtering noise, normalizing it, etc., to convert it into a form suitable for image analysis.

[1234] Step 6:

[1235] The server inputs the preprocessed image data into an AI analysis module, where a neural network analyzes the food's color, surface condition, shape, and other characteristics to extract specific features.

[1236] Step 7:

[1237] The AI ​​analysis module determines the freshness of perishable foods based on the extracted features. For example, in the case of grapes, the color of the skin, the presence or absence of bloom, and the color of the branch are indicators of freshness. These features are combined to calculate a freshness score.

[1238] Step 8:

[1239] The server packages the freshness score and detailed analysis results in JSON format or similar and sends them to the device. The data is returned using an HTTP response.

[1240] Step 9:

[1241] The device decodes the analysis results it receives and displays them on the user interface. Specifically, it displays a message such as "These grapes are fresh. Freshness score: 80 / 100" to provide the user with a visual result.

[1242] Step 10:

[1243] Users can then make a purchasing decision based on the displayed freshness score, or continue analyzing other fresh foods using other features within the app.

[1244] Example 1

[1245] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1246] When purchasing fresh food, determining its freshness is extremely important and presents a major challenge for consumers. However, it can be difficult to accurately judge freshness using visual information alone. This challenge is particularly severe for consumers who lack specialized knowledge or experience in assessing freshness. Currently, there is no reliable method for determining freshness, so there is a need to eliminate the uncertainty consumers have when selecting high-quality fresh food.

[1247] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1248] In this invention, the server includes a means for determining the freshness of the food using image analysis technology, a means for the server to analyze specific features of the food using a neural network model and calculate a freshness score, and a means for transmitting the freshness score to a user's device in JSON format, thereby enabling users to easily determine the freshness of fresh food and make purchasing decisions based on reliable information.

[1249] "User" refers to the consumer who uses the dedicated app to operate the system and determine the freshness of perishables.

[1250] "Terminal" refers to an electronic device used by a user, such as a smartphone or tablet, that acquires images and communicates with a server.

[1251] "Image acquisition means" refers to a function for taking images of fresh food using the camera function of the terminal.

[1252] "Transmission means" refers to a function for transmitting image data and additional information acquired from a terminal to a server via the Internet.

[1253] "Server" refers to a computer system that processes received image data using image analysis techniques to determine freshness.

[1254] "Image analysis technology" refers to technology that uses software algorithms running on a server to extract specific features from images of fresh food and determine its freshness.

[1255] A "neural network model" is a model that uses artificial intelligence technology that runs on a server and is used to analyze food characteristics such as color, surface condition, and shape.

[1256] "Freshness score" refers to a numerical indicator of the freshness of fresh food, calculated based on image analysis technology and neural network models.

[1257] "JSON format" is a type of data format used when sending analysis results and freshness scores to devices, and refers to a lightweight, structured data exchange format.

[1258] "Display means" refers to a function for visually showing the freshness score and analysis results received on the terminal to the user.

[1259] The present invention relates to a system for users to determine the freshness of fresh food. Implementing this system involves a series of processes: acquiring an image of the food based on user operations, transmitting the image to a server via the Internet, and the server using image analysis technology to determine the freshness and providing the analysis results to the user.

[1260] Hardware and software used

[1261] 1. The device used by the user is an electronic device such as a smartphone or tablet. Specific examples include an iPhone 12 or an Android-based device. A dedicated app (e.g., FreshCheck App) is installed on the device, and by launching this app, the user can use the camera function to take a picture of the food.

[1262] 2. The device encodes the captured image in JPEG format, adds user identification information and food category information (e.g., fruit, vegetable), and sends it to the server via HTTPS protocol.

[1263] 3. The server is a high-performance computer system equipped with software that performs image analysis techniques, specifically a neural network model using the TensorFlow library, which analyzes specific features of food, such as color, surface condition, and shape, to calculate a freshness score.

[1264] Example of operation

[1265] While selecting tomatoes to purchase at the supermarket, a user launches a dedicated app and takes a picture of the tomato. The device encodes this image into JPEG format, adds the information "User ID: 67890" and "Food category: Vegetables", and sends it to the server. When the server receives the image data, it inputs the image into a neural network model using TensorFlow, which analyzes the color, surface condition, etc. of the tomato. The freshness score calculated as a result of the analysis (e.g., 85 / 100) is formatted in JSON format and sent to the user's device. The device then displays the received result on the app interface as "This tomato is fresh. Freshness score: 85 / 100." The user can use this information to select fresh tomatoes.

[1266] Prompt Sentence Examples

[1267] An example of a prompt in text format is shown below.

[1268] The user launches the dedicated app "FreshCheck App" on their smartphone and takes a picture of the tomato they plan to purchase. The image data is sent to the server, which uses TensorFlow to analyze the color and surface condition. The analysis result is displayed on the user's device as "This tomato is fresh. Freshness score: 85 / 100." Based on this information, the user can select fresh tomatoes.

[1269] Thus, the system according to the present invention is a powerful tool that helps users to select fresh foods in their daily shopping.

[1270] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1271] Processing Steps

[1272] Step 1:

[1273] The user launches a dedicated app and takes a picture of the fresh food they plan to purchase.

[1274] Specific operation: The user launches the "FreshCheck App" on their smartphone and takes a picture of a tomato using the app's camera function.

[1275] Input: A camera activated by user action, an image of a tomato

[1276] Output: Image data of the photographed tomato

[1277] Step 2:

[1278] The device encodes the captured image data and sends it to a server via the Internet.

[1279] Specific operation: The device encodes the captured image into JPEG format, and then adds user identification information (e.g., UserID: 67890) and food category information (e.g., vegetables).

[1280] Input: Tomato image, user identification information, food category information

[1281] Output: Encoded JPEG image data and additional information

[1282] Step 3:

[1283] The terminal transmits the encoded image data and additional information to the server using the HTTPS protocol.

[1284] Specific operation: The terminal sends the encoded image data and additional information to the server as an HTTPS request.

[1285] Input: JPEG image data and additional information

[1286] Output: Image data and additional information are sent to the server as an HTTPS request.

[1287] Step 4:

[1288] The server performs image analysis using a neural network based on the image data and additional information received.

[1289] Specific operation: The server uses the TensorFlow library to input image data into a neural network model and analyzes features such as the color and surface condition of the tomato.

[1290] Input: Received JPEG image data and additional information

[1291] Output: Analyzed feature data (e.g., RGB values ​​of color, edge information of surface condition)

[1292] Step 5:

[1293] The server calculates a freshness score based on the analysis results and sends it to the user's device in JSON format.

[1294] Specific operation: The server calculates a freshness score (e.g., 85 / 100) based on the analyzed feature data and formats the result in JSON format.

[1295] Input: Analyzed feature data

[1296] Output: JSON format data containing the freshness score.

[1297] Step 6:

[1298] The server sends the parsed results in JSON format to the user's device.

[1299] Specific operation: The server sends the analysis result in JSON format, including the freshness score, to the user's device as an HTTPS response.

[1300] Input: JSON formatted data containing freshness scores

[1301] Output: Analysis results are sent to the device as an HTTPS response

[1302] Step 7:

[1303] The terminal displays the analysis results received to the user.

[1304] Specific operation: The device reads the JSON format analysis results received from the server and displays "This tomato is fresh. Freshness score: 85 / 100" on the app interface.

[1305] Input: Parsed result of received JSON format

[1306] Output: Freshness score and message displayed in the app

[1307] This flow of steps allows users to easily determine the freshness of perishable foods and make purchasing decisions based on reliable information.

[1308] (Application example 1)

[1309] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1310] Existing systems lack the means to not only determine the freshness of fresh food, but also to confirm its safety and quality. Furthermore, they do not meet the needs of users who want to detect counterfeit food and illegally distributed products. To solve this problem, a system is needed that can not only determine the freshness and quality of food, but also verify its authenticity using security tags.

[1311] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1312] In this invention, the server includes means for using image analysis technology to determine the freshness and quality of the item, means for verifying the security tag, and means for providing the results of the analysis and verification to the user, thereby enabling the user to verify not only the freshness and quality of the food they are purchasing, but also whether the food is authentic.

[1313] "Item" refers to a specific product or item that is the subject of analysis using a photograph or image.

[1314] "Freshness" is an index that indicates the degree of freshness that an item maintains and the quality of its preservation.

[1315] "Quality" refers to the characteristics and condition that make up the overall evaluation of a product, including its value and safety to the consumer.

[1316] "Image analysis technology" is a technology for analyzing the characteristics of an item based on a photograph or image to determine its freshness and quality.

[1317] A "security tag" is identification information, such as a QR code or barcode, attached to an item to prove its authenticity.

[1318] "User" means any individual or entity that uses the system to verify the freshness, quality, and authenticity of goods.

[1319] "Via the Internet" refers to a method of using public lines to send and receive data between a user's device and a server.

[1320] A "server" is a computer system that receives data sent from a user, analyzes it, and returns the results.

[1321] "Transmitting" refers to the act of sending images and related information from a user's device to a server via telecommunications means.

[1322] "Providing" refers to the act of presenting the results analyzed and confirmed by the server in the form of a display on the user's device.

[1323] System Program

[1324] The system program for this application example begins with a user taking an image of an item using a device such as a smartphone, smart glasses, or head-mounted display and sending it to a server via the Internet. The server analyzes the received image data, determines its freshness and quality, and then checks the security tag based on that information. The results of the analysis and check are then provided back to the user's device, and the user can use that information to determine the item's freshness and authenticity.

[1325] Natural language explanation of the process

[1326] This system is realized using the following hardware and software.

[1327] The hardware used includes devices such as smartphones, smart glasses, and head-mounted displays, and the software used is Python, OpenCV, Keras (a library for neural networks), and requests (a library for HTTP communication).

[1328] 1. Image capture: The user takes an image of an object using the device's camera, which can be done through an interface such as a smartphone, smart glasses, or a head-mounted display.

[1329] 2. Data transmission: The acquired image data is encoded by the device and sent to the server using the HTTP protocol. At this time, user identification information, item category information, and other information are sent along with the image data.

[1330] 3. Image analysis: The server inputs the received image data into a neural network model to analyze specific features (e.g., color, surface condition, shape, etc.) to determine freshness and quality. This uses a neural network built using Python and Keras.

[1331] 4. Security Verification: The server detects and analyzes the item's security tag (e.g., QR code, barcode, etc.) to verify that the item is authentic. In this step, an HTTP request is made to an external security tag verification service.

[1332] 5. Providing results: The analysis and verification results are returned to the user's device in a data format such as JSON. The analysis results are then visually displayed on the user's device. For example, "This item is fresh. Freshness score: 80 / 100. This item is authentic."

[1333] Specific examples

[1334] For example, if a user is looking to buy grapes at a supermarket, they launch a dedicated app and take a picture of the grapes. The captured image is sent to a server, where a neural network is used to analyze the grapes' color, surface condition, and branch condition to calculate a freshness score. At the same time, the QR code attached to the grapes' packaging is scanned to confirm their authenticity. The result is a message displayed on the smartphone saying, "These grapes are fresh. Freshness score: 85 / 100. These grapes are authentic."

[1335] Example prompts to be input to the generative AI model

[1336] "Analyze the features of fresh produce, such as color, surface condition, and shape, to calculate a freshness score. Also, use security tags to verify whether the food is authentic."

[1337] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1338] Step 1:

[1339] Image acquisition

[1340] input:

[1341] The user activates the camera on their smart device and takes a picture of an item (e.g., grapes).

[1342] Specific behavior:

[1343] The user activates the camera function on a smartphone or smart glasses interface and takes a picture of the item, which generates a high-resolution image of the item.

[1344] output:

[1345] Acquired item image data.

[1346] Step 2:

[1347] Sending data

[1348] input:

[1349] The item image data, user identification information, and item category information obtained in step 1.

[1350] Specific behavior:

[1351] The terminal transmits the acquired image data, user identification information (e.g., user ID, store ID), and item category information (e.g., fruit, vegetable) to the server as an HTTP request.

[1352] output:

[1353] Image data and related information sent to the server.

[1354] Step 3:

[1355] Image analysis

[1356] input:

[1357] The product image data and related information are sent to the server.

[1358] Specific behavior:

[1359] The server inputs the received image data into a neural network model built using Python and Keras to analyze features such as color, surface condition, and shape. For example, in the case of grapes, the skin color and surface gloss are analyzed. The neural network evaluates these features using a trained generative AI model and calculates a freshness score.

[1360] output:

[1361] Parsed freshness score and other feature data.

[1362] Step 4:

[1363] Check the security tag

[1364] input:

[1365] Item image data and related information obtained after analysis.

[1366] Specific behavior:

[1367] The server detects the security tag attached to the item (e.g., QR code or barcode) from the image and uses it to query an external security tag verification service. For example, it scans the QR code, extracts its content, and checks whether the item is authentic based on that content.

[1368] output:

[1369] Security tag verification results (genuine / non-genuine).

[1370] Step 5:

[1371] Providing analysis and verification results

[1372] input:

[1373] Freshness score and security tag validation results.

[1374] Specific behavior:

[1375] The server compiles the analysis results and security tag verification results in a data format such as JSON and returns them to the user's device. At that time, to make the results easier to understand, it generates a specific message such as "This item is fresh. Freshness score: 85 / 100. This item is authentic."

[1376] output:

[1377] Analysis and verification results are displayed on the user's device.

[1378] Step 6:

[1379] Displaying the results

[1380] input:

[1381] Analysis and verification result data returned from the server.

[1382] Specific behavior:

[1383] The device analyzes the received data and visually displays it on the user interface. For example, a message such as "These grapes are fresh. Freshness score: 85 / 100. These grapes are authentic" could be displayed on the smartphone screen, making the results easy for the user to understand.

[1384] output:

[1385] User perception of the results and purchasing decisions based on them.

[1386] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1387] The system according to the present invention has the function of not only determining the freshness of perishable foods but also recognizing the user's emotions and providing information based on those emotions. Specific embodiments of the system will be described below.

[1388] First, a user launches a dedicated smartphone app and takes a picture of the fresh food. For example, if a user is looking to buy tomatoes at a supermarket, they can use the app's camera to take a picture of the tomato. The captured image data is encoded by the device and sent to a server via the Internet.

[1389] The server receives the image data sent from the device and performs preprocessing, which includes image resizing, noise filtering, and normalization. The processed image is then input into a neural network to extract features such as the food's color, surface condition, and shape, and calculate a freshness score.

[1390] The newly added emotion engine feature uses facial recognition and voice analysis to determine a user's current emotional state. For example, if a user smiles into the smartphone camera or vocally expresses joy, that emotional data is collected.

[1391] The server analyzes the detected freshness score by combining it with the user's emotional data. If the emotional data is positive, it provides the user with a result that is more likely to be recommended. Conversely, if negative emotions are detected, it provides the user with appropriate feedback, such as detailed analysis information or other options.

[1392] For example, if a user is checking the freshness of a tomato and the emotion engine detects that the user is smiling, it will highlight the high freshness score. If the user looks dissatisfied, it will suggest, "Try looking for a fresher tomato."

[1393] The analysis results and emotional feedback are sent to the device and displayed visually within the app, allowing users to make informed purchasing decisions about fresh food.

[1394] Furthermore, the emotion engine can learn a user's emotional patterns based on their purchasing history and use them in future judgments. For example, if past data shows that a user tends to prefer foods of a certain color or shape, the engine can adjust the freshness judgment results based on that tendency and provide more personalized information.

[1395] In this way, the system of the present invention aims to provide users with a more appropriate and satisfying purchasing experience by integrating freshness determination and emotion recognition, thereby improving users' constant satisfaction and providing highly accurate freshness determination.

[1396] The processing flow will be explained below.

[1397] Step 1:

[1398] The user launches a dedicated app on their device and takes a picture of the fresh food (e.g., a tomato) they are considering purchasing. The user then uses the app's camera function to capture a clear image of the food under appropriate lighting.

[1399] Step 2:

[1400] The device encodes the captured image data. Specifically, it compresses the image into JPEG or PNG format to reduce the data size. The encoded image data is temporarily saved.

[1401] Step 3:

[1402] The device packages the encoded image data with user identification information and food category information, and sends the packaged data to the server via the Internet. The image data and accompanying information are sent using an HTTP POST request.

[1403] Step 4:

[1404] The server receives the image data sent from the device. The server analyzes the received data and stores the image data, user identification information, food category information, etc. in the respective variables.

[1405] Step 5:

[1406] The server preprocesses the image data stored on the server, specifically by resizing the image, filtering noise, normalizing it, etc., to convert it into a form suitable for image analysis.

[1407] Step 6:

[1408] The server inputs the preprocessed image data into an AI analysis module, where a neural network analyzes the food's color, surface condition, shape, and other characteristics to extract specific features.

[1409] Step 7:

[1410] The AI ​​analysis module determines the freshness of fresh food based on the extracted features. For example, in the case of tomatoes, the color of the skin and the presence or absence of blemishes or scratches on the surface are indicators of freshness. These features are combined to calculate a freshness score.

[1411] Step 8:

[1412] The device captures the user's face and voice in real time and sends them to the emotion engine, which then uses facial recognition and voice analysis to determine the user's emotional state. Emotional data is generated, such as whether the user is smiling, angry, or sad.

[1413] Step 9:

[1414] The server receives the emotion data from the emotion engine and makes a comprehensive judgment based on the freshness score. If the emotion data is positive, it returns a highly recommended result, and if it is negative, it makes adjustments such as presenting a detailed analysis.

[1415] Step 10:

[1416] The server packages the final analysis results in JSON format or similar and sends them to the terminal, returning the data using an HTTP response.

[1417] Step 11:

[1418] The device decodes the analysis results it receives and displays them on the user interface. Specifically, it displays a message such as "This tomato is fresh. Freshness score: 85 / 100" so that the user can visually confirm the results.

[1419] Step 12:

[1420] Users make purchasing decisions based on the displayed freshness score and their own emotional feedback. Users can also use other features within the app to continue analyzing other fresh foods.

[1421] Step 13:

[1422] The emotion engine learns the user's purchasing history and emotional tendencies, and reflects this in future analyses. Based on past data, a more personalized freshness assessment is made.

[1423] Example 2

[1424] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1425] In recent years, technology for evaluating the quality and freshness of fresh food has advanced, but systems that provide feedback that takes user emotions into account are still limited. Furthermore, conventional systems are limited to determining freshness through image analysis, making it difficult to make personalized recommendations based on the user's emotions and preferences. This makes it difficult to improve the user experience and support satisfying purchasing decisions.

[1426] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1427] In this invention, the server includes means for determining the freshness of the food using image analysis technology, means for performing face recognition and voice analysis to recognize the user's emotions, and means for performing analysis based on the determination results and emotion data and providing information to the user. This enables analysis that integrates freshness determination and user emotions, making it possible to provide a more appropriate and satisfying purchasing experience.

[1428] "User" refers to any individual or legal entity that uses the System.

[1429] An "operation" refers to an action taken by a user to achieve a specific goal using the system.

[1430] "Image" refers to visual data captured by a user using a device's camera or the like.

[1431] "Internet" refers to the technology that connects computers and databases around the world via information and communications networks.

[1432] "Server" refers to a set of hardware or software that receives requests and processes data.

[1433] "Image analysis technology" refers to software and algorithms for extracting specific information from image data.

[1434] "Freshness" refers to an indication of the appropriate quality and state of preservation of an item such as food.

[1435] "Facial recognition" refers to the technology that detects and recognizes the faces of people captured on camera.

[1436] "Voice analysis" refers to the technology of analyzing voice data and extracting information.

[1437] "Determination result" refers to a conclusion reached using image analysis techniques or other analytical methods.

[1438] "Emotional data" refers to the results of collecting and analyzing data that indicates the emotional state of a user.

[1439] "Analysis" refers to the process of deriving information and patterns from collected data.

[1440] "Providing information" refers to the act of communicating analysis results and recommendations to users.

[1441] "Visually displaying" refers to the act of graphically showing analytical results or information on a computer screen or device display.

[1442] "Goods" refers to a broad definition of the subject matter, including food and other subject items.

[1443] "Feedback" refers to the analysis results and recommendations provided to the user.

[1444] A "machine learning model" refers to an algorithm that learns patterns from data and makes predictions or classifications for new data.

[1445] The system of the present invention not only determines the freshness of perishable food, but also has the ability to recognize the user's emotions and provide appropriate feedback based on them.

[1446] Hardware and Software Configuration

[1447] The implementation of this system utilizes the following hardware and software:

[1448] Hardware:

[1449] Smartphone or tablet: Use the camera function to take a picture.

[1450] Server: Receives data, analyzes it, and generates feedback.

[1451] software:

[1452] Dedicated app: An application that provides an interface for users to take images and input emotions.

[1453] Image analysis software: Use libraries such as OpenCV to preprocess images.

[1454] Machine learning model: We use TensorFlow and PyTorch to build a neural network to extract food features and calculate a freshness score.

[1455] Emotion Recognition Software: Analyze user emotions using Emotion API or our own emotion recognition model.

[1456] Specific functions of the system

[1457] 1. Image capture and transmission:

[1458] Users launch a dedicated smartphone app and take a picture of the fresh food. The captured image data is encoded by the device and sent to a server via the Internet.

[1459] 2. Image preprocessing:

[1460] The server receives image data sent from the device and performs preprocessing such as resizing, noise filtering, and normalization, using an image processing library such as OpenCV.

[1461] 3. Feature extraction and freshness determination:

[1462] The preprocessed images are input into a neural network to extract features such as food color, surface condition, and shape, using a machine learning model (for example, TensorFlow or PyTorch).

[1463] 4. Emotion recognition:

[1464] When a user smiles at the camera or expresses emotion through voice, the device sends the data to the server, which then performs facial recognition and voice analysis to determine the user's emotional state, using the Emotion API or a proprietary emotion recognition model.

[1465] 5. Data Integration and Analysis:

[1466] The server integrates the freshness score and the user's emotional data and performs an analysis based on the results. If the emotional data is positive, the server provides positive feedback, and if it is negative, the server presents detailed analysis information or other options to the user.

[1467] 6. Providing Feedback:

[1468] The analysis results and emotional feedback are sent to the device and displayed visually within a dedicated app, allowing users to make purchasing decisions about fresh food based on this information.

[1469] 7. Learning Emotional Patterns:

[1470] The emotion engine learns the user's emotional patterns based on their purchasing behavior history and uses them to make decisions for future purchases. It also extracts the user's preferences from past data and provides personalized information.

[1471] Specific examples

[1472] For example, if a user is checking the freshness of tomatoes at a supermarket and the emotion engine detects that the user is smiling, it will highlight the high freshness score. If the user looks dissatisfied, it will suggest, "Try looking for a fresher tomato."

[1473] Example prompts for generative AI models

[1474] "I want to determine the freshness of a particular food item from an image. Can you please tell me some concrete ways to build a system that recognizes the user's emotions and provides feedback based on their reaction?"

[1475] By integrating freshness assessment and emotion recognition, this system aims to provide users with a more relevant and satisfying purchasing experience.

[1476] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1477] Step 1:

[1478] The user launches the app and takes a picture

[1479] Input: Fresh produce the user wants to buy (e.g. tomatoes)

[1480] Output: Image data taken with a smartphone

[1481] How it works: The user launches a dedicated app on their smartphone and takes a picture of the fresh food. The app uses the camera function to capture a high-resolution image.

[1482] Step 2:

[1483] The device encodes and transmits the image data.

[1484] Input: Photographed image data

[1485] Output: Encoded image data (e.g. JPEG, PNG format)

[1486] What it does: The device encodes the captured image and sends it over the Internet to a server, using the HTTPS protocol for secure data transfer.

[1487] Step 3:

[1488] The server receives the image data and performs preprocessing.

[1489] Input: Encoded image data sent from the device

[1490] Output: Preprocessed image data

[1491] Specific operation: The server performs preprocessing on the received image data, such as resizing (e.g., changing to 224x224 pixels), noise filtering, and normalization. This processing uses an image processing library such as OpenCV.

[1492] Step 4:

[1493] The server extracts features using a neural network

[1494] Input: Preprocessed image data

[1495] Output: Extracted features

[1496] Specific operation: The server inputs the preprocessed images into a neural network (e.g., using TensorFlow or PyTorch) to extract features such as the color, surface condition, and shape of the food.

[1497] Step 5:

[1498] The server calculates a freshness score based on the features.

[1499] Input: extracted features

[1500] Output: Freshness score

[1501] How it works: The server uses the extracted features to calculate the freshness score of the fresh produce, using a pre-trained model.

[1502] Step 6:

[1503] The user expresses their emotions through the camera and microphone

[1504] Input: User's facial expressions and voice

[1505] Output: Facial image data and audio data

[1506] Specific actions: The user smiles into the smartphone camera or expresses an emotion through voice.

[1507] Step 7:

[1508] The device sends the facial image data and voice data to the server.

[1509] Input: Facial image data and audio data

[1510] Output: Facial image data and audio data sent from the device

[1511] Specific operation: The device encodes the collected facial image data and voice data and sends them to the server. The data is transferred securely using the HTTPS protocol.

[1512] Step 8:

[1513] The server performs emotion recognition

[1514] Input: Facial image data and audio data

[1515] Output: Emotion data

[1516] How it works: The server performs facial recognition and voice analysis to determine the user's current emotional state, using an emotion recognition model (e.g., Emotion API or a proprietary emotion recognition model).

[1517] Step 9:

[1518] The server combines the freshness score and sentiment data for analysis.

[1519] Input: Freshness score and sentiment data

[1520] Output: Analysis results and feedback

[1521] How it works: The server combines the freshness score with the user's sentiment data and generates feedback based on the results. If the sentiment is positive, it provides positive feedback, and if it is negative, it provides detailed analysis information or other options.

[1522] Step 10:

[1523] The server sends the analysis results to the device.

[1524] Input: Analysis results and feedback

[1525] Output: Analysis results and feedback to the device

[1526] Specific operation: The server sends the analysis results and feedback to a dedicated app and then to the device for visual display.

[1527] Step 11:

[1528] The terminal displays the results visually.

[1529] Input: Analysis results and feedback sent from the server

[1530] Output: Analysis results and feedback displayed visually within a dedicated app

[1531] Specific operation: The device visually displays the analysis results and feedback within a dedicated app so that the user can check them.

[1532] Step 12:

[1533] The server learns emotional patterns

[1534] Input: User's past purchasing behavior history and emotional data

[1535] Output: Learned emotion patterns

[1536] Specific operation: The server uses an emotion engine to learn the user's emotional patterns based on their purchasing behavior history and utilizes them for future judgments. It also extracts the user's preferences from past data and optimizes freshness judgment and feedback based on them.

[1537] (Application example 2)

[1538] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1539] Conventional freshness assessment systems for fresh food simply assess freshness based on image data of the food, without providing feedback that takes into account the user's emotions or purchasing experience. As a result, even if a user confirms the freshness of the food, their satisfaction with the purchase may be low. Furthermore, because they are unable to provide detailed responses based on the user's emotions, it is difficult to influence the user's purchasing behavior.

[1540] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1541] In this invention, the server includes means for acquiring images of fresh food based on user operations, means for transmitting the images to the server via the Internet, means for determining the freshness of the fresh food using image analysis technology in the server, means for providing the results of the image analysis to the user, means for performing facial recognition and voice analysis to recognize the user's emotions, and means for adjusting the results of the image analysis based on the user's emotional state. This makes it possible to provide appropriate feedback that takes the user's emotions into consideration, thereby improving the user's purchasing experience and satisfaction.

[1542] "User operations" refers to a series of actions a user takes to use a system or application.

[1543] "Fresh food" refers to food for which freshness and quality are important and which has a limited shelf life.

[1544] "Means for acquiring an image" refers to a method for acquiring visual information as a digital image using a device such as a camera or scanner.

[1545] "Transmission via the Internet" refers to the technology that utilizes Internet protocols to transmit data to a remote server.

[1546] A "server" refers to a computer system that processes various data and provides services via a network.

[1547] "Image analysis technology" refers to methods and algorithms for extracting useful information from digital image data.

[1548] "Means for determining freshness" refers to methods for evaluating the freshness of fresh food based on various indicators and analytical results.

[1549] "Means of providing" refers to the method of notifying or displaying the analysis results or information to the user.

[1550] "Emotion recognition" refers to analyzing a person's facial expressions and voice data to determine their emotional state.

[1551] "Facial recognition" refers to the technology of identifying facial features in digital images or video streams to recognize and identify individuals.

[1552] "Voice analysis" refers to the technology of analyzing voice data and extracting information such as content, emotions, and speaker identification.

[1553] "Emotional state" refers to the user's mental and emotional state as indicated by the analyzed results.

[1554] "Adjustment means" refers to a method for changing or adjusting the analysis results or provided information according to the emotional state of the user.

[1555] "Feedback" refers to the reaction or information that a system provides to a user.

[1556] "Purchasing experience" refers to the series of experiences and emotions a user feels during the process of purchasing a product.

[1557] The system of the present invention aims to provide a more satisfying shopping experience by combining freshness assessment of fresh food in a physical store with user emotion recognition. This system is implemented using smart glasses.

[1558] First, a user wears smart glasses and makes a purchase in a physical store. The user picks up a fresh food item they are considering purchasing and takes a picture of it with the smart glasses' camera. The captured image is then captured as digital data by the glasses' built-in camera system. This captured image data is temporarily processed by a processor in the smart glasses and sent to a remote server via the Internet.

[1559] The server preprocesses the received image data and uses image analysis technology to determine the freshness of the fresh food. This preprocessing mainly includes image resizing, noise filtering, and normalization. The preprocessed image data is input into a neural network model to extract features such as the food's color, surface condition, and shape. This allows a freshness score to be calculated.

[1560] Next, a process to recognize the user's emotions is carried out. The camera and microphone attached to the smart glasses capture the user's facial expressions and voice in real time. The server receives this data and uses facial recognition and voice analysis technologies to determine the user's emotional state. Facial recognition reads emotions from the user's facial expressions, while voice analysis analyzes emotions from the user's tone and intonation of voice.

[1561] The server analyzes the freshness score by combining it with the user's emotional data. If the user is in a positive emotional state (e.g., smiling, happy voice), the system highlights the freshness score and provides feedback that increases the recommendation. Conversely, if a negative emotional state (e.g., dissatisfied face, flat voice) is detected, the system provides feedback such as detailed analysis information or other options.

[1562] The analysis results and emotional data feedback are then sent to the smart glasses via the internet and visually displayed on the screen, allowing users to make purchasing decisions about fresh food based on this information.

[1563] As a specific example, a user picks up a tomato in a supermarket and takes a picture of it with the smart glasses' camera. Image analysis determines that the captured image has a high freshness score, and the server feeds that score back to the user. At the same time, if the user smiles, the smart glasses' display will show "This product is fresh! Freshness score: 0.85." On the other hand, if the user looks dissatisfied, the display will show more detailed information, such as "Please try looking for a fresher product. Freshness score: 0.65."

[1564] Examples of prompts using generative AI models for emotion recognition and freshness determination include:

[1565] "A user picks up a tomato in a supermarket and uses smart glasses to check its freshness. If the user smiles, provide feedback highlighting a high freshness score. If the user looks dissatisfied, provide detailed analytics or alternative options."

[1566] As described above, the system of the present invention improves the user's purchasing experience and satisfaction by providing feedback to the user by integrating the freshness determination of perishable foods with the recognition of the user's emotions.

[1567] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1568] Step 1:

[1569] A user wears smart glasses and picks up fresh produce in a brick-and-mortar store. The smart glasses' camera is used to capture an image of the food, where the input is the physical appearance of the fresh produce and the output is digital image data. This digital image data is sent to the system for analysis.

[1570] Step 2:

[1571] The terminal (smart glasses) sends the captured image data to a server via the Internet. The input is the digital image data acquired in step 1, and the output is the image data transferred to the server. A network communication protocol is used for data transfer.

[1572] Step 3:

[1573] The server preprocesses the received image data by resizing, noise filtering, normalizing, and converting it into a format suitable for analysis. The input is the image data of the fresh food sent to the server, and the output is the preprocessed image data.

[1574] Step 4:

[1575] The server inputs the preprocessed image data into a neural network model to extract features such as the color, surface condition, and shape of the food, which then calculates a freshness score. The input is the preprocessed image data, and the output is the freshness score.

[1576] Step 5:

[1577] The device (smart glasses) captures the user's facial expressions and voice in real time. It uses the camera and microphone of the smart glasses to obtain facial image data and voice data. The input is the user's facial expressions and voice, and the output is facial image data and voice data.

[1578] Step 6:

[1579] The server analyzes the received facial image data and voice data to recognize the user's emotional state. Using facial recognition and voice analysis technologies, it determines whether the user's emotion is positive or negative. The input is facial image data and voice data, and the output is the user's emotional state.

[1580] Step 7:

[1581] The server performs an analysis by combining the calculated freshness score with the user's emotional state. If the emotional state is positive, it highlights the freshness score and generates feedback that increases the recommendation level. Conversely, if the emotional state is negative, it generates feedback that presents detailed analysis information and other options. The input is the freshness score and the emotional state, and the output is feedback information.

[1582] Step 8:

[1583] The server sends the generated feedback information to the terminal (smart glasses) via the Internet. The input is the feedback information, and the output is the feedback information displayed on the display of the smart glasses.

[1584] Step 9:

[1585] The user makes a purchasing decision on fresh food based on the feedback information displayed on the smart glasses display. The input is the feedback information, and the output is the purchase decision on fresh food.

[1586] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1587] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1588] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1589] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1590] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1591] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1592] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1593] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1594] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1595] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1596] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1597] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1598] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1599] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1600] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1601] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1602] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1603] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1604] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1605] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1606] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1607] The following is further disclosed regarding the above embodiment.

[1608] (Claim 1)

[1609] means for acquiring an image of food based on a user's operation;

[1610] means for transmitting the image to a server via the Internet;

[1611] a means for determining the freshness of the food using image analysis technology in the server;

[1612] A system including means for providing results of said image analysis to a user.

[1613] (Claim 2)

[1614] 10. The system of claim 1, further comprising means for using specific features to determine the freshness of food selected by a user.

[1615] (Claim 3)

[1616] 2. The system of claim 1, wherein the image analysis technology comprises means for analyzing the color, surface condition, and shape of food using a neural network.

[1617] "Example 1"

[1618] Claims

[1619] (Claim 1)

[1620] means for acquiring an image of food based on a user's operation;

[1621] means for transmitting the image to a server via the Internet;

[1622] a means for determining the freshness of the food using image analysis technology in the server;

[1623] means for providing a user with the results of said image analysis;

[1624] a means for the server to analyze specific features of the food using a neural network model and calculate a freshness score;

[1625] means for transmitting the freshness score in JSON format to a user's terminal;

[1626] means for displaying the received freshness score to a user;

[1627] A system including:

[1628] (Claim 2)

[1629] 10. The system of claim 1, wherein the neural network model comprises means for analyzing the color, surface condition, and shape of the food product.

[1630] (Claim 3)

[1631] 10. The system of claim 1, further comprising means for transmitting user identification information and food category information to the server.

[1632] "Application Example 1"

[1633] (Claim 1)

[1634] means for acquiring an image of an article based on a user's operation;

[1635] means for transmitting the image to a server via the Internet;

[1636] means, in the server, for determining the freshness and quality of said items using image analysis techniques;

[1637] A means for verifying the security tag;

[1638] and means for providing results of said image analysis and security tag verification to a user.

[1639] (Claim 2)

[1640] 10. The system of claim 1, further comprising means for using specific features to determine the freshness and quality of an item selected by a user.

[1641] (Claim 3)

[1642] 10. The system of claim 1, wherein the image analysis technology comprises means for analyzing the color, surface condition, and shape of the object using a neural network.

[1643] "Example 2: Combining Emotion Engines"

[1644] (Claim 1)

[1645] means for acquiring an image based on a user's operation;

[1646] means for transmitting the image to a server via the Internet;

[1647] a means for determining the freshness of the food using image analysis technology in the server;

[1648] means for performing facial recognition and voice analysis to recognize the user's emotions;

[1649] means for performing analysis based on the determination result and emotion data and providing information to a user;

[1650] The system includes means for visually displaying the analysis results and emotion data to a user.

[1651] (Claim 2)

[1652] 2. The system according to claim 1, further comprising means for using the feature to determine the freshness of an item selected by a user, and for providing feedback based on the result of the determination of the item.

[1653] (Claim 3)

[1654] 10. The system of claim 1, wherein the image analysis technology comprises means for analyzing the color, surface condition, and shape of an object using machine learning models.

[1655] "Application example 2 when combining emotion engines"

[1656] (Claim 1)

[1657] means for acquiring an image of fresh food based on a user's operation;

[1658] means for transmitting the image to a server via the Internet;

[1659] a means for determining the freshness of the perishable food using image analysis technology in the server;

[1660] means for providing a user with the results of said image analysis;

[1661] means for performing facial recognition and voice analysis to recognize the user's emotions;

[1662] A system including means for adjusting the results of said image analysis based on the emotional state of a user.

[1663] (Claim 2)

[1664] 2. The system according to claim 1, further comprising means for using specific features to determine the freshness of fresh food selected by a user.

[1665] (Claim 3)

[1666] 2. The system of claim 1, wherein the image analysis technology comprises means for analyzing the color, surface condition, and shape of fresh food using a neural network. [Explanation of symbols]

[1667] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for acquiring an image of food based on a user's operation; means for transmitting the image to a server via the Internet; a means for determining the freshness of the food using image analysis technology in the server; A system including means for providing results of said image analysis to a user.

2. 10. The system of claim 1, further comprising means for using specific features to determine the freshness of a food item selected by a user.

3. 2. The system of claim 1, wherein the image analysis technology comprises means for analyzing the color, surface condition, and shape of the food using a neural network.

Images