Food material identification method and device, intelligent refrigerator, electronic equipment and storage medium

Abstract

The invention provides a food material identification method and device, an intelligent refrigerator, electronic equipment and a storage medium, and the method comprises the steps: obtaining target environment information of a food material storage region, and obtaining target model parameters matched with the target environment information; according to the target model parameters and the target environment information, adjusting a preset basic model to obtain a target model; and acquiring target image data collected for the target food material, and identifying the target image data according to the target model so as to identify the target food material. According to the embodiment of the invention, the accuracy of food material identification can be improved by constructing the exclusive models of different users.

Description

Technical Field

[0001] This application belongs to the technical field of food identification, specifically relating to a method, device, smart refrigerator, electronic device, and storage medium for identifying food. Background Technology

[0002] In related technologies, image recognition can be used to identify the food items in a refrigerator to determine what items are stored there. However, current recognition methods have relatively poor accuracy in identifying food items. Summary of the Invention

[0003] In view of the above problems, a method, apparatus, smart refrigerator, electronic device, and storage medium for identifying food ingredients are proposed to overcome or at least partially solve the above problems, including: A method for identifying an ingredient, the method comprising: Obtain target environmental information of the food storage area, and obtain target model parameters that match the target environmental information; Based on the target model parameters and the target environment information, the preset basic model is adjusted to obtain the target model; Acquire target image data collected for the target ingredient, and identify the target image data according to the target model to identify the target ingredient.

[0004] In some embodiments, obtaining the target model parameters that match the target environment information includes: Obtain a preset knowledge base, which stores model parameters of different food identification models trained based on different environmental information; Target model parameters that match the target environment information are determined from the preset knowledge base.

[0005] In some embodiments, the method further includes: Receive user feedback on the food identification results; The target model is adjusted based on the feedback information.

[0006] In some embodiments, the target model is infused with an allergen attention mechanism; the step of identifying the target image data based on the target model to identify the target food ingredient includes: The target image data is input into the target model; the target model is used to identify the target image data to determine the ingredient information and allergen information of the target food.

[0007] In some embodiments, the method further includes: Acquire chemical characteristic information and optical character recognition characteristic information for the target food ingredient; Based on the food identification results output from the target image data, as well as the chemical feature information and the optical character recognition feature information, the allergen risk information of the target food is determined; Risk warnings are issued based on the allergen risk information.

[0008] In some embodiments, determining the allergen risk information of the target food ingredient based on the food ingredient recognition result output from the target image data, as well as the chemical feature information and the optical character recognition feature information, includes: Obtain preset weight information; Based on the preset weight information, the food ingredient identification results, the chemical feature information, and the optical character recognition feature information, the allergen risk information of the target food ingredient is determined; the allergen risk information includes a multimodal allergy risk score.

[0009] In some embodiments, the risk warning based on the allergen risk information includes: Based on the multimodal allergy risk score, an early warning strategy is determined; Based on the aforementioned early warning strategy, risk warnings are issued for the target food ingredients.

[0010] In some embodiments, the method further includes: To determine if there is a demand for the target ingredient; When it is determined that there is a demand for the target ingredient, the step of issuing a risk warning for the target ingredient according to the warning strategy is executed.

[0011] This application embodiment also provides a food ingredient identification device, the device comprising: The acquisition module is used to acquire target environmental information of the food storage area and acquire target model parameters that match the target environmental information. The adjustment module is used to adjust the preset basic model according to the target model parameters and the target environment information to obtain the target model; The recognition module is used to acquire target image data collected for the target ingredient, and to recognize the target image data according to the target model in order to identify the target ingredient.

[0012] In some embodiments, the acquisition module is used to acquire a preset knowledge base, which stores model parameters of different food identification models trained based on different environmental information; and to determine target model parameters that match the target environmental information from the preset knowledge base.

[0013] In some embodiments, the adjustment module is further configured to receive feedback from the user regarding the food ingredient recognition results; and adjust the target model based on the feedback.

[0014] In some embodiments, the target model is infused with an allergen attention mechanism; the identification module is used to input the target image data into the target model; the target model is used to identify the target image data to determine the food information and allergen information of the target food.

[0015] In some embodiments, the apparatus further includes: The early warning module is used to acquire chemical characteristic information and optical character recognition characteristic information of the target food ingredient; determine the allergen risk information of the target food ingredient based on the food ingredient recognition result output based on the target image data, as well as the chemical characteristic information and the optical character recognition characteristic information; and issue a risk warning based on the allergen risk information.

[0016] In some embodiments, the warning module is used to acquire preset weight information; and determine the allergen risk information of the target food ingredient based on the preset weight information, the food ingredient identification result, the chemical feature information, and the optical character recognition feature information; the allergen risk information includes a multimodal allergy risk score.

[0017] In some embodiments, the warning module is configured to determine a warning strategy based on the multimodal allergy risk score; and to issue a risk warning for the target food ingredient based on the warning strategy.

[0018] In some embodiments, the warning module is further configured to detect whether there is a demand for the target ingredient; when it is determined that there is a demand for the target ingredient, the module executes a step of issuing a risk warning for the target ingredient according to the warning strategy.

[0019] This application also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the above-described method for identifying food ingredients.

[0020] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for identifying food ingredients.

[0021] This application also provides a smart refrigerator, which may include the above-mentioned food identification device, and / or, the above-mentioned electronic device, and / or, the above-mentioned computer-readable storage medium, and / or, the above-mentioned food identification method.

[0022] The embodiments of this application have the following advantages: In this embodiment, target environmental information of the food storage area is obtained, and target model parameters matching the target environmental information are obtained. Based on the target model parameters and the target environmental information, a preset basic model is adjusted to obtain the target model. Target image data collected for the target food is obtained, and the target image data is recognized based on the target model to identify the target food. Through this embodiment, the accuracy of food identification can be improved by constructing exclusive models for different users. Attached Figure Description

[0023] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart illustrating the steps of a method for identifying food ingredients according to an embodiment of this application; Figure 2 This is a flowchart illustrating the steps of another method for identifying food ingredients according to an embodiment of this application; Figure 3 This is a diagram illustrating the system top-level architecture and data flow of an embodiment of this application; Figure 4 This is a flowchart of the generative meta-learning adaptive recognition system according to an embodiment of this application; Figure 5 This is the architecture of the generative meta-learning engine in the embodiments of this application; Figure 6 This is a flowchart of the allergen deep integration optimization process according to an embodiment of this application; Figure 7 This is a schematic diagram of the protection decision tree according to an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a food ingredient identification device according to an embodiment of this application. Detailed Implementation

[0024] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0025] In related technologies, food identification solutions all employ general deep learning techniques, failing to consider the influence of factors such as the different home environments of users, leading to low accuracy in food identification. Therefore, this application provides a food identification method that improves accuracy by constructing user-specific models. For details, please refer to... Figure 1 , Figure 1 A flowchart illustrating the steps of a method for identifying food ingredients according to an embodiment of this application is shown.

[0026] like Figure 1 As shown, the method for identifying this ingredient may include the following steps: Step 101: Obtain the target environment information of the food storage area and obtain the target model parameters that match the target environment information.

[0027] In this embodiment of the application, target environmental information of the food storage area used for storing food can be obtained first; the target environmental information may include the brightness, humidity, temperature, etc. of the food storage area, and may also include the packaging, placement angle, etc. of the food in the food storage area.

[0028] The food storage area can be a region within a food storage device; the food storage device can be a smart refrigerator or other device capable of storing food.

[0029] In some embodiments, sensors may be deployed in the food storage device to acquire target environmental information of the food storage area.

[0030] After obtaining the target environment information, target model parameters matching the target environment information can be acquired. For example, these target model parameters can be pre-set model parameters for a model that identifies food ingredients in scenarios with similar environmental information to the target environment information. In this embodiment, different model parameters are set for different environmental information, and environmental information is comprehensively considered when identifying food ingredients; by considering the impact of different environments on food ingredient identification, this application can improve the accuracy of the model in identifying food ingredients.

[0031] Step 102: Adjust the preset basic model according to the target model parameters and target environment information to obtain the target model.

[0032] After determining the target model parameters, the preset base model can be adjusted according to the target model parameters to obtain a user-specific model, namely the target model.

[0033] For example, if the target environment information completely matches an environment, the model parameters corresponding to that environment information can be directly obtained as the target model parameters, and the preset basic model can be adjusted based on the target model parameters to obtain the target model.

[0034] In another example, if the target environment information partially matches an environment information, the preset base model can be adjusted based on the obtained target model parameters and target environment information to obtain the target model.

[0035] Step 103: Acquire target image data for the target ingredient, and identify the target image data according to the target model to identify the target ingredient.

[0036] After obtaining the target model, it can be stored; after the target ingredient is placed in the ingredient storage area, the target image data collected for the target ingredient can be obtained first.

[0037] For example, the target ingredient can be food that is about to be placed in the ingredient storage area, or it can be food that has already been placed in the ingredient storage area. This application embodiment does not limit this.

[0038] After obtaining the target image data, the target image data can be input into the target model; the target model can recognize it, thereby completing the recognition of the target food ingredient and outputting the food ingredient recognition result; the food ingredient recognition result may include the name of the target food ingredient, as well as the freshness of the target food ingredient, the quantity of the target food ingredient, etc., which are not limited in this embodiment of the application.

[0039] In this embodiment, target environmental information of the food storage area is obtained, and target model parameters matching the target environmental information are obtained. Based on the target model parameters and the target environmental information, a preset basic model is adjusted to obtain the target model. Target image data collected for the target food is obtained, and the target image data is recognized based on the target model to identify the target food. Through this embodiment, the accuracy of food identification can be improved by constructing exclusive models for different users.

[0040] Reference Figure 2 The diagram illustrates a flowchart of another method for identifying food ingredients according to an embodiment of this application, which may include the following steps: Step 201: Obtain the target environmental information of the food storage area.

[0041] In this embodiment of the application, target environmental information of the food storage area used to store food ingredients can be obtained first; for example, sensors can be deployed in the food storage device to obtain target environmental information of the food storage area.

[0042] Step 202: Obtain a preset knowledge base, which stores model parameters of different food recognition models trained based on different environmental information.

[0043] In some embodiments, a pre-built preset knowledge base can be obtained; the preset knowledge base can store model parameters of different food identification models trained with different environmental information; in practical applications, by taking environmental information into account, the accuracy of food identification can be improved.

[0044] In practical applications, different food image datasets can be collected for model training to obtain model parameters for different food recognition models. The image data in these different food image datasets can correspond to different environmental information, such as different lighting, packaging, and placement angles. This application embodiment does not impose any limitations on this.

[0045] Step 203: Determine the target model parameters that match the target environment information from the preset knowledge base.

[0046] After obtaining the target environment information and the preset knowledge base, the matching degree between the target environment information and each environment information in the preset knowledge base can be calculated respectively, and the model parameters corresponding to the environment information with the highest matching degree with the target environment information in the preset knowledge base can be used as the target model parameters.

[0047] Step 204: Adjust the preset basic model according to the target model parameters and target environment information to obtain the target model.

[0048] After determining the target model parameters, the preset base model can be adjusted according to the target model parameters to obtain a user-specific model, namely the target model.

[0049] For example, if the target environment information completely matches an environment, the model parameters corresponding to that environment information can be directly obtained as the target model parameters, and the preset basic model can be adjusted based on the target model parameters to obtain the target model.

[0050] In another example, if the target environment information partially matches an environment information, the preset base model can be adjusted based on the obtained target model parameters and target environment information to obtain the target model.

[0051] Step 205: Acquire target image data collected for the target ingredient, and identify the target image data according to the target model to identify the target ingredient.

[0052] After obtaining the target model, it can be stored; after the target ingredient is placed in the ingredient storage area, the target image data collected for the target ingredient can be obtained first.

[0053] For example, the target ingredient can be food that is about to be placed in the ingredient storage area, or it can be food that has already been placed in the ingredient storage area. This application embodiment does not limit this.

[0054] In some embodiments of this application, the above method may further include the following steps: Receive user feedback on the food identification results; adjust the target model based on the feedback.

[0055] In some embodiments, after identifying the target ingredient, the ingredient identification result can be displayed to the user; after seeing the ingredient identification result, the user can provide feedback and input feedback information; the feedback information can be used to indicate whether the ingredient identification result is accurate.

[0056] Based on the feedback information, this application can adjust the target model; for example, when the feedback information indicates that the food identification result is incorrect, the target model can be adjusted to improve the accuracy of the target model identification; when the feedback information indicates that the food identification result is correct, the target model can be saved. This application does not limit this.

[0057] In some embodiments of this application, an allergen attention mechanism is injected into the target model; based on this, step 205 can be implemented in the following manner: The target image data is input into the target model; the target model is used to identify the target image data in order to determine the food information and allergen information of the target food.

[0058] In related technologies, the analysis of nutritional components and the provision of reasonable suggestions are mainly based on the results of food identification, or recipe recommendations and health risk assessments are provided, without detecting and identifying allergens in the food. Based on this, the embodiments of this application can inject an allergen attention mechanism into the target model so that the target model can output the allergen information of the target food when identifying the target food.

[0059] For example, after obtaining the target image data, the target image data can be input into the target model.

[0060] The target model can identify the target food based on the target image data to determine what kind of food it is.

[0061] The target model also learns the relationship between food ingredients and allergens. Based on this, after determining the target food ingredient, the model can also identify the allergens present in that ingredient, and then output both the food ingredient information and the allergen information. The food ingredient information can refer to the type, freshness, and quantity of the food ingredient; the allergen information can refer to the allergens that the food ingredient may contain.

[0062] In some embodiments, it can be first determined whether there are allergens in the household where the food storage device is located; if so, an allergen attention mechanism can be injected into the food corresponding to these allergens; that is, the target model will only output the corresponding allergen information when it recognizes these food ingredients; when other food ingredients are recognized, it will not output allergen information.

[0063] In other embodiments, the injection can also target all food ingredients that may cause allergies in humans; thus, as long as a food ingredient may contain allergens, the target model will output allergen information when it identifies such food ingredient. This application does not limit this.

[0064] In some embodiments of this application, based on the allergen attention mechanism, the above method may further include the following steps: Acquire chemical and optical character recognition information of the target food ingredient; determine the allergen risk information of the target food ingredient based on the food ingredient recognition results output from the target image data, as well as the chemical and optical character recognition information; and issue a risk warning based on the allergen risk information.

[0065] In some embodiments, the food storage device may also be equipped with a chemical sensor that can output chemical characteristic information of the target food. For example, the chemical sensor may be integrated with a 16-channel nano-gold electrode sensor that can detect characteristic peaks of volatile organic compounds (VOCs) and protein allergens (such as the mass spectrometry peak of peanut protein Ara h1 at 156.7 m / z).

[0066] In addition, food storage devices can also be equipped with a text recognition module based on the PaddleOCR engine, supporting multi-angle text extraction from packaging and featuring a built-in allergen keyword library (such as "peanut" and "gluten"). It can generate optical character recognition feature information from the text on the packaging of the target food ingredient.

[0067] After obtaining chemical and optical character recognition information, the food identification results output from the target image data, along with the chemical and optical character recognition information, can be combined to comprehensively determine the allergen risk information of the target food.

[0068] After identifying allergen risk information, risk warnings can be issued based on this information to prevent users from consuming potentially allergenic foods and developing allergies.

[0069] In some embodiments of this application, determining allergen risk information can be achieved through the following sub-steps: Sub-step 11: Obtain preset weight information.

[0070] In some embodiments, preset weight information can be set in advance for the food identification results, chemical feature information, and optical character recognition feature information, respectively.

[0071] Sub-step 12: Based on the preset weight information, as well as the food identification results, chemical feature information, and optical character recognition feature information, determine the allergen risk information of the target food; the allergen risk information includes a multimodal allergy risk score.

[0072] After obtaining the preset weight information, a multimodal allergy risk score for the target food ingredient that may contain allergens can be calculated based on the preset weight information, as well as the food ingredient recognition results, chemical feature information, and optical character recognition feature information. The higher the score, the more likely the target food ingredient is to cause allergies in users.

[0073] In some embodiments of this application, risk warning can be performed based on a multimodal allergy risk score through the following sub-steps: Sub-step 21: Determine the early warning strategy based on the multimodal allergy risk score.

[0074] In some embodiments, after determining the multimodal allergy risk score, a corresponding early warning strategy can be determined based on the multimodal allergy risk score; for example, corresponding early warning strategies can be pre-set for scores in different score ranges.

[0075] Sub-step 22: Based on the early warning strategy, issue risk warnings for the target ingredients.

[0076] After determining the early warning strategy, risk warnings can be issued for target ingredients based on the early warning strategy; for example, playing voice prompts, pushing alarm information, etc., to avoid users consuming ingredients that cause allergies. This application embodiment does not limit this.

[0077] In some embodiments of this application, the above method may further include the following steps: Detect whether there is a demand for the target ingredient; when it is determined that there is a demand for the target ingredient, execute the steps of issuing a risk warning for the target ingredient according to the warning strategy.

[0078] In some embodiments, it may be possible to first detect whether there is a demand for the target ingredient; for example, if it is detected that the ingredient storage device is opened, or the target ingredient in the ingredient storage device is taken out, or the user is staring at the target ingredient, it can be determined that there is a demand for the target ingredient.

[0079] If it is determined that there is a demand for the target ingredient, the step of issuing a risk warning for the target ingredient based on the warning strategy can be executed; otherwise, if there is no demand for the target ingredient, the step of issuing a risk warning for the target ingredient based on the warning strategy can be skipped. This application does not limit this.

[0080] In this embodiment, target environmental information of the food storage area is obtained; a preset knowledge base is obtained, which stores model parameters of different food recognition models trained based on different environmental information; target model parameters matching the target environmental information are determined from the preset knowledge base; the preset basic model is adjusted according to the target model parameters and the target environmental information to obtain the target model; target image data collected for the target food is obtained, and the target image data is recognized according to the target model to identify the target food. Through this embodiment, the accuracy of food recognition can be improved by constructing exclusive models for different users.

[0081] The following section uses a smart refrigerator system as an example, combined with... Figure 3 - Figure 7 The identification methods for the above-mentioned ingredients will be further explained below: This intelligent refrigerator system integrates user refrigerator video stream analysis, generative meta-learning adaptive large-scale models, and deep integration and optimization of allergen information. Through environmental adaptive feature generation, allergen perception decision-making, and a continuous evolution mechanism, the system achieves personalized identification of refrigerator food and dynamic protection against allergy risks. The system consists of three layers: (1) Perception layer: Information about ingredients is obtained through video streams, chemical sensors and text recognition (PaddleOCR); (2) Cognitive layer: Generative meta-learning engine, which combines user allergen information for adaptive learning, can output food identification results, which can include food information and allergen information; (3) Decision layer: Dynamic protection strategy based on reinforcement learning.

[0082] Figure 3 The system top-level architecture and data flow diagram of an embodiment of this application are shown below: Functional Positioning: The system's top-level framework. It outlines the macro-level blueprint for "where the data comes from, how decisions are made, and how the system evolves." Implementation Process: The multimodal perception layer is responsible for collecting data; after the data is sent to the generative learning engine for identification, it is processed by the allergen decision layer.

[0083] The dynamic protection strategies generated by the allergen decision-making layer are executed at the execution control layer, translating into specific reminders or notifications. Feedback from the user interaction layer can adjust the allergen decision-making layer.

[0084] User interactions are collected by the system as user feedback and used to update the meta-knowledge base, target model, and allergen decision layer, thereby achieving closed-loop evolution of the system.

[0085] In the generative meta-learning engine, environmental features and allergen features can be embedded. Based on information from various features (e.g., features corresponding to image data, chemical feature information, optical character recognition feature information), dynamic protection strategies (i.e., early warning strategies) can be generated. Figure 4 A flowchart of the generative meta-learning adaptive recognition system according to an embodiment of this application is shown: Functional positioning: Model personalization and optimization engine. Its core task is to "forge" an intelligent recognition model that can understand specific user environments and health risks.

[0086] Implementation process: The process begins with a strong underlying big model (such as CLIP-ViT).

[0087] The generative meta-learning engine, as the core processor, receives two key inputs: (1) Environmental features: features of environmental information extracted from the user's refrigerator environment. (2) User feedback information: corrections or confirmations by the user regarding historical food identification results.

[0088] The engine uses a meta-learning mechanism to inject environmental features into the base model and uses feedback to fine-tune it, dynamically generating a highly customized user-specific model, namely the target model.

[0089] This proprietary model no longer outputs ordinary visual features, but rather risk perception features that incorporate risk awareness, and delivers them directly to the allergen decision-making level.

[0090] Figure 5 The architecture of the generative meta-learning engine according to an embodiment of this application is shown: it consists of a series of functional modules that work together to quickly adapt a general base model into a user-specific model. The responsibilities and data interaction relationships of each module are as follows: 1. Environmental scanning module; Function: This module is deployed inside the refrigerator in the user's home and is responsible for actively collecting information about the refrigerator's environment, including lighting conditions, storage layout, and common packaging materials and styles.

[0091] 2. Feature encoder module; Function: Receives environmental information from the refrigerator module, and performs feature extraction and dimensionality reduction on it through a deep neural network.

[0092] Output: Generate a low-dimensional, digitized environmental signature (i.e., environmental feature vector) and send it to the meta-learning controller. This signature is a unique digital identifier for the home environment.

[0093] 3. The module of the meta-learning controller; function: the command center of the engine. It receives the environment signature sent by the feature encoder and performs two core operations based on the signature: (1) Knowledge retrieval: Initiates a query to the neural knowledge base to request the prior knowledge (i.e. model parameters) most similar to the current environment signature. (2) Instruction construction: After receiving the returned result, it integrates the environment signature and the retrieved knowledge to generate specific adaptation instructions and sends them to the generator.

[0094] 4. The Neural Knowledge Base section; Function: A pre-trained, massive database storing "environment signature - model parameter" pairs from numerous families.

[0095] Output: In response to a query from the meta-learning controller, return a set of the most similar parameter groups (i.e., model parameters) to the meta-learning controller.

[0096] 5. Generator; Function: Typically a conditional generative adversarial network. It receives adaptation instructions and environment signatures from the meta-learning controller.

[0097] Output: Based on the instructions and signature, generate enhanced food feature variants that conform to the characteristics of the current home environment for subsequent model fine-tuning.

[0098] 6. Base Model; Function: Refers to a general, pre-trained large-scale visual model (such as CLIP). It is the foundation and starting point of the algorithm.

[0099] Input / Processing: Receives feature variants from the generator and injects environmental signatures from the feature encoder through attention mechanisms and other methods. Simultaneously, it accepts meta-model parameters from the neural knowledge base for fusion and rapid fine-tuning.

[0100] Output: The final output is a customized model that is deeply optimized based on the current user's home environment, food preferences, and allergen information.

[0101] Figure 5 This paper describes a complete technology chain that starts with environmental scanning and feature encoding, uses a meta-learning controller to schedule a neural knowledge base and generator, and ultimately personalizes a basic large model to generate a custom model. Each module performs its specific function, forming an efficient and adaptive model personalization generation system.

[0102] Figure 6 A flowchart illustrating the allergen deep integration optimization process according to an embodiment of this application is shown: Functional positioning: The core algorithm process within the decision-making level. It is a... Figure 3 Detailed expansion and implementation of some parts.

[0103] Implementation process: It receives three streams of raw information from the perception layer: visual features, protein matching results from the chemical sensor, and keyword detection results from OCR.

[0104] These three pieces of information are then fused together with the user's allergy profile (after passing through a feature encoder).

[0105] The fused features are quantitatively evaluated to calculate a multimodal allergy risk score.

[0106] Ultimately, the protection decision tree selects and outputs the most appropriate dynamic protection strategy (such as different levels of early warning strategy alerts) based on this multimodal allergy risk score.

[0107] like Figure 7 The diagram below illustrates a protection decision tree according to an embodiment of this application: Based on the multimodal allergy risk score R, different layers are set: when R < 0.3, it is in the safety layer and no operation is performed; when 0.3 ≤ R < 0.6, it is in the warning layer, which corresponds to the warning strategies of "ambient light reminder" and / or "vibration warning"; when 0.6 ≤ R < 0.9, it is in the blocking layer, which corresponds to the warning strategies of "voice warning" and / or "door lock delay"; when 0.9 ≤ R, it is in the emergency layer, which corresponds to the warning strategies of "biological locking" and / or "medical linkage".

[0108] Figure 4 yes Figure 3 , Figure 6 Intelligent driving force: Figure 4 The generated proprietary model and risk perception features are Figure 3 , Figure 6 This is the fundamental guarantee for the efficient operation of "visual feature" extraction and "multimodal engine" in China.

[0109] Figure 3 , Figure 6 yes Figure 4 The path to value realization: Figure 4 The generated intelligent features, through Figure 3 , Figure 6 , Figure 7 Ultimately, this translates into practical actions to protect users, namely, pre-defined strategies.

[0110] Closed-loop cycle: Figure 3 , Figure 6User feedback generated after the strategy was implemented flowed back to Figure 4 Its meta-learning engine drives its continuous evolution, forming an ever-enhancing intelligent closed loop.

[0111] Figure 4 Responsible for creating a precise "brain". Figure 3 , Figure 6 It is responsible for building sensitive "sensory organs" and decisive "reactive nerves," which work together to form an organic intelligent system that integrates perception, cognition, decision-making, and evolution.

[0112] Step 1: The perception layer acquires multimodal data. Video stream analysis: A wide-angle high-definition camera (120° FOV, 30fps) is used to capture the food storage and retrieval process in real time. Multi-frame fusion technology is used to solve the dynamic blur problem. At the same time, based on the acquired key frame images, a large model is called (Step 2) to identify food information, such as food name, quantity, nutritional components, freshness, etc.

[0113] Chemical sensor array: Integrates a 16-channel nano-gold electrode sensor that can detect characteristic peaks of volatile organic compounds (VOCs) and protein allergens (such as the mass spectrometry peak of peanut protein Arah1 at 156.7 m / z).

[0114] Text recognition module: Based on the PaddleOCR engine, it supports multi-angle text extraction and has a built-in allergen keyword library (such as "peanut" and "gluten").

[0115] To solve the problem of dynamic blur, the following steps can be taken: Step 1.1: High-speed image sequence acquisition. Using a 30fps high-definition camera, N frames (e.g., 60 frames) of raw image sequence are continuously captured during the user's food storage and retrieval cycle, forming the basic data source for deblurring processing. The number of frames N and the number of iterations are adaptively adjusted.

[0116] Step 1.2: Inter-frame motion vector estimation is based on techniques such as optical flow or feature point matching. It calculates global and local motion vectors between consecutive frames, establishes pixel-level displacement mapping relationships between frames, and completes image alignment (motion compensation).

[0117] Step 1.3: Blur Kernel Modeling and Optimization Based on the aligned image sequence, a blind deconvolution algorithm is used to iteratively estimate the blur kernel (point spread function) of each frame, and the accurate blur degradation model is obtained through cross-validation of information from multiple frames.

[0118] Step 1.4: Clear Image Reconstruction and Output. Using the estimated blur kernel, a non-blind zone convolution operation is performed on the original image sequence. The effective information of each frame is fused through maximum a posteriori probability estimation to reconstruct and output a globally clear, high-quality image for subsequent large-scale model recognition.

[0119] Step 2: Generative Meta-Learning Adaptive Recognition Model: A personalized recognition system capable of self-evolving based on the user's home environment and personal allergen profile. Its functions can be specifically broken down as follows: Core identification task: Accurately identify basic information such as the name, category, and quantity of ingredients.

[0120] Status assessment task: Further analyze the freshness, maturity, and expected shelf life of the ingredients.

[0121] Attribute association task: While identifying allergens, automatically associate and activate their built-in allergen knowledge base to determine whether the food contains common allergens (such as peanuts, soybeans, seafood, etc.), providing data support for subsequent risk decisions.

[0122] Step 21. Basic Model Construction: Model selection: CLIP-ViT-L / 14 was used as the base model (other large vision models are also acceptable); Pre-training data: (1) Tens of millions of food images (including 5,000+ food categories); (2) Cross-environment samples: different lighting, packaging, and placement angles; (3) Allergen association annotation: food-allergen correspondence.

[0123] "Allergen association annotation" refers to the systematic annotation of known common allergens that may be contained in each food image or category label in the pre-trained dataset, thus constructing a crucial "food-allergen" mapping knowledge base.

[0124] Specifically: Label content: This label clearly records the correspondence between a certain food ingredient (e.g., "shrimp") and a specific allergen (e.g., "tropomyosin"). Data format: It usually exists in the form of a multi-dimensional vector, for example: [Food ID: Shrimp, Allergen: ["Crustaceans", "tropomyosin"], Risk level: High] [Food ID: Milk, Allergen: ["Whey protein", "Lactoglobulin"], Risk level: Medium] [Food ID: Peanut, Allergen: ["Peanut protein Arah1", "Arah2"], Risk level: High].

[0125] Model Training: During the pre-training phase, the model not only learns to recognize the visual features of food ingredients but also learns to establish potential associations between visual features and allergen risks. For example, when the model learns to recognize the appearance of "peanuts," it will simultaneously activate alerts for "high allergy risk." Real-time Query: After deployment, when the model recognizes "peanuts" in an image, the system can immediately query its built-in knowledge base to find the associated allergen as "peanut protein," and, combined with the user's personal allergy profile, trigger the corresponding warning mechanism.

[0126] Allergen association labeling is a safety label added to the training data. It upgrades a simple visual recognition problem into a perception-cognition integrated task closely related to user health and safety. It is a prerequisite and data foundation for achieving subsequent deep integration and optimization of allergens.

[0127] Step 22. Generative Meta-Learning Engine Mechanism.

[0128] Step 23. Dedicated model generation mechanism.

[0129] (1) Injection of environmental features; Home environment characteristics (lighting patterns, storage layout, packaging preferences) are encoded as 128-dimensional vectors; Injecting into the intermediate layers of the base model via an attention gate mechanism: ; In the formula, F base Based on the characteristics of the model, E env This is an encoding vector for environmental information. This is the Sigmoid activation function.

[0130] (2) Dynamic feature generation: When the target ingredient is detected (confidence level < 0.7): the conditional generation network is activated and 10 feature variants are generated based on environmental information. The optimal feature representation is selected through contrastive learning.

[0131] (3) Meta-knowledge transfer: The neural knowledge base stores 100,000+ family configuration options. Similarity retrieval algorithm: ; In the formula, E i E j : These represent the feature vectors of the current user's home environment information to be retrieved and the feature vector of the j-th home's environment information in the knowledge base, respectively.

[0132] D i D j: These represent the histograms of food distribution for the current user's household and the j-th household in the knowledge base, respectively. These histograms statistically analyze the frequency of occurrence of various food items.

[0133] α and β are preset weighting coefficients (α+β=1), used to balance the importance of environmental similarity and food distribution similarity in the overall similarity calculation.

[0134] KL: Represents the KL divergence, used here to measure the divergence between two food distribution histograms D. i and D j The statistical differences between the two families. The smaller the KL divergence, the more similar the food preferences of the two families are.

[0135] The core function of the similarity retrieval algorithm is to intelligently filter out the family experiences most similar to the current user from 100,000+ family configuration patterns, providing an accurate reference benchmark for quickly building personalized models.

[0136] 1. Multi-dimensional similarity calculation: Similarity = α × Environmental Similarity + β × Food Preference Similarity; Environmental similarity: based on the similarity of environmental feature vectors E (such as lighting, layout, packaging habits); Food Preference Similarity: Statistical similarity based on food distribution histogram D; α and β weights: to balance the influence of environmental factors and food preferences on the model's adaptability; 2. Environmental feature matching; Calculate the similarity (e.g., cosine similarity) between environmental feature vectors E. Objective: To find families with similar visual environments inside their refrigerators; Value: Ensuring the environmental adaptability of visual recognition models; 3. Matching of ingredient distribution; Calculate the KL divergence between the histograms D of the food distribution; Objective: To identify families with similar food purchasing and consumption habits; Value: Ensure the model focuses on the food categories frequently used by this user group; 4. Comprehensive similarity assessment; Dynamic weight adjustment: α and β are dynamically adjusted based on user type; Ranking and filtering: Select the top K families with the highest similarity as a reference; Quality assurance: Ensure that the retrieved experiences are both relevant to the environment and match food preferences; Parameter fusion formula: ; Δθnew The final parameter update vector generated by the fusion represents the optimal adjustment direction extracted from multiple similar family experiences.

[0137] ω k The k-th weight is the normalized weight assigned to the k-th meta-model. It is calculated using the Softmax function and its value is between (0, 1) and its sum is 1.

[0138] Δθ k : The update amount of the k-th parameter, the change in parameters of the k-th similarity model relative to the base model.

[0139] R: The number of similar families selected based on the number of reference families determines the number of meta-models to participate in the fusion.

[0140] γ: Temperature parameter. A key parameter that controls the "sharpness" of the weight distribution: • Larger γ value → sharper weight distribution → families with the highest similarity receive the largest weight. • Smaller γ value → smoother weight distribution → more even weight allocation. sim k : The k-th similarity score, the similarity score between the k-th metamodel family and the new user family, derived from the previous similarity retrieval algorithm.

[0141] sim j : The j-th similarity, summing the similarity variables in the denominator, where j iterates through all reference families from 1 to R.

[0142] Through the generative meta-learning techniques described above, using a basic large model, and optimizing the model based on the different family ingredients of different users, a personalized model is built, resulting in increasingly better detection accuracy.

[0143] Step 3: Deep integration and optimization of allergens: 1. Allergen attention mechanism High-risk ingredients automatically receive a +30% attention weight; Establish an "allergy rejection vector" in the feature space: ; In the formula, f base The feature vector initially extracted by a visual recognition model (such as CLIP) represents the original visual information (shape, color, texture) of the food. It is inherently a neutral and objective risk.

[0144] f risk New food ingredients, after being modified by the allergen attention mechanism, are characterized by carrying high allergy risk information.

[0145] The larger the λ value, the higher the risk of user allergies, the larger the adjustment step size, and the further the final feature frisk deviates from the original feature fbase; λ is the risk sensitivity factor (0.1-0.5 for levels 1-5).

[0146] L: Represents a predefined allergen loss function (contrastive loss function). This function is designed such that the loss value L will be very large when the model misclassifies an allergenic food as a safe food.

[0147] L / f: This gradient indicates in which direction feature f should change to increase the loss L (i.e., to make the model more likely to misclassify). Therefore, -( L / f) specifies the direction in which feature f should be moved in order to reduce the loss L (i.e., to make the model more accurately identify the risk of allergies). This direction is called the "allergy rejection direction".

[0148] 2. Dynamic Risk Matrix: Different weights are assigned to information from different risk sources; see Table 1 below for reference: Table 1:

[0149] Specifically, targeted and in-depth optimization mechanisms were implemented for the underlying data generation technology and information quality used for each risk source (visual, chemical, textual, user allergy level) in the matrix to strengthen and enhance them. The goal is to ensure that every piece of evidence input into the fusion decision-making process is of the highest confidence and most reliable, thereby guaranteeing the accuracy of the final decision from the source. This constructs a multi-layered, multi-chain of evidence cross-validation system. Main recognition layer (vision): High-precision, normalized recognition is provided by an adaptive model.

[0150] Molecular verification layer (chemistry): Provides irrefutable molecular-level evidence when visual identification is uncertain or risky.

[0151] Ingredient support layer (text): Provides direct risk warnings through packaging text, serving as a double layer of protection.

[0152] Decision calibration layer (user): Ultimately, all objective evidence is weighted and interpreted using user profiles.

[0153] Here's an example illustrating the collaborative work of the optimization mechanism: Taking a user's peanut allergy as an example: Scenario A (unpackaged peanuts): The visual model (environmentally optimized) accurately identifies "peanuts," even in dim lighting. The system immediately triggers a high-risk alert. (The chemical and text layers are not activated, but the visual layer is sufficient.) Scenario B (peanut butter cookies): The visual model may only identify "cookies," with moderate confidence and no allergy alert triggered. However, OCR text recognition extracts the word "peanut" from the ingredient list, immediately contributing a risk score and triggering an alert. Scenario C (pastries without peanut butter label): Neither the visual nor text layers can confirm this. At this point, the chemical sensor detects the characteristic peaks of peanut protein, providing crucial evidence and preventing a potential misjudgment.

[0154] The "optimization mechanism" of the dynamic risk matrix is ​​essentially a series of cutting-edge and specialized technologies that deeply enhance the information quality of each risk source, so that subsequent weighted fusion decisions are based on a solid and reliable data foundation, thereby achieving a leap from "potentially accurate" to "extremely reliable".

[0155] 3. Tiered Response Strategy A four-level dynamic response strategy is implemented based on a real-time calculated multimodal allergy risk score (R = visual confidence × 0.5 + chemical matching × 0.3 + text risk × 0.15 + user allergy level × 0.05). The user allergy level can be set by the user, and this embodiment does not impose any restrictions on this.

[0156] For example: Suppose a child in the family is allergic to shrimp, and an elderly family member is unaware of the seafood allergy. They want to take seafood from the refrigerator to cook for the child to supplement their nutrition. At this moment, based on frame-by-frame images from the refrigerator's video stream, the model detects that shrimp has been taken out and issues a voice reminder to the unsuspecting grandparent: "The child is allergic to seafood; please do not let the child eat it." Alternatively, an app or text message may notify the parents that they have opened the refrigerator door to take out seafood to prevent unnecessary consumption and potential harm. Different actions are triggered depending on the specific situation and the different input sensing layers.

[0157] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0158] Reference Figure 8 The diagram shows a structural schematic of a food ingredient identification device according to an embodiment of this application, which may include the following modules: The acquisition module 801 is used to acquire target environmental information of the food storage area and acquire target model parameters that match the target environmental information. The adjustment module 802 is used to adjust the preset basic model according to the target model parameters and target environment information to obtain the target model; The recognition module 803 is used to acquire target image data collected for the target food ingredient, and to recognize the target image data according to the target model in order to identify the target food ingredient.

[0159] In one optional embodiment of this application, the acquisition module 801 is used to acquire a preset knowledge base, which stores model parameters of different food identification models trained based on different environmental information; and to determine the target model parameters that match the target environmental information from the preset knowledge base.

[0160] In one optional embodiment of this application, the adjustment module 802 is further configured to receive feedback information from the user regarding the food identification results; and adjust the target model based on the feedback information.

[0161] In one optional embodiment of this application, an allergen attention mechanism is injected into the target model; the recognition module 803 is used to input target image data into the target model; the target model is used to recognize the target image data to determine the food information and allergen information of the target food.

[0162] In an optional embodiment of this application, the apparatus further includes: The early warning module is used to acquire chemical characteristic information and optical character recognition characteristic information of the target food; based on the food identification results output from the target image data, as well as the chemical characteristic information and optical character recognition characteristic information, it determines the allergen risk information of the target food; and based on the allergen risk information, it issues a risk warning.

[0163] In one optional embodiment of this application, the early warning module is used to obtain preset weight information; and determine the allergen risk information of the target food ingredient based on the preset weight information, as well as the food ingredient identification results, chemical feature information, and optical character recognition feature information; the allergen risk information includes a multimodal allergy risk score.

[0164] In one optional embodiment of this application, the early warning module is used to determine an early warning strategy based on a multimodal allergy risk score; and to issue a risk warning for the target food ingredient based on the early warning strategy.

[0165] In an optional embodiment of this application, the early warning module is further configured to detect whether there is a demand for the target ingredient; when it is determined that there is a demand for the target ingredient, the module executes a step of issuing a risk warning for the target ingredient according to the early warning strategy.

[0166] In this embodiment, target environmental information of the food storage area is obtained, and target model parameters matching the target environmental information are obtained. Based on the target model parameters and the target environmental information, a preset basic model is adjusted to obtain the target model. Target image data collected for the target food is obtained, and the target image data is recognized based on the target model to identify the target food. Through this embodiment, the accuracy of food identification can be improved by constructing exclusive models for different users.

[0167] This application also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the above-mentioned method for identifying food ingredients.

[0168] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for identifying food ingredients.

[0169] This application also provides a smart refrigerator, which may include the above-mentioned food identification device, and / or, the above-mentioned electronic device, and / or, the above-mentioned computer-readable storage medium, and / or, the above-mentioned food identification method.

[0170] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0171] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0172] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0173] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0174] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0176] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0177] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0178] The above provides a detailed description of a method, apparatus, smart refrigerator, electronic device, and storage medium for identifying food ingredients. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for identifying food ingredients, characterized in that, The method includes: Obtain target environmental information of the food storage area, and obtain target model parameters that match the target environmental information; Based on the target model parameters and the target environment information, the preset basic model is adjusted to obtain the target model; Acquire target image data collected for the target ingredient, and identify the target image data according to the target model to identify the target ingredient.

2. The method according to claim 1, characterized in that, The step of obtaining target model parameters that match the target environment information includes: Obtain a preset knowledge base, which stores model parameters of different food identification models trained based on different environmental information; Target model parameters that match the target environment information are determined from the preset knowledge base.

3. The method according to claim 1, characterized in that, The method further includes: Receive user feedback on the food identification results; The target model is adjusted based on the feedback information.

4. The method according to claim 1, characterized in that, The target model incorporates an allergen attention mechanism. The step of identifying the target image data based on the target model to identify the target food ingredient includes: Input the target image data into the target model; The target model is used to identify the target image data in order to determine the ingredient information and allergen information of the target food.

5. The method according to claim 4, characterized in that, The method further includes: Acquire chemical characteristic information and optical character recognition characteristic information for the target food ingredient; Based on the food identification results output from the target image data, as well as the chemical feature information and the optical character recognition feature information, the allergen risk information of the target food is determined; Risk warnings are issued based on the allergen risk information.

6. The method according to claim 5, characterized in that, The step of determining the allergen risk information of the target food ingredient based on the food ingredient recognition result output from the target image data, as well as the chemical feature information and the optical character recognition feature information, includes: Obtain preset weight information; Based on the preset weight information, the food ingredient identification results, the chemical feature information, and the optical character recognition feature information, the allergen risk information of the target food ingredient is determined; the allergen risk information includes a multimodal allergy risk score.

7. The method according to claim 6, characterized in that, The risk warning based on the allergen risk information includes: Based on the multimodal allergy risk score, an early warning strategy is determined; Based on the aforementioned early warning strategy, risk warnings are issued for the target food ingredients.

8. The method according to claim 7, characterized in that, The method further includes: To determine if there is a demand for the target ingredient; When it is determined that there is a demand for the target ingredient, the step of issuing a risk warning for the target ingredient according to the warning strategy is executed.

9. A food ingredient identification device, characterized in that, The device includes: The acquisition module is used to acquire target environmental information of the food storage area and acquire target model parameters that match the target environmental information. The adjustment module is used to adjust the preset basic model according to the target model parameters and the target environment information to obtain the target model; The recognition module is used to acquire target image data collected for the target ingredient, and to recognize the target image data according to the target model in order to identify the target ingredient.

10. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the food identification method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the food identification method as described in any one of claims 1 to 8.

12. A smart refrigerator, characterized in that, Includes the apparatus as described in claim 9, and / or the electronic device as described in claim 10, and / or the computer-readable storage medium as described in claim 11, and / or, applied with the food identification method as described in any one of claims 1-8.

Images