Method for illusion suppression of refrigerator food recognition result and refrigerator

By acquiring multimodal data from inside the refrigerator for food identification, and using the matching relationship between image and voice data for verification and correction, the illusion phenomenon of food identification models in smart refrigerators has been solved, improving recognition accuracy and user experience.

CN123072002APending Publication Date: 2026-11-10QINDAO HAIER REFRIGERATOR CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610993888.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-03
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

In existing technologies, the food recognition models of smart refrigerators are prone to illusions, leading to recognition errors and affecting accuracy and user experience.

Method used

By acquiring multimodal data (image and voice data), initial food identification and image recognition are performed separately. The matching relationship between the two is used for illusion verification and correction to ensure the accuracy of the identification results.

Benefits of technology

It effectively suppressed model illusions, improved the accuracy of food identification in the refrigerator, and enhanced the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123072002A_ABST
    Figure CN123072002A_ABST
Patent Text Reader

Abstract

The application discloses a refrigerator food material recognition result illusion suppression method and a refrigerator, and belongs to the technical field of household appliances. The refrigerator food material recognition result illusion suppression method comprises the following steps: acquiring multi-modal data of a food material to be recognized, wherein the multi-modal data at least comprises image data and voice data; the voice data is used for extracting food material feature information; food material recognition is performed based on the food material feature information, and an initial food material recognition result is obtained; food material recognition is performed based on the image data, and a food material image recognition result is obtained; illusion verification is performed on the initial food material recognition result based on a matching relationship between the initial food material recognition result and the food material image recognition result; and in the case that the illusion verification result represents that the initial food material recognition result has an illusion error, the initial food material recognition result is corrected based on the food material image recognition result, and a target food material recognition result is obtained. The application improves the recognition accuracy of food materials in the refrigerator.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the field of home appliance technology, and in particular relates to a method for suppressing illusions in refrigerator food identification results and a refrigerator. Background Technology

[0002] With the development of smart home appliance technology, language models and multimodal models are widely used in the fields of food identification and dietary health management.

[0003] In related technologies, food identification typically employs methods such as cue words, supervised learning, or reinforcement learning to improve the accuracy of the model's output. However, in practical applications, this approach suffers from problems such as discrepancies between the model's output and reality, and identification errors, making it difficult to achieve high-accuracy food identification and impacting user experience. Summary of the Invention

[0004] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, this invention proposes a method for suppressing illusions in refrigerator food identification results and a refrigerator itself, in order to improve the accuracy of food identification in the refrigerator.

[0005] In a first aspect, this application provides a method for suppressing hallucinations in refrigerator food identification results, the method comprising: Acquire multimodal data of the food to be identified, including at least image data and speech data; the speech data is used to extract the food feature information. Based on the food feature information, food identification is performed to obtain the initial food identification result; Food ingredient recognition is performed based on image data to obtain food ingredient image recognition results; Based on the matching relationship between the initial food identification results and the food image identification results, an illusion verification is performed on the initial food identification results; If the illusion verification result indicates that there is an illusion error in the initial identification result of the food ingredient, the initial identification result of the food ingredient is corrected based on the food ingredient image recognition result to obtain the target food ingredient identification result.

[0006] According to one embodiment of this application, food identification based on food feature information specifically includes: extracting core feature information and related feature information from the food feature information; mapping the core feature information to implicit space and explicit space respectively to obtain implicit feature information and explicit feature information; fusing implicit feature information, explicit feature information and related feature information, and identifying food based on the fused feature information.

[0007] According to one embodiment of this application, food identification based on food feature information specifically includes: generating scene prompt words for food feature information, wherein the scene prompt words include at least one of basic scene prompt words, refined scene prompt words, and specific scene prompt words; enhancing the food feature information based on the scene prompt words to obtain enhanced food feature information; and identifying food based on the enhanced food feature information.

[0008] According to one embodiment of this application, food ingredient recognition is obtained based on a trained food ingredient text recognition model. The steps of training the food ingredient text recognition model include: acquiring training data; the training data includes food ingredient feature samples; training a teacher model based on the training data to obtain teacher model parameters; transferring the teacher model parameters to a student model; adjusting the student model so that the difference between the parameters of the student model and the parameters of the teacher model is constrained within the difference range; and using the adjusted student model as the food ingredient text recognition model.

[0009] According to one embodiment of this application, based on the matching relationship between the initial food identification result and the food image identification result, a hallucination verification is performed on the initial food identification result, specifically including: determining the initial food identification vector of the initial food identification result; determining the food image identification vector of the food image identification result; obtaining the similarity between the initial food identification vector and the food image identification vector; and performing a hallucination verification on the initial food identification result based on the similarity.

[0010] According to one embodiment of this application, the initial identification result of food ingredients is corrected based on the food ingredient image recognition result, specifically including: replacing the initial identification result of food ingredients with the food ingredient image recognition result; or fusing the food ingredient image recognition result with the initial identification result of food ingredients to obtain a fused identification result, and replacing the initial identification result of food ingredients with the fused identification result.

[0011] According to one embodiment of this application, the method further includes: determining a task to be executed based on user interaction actions; the task to be executed includes at least one of a food ingredient management task, a recipe generation task, and a food ingredient health management task; determining the execution result of the task to be executed based on the target food ingredient identification result, and updating the food ingredient display information in the refrigerator display interface based on the execution result.

[0012] According to one embodiment of this application, when the task to be performed is a food management task, the execution result of the task to be performed is determined based on the target food identification result, specifically including: obtaining the food location identification result and the food quantity identification result based on the target food identification result; matching the food location identification result with the preset food storage area inside the refrigerator to obtain food storage information; updating the food storage information based on the food quantity identification result; and determining the execution result of the food management task based on the updated food storage information.

[0013] According to one embodiment of this application, determining the execution result of the food management task based on the updated food storage information specifically includes: when the updated food storage information indicates that the food inventory in the refrigerator is insufficient, generating a replenishment list based on the food quantity identification result and the target food identification result, and using the replenishment list as the execution result of the food management task; and / or when the updated food storage information indicates that the food in the refrigerator meets the expiration reminder conditions, generating a time-limited food list, and using the time-limited food list as the execution result of the food management task.

[0014] According to one embodiment of this application, when the task to be performed is a recipe generation task, the execution result of the task to be performed is determined based on the target ingredient identification result, specifically including: generating recipe information based on the user's health data and the target ingredient identification result; and using the recipe information as the execution result of the recipe generation task.

[0015] According to one embodiment of this application, when the task to be performed is a food health management task, the execution result of the task to be performed is determined based on the target food identification result, specifically including: determining the food taken by the user based on the user's interaction action and the target identification result; generating health reminder information based on the user's health data and the food taken by the user; and using the health reminder information as the execution result of the food health management task.

[0016] In a second aspect, this application provides a refrigerator, including a controller configured to perform a method for suppressing illusions based on the refrigerator food identification results as described in the first aspect above.

[0017] Thirdly, this application provides a device for suppressing hallucinations based on refrigerator food identification results, the device comprising: The acquisition module is used to acquire multimodal data of the food to be identified. The multimodal data includes at least image data and voice data; the voice data is used to extract the feature information of the food. The first identification module is used to identify ingredients based on their characteristic information and obtain initial identification results. The second recognition module is used to identify ingredients based on image data and obtain ingredient image recognition results. The verification module is used to perform illusion verification on the initial food identification results based on the matching relationship between the initial food identification results and the food image identification results; The correction module is used to correct the initial identification result of the food ingredient based on the food ingredient image recognition result when the illusion verification result indicates that there is an illusion error in the initial identification result of the food ingredient, so as to obtain the target food ingredient identification result.

[0018] Fourthly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the illusion suppression method for refrigerator food identification results as described in the first aspect above.

[0019] Fifthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the illusion suppression method for refrigerator food identification results as described in the first aspect above.

[0020] In a sixth aspect, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the illusion suppression method for refrigerator food identification results as described in the first aspect.

[0021] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the illusion suppression method for refrigerator food identification results as described in the first aspect above.

[0022] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects: By acquiring multimodal data including image and voice data of the food items to be identified, and extracting food feature information from the voice data, the attributes of the food items inside the refrigerator are obtained from both voice / text and image dimensions, providing richer semantic information for subsequent food identification and avoiding identification errors caused by incomplete information from a single modality. Subsequently, food identification is performed based on the food feature information to obtain initial identification results, and then based on the image data to obtain image identification results, achieving parallel food identification along different modal paths. Reasoning is performed on the food items inside the refrigerator along both the voice and image paths, providing independent and different identification criteria for subsequent verification. After this, based on the food items... The matching relationship between the initial recognition result and the food image recognition result is used to perform illusion verification on the initial recognition result of the food. This can identify erroneous outputs caused by model illusions in the initial recognition result of the food, thereby performing factual verification on the model output result and effectively suppressing the illusion phenomenon in the food recognition result. Finally, when the illusion verification result indicates that there are illusion errors in the initial recognition result of the food, the initial recognition result of the food is corrected based on the food image recognition result. The corrected recognition result is used as the final output, which can effectively eliminate recognition errors caused by model illusions and improve the recognition accuracy of food in the refrigerator.

[0023] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0024] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart illustrating the method for suppressing illusions based on refrigerator food identification results provided in some embodiments of this application; Figure 2 This is a schematic diagram illustrating the process of correcting the initial identification result of food ingredients based on the food ingredient image recognition result provided in some embodiments of this application; Figure 3 This is a schematic diagram of the process for food identification based on food feature information provided in some embodiments of this application; Figure 4 This is a flowchart illustrating food identification based on food feature information provided in other embodiments of this application; Figure 5 This is a schematic diagram illustrating the training process of the food ingredient text recognition model provided in some embodiments of this application; Figure 6 This is a schematic diagram of the multimodal model structure provided in some embodiments of this application; Figure 7 This is a schematic diagram of the structure of the illusion suppression device for refrigerator food identification results provided in some embodiments of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in some embodiments of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0026] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0027] In smart refrigerator food recognition applications, visual language models (VLM) or multi-modal large language models (M-LLM) are typically used to process image and voice data to identify information such as food type, quantity, and location. However, these models commonly exhibit "illusion" phenomena in refrigerator food recognition scenarios, meaning the model outputs results that do not match reality. For example, they might identify packaged "crab sticks" as "beef," "salmon" as "fish oil," packaged "goji berries" as incorrect text with extra or missing characters, misidentify mixed Chinese and English labels on alcoholic beverages, misidentify "celery" as "leeks," and identify frozen "steak" as "lamb." These illusions severely impact the accuracy of food recognition, causing the refrigerator to display incorrect food information and degrading the user experience.

[0028] In related technologies, methods for suppressing hallucinations mainly include cue word engineering, supervised fine-tuning, and reward-based reinforcement learning. However, relying solely on cue word methods is difficult to effectively suppress hallucinations and suffers from content constraints and conflict issues; supervised fine-tuning is prone to cognitive biases, leading to the forgetting of old knowledge; and reward-based reinforcement learning only provides scores without providing the scoring criteria and reasoning process, making it difficult to fundamentally solve the problems of catastrophic forgetting and hallucinations.

[0029] In view of this, this application provides a method for suppressing illusions in refrigerator food identification results, which aims to improve the accuracy of food identification in the refrigerator. By performing parallel identification based on data of different dimensions, and performing illusion verification and correction on the initial food identification results based on the matching relationship between the two identification results, the method effectively suppresses food identification errors caused by model illusions.

[0030] The following description, in conjunction with the accompanying drawings, details the method for suppressing illusions in refrigerator food identification results and the refrigerator provided in this application, through specific embodiments and application scenarios.

[0031] The illusion suppression method for refrigerator food identification results provided in this application embodiment can be implemented by an electronic device or a functional module or entity within an electronic device capable of implementing the illusion suppression method for refrigerator food identification results. The electronic device mentioned in this application embodiment includes, but is not limited to, a terminal or a server.

[0032] Among them, the terminals include, but are not limited to, one or more of the following: refrigerators, smart home terminals, smartphones, tablets, computers, etc.; refrigerators include, but are not limited to, refrigerators, freezers, and vehicle refrigerators in household refrigeration equipment, display freezers, large commercial refrigerators, ice makers, cold storage, etc. in commercial refrigeration equipment, and refrigerated containers in industrial refrigeration equipment.

[0033] A server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0034] In the embodiments of this application, the method for suppressing illusions of refrigerator food identification results can be executed by the local controller of the refrigerator alone, or by a cloud server that communicates with the refrigerator, or by both in combination. This application does not limit this.

[0035] The following describes the illusion suppression method for refrigerator food identification results provided in this application embodiment, taking the controller as the execution subject as an example. Exemplarily, the controller may be one or more of the following: a central processing unit (CPU), a microcontroller unit (MCU), or a processing chip or module with data processing and instruction execution capabilities.

[0036] Figure 1 This is a flowchart illustrating a method for suppressing hallucinations based on refrigerator food identification results provided in some embodiments of this application. For example... Figure 1 As shown, the method for suppressing hallucinations in the refrigerator food identification results includes steps 110 to 150.

[0037] Step 110: Obtain multimodal data of the food to be identified. The multimodal data includes at least image data and voice data. The voice data is used to extract the feature information of the food.

[0038] Multimodal data refers to data that contains at least two different types of information sources. For example, multimodal data includes, but is not limited to, one or more of image data, audio data, video data, and text data.

[0039] In feasible implementations, multimodal data includes at least image data and voice data, wherein image data refers to one or more of the following: static images or single frames from dynamic videos acquired by an image acquisition device. For example, the image acquisition device may be one or more of the following: a camera, an infrared camera, or a depth camera.

[0040] Voice data refers to one or more of the following: user voice audio, signals, etc., acquired through a voice acquisition device. For example, the voice acquisition device may be one or more of the following: a microphone, a microphone array, a receiver, etc. In feasible implementations, voice data can also be obtained from user interaction audio and video data acquired by the controller based on audio separation.

[0041] Furthermore, food feature information refers to semantic information extracted from speech data to characterize the attributes of food ingredients. For example, food feature information may be descriptive features such as food ingredient name, quantity, color, and shape, or it may be one or more of the following: text-based feature vectors or structured semantic tags.

[0042] The controller acquires image data of the food to be identified through an image acquisition device, acquires voice data of the user through a voice acquisition device, and extracts food feature information from the voice data.

[0043] Through the above steps, the controller acquires image data, voice data, and ingredient feature information of the ingredients, providing input data for subsequent ingredient recognition based on image data and ingredient feature information.

[0044] Step 120: Identify ingredients based on their characteristic information to obtain initial identification results.

[0045] The initial ingredient identification result refers to the preliminary determination of the ingredient category or name output by the identification model based on the ingredient feature information, representing the ingredient-related information inferred from the speech and semantic dimension. For example, the initial ingredient identification result includes, but is not limited to, one or more of the following: ingredient name, ingredient category, and ingredient attribute label.

[0046] The controller performs food identification on the obtained food feature information to obtain the initial food identification result.

[0047] In feasible implementations, the controller can be based on a food text recognition model, such as a text classification model based on a Transformer architecture, or a natural language processing model based on an encoder-decoder structure.

[0048] In some implementations, the food text recognition model may be, for example, a vertical domain recognition model that has been pre-trained and fine-tuned, specifically designed for food recognition inside a refrigerator.

[0049] In some other implementations, the food ingredient text recognition model can be a text recognition model, whose input is food ingredient feature information in text form, and whose output is food ingredient category labels. This text recognition model can be trained using a self-distillation reinforcement learning method, thereby further suppressing the illusion of food ingredient recognition results during the text recognition process.

[0050] In other implementations, the food feature information may include multiple semantic tags; the controller combines the multiple tags and inputs them into the food text recognition model to obtain a comprehensive initial food recognition result.

[0051] Through the above steps, the controller completes the initial identification of the food based on the food feature information from the voice source, obtains the initial identification result of the food, and then verifies it based on the image recognition result to suppress the illusion phenomenon of the initial identification result of the food.

[0052] Step 130: Based on the image data, identify the ingredients to obtain the ingredient image recognition results.

[0053] The food image recognition result refers to the determination result of the food category or name, etc., output by the image recognition model based on image data. For example, the food image recognition result includes, but is not limited to, one or more of the following: food name, food category, food location information, and food quantity information.

[0054] The controller inputs the acquired image data into the image recognition model, which extracts and classifies the image data, and outputs the corresponding food image recognition results. The food image recognition results represent the type of food inferred from the visual dimension.

[0055] In feasible implementations, the image recognition model can be, for example, an object detection model based on a Convolutional Neural Network (CNN), or a vision model based on a Transformer architecture, such as the Vision Transformer (ViT) model, or one or more thereof. The controller inputs image data into the image recognition model and obtains one or more of the following from the output of the image recognition model: category labels, quantity statistics, and location boxes of the food items in the image.

[0056] It should be noted that in other feasible implementations, when the controller detects that the image data is occluded, has a resolution lower than a preset threshold, or belongs to a long-tail distribution (e.g., rare ingredients, severe angular deflection, low lighting, etc.), the controller can perform occlusion compensation, feature enhancement, or data augmentation on the image data. Occlusion compensation includes feature completion using feature information from adjacent video frames; feature enhancement includes denoising by fusing shallow and deep features; and data augmentation includes expansion operations such as rotation, scaling, and brightness adjustment on long-tail samples to improve the model's robustness in recognizing low-resource-category ingredients.

[0057] In some implementations, the image recognition model can be, for example, a pre-trained and fine-tuned vertical domain detection model used for food identification inside the refrigerator. This model can identify common food items such as frozen meats, packaged foods, fruits and vegetables, and alcoholic beverages.

[0058] In other implementations, the image data may contain multiple food objects, and the image recognition model may output multiple food image recognition results, each corresponding to one or more detected food objects.

[0059] In some other implementations, the image recognition model can also output the morphological features of the food ingredients, such as auxiliary information like "packaged," "bulk," and "sliced," to enrich the semantic content of the food ingredient image recognition results.

[0060] In some other implementations, the image recognition model can also be a food packaging recognition model, which identifies one or more of the following in the image output by recognizing text information on food packaging: food category labels, quantity statistics, and location boxes.

[0061] Through the above steps, the controller completes the visual recognition of the ingredients based on the image data, obtains the ingredient image recognition result, and uses it for subsequent ingredient recognition result verification.

[0062] Step 140: Based on the matching relationship between the initial food identification result and the food image identification result, perform illusion verification on the initial food identification result.

[0063] Illusion verification refers to the process of detecting whether the initial identification result of the food ingredient is inconsistent with the facts. In feasible implementations, illusion verification can be based on the matching relationship between the initial identification result and the food ingredient image recognition result. Specifically, the matching relationship characterizes the consistency or similarity between the initial identification result and the food ingredient image recognition result at the semantic level. For example, the matching relationship can be expressed as one or more of semantic similarity, feature vector distance, or category affiliation.

[0064] The controller compares the initial food identification result with the food image identification result and calculates the degree of matching between the two. Then, the controller determines whether the initial food identification result contains a hallucination error based on the calculated matching degree. If the initial food identification result does not match the food image identification result, it is determined that the initial food identification result contains a hallucination.

[0065] In some implementations, the controller can convert the initial food identification result and the food image identification result into vector representations respectively, and calculate the cosine similarity or Euclidean distance between the two vectors. When the similarity is lower than a preset threshold or the distance is greater than a preset threshold, it is determined that the initial food identification result does not match the food image identification result, and the initial food identification result is a hallucination.

[0066] In other implementations, the controller can determine the matching relationship between the initial food identification result and the food image identification result using a semantic classification model. For example, the controller can concatenate the initial food identification result and the food image identification result, and then input the concatenated result into a binary classification model to directly obtain the matching or non-matching labels.

[0067] In some other implementations, the controller can convert the initial food identification result and the food image identification result into category labels (such as "beef" and "lamb"), and the controller determines whether the two belong to the same category or have a hierarchical relationship. For example, "steak" and "beef" can be considered a match, while "crab sticks" and "beef" are considered a mismatch.

[0068] In some other implementations, the controller can also make a judgment based on the confidence levels of the initial food identification result and the food image identification result. For example, when the confidence level of the food image identification result is higher than a preset corresponding threshold, while the confidence level of the initial food identification result is lower than a preset corresponding threshold, and the two are inconsistent, it is determined that the initial food identification result is a hallucination.

[0069] Through the above steps, the controller can use the image recognition results to perform factual verification on the initial food identification results obtained from the speech recognition, thereby determining whether there are any illusion errors in the initial food identification results, and providing a basis for judgment for subsequent correction operations.

[0070] Step 150: If the illusion verification result indicates that there is an illusion error in the initial identification result of the food ingredient, the initial identification result of the food ingredient is corrected based on the food ingredient image recognition result to obtain the target food ingredient identification result.

[0071] Among them, correction refers to the process of replacing or fusing the initial recognition result of the food ingredient with the image recognition result of the food ingredient when the controller determines that there is a phantom error in the initial recognition result of the food ingredient, so as to generate a more accurate recognition result.

[0072] The target ingredient recognition result refers to the ingredient recognition result obtained after correction, which can be used for subsequent ingredient information management or interface display, etc.

[0073] Based on the illusion verification result obtained in step 140, if the controller determines that the initial food identification result contains an illusion error, it corrects the initial food identification result based on the food image identification result. In feasible implementations, the correction method includes, but is not limited to, replacing the initial food identification result with the food image identification result, or semantically fusing the two to generate a new identification result. The corrected result is the target food identification result.

[0074] In some implementations, when the controller simultaneously identifies multiple food objects in the image data, such that the food image recognition result contains multiple candidate labels and corresponding confidence levels, the controller can select the candidate label with the highest confidence level to correct the initial food recognition result.

[0075] In other implementations, the controller can also determine whether the confidence level of the food image recognition result is higher than a preset threshold before making corrections. The controller will only perform the correction operation when the confidence level of the food image recognition result meets the requirements; otherwise, it can perform operations such as retaining the initial food recognition result or requesting user confirmation.

[0076] In some other implementations, if the illusion verification result obtained by the controller indicates that there is no illusion error in the initial identification result of the food ingredient, the initial identification result of the food ingredient can be directly used as the target food ingredient identification result without performing a correction operation.

[0077] Through the above steps, the controller corrects the initial food identification results that may be hallucinatory to food identification results that are consistent with or related to visual facts, thereby improving the overall accuracy of food identification.

[0078] The illusion suppression method for refrigerator food identification results provided in this application embodiment acquires multimodal data including image data and voice data of the food to be identified, and extracts food feature information from the voice data. It obtains the attributes of the food inside the refrigerator from both voice and image dimensions, providing richer semantic information for subsequent food identification and avoiding identification bias caused by incomplete information in a single modality. Subsequently, it performs food identification based on the food feature information to obtain initial food identification results, and performs food identification based on the image data to obtain food image identification results. This achieves parallel food identification along different modal paths, performing inference on the food inside the refrigerator along both the voice and image paths, providing mutually independent information for subsequent verification. The identification process is based on several steps. First, based on the matching relationship between the initial food identification result and the food image identification result, a hallucination check is performed on the initial food identification result. This check identifies erroneous outputs caused by model hallucinations in the initial food identification result, thus verifying the model output and effectively suppressing hallucination phenomena in the food identification result. Finally, if the hallucination check result indicates that the initial food identification result contains hallucination errors, the initial food identification result is corrected based on the food image identification result. The corrected identification result is then used as the final output, effectively eliminating identification errors caused by model hallucinations and improving the accuracy of food identification in the refrigerator.

[0079] In some implementations, the multimodal data also includes video data. The controller performs hand detection based on the video data, identifies the hand area and the food object being held, and detects whether the food object is entering the refrigerator drawer or freezer drawer. When it enters, the controller locates the packaged food area held by the food object and extracts the packaging text information from the packaged food area as food feature information.

[0080] The multimodal data includes, but is not limited to, real-time and offline voice, image, video, and audio-visual data from smart devices or home appliances. Acquisition channels include various transmission methods such as wireless networks, cellular networks, 4G / 5G / 6G networks, microphones, mobile phones, Bluetooth, large screens, and in-screen displays. If the hardware acquisition device is an independent peripheral, the data is acquired by the external device and then transmitted to the controller.

[0081] After acquiring video data, the controller first detects the hand area. In feasible implementations, the controller can use a deep learning-based hand detection model to locate the hand bounding box in the video frame, and further use a target tracking algorithm to continuously track the hand area and the food object held by the hand during hand movement. The controller detects whether the food object held by the hand enters the refrigerator's refrigerator drawer or freezer drawer. For example, the entry status can be determined by judging whether the bounding box of the hand and the held food object intersects with or crosses the boundary line of a preset drawer area (pre-marked by modeling the internal space of the refrigerator). For example, if a user moves a bag of packaged food from outside the refrigerator to inside the refrigerator drawer, the controller analyzes consecutive frames and determines "entry" when it detects that the hand and food have crossed the virtual boundary line of the drawer opening. In some implementations, the controller constructs a target tracking and detection model with the hand and drawer area as the scope of interest. This model can use a deep convolutional neural network, a transformer architecture, a diffusion network model, or a fusion structure of multiple networks. Using the drawer frame as a boundary, the model identifies and classifies objects within the drawer frame, determining their specific categories and location information. For objects outside the drawer frame, the model continuously tracks them via video to acquire the motion trajectory of the held food or target object. The posture of the held food object includes flat grip, angled grip, and various other angles. The model tracks and acquires the positional information and bounding box of the held object. The positional coordinates can be obtained using the coordinates of the top-left and bottom-left corners or keypoint detection boxes, and the model continuously tracks the target feature information of the held object.

[0082] After the object is identified as having entered the packaging, the controller precisely locates the area where the handheld food object is positioned, locking onto the packaged food area (i.e., the front of the packaging bag or box). The controller then separates the packaging area from the hand area using image segmentation or object detection models. Next, the controller extracts text information from the packaging area image using Optical Character Recognition (OCR) technology or a deep learning-based end-to-end text recognition model. This text information includes the food name (e.g., "crab sticks," "salmon"), net weight, and production date. The extracted text information is then added to the food feature information for subsequent food recognition.

[0083] After obtaining the initial food identification results, in some implementations, the controller can also use a self-play deep reinforcement learning method to perform multi-step reasoning verification on the initial food identification results. Specifically, the controller engages in a self-play exercise with the contextual information (such as location, quantity, and recent retrieval records) of other food items in the refrigerator, simulating the matching degree of the identification results under different reasoning paths, and evaluates the consistency and correctness through a reward function. Subsequently, the identification strategy is updated based on the evaluation results.

[0084] In other implementations, to prevent spelling errors during speech recognition or model generation, the controller can also perform multilingual text correction on the initial food identification results. For example, the controller can use Bidirectional Encoder Representations from Transformers (BERT) or a BERT-based spelling correction model to correct mixed Chinese and English text, special characters (such as English letters in wine labels), and out-of-vocabulary words.

[0085] In a feasible implementation, during the process of obtaining food image recognition results based on image data, the controller can employ a fine-grained image classification network to better distinguish easily confused foods with similar appearances. This network first adds a deep classification subnetwork before the neck network of the target detection model to identify the major category to which the food belongs (e.g., "vegetables"); then, it performs secondary recognition of the subcategories within the major category (e.g., "celery") in the detection head or dedicated classification head. This significantly reduces category confusion and improves the accuracy of fine-grained recognition.

[0086] After obtaining the target ingredient recognition result, in some embodiments, the controller can also generate result video data for presentation on the refrigerator display interface. For example, the video data can be generated through a full attention module, which is based on the Transformer architecture and includes a full attention mechanism, convolutional layers, activation function layers, and layer normalization layers. Specifically, the full attention mechanism is used to capture the internal correlations of the input features themselves and the global dependencies between different features, forming a unified high-level semantic representation; then, it is sequentially decoded and generated into a video frame sequence through convolution operations, activation function activation, and layer normalization processing. The video generated in this way can dynamically display the ingredient recognition process, packaging text highlighting, location markings, and other information, thereby improving the intuitiveness of user interaction.

[0087] In some embodiments, based on the matching relationship between the initial food identification result and the food image identification result, a hallucination verification is performed on the initial food identification result, specifically including: determining the initial food identification vector of the initial food identification result; determining the food image identification vector of the food image identification result; obtaining the similarity between the initial food identification vector and the food image identification vector; and performing a hallucination verification on the initial food identification result based on the similarity.

[0088] Here, the initial ingredient recognition vector refers to the numerical vector representation converted from the initial ingredient recognition result (text form) through the embedding model. The ingredient image recognition vector refers to the numerical vector representation converted from the ingredient image recognition result (class label in text form). Similarity refers to the closeness of two vectors in the vector space, used to measure the semantic matching degree between the initial ingredient recognition result and the ingredient image recognition result. For example, similarity can be cosine similarity, calculated as the dot product of the two vectors divided by the product of their magnitudes, with a value ranging from -1 to 1; a larger value indicates greater similarity. It can also be the reciprocal of the Euclidean distance or a normalized form, etc.

[0089] The controller first inputs the initial food identification results into the embedding model to obtain the corresponding initial food identification vector. Here, the embedding model refers to a model that maps discrete text or labels to continuous vectors. For example, the embedding model includes, but is not limited to, one or more of the following: BERT-based encoder models, word vector models, global vector models, or fast text models.

[0090] Simultaneously, the food image recognition results from the controller are input into the same embedding model to obtain food image recognition vectors. It should be noted that using the same embedding model ensures that the two vectors reside in the same semantic space, thus making them comparable.

[0091] Subsequently, the controller calculates the similarity between the initial food identification vector and the food image identification vector, and compares the calculated similarity with a preset similarity threshold. If the similarity is greater than or equal to the preset threshold, it is determined that the initial food identification result matches the food image identification result, and there is no illusion error; if the similarity is lower than the threshold, it is determined that there is an illusion error.

[0092] For example, the similarity threshold can be set to 0.7. When the cosine similarity between the initial food identification vector and the food image identification vector is 0.85 (higher than 0.7), a match is determined; when the cosine similarity between the initial food identification vector and the food image identification vector is 0.3 (lower than 0.7), a mismatch is determined, indicating an illusion error.

[0093] In the above embodiments, by determining the initial identification vector and the image identification vector of the ingredients respectively, calculating the similarity between the two, and performing illusion verification based on the similarity, the semantic similarity in the vector space is used to replace simple string matching, which can correctly identify semantic equivalence such as synonyms and hyponyms, thus improving the robustness and accuracy of the verification. In addition, the vector representation generated by the embedding model can capture deep semantic associations, thereby more accurately detecting illusion errors.

[0094] In some embodiments, the initial identification result of the food ingredient is corrected based on the food ingredient image recognition result, specifically including: replacing the initial identification result of the food ingredient with the food ingredient image recognition result; or fusing the food ingredient image recognition result with the initial identification result of the food ingredient to obtain a fused identification result, and replacing the initial identification result of the food ingredient with the fused identification result.

[0095] Fusion refers to the process of combining two or more recognition results into a new recognition result according to certain rules. The fused recognition result is the result obtained after the fusion operation, which may include the food name from the food image recognition result, while retaining auxiliary information (such as quantity, location, freshness, etc.) from the initial food recognition result. For example, the fusion method could be to concatenate the food name from the image recognition result with the quantity description from the speech recognition result.

[0096] When the illusion verification result determines that there is an illusion error in the initial identification result of the food, the controller corrects the initial identification result of the food based on the food image recognition result.

[0097] In some implementations, the controller directly replaces the initial food identification result with the food image identification result.

[0098] In other implementations, the controller fuses the food image recognition result with the initial food recognition result to obtain a fused recognition result, and then replaces the initial food recognition result with the fused recognition result.

[0099] In some other implementations, the controller can automatically select a replacement or fusion method based on a preset strategy (e.g., the confidence level of the recognition result, the presence of additional information fields). For example, if the initial food identification result does not contain any auxiliary information other than the food name, the method of directly replacing the initial food identification result with the food image recognition result is selected; if the initial food identification result contains auxiliary information other than the food name, the method of fusing the food image recognition result with the initial food identification result is selected.

[0100] In feasible implementations Figure 2 This is a schematic diagram illustrating the process of correcting the initial identification result of food ingredients based on the food ingredient image recognition result, as provided in some embodiments of this application. For example... Figure 2As shown, firstly, the controller acquires a pre-constructed image-text pair verification set, which contains multiple pairs of paired image features and corresponding text embeddings. When the illusion verification result determines that the initial food identification result contains an illusion error, the controller retrieves the image features that match the current food image data from the verification set; that is, it uses the currently acquired food image data to retrieve the most similar image features from the verification set. After this, the controller obtains the text embedding vectors corresponding to the image features. Finally, the controller calculates the weighted average of one or more of the most similar neighboring text embedding vectors, or sets a similarity threshold, using the weighted average or text that meets the threshold as reference text, and corrects the initial food identification result to the reference text, thereby obtaining the target food identification result.

[0101] It should be noted that since the image features and text embedding vectors in the validation set have been pre-aligned, once a matching image feature is retrieved, one or more candidate text embedding vectors associated with it can be obtained.

[0102] In the above embodiments, by replacing the initial food identification result with the food image identification result, the erroneous result caused by model illusion can be quickly corrected to the food category consistent with the visual facts of the image, thereby improving the accuracy of the food identification result; by fusing the initial food identification result with the food image identification result, the auxiliary information that cannot be obtained from the image data in the initial food identification result can be retained, thereby reducing information loss while suppressing illusion errors, so that the final target food identification result has both accuracy and richness.

[0103] In related technologies, food feature information extracted from speech is usually directly input into text classification models for recognition. However, speech data often contains complex semantic layers: there are core keywords that directly describe the food (such as "salmon"), as well as related descriptions of the food's attributes, state, and location (such as "the one in the upper left corner of the refrigerator" or "with packaging"). Related text recognition models struggle to effectively distinguish between core semantics and related semantics, and are prone to losing deep semantic information. This leads to the recognition results being interfered with by redundant information, and may even produce illusions, thus limiting the improvement of food recognition accuracy.

[0104] Based on this, in some embodiments, food identification is performed based on food feature information, specifically including: extracting core feature information and related feature information from the food feature information; mapping the core feature information to implicit space and explicit space respectively to obtain implicit feature information and explicit feature information; fusing implicit feature information, explicit feature information and related feature information, and performing food identification based on the fused feature information.

[0105] Among them, core feature information refers to keywords or phrases extracted from the food feature information that directly indicate the name or category of the food, such as "beef", "salmon", "apple", etc.

[0106] Associated feature information refers to semantic units that modify, describe, or provide auxiliary information in the context of ingredients, such as "frozen," "packaged," or "in the second layer."

[0107] Furthermore, the explicit space refers to an interpretable vector representation space obtained through word embeddings or pre-trained language models, where each dimension typically corresponds to certain semantic features. For example, a pre-trained language model could be the BERT model.

[0108] The implicit space refers to the vector representation space automatically learned through the hidden layers of a neural network. In the implicit space, feature vectors lack direct interpretability; that is, the semantic meaning of each dimension cannot be intuitively seen. However, this space can capture the deep semantic relationships between words or sentences. For example, when the word "beef" is input into a Transformer model, the feature vector output by its deep network is its representation in the implicit space. Although this representation is difficult to interpret directly, it can reflect the semantic proximity between "beef" and words like "steak" and "lamb."

[0109] The controller first performs semantic parsing on the obtained food feature information to extract core feature information and related feature information.

[0110] In some implementations, the controller can extract core feature information through Named Entity Recognition (NER). Named Entity Recognition refers to the technology of identifying entities with specific meanings from text. Entities include, but are not limited to, categories such as food names, quantities, times, and locations. For example, performing Named Entity Recognition on the text "two pieces of salmon" can identify "salmon" as a food name entity, thus extracting it as core feature information.

[0111] In other implementations, the controller can extract association feature information through dependency parsing. Dependency parsing is a technique for analyzing the dependency relationships between words in a sentence, used to identify the grammatical relationship between a core word and its modifiers. For example, performing dependency parsing on the text "frozen beef" can identify "beef" as the core noun and "frozen" as the adjective modifier, with a noun-head relationship between them, thus extracting "frozen" as the association feature information.

[0112] Following this, the controller maps the core feature information to both the implicit and explicit spaces. The mapping in the explicit space can be achieved using a pre-trained BERT model or word embeddings to obtain explicit feature vectors; while the mapping in the implicit space can be achieved by inputting the core feature information into the deep network of the Transformer model and extracting its hidden layer output as implicit feature vectors.

[0113] It should be noted that the dimensions of explicit feature vectors and implicit feature vectors can be the same or different, and their dimension design can be selected according to the actual model.

[0114] Next, the controller fuses the implicit, explicit, and associated feature information. In feasible implementations, the fusion method can be vector concatenation, weighted summation, or combining the three information after calculating their weights using an attention mechanism, thus obtaining the fused feature information. The fused feature information includes explicit and implicit representations of the core features, as well as contextual information, resulting in stronger semantic expressive power.

[0115] Finally, the controller obtains the initial food identification result based on the fused feature information.

[0116] In some embodiments, the controller converts food feature information into a word sequence; for each word in the word sequence, it obtains its state representation in the implicit space and its vector representation in both the implicit and explicit spaces; it calculates the score of the words in the preceding and following sequences, where the score is used to characterize the importance of each word in the semantic sequence; finally, the controller performs food identification based on the features fused from the state representation, vector representation, and score.

[0117] The controller first segments the food feature information into words or sub-words, converting it into a sequence of word units. The controller processes each word unit in the sequence, obtaining its state representation in the implicit space and its vector representation in the explicit space (i.e., obtaining implicit and explicit feature information respectively). Next, the controller calculates the scores of word units in the preceding and following sequences, such as calculating the semantic similarity or attention score of each word unit with its adjacent word units, thus reflecting the importance of that word unit in the sequence. Finally, the controller performs a weighted fusion of the implicit state representation, implicit / explicit vector representation, and sequence score of each word unit to obtain a fused feature vector. The fusion method can be vector concatenation followed by linear transformation, or weighted summation using scores as weights. Based on the fused features, the controller obtains the initial food identification result.

[0118] Figure 3 This is a flowchart illustrating the process of food identification based on food feature information provided in some embodiments of this application, such as... Figure 3As shown, the controller converts food feature information into tokens per unit time and obtains each token in the token sequence, determining the state representation of each token in the implicit space. Simultaneously, the controller obtains the vector representations of each token in both the implicit and explicit spaces (i.e., obtaining implicit and explicit feature information respectively). Further, the controller calculates the scores of tokens in the preceding and following sequences; these scores characterize the importance of each token in the semantic sequence and its contribution to the recognition result. In other words, the state representation in the implicit space, the implicit / explicit dual-space vector representation, and the sequence score can be fused into an enhanced feature representation. Finally, the controller performs food recognition based on the enhanced feature representation to obtain the initial food recognition result.

[0119] In some embodiments, the model used for food identification based on food feature information can be a Mixture of Experts (MoE) model. This model includes a shared expert and multiple routing experts, each corresponding to a different food category identification scenario, such as an expert on frozen meats, fruits and vegetables, packaged foods, and wines. The controller inputs the food feature information into the routing network, which dynamically selects one or more of the most suitable experts for processing, while the shared expert provides general features. This significantly improves identification efficiency and accuracy.

[0120] In the above embodiments, by extracting core feature information and associated feature information, and mapping the core features to implicit and explicit spaces respectively, and then performing recognition after fusing the three, the core and associated semantics can be effectively distinguished, reducing the negative impact of redundant information and interference words on the recognition results, and enabling the model to focus on the key descriptions of ingredients. In addition, the explicit and implicit dual-space mapping can simultaneously preserve the semantic interpretability of explicit information and the deep association of implicit information, making up for the information loss of single-space representation, enhancing the model's ability to capture complex semantic relationships such as synonyms and hyponyms, thereby effectively improving the accuracy of the initial recognition results of ingredients, reducing the illusion phenomenon caused by insufficient semantic understanding from the source, and providing a more reliable initial input for subsequent illusion verification steps.

[0121] In other embodiments, food identification based on food feature information specifically includes: generating scene prompts for food feature information, wherein the scene prompts include at least one of basic scene prompts, refined scene prompts, and specific scene prompts; enhancing the food feature information based on the scene prompts to obtain enhanced food feature information; and identifying food based on the enhanced food feature information.

[0122] Contextual prompts refer to supplementary textual information used to enhance the semantic expression of food feature information. In feasible implementations, contextual prompts include, but are not limited to, at least one of basic contextual prompts, refined contextual prompts, and specific contextual prompts. Specifically, basic contextual prompts are used to describe general refrigerator environment or food category prompts (e.g., "food in the refrigerator," "food that needs to be identified"); refined contextual prompts are used to describe specific attributes or states of food (e.g., "frozen state," "packaged food," "second shelf of the refrigerator compartment"); and specific contextual prompts refer to prompts customized for specific food or user habits (e.g., "brands frequently purchased by the user," "seasonal food").

[0123] The controller first generates at least one scene prompt word based on the current recognition scenario and the characteristics of the ingredients. In a feasible implementation, the generation rules for the scene prompt words can be based on a preset template library or implemented through a lightweight prompt word generation model.

[0124] For example, if the food ingredient information contains the word "beef", the controller can supplement the basic scene prompt "food in the freezer compartment of the refrigerator" to limit the recognition range; at the same time, it can generate a refined scene prompt "frozen status" based on environmental information (such as temperature zones) obtained from sensors inside the refrigerator; if it recognizes the user's historical preferences, it can also generate specific scene prompts such as "beef cuts suitable for grilling".

[0125] The controller then fuses the generated scene prompts with the original food feature information to obtain enhanced food feature information. For example, the fusion method can be one or more of the following: direct splicing, weighted combination, or fusion through an attention mechanism.

[0126] Finally, the controller obtains the initial food identification result based on the enhanced food feature information.

[0127] In some implementations, the controller identifies ingredients based on their characteristic information, and enhances the characteristic information by generating and fusing multi-level scene prompts before identification. Figure 4 This is a flowchart illustrating the process of food identification based on food feature information provided in other embodiments of this application, such as... Figure 4As shown, the controller first assesses the current recognition scenario to determine the type of prompt words to be generated. The prompt word types include at least one of A / first scenario prompts, B / second scenario prompts, and specific prompts. A / first scenario prompts correspond to basic scenario prompt words, used to describe a general refrigerator environment or food category; B / second scenario prompts correspond to refined scenario prompt words, used to describe specific attributes or storage locations of food items; and specific prompts correspond to specific scenario prompt words, used to describe user preferences or special situations. After determining the prompt word type, the controller generates the corresponding scenario prompt words and fuses them with the original food feature information. To improve processing efficiency, the controller can also employ a flash attention mechanism (flash prompts), which accelerates the computation of the prompt vector to quickly obtain the enhanced prompt representation.

[0128] In the above embodiments, by generating scene prompts for food feature information and fusing the prompts with the original features for recognition, additional contextual constraints are provided for food recognition, which effectively alleviates recognition errors caused by incomplete or ambiguous descriptions. In addition, multi-level scene prompts can adapt to contextual requirements of different granularities, ensuring coverage of general scenarios while supporting optimization of personalized scenarios, thus improving the model's adaptability.

[0129] In related technologies, text models used for food identification are typically trained using supervised fine-tuning (SFT), which involves updating the parameters of a pre-trained model on new data. However, this training method suffers from a catastrophic forgetting problem: when the model learns new food categories, it often forgets previously learned categories. Furthermore, while traditional distillation training can transfer knowledge, it doesn't effectively constrain the parameter update space of the student model, leading to overfitting to the new task and deviating from the distribution of old knowledge. This makes it difficult for the student model to efficiently learn new knowledge while retaining old knowledge, limiting its ability to continuously update in refrigerator food identification scenarios.

[0130] Based on this, in some embodiments, food ingredient recognition is obtained based on a trained food ingredient text recognition model; the steps of training the food ingredient text recognition model include: acquiring training data; the training data includes food ingredient feature samples; training a teacher model based on the training data to obtain teacher model parameters; transferring the teacher model parameters to a student model; adjusting the student model so that the difference between the parameters of the student model and the parameters of the teacher model is constrained within the difference range; and using the adjusted student model as the food ingredient text recognition model.

[0131] In this context, the teacher model refers to a pre-trained model that provides the knowledge source, typically with a large number of parameters and high recognition accuracy. The student model, on the other hand, is the target model to be trained, with fewer parameters or a simpler structure, gaining recognition ability by transferring knowledge from the teacher model. Furthermore, model parameters refer to the weights and biases of each layer in the model, which determine the input-output mapping relationship.

[0132] The constraint of parameter differences within a certain range refers to limiting the degree of difference between student model parameters and teacher model parameters through a specific loss function or distance metric, confining it to a predetermined range. For example, the loss function could be the relative entropy loss function (Kullback-Leibler Divergence, KL divergence), and the distance metric could be Euclidean distance, etc.

[0133] In feasible implementations, the range of differences can be defined using the shortest distance space or the shortest path method, for example, by constraining the absolute value of the offset of the student model parameters relative to the teacher model parameters to be less than a preset threshold.

[0134] It should be noted that the training of the food text recognition model can be performed in the refrigerator controller, or it can be trained on a cloud server and then deployed in the refrigerator controller. This application does not limit this.

[0135] The controller first acquires training data, which includes food feature samples and their corresponding annotations. For example, food feature samples may be one or more of the following: speech-to-text, user historical input, or publicly available food corpora. The controller uses this training data to train the teacher model, thus obtaining a converged teacher model and its parameters.

[0136] In some implementations, the controller uses both offline and online training data for training the teacher model.

[0137] Subsequently, the controller assigns the parameters of the teacher model as initial parameters to the student model (i.e., parameter transfer). Based on this, the controller further adjusts the student model using the same training data or new training data. In some implementations, the controller introduces a constraint term during the adjustment process to calculate the distance (e.g., Euclidean distance or KL divergence) between the current parameters of the student model and the parameters of the teacher model. By optimizing the objective function, the controller ensures that the parameters of the student model adapt to the new task without deviating from the teacher model parameters by more than a preset range.

[0138] When the parameter differences exceed a preset range, the controller forces a pullback by reducing the learning rate or increasing the weights of constraint terms, ultimately resulting in an adjusted student model. This student model is the food text recognition model used for food identification.

[0139] In other implementations, the range of parameter differences can be constrained by the shortest distance space or the shortest path method. For example, the Euclidean distance between the student model parameter vector and the teacher model parameter vector can be set to be less than a threshold, or the absolute value of the change in weights of each layer can be set to be less than a preset threshold.

[0140] Figure 5 This is a schematic diagram illustrating the training process of the food ingredient text recognition model provided in some embodiments of this application. For example... Figure 5 As shown, the controller first acquires training data and trains the teacher model based on the training data to obtain the trained teacher model and its parameters. Second, the controller transfers the teacher model parameters to the student model, using the teacher model parameters as the initial parameters of the student model. In the fine-tuning phase, the controller uses an expert preference method to fine-tune the student model. The expert preference method refers to using expert-annotated preference data (e.g., including preference criteria and standards, such as which recognition results are correct and which are reasonable and acceptable) to guide the direction of model optimization. During training, the controller simultaneously calculates the alignment between the student model's output and the expert preferences and constrains this alignment using a loss function. Furthermore, the controller constrains the parameter update space of the student model to minimize the distance or similarity between the student model and the teacher model. In feasible implementations, the magnitude and direction of parameter updates can be constrained by minimizing the Euclidean distance between the student model parameters and the teacher model parameters, or by maximizing their cosine similarity. In other feasible implementations, the controller can also use a shortest path method to optimize the parameter update trajectory, ensuring that the student model starts from the teacher model's parameter space and reaches the target parameter space along the shortest path, thereby avoiding significant deviations during the update process.

[0141] In the above embodiments, the teacher model is trained by acquiring training data, the parameters of the teacher model are transferred to the student model, and the differences between the parameters of the student model and the teacher model are constrained. This allows the student model to inherit the old knowledge already learned by the teacher model, avoiding the data and computational overhead of training from scratch. By constraining the parameter differences within a preset range, the catastrophic forgetting of old knowledge by the student model when learning new food categories is effectively suppressed, thereby maintaining the model's stable recognition ability for various food categories during continuous updates. In addition, compared with traditional unconstrained distillation or simple fine-tuning, this method can achieve a better balance between learning new tasks and retaining old tasks, significantly improving the robustness of the food text recognition model in incremental learning scenarios and fundamentally reducing recognition illusions caused by knowledge forgetting.

[0142] In related technologies, refrigerator food identification results are typically only used to display the food names, lacking intelligent interactive capabilities that integrate with the user's actual needs. For example, after identifying the food, the user still needs to manually record the quantity, manually set an expiration reminder, or search for recipes separately, which is cumbersome and easy to forget. Furthermore, different users have significantly different usage scenarios, making it difficult for related technologies to correlate food identification results with the user's interactive intent, resulting in low utilization efficiency of the identification results and a poor user experience.

[0143] Based on this, in some embodiments, the controller determines the task to be executed based on user interaction actions; the task to be executed includes at least one of food management task, recipe generation task and food health management task; the execution result of the task to be executed is determined based on the target food identification result, and the food display information in the refrigerator display interface is updated based on the execution result.

[0144] User interaction refers to the interactive behaviors between the user and the refrigerator, including but not limited to one or more of the following: voice commands, touchscreen operation, gesture recognition, and button pressing. For example, a user can trigger a corresponding task by saying voice commands such as "manage ingredients" or "recommend a recipe".

[0145] Tasks to be performed refer to specific operational tasks that the refrigerator needs to perform, determined based on user interaction actions. These include, but are not limited to, at least one of the following: food management tasks, recipe generation tasks, and food health management tasks. Specifically, food management tasks involve operations such as entering, modifying, deleting, querying, or reminding users to replenish food in the refrigerator; recipe generation tasks involve recommending or generating recipes for the user based on the current ingredients; and food health management tasks involve providing dietary health advice based on food information and user health data (such as medical examination reports).

[0146] Furthermore, the refrigerator display interface refers to the user interface presented by the display screen on the refrigerator, which is used to display one or more of the following: food information, task execution results, and various prompts.

[0147] The controller first identifies the user's interactive actions. For example, the user might say "Help me manage the food in the refrigerator," or tap the "Food Management" button on the touchscreen, or gesture to a specific area of ​​the refrigerator, or the controller might detect the user performing one of the following actions: taking out or storing food. The controller then interprets the user's intent based on these interactive actions to determine the task to be performed.

[0148] In some embodiments, when the task to be performed is a food management task, the execution result of the task to be performed is determined based on the target food identification result, including: obtaining the food location identification result and the food quantity identification result based on the target food identification result; matching the food location identification result with the preset food storage area inside the refrigerator to obtain food storage information; updating the food storage information based on the food quantity identification result; and determining the execution result of the food management task based on the updated food storage information.

[0149] The food location identification result refers to the information about the spatial area of ​​the food to be identified within the refrigerator, obtained based on image or video data analysis. For example, the food location identification result may include "third shelf in the refrigerator compartment", "left side of the freezer drawer", "second shelf on the door shelf", etc.

[0150] The food quantity identification result refers to the number or weight information of the food items to be identified, obtained based on the analysis of image or video data. For example, the food quantity identification result may include "3 items", "approximately 500 grams", "half a bag", etc. Furthermore, food storage information refers to the data structure recorded inside the refrigerator that describes the type, quantity, and location of food items in each storage area, which can be stored in the refrigerator's memory in the form of a list, table, or tag cloud.

[0151] When the task to be performed is food management, the controller first extracts the food location and quantity identification results from the target food identification results. Next, the controller matches the food location identification results with the preset food storage areas inside the refrigerator. These preset food storage areas can be one or more predefined areas, such as: refrigerator compartments (first, second, and third shelves), freezer compartments (drawers and shelves), door shelves (left and right), etc. The controller obtains the specific storage area corresponding to that location through semantic matching or area mapping.

[0152] Then, the controller updates the food storage information for that storage area based on the food quantity identification result. In some implementations, if the controller detects that it is a newly added food, it adds a record to the food list for that area, recording the food name and quantity; if it is an existing food (such as food that has been taken out and put back in by the user), it updates the quantity field of the corresponding food in that area according to the quantity identified this time (e.g., overwrite or accumulate).

[0153] Finally, the controller determines the execution result of the food management task based on the updated food storage information. In some implementations, the execution result may be, for example, a summary food inventory list or a "Food updated" message. The controller presents this execution result to the user on the refrigerator display interface, for example, by displaying the names and quantities of food in each storage area in a list format.

[0154] In other implementations, the controller can also perform difference comparisons based on historical records. When a decrease in the quantity of a certain ingredient is detected, the controller can automatically record the removal behavior and generate a consumption log.

[0155] In the above embodiments, by obtaining the food location recognition result and the food quantity recognition result based on the target food recognition result, matching the location result with the preset storage area inside the refrigerator, and updating the storage information based on the quantity result, the automatic perception of the food location and quantity is realized, eliminating the need for manual input by the user, simplifying the operation process, improving the efficiency of food management, and ensuring the accuracy of food management information.

[0156] In some embodiments, the multimodal data also includes video data. The controller extracts keyframes from the video data as image data, performs food target tracking based on the video data, obtains position change information of the food to be identified, and determines the food position recognition result based on the position change information.

[0157] In some implementations, the controller uses temporal contrastive learning to compare features of consecutive frames in the video data and extract the best matching feature frame as the keyframe.

[0158] The controller acquires video data of user interaction through a built-in or external image acquisition device in the refrigerator. First, the controller extracts keyframes from the video data, using these keyframes as input for subsequent image recognition. In feasible implementations, the controller employs temporal contrastive learning to compare features of adjacent or spaced frames in the video, calculating the feature similarity or difference between frames, and selecting the keyframe that best matches the features of preceding and following frames. For example, when a user slowly places food into a drawer, the video contains multiple frames; the controller uses temporal contrastive learning to select the frame with the most stable food posture and the least occlusion as the keyframe.

[0159] Subsequently, the controller tracks the food object based on the video data. Using target tracking algorithms (such as Kalman filtering, optical flow, or deep learning-based trackers), the controller locates the bounding box of the food object to be identified in consecutive frames, obtaining its positional change information, such as the trajectory of the food moving from outside the refrigerator to inside the refrigerator drawer. Based on this positional change information, the controller determines the area where the food finally resides, thus determining the food location identification result.

[0160] In some embodiments, determining the execution result of the food management task based on the updated food storage information specifically includes: generating a replenishment list based on the food quantity identification result and the target food identification result when the updated food storage information indicates that the food inventory in the refrigerator is insufficient, and using the replenishment list as the execution result of the food management task; and / or generating a time-limited food list when the updated food storage information indicates that the food in the refrigerator meets the expiration reminder conditions, and using the time-limited food list as the execution result of the food management task.

[0161] In this context, "insufficient inventory" refers to the remaining quantity of food ingredients falling below a preset inventory threshold, while "replenishment list" refers to a suggested purchase list generated by the controller when the remaining quantity of one or more types of food ingredients in the refrigerator falls below the preset inventory threshold, serving as a reminder to the user to replenish the corresponding food ingredients.

[0162] Expiration reminder conditions refer to the remaining shelf life of food items being less than a preset number of days (e.g., "less than 3 days until expiration" or already expired); the expiration date food list refers to a list of food items that the controller generates to prioritize or process when certain food items in the refrigerator meet the expiration reminder conditions, reminding the user to process them in a timely manner.

[0163] In some implementations, the controller obtains the current quantity of each ingredient from the updated food storage information and compares it with a preset inventory threshold. In feasible implementations, the inventory threshold can be preset by the user or automatically generated by the controller based on historical consumption rates. When the current quantity of a certain ingredient is detected to be lower than the inventory threshold, the controller determines that the inventory is insufficient. Finally, the controller generates a replenishment list based on the ingredient quantity identification result and the target ingredient identification result. For example, the replenishment list may include one or more of the following: ingredient name, suggested purchase quantity, etc. Finally, the controller displays the replenishment list as the execution result of the food management task on the refrigerator display interface.

[0164] In other implementations, the controller obtains the shelf-life information of each ingredient from the updated food storage information. In feasible implementations, the shelf-life information can be determined through methods such as manual user input, image recognition of the production date and shelf-life on the packaging, or querying the default shelf-life from a food database. Based on this, the controller calculates the remaining days between the current date and the expiration date and compares them with a preset expiration reminder threshold. When the remaining days are less than or equal to the threshold, or the shelf-life has expired, the controller determines that the expiration reminder condition is met and generates a list of expired ingredients. For example, the list of expired ingredients may include one or more of the following: ingredient name, remaining days, storage location, and suggested handling method. Finally, the controller displays the list of expired ingredients as the result of the food management task on the refrigerator display interface.

[0165] In some other implementations, the controller can generate both a replenishment list and a time-limited food list simultaneously, and display them either together or separately.

[0166] In the above embodiments, by generating a replenishment list when food inventory is insufficient and a time-limited food list when the expiration reminder conditions are met, users can be proactively reminded to purchase scarce food items, avoiding inconvenience caused by forgetting and improving user convenience. In addition, the time-limited food list can remind users to prioritize the disposal of food items that are about to expire, reducing food waste, and alert users to expired food items, ensuring food safety. Moreover, it does not require manual settings by users, realizing full automation and intelligence in food management.

[0167] In some embodiments, when the task to be performed is a recipe generation task, the execution result of the task to be performed is determined based on the target ingredient identification result, specifically including: generating recipe information based on the user's health data and the target ingredient identification result; and using the recipe information as the execution result of the recipe generation task.

[0168] Health data refers to information related to a user's physical condition, dietary restrictions, or health goals. For example, health data includes, but is not limited to, one or more of the following: a user's physical examination report (such as indicators of blood lipids, blood sugar, blood pressure, uric acid, etc.), known allergens (such as allergies to seafood or peanuts), chronic disease information (such as diabetes, hypertension, gout), personal health goals (such as weight loss, muscle gain, cholesterol control), and dietary habits and preferences (such as vegetarianism, low salt, low sugar).

[0169] In some implementations, users can upload medical examination reports via a mobile application or directly enter health information on the refrigerator interface.

[0170] Recipe information refers to a recipe preparation plan suitable for the user's consumption, generated based on the ingredients currently in the refrigerator and the user's health data. For example, recipe information includes, but is not limited to, one or more of the following: dish name, required ingredients and quantities, preparation steps, cooking time, nutrition label, and health tips.

[0171] When the task to be performed is recipe generation, the controller first obtains the user's health data. This health data can be pre-stored in the refrigerator's memory (e.g., the user's last uploaded medical report) or obtained in real time through user interaction.

[0172] The controller acquires the target ingredient identification results and matches and filters the user's health data against available ingredients. For example, if the user's health data shows high blood lipids, the controller will prioritize low-fat ingredients (such as chicken breast instead of fatty beef) and avoid generating recipes containing high-cholesterol ingredients (such as excessive egg yolks). If the user has diabetes, the controller will avoid generating recipes high in sugar (such as sugary drinks and desserts) and prioritize recommending combinations of ingredients with a low glycemic index.

[0173] The controller calls the local recipe database or a cloud-based recipe generation service to generate recipe information that meets the requirements. The controller then uses this generated recipe information as the result of the recipe generation task and displays it on the refrigerator's display interface.

[0174] In some implementations, the content displayed may include one or more of the following: recipe name, a list of required ingredients (indicating whether all are in the refrigerator and which are missing), preparation steps, a link to a cooking video, and a health rating.

[0175] In other implementations, the controller can generate multiple candidate recipes, sorted by health suitability or user preference, for the user to choose from.

[0176] In the above embodiments, recipe information is generated based on the user's health data and the target ingredient identification results. Individual health differences are incorporated into the recipe recommendations, avoiding the health risks that may be brought about by general recipes and realizing personalized dietary guidance. In addition, by combining structured health data such as physical examination reports, it is possible to accurately avoid ingredients that users cannot tolerate and proactively recommend healthy food combinations, which helps users improve their dietary structure and health status, and upgrades the refrigerator from a simple storage device to a personalized health diet management terminal.

[0177] In some embodiments, when the task to be performed is a food health management task, the execution result of the task to be performed is determined based on the target food identification result, specifically including: determining the food taken by the user based on the user's interaction actions and the target identification result; generating health reminder information based on the user's health data and the food taken by the user; and using the health reminder information as the execution result of the food health management task.

[0178] When the task to be performed is food health management, the controller first detects the user's interaction and, combined with the target food identification result, determines the specific food the user is currently taking. For example, if a user takes a bottle of beverage from the refrigerator, the camera or sensor inside the refrigerator detects this action, and the controller, combined with the target food identification result, determines that the food the user has taken is a beverage.

[0179] The controller then retrieves pre-stored user health data and matches it with the food the user chooses, generating health tips. For example, if a user has diabetes and chooses a beverage, the system might generate a tip such as "It is recommended to control blood sugar."

[0180] The controller uses health alerts as the result of food health management tasks and displays them in real time on the refrigerator's display interface, or it can issue reminders to users via voice broadcast.

[0181] In some implementations, after receiving the execution result, the controller can generate video data for the refrigerator display interface. Specifically, for the video output data (such as short videos), the controller uses a transformer deep network model consisting of convolutions, activation functions, and layer normalization of a fully attention-based module to form an attention convolution module to complete the decoding task. First, the full attention mechanism is used to uniformly model the features themselves and the relationships between features, forming a high-level unified feature vector or expression; then, convolution operations, activation calculations, and layer normalization are performed sequentially to obtain high-level semantic features with long-distance dependencies, and finally, the corresponding video data is decoded and generated.

[0182] In addition, the controller supports outputting execution results in text, image, or voice formats. The output results include, but are not limited to, the name of the food ingredient, location information, freshness status, and expiration information, and can be displayed in a visually appealing manner with both text and images.

[0183] In the above embodiments, the ingredients taken by the user are determined based on the user's interaction actions and target recognition results. Targeted health tips are generated by combining the user's health data. This allows for real-time perception of the user's taking behavior and proactive intervention, preventing the user from making unfavorable choices due to a lack of understanding of the health effects of the ingredients. Furthermore, the tips are customized based on individual health data, ensuring high accuracy and avoiding the ineffectiveness of general tips.

[0184] In refrigerator food identification scenarios, traditional single-modal recognition models (such as those relying solely on images or solely on speech) struggle to handle complex food identification scenarios due to their limited information dimensions. This is especially true when speech descriptions are vague or packaging appearances are similar, leading to the model easily producing illusionary results that contradict reality. Furthermore, existing multimodal model architectures typically involve simply splicing or later fusing data from different modalities, resulting in insufficient illusion suppression capabilities.

[0185] Figure 6 This is a schematic diagram of a multimodal model structure provided in some embodiments of this application. For example... Figure 6 As shown in the embodiments of this application, the multimodal model includes a data acquisition layer, an aggregation layer, a perception layer, and a multimodal fusion layer. The data acquisition layer is used to acquire voice data supporting wake-up and wake-free modes, as well as video data with images and audio. The aggregation layer receives the voice data, extracts voice features, generates an aggregation, and then processes it through a voice converter (decoder with max...). The aggregation layer converts the token (and other illusion control mechanisms) into text; simultaneously, it receives video data, samples video features, detects existing features, extracts and stores frame information, and then processes it through a video extractor. The text and video features output from the aggregation layer enter the perception layer: the perception layer extracts keyframes based on temporal comparison learning between consecutive frames, and distinguishes keyframe details for different tasks through a keyframe task adapter (fusion of image tokens and task IDs). Finally, the multimodal fusion layer receives the keyframe features output from the perception layer and the text output from the aggregation layer, which are processed by the LLM text model, image / object classification model, video model, food object detection model, and packaged food ingredient recognition model, respectively. The processed results are then bidirectionally aligned and verified by the language model error correction and rule correction module (spelling, grammar error correction, and rule correction) and the detection result validity verification module, thereby completing the complete inference from multimodal input to target food ingredient recognition results and suppressing the illusion of food identification inside the refrigerator.

[0186] In some embodiments, after acquiring speech data, the controller first extracts speech features. Then, the controller determines whether the speech features are unknown, i.e., whether they are unlearned speech features. Specifically, the controller can construct fine-grained judgment factors for multiple dimensions of the speech features, such as speech frame information and speech attribute distribution, and make a comprehensive judgment based on these factors. If the speech features are known features, the extracted speech features are input to the speech decoder; if the speech features are unknown features, the speech data is sampled and new speech features are dynamically generated, and then input to the decoder with hallucination suppression functionality. The decoder controls the maximum word size (max)... The token speech feature sequence is used to threshold the new token speech features and maximize token compression, thereby improving the decoding efficiency of speech features. At the same time, it effectively suppresses the hallucinations that may occur in the speech data at the decoder level, and finally outputs the specific text content.

[0187] It should be noted that in practical applications, the food packaging model can run independently or be routed as an expert module in the aforementioned multimodal fusion layer (e.g., the "packaged food ingredient recognition model" in the hybrid expert model). The packaging recognition results output by this model (including packaging text information and appearance category) can be integrated into the ingredient feature information or ingredient image recognition results for subsequent illusion verification and correction.

[0188] In other embodiments, to improve the accuracy of identifying packaged food items (such as bagged snacks, boxed milk, bottled beverages, vacuum-packed meat, etc.), the multimodal fusion layer includes a packaged food item recognition model. In some embodiments, the packaged food item recognition model includes a packaging area detection module, a packaging text recognition module, and a packaging appearance classification module. First, the packaging area detection module locates the position of the video packaging area from the food image data or keyframes based on a target detection network, filtering out non-packaging backgrounds. For scenarios where packaged food is held and placed in a refrigerator drawer, this module can also combine the aforementioned hand tracking results to accurately locate the packaging before and after it crosses the drawer boundary. Second, the packaging text recognition submodule uses optical character recognition or an end-to-end text recognition model based on deep learning to extract key information such as the food name, brand, net content, production date, and shelf life from the located packaging area. For packaging text with mixed Chinese and English text, artistic fonts, or low-light conditions, this submodule can pre-enhance the image (such as contrast stretching and binarization) and adapt it to multiple language character sets. Furthermore, the packaging appearance classification submodule utilizes convolutional neural networks and other technologies to classify the packaging's material, shape, color, and other appearance features, serving as a supplement to the text recognition results. When the packaging text is blurred or missing, the appearance classification results can be used as the primary basis for recognition.

[0189] In some other embodiments, the controller completes the keyframe extraction process through the following steps: First, it uses temporal contrastive learning to compare and learn the features of consecutive frames in the video data, and extracts the optimal matching feature frame as the keyframe; then, it obtains the feature task adapter information of the keyframe, uses the fusion mode of image lexical and task identifier to distinguish the detailed information of the keyframe, and judges the combined features of lexical and task identifier based on this; then, it obtains the most critical frame feature data by calculating the similarity of the combined features; finally, it stores the optimal keyframe data in a distributed data storage space or object storage space so that it can be accessed in a short time.

[0190] For example, text generation and understanding can be accomplished using a language model. This model supports a decoder. Only, encoder Only) and end-to-end codec (Encoder) This paper proposes a hybrid modular language model system that utilizes three network structures: Decoder, Decoder, and Decoder. A modular approach is employed to optimize and improve the core algorithm. For example, the attention mechanism can be specifically improved by using multi-head attention. Head Attention (MHA); a feedforward network is applied to the decoder according to the target task. Forward Network (FFN) and normalized residual direct connection are used. The model architecture incorporates a Mixture of Experts (MoE) model, for example, using one shared expert and 255 routing experts, each corresponding to different food categories or application scenarios. This collaborative work through routing division of labor improves interaction efficiency. Furthermore, model compression and multimodal multi-term prediction (Multi-term prediction) can be integrated into the model. One or more of the following can be used: Token Prediction (MTP), Rotary Position Embedding (RoPE), context learning, and reinforcement learning improvement strategies (such as Distributed Proximal Policy Optimization, DPRO). In the inference optimization module, Process Supervision Learning (PSL) can also be employed, combined with sampling training strategies such as Proximal Policy Optimization (PPO) or Direct Preference Optimization (DPO), and synthetic data obtained from knowledge retrieval can be introduced to train preference optimization, thereby obtaining an inference model for complex tasks.

[0191] It should be noted that language models can be constructed using various neural network architectures. Applicable models include, but are not limited to, language models with Graph Neural Networks (GNNs), attention mechanisms, Convolutional Neural Networks (CNNs), Mamba architecture, Transformers, and their variations or improvements; distilled network language models; latent diffusion models or diffusion models; deep network models such as U-Nets; and fused deep network models, such as those based on CNN / Mamba / Transformer deep convolutional networks and their derivative variants, such as Bidirectional Long Short-Term Memory (BiLSTM) + Multi-Modal Convolutional Neural Network (MMCNN) + Recurrent Neural Network (RNN), BiLSTM + MMCNN + attention mechanism, MMCNN + BiLSTM + attention mechanism fusion models, and gated recurrent units. One or more of the following: GRU+CNN, GRU+CNN+attention mechanism models; deep reinforcement learning and reward models; Retrieval-Augmented Generation (RAG) series of language models; and Gaussian mixture deep neural network models.

[0192] The method for suppressing illusions related to refrigerator food identification results provided in this application can be implemented by a device for suppressing illusions related to refrigerator food identification results. This application uses an example of a device for suppressing illusions related to refrigerator food identification results executing the method for suppressing illusions related to refrigerator food identification results to illustrate the device provided in this application.

[0193] This application also provides a device for suppressing illusions in refrigerator food identification results. Figure 7 This is a schematic diagram of the structure of a device for suppressing illusions related to refrigerator food identification results provided in some embodiments of this application. For example... Figure 7 As shown, the illusion suppression device for the refrigerator food identification results includes: an acquisition module 701, a first identification module 702, a second identification module 703, a verification module 704, and a correction module 705.

[0194] The acquisition module 701 is used to acquire multimodal data of the food to be identified. The multimodal data includes at least image data and voice data. The voice data is used to extract the feature information of the food. The first identification module 702 is used to identify ingredients based on ingredient feature information and obtain initial ingredient identification results. The second recognition module 703 is used to recognize food ingredients based on image data and obtain food ingredient image recognition results; The verification module 704 is used to perform illusion verification on the initial food identification result based on the matching relationship between the initial food identification result and the food image identification result; The correction module 705 is used to correct the initial identification result of the food ingredient based on the food ingredient image recognition result when the illusion verification result indicates that there is an illusion error in the initial identification result of the food ingredient, so as to obtain the target food ingredient identification result.

[0195] The illusion suppression device for refrigerator food identification results provided in this application embodiment acquires multimodal data including image data and voice data of the food to be identified, and extracts food feature information from the voice data. It obtains the attributes of the food inside the refrigerator from both voice and image dimensions, providing richer semantic information for subsequent food identification and avoiding identification bias caused by incomplete information in a single modality. Subsequently, it performs food identification based on the food feature information to obtain initial food identification results, and performs food identification based on the image data to obtain food image identification results. This achieves parallel food identification along different modal paths, performing inference on the food inside the refrigerator along both the voice and image paths, providing mutually independent information for subsequent verification. The identification process is based on several steps. First, based on the matching relationship between the initial food identification result and the food image identification result, a hallucination check is performed on the initial food identification result. This check identifies erroneous outputs caused by model hallucinations in the initial food identification result, thus verifying the model output and effectively suppressing hallucination phenomena in the food identification result. Finally, if the hallucination check result indicates that the initial food identification result contains hallucination errors, the initial food identification result is corrected based on the food image identification result. The corrected identification result is then used as the final output, effectively eliminating identification errors caused by model hallucinations and improving the accuracy of food identification in the refrigerator.

[0196] In some embodiments, the first identification module is further configured to extract core feature information and associated feature information from the food feature information; map the core feature information to the implicit space and the explicit space respectively to obtain implicit feature information and explicit feature information; fuse the implicit feature information, explicit feature information and associated feature information, and perform food identification based on the fused feature information.

[0197] In some embodiments, the first identification module is further configured to generate scene prompt words for the food feature information, the scene prompt words including at least one of basic scene prompt words, refined scene prompt words and specific scene prompt words; enhance the food feature information based on the scene prompt words to obtain enhanced food feature information; and identify the food based on the enhanced food feature information.

[0198] In some embodiments, the first recognition module is further configured to acquire training data; the training data includes food feature samples; train the teacher model based on the training data to obtain teacher model parameters; transfer the teacher model parameters to the student model; adjust the student model so that the difference between the parameters of the student model and the parameters of the teacher model is constrained within the range of difference; and use the adjusted student model as the food text recognition model.

[0199] In some embodiments, the verification module is further configured to determine the initial identification vector of the initial identification result of the food ingredient; determine the food image identification vector of the initial identification result of the food ingredient; obtain the similarity between the initial identification vector of the food ingredient and the food image identification vector; and perform illusion verification on the initial identification result of the food ingredient based on the similarity.

[0200] In some embodiments, the correction module is further configured to replace the initial food identification result with the food image identification result; or to fuse the food image identification result with the initial food identification result to obtain a fused identification result, and then replace the initial food identification result with the fused identification result.

[0201] In some embodiments, the correction module is further configured to determine the task to be executed based on user interaction actions; the task to be executed includes at least one of a food ingredient management task, a recipe generation task, and a food ingredient health management task; determine the execution result of the task to be executed based on the target food ingredient identification result, and update the food ingredient display information in the refrigerator display interface based on the execution result. In some embodiments, the correction module is further configured to obtain food location identification results and food quantity identification results based on the target food identification results; match the food location identification results with the preset food storage areas inside the refrigerator to obtain food storage information; update the food storage information based on the food quantity identification results; and determine the execution result of the food management task based on the updated food storage information.

[0202] In some embodiments, the correction module is further configured to generate a replenishment list based on the food quantity identification result and the target food identification result when the updated food storage information indicates that the food inventory in the refrigerator is insufficient, and to use the replenishment list as the execution result of the food management task; and / or to generate a time-limited food list when the updated food storage information indicates that the food in the refrigerator meets the expiration reminder conditions, and to use the time-limited food list as the execution result of the food management task.

[0203] In some embodiments, the correction module is further configured to generate recipe information based on the user's health data and the target ingredient identification results; and to use the recipe information as the execution result of the recipe generation task.

[0204] In some embodiments, the correction module is further configured to determine the ingredients taken by the user based on the user's interaction actions and target recognition results; generate health reminder information based on the user's health data and the ingredients taken by the user; and use the health reminder information as the execution result of the ingredient health management task.

[0205] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0206] The illusion suppression device for refrigerator food identification results in this application embodiment can be an electronic device or a component of an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal, such as a server.

[0207] The illusion suppression device for refrigerator food identification results provided in this application embodiment can realize the various processes implemented in the above-described embodiment of the illusion suppression method for refrigerator food identification results. To avoid repetition, it will not be described again here.

[0208] Based on the same inventive concept, this application also provides a refrigerator, including a controller. The controller is used to implement the various processes of the above-described embodiment of the illusion suppression method for identifying food ingredients in the refrigerator, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0209] It should be noted that the refrigerator provided in this application embodiment supports multimodal interaction, including various input methods such as gestures, voice, text, audio and video, image acquisition, microphone arrays, and radar signal acquisition, as well as extended tools such as optical character recognition, virtual reality, and augmented reality; it supports communication protocols such as WiFi and Bluetooth, and network connection methods such as 5G, WiFi, wired networks, and 6G; the food recognition results can be presented through applications, personal computers, web pages, mini-programs, and large screens, and the displayed content includes, but is not limited to, the food name, location information, freshness status, and expiration information, and can be displayed with pictures and text or dynamic video. This refrigerator can be applied to household, hotel, industrial, indoor and outdoor, military, and aerospace fields, and the embedding methods include flush mounting, vehicle embedding, and integrated installation.

[0210] In some embodiments, Figure 8 This is a schematic diagram of the structure of an electronic device provided in some embodiments of this application. For example... Figure 8 As shown, this application embodiment also provides an electronic device 800, including a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the program is executed by the processor 801, it implements the various processes of the above-described embodiment of the illusion suppression method for identifying refrigerator food results and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0211] This application provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described embodiment of the illusion suppression method for identifying refrigerator food ingredients, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0212] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable media, such as computer read-only memory (ROM), random-access memory (RAM), magnetic disks, or optical disks.

[0213] The computer-readable storage medium may include: read-only memory (ROM), random-access memory (RAM), magnetic disk or optical disk, etc.

[0214] This application provides a computer program product, including a computer program that, when executed by a processor, implements the above-mentioned method for suppressing illusions related to refrigerator food identification results.

[0215] This application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described embodiment of the illusion suppression method for refrigerator food identification results, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0216] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0217] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0218] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0219] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0220] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0221] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for suppressing hallucinations based on refrigerator food identification results, characterized in that, include: Acquire multimodal data of the food ingredient to be identified, wherein the multimodal data includes at least image data and speech data; the speech data is used to extract feature information of the food ingredient. Based on the food feature information, food identification is performed to obtain the initial food identification result; Based on the image data, food ingredients are identified to obtain food ingredient image recognition results; Based on the matching relationship between the initial food identification result and the food image identification result, an illusion verification is performed on the initial food identification result; If the illusion verification result indicates that the initial identification result of the food ingredient contains an illusion error, the initial identification result of the food ingredient is corrected based on the food ingredient image identification result to obtain the target food ingredient identification result.

2. The method for suppressing hallucinations based on refrigerator food identification results according to claim 1, characterized in that, The process of identifying ingredients based on the ingredient feature information includes: Extract the core and related feature information from the food ingredient feature information; The core feature information is mapped to the implicit space and the explicit space respectively to obtain implicit feature information and explicit feature information; The implicit feature information, the explicit feature information, and the associated feature information are fused together, and food ingredients are identified based on the fused feature information.

3. The method for suppressing hallucinations based on refrigerator food identification results according to claim 1, characterized in that, The process of identifying ingredients based on the ingredient feature information includes: Generate scene prompts for the food feature information, wherein the scene prompts include at least one of basic scene prompts, refined scene prompts, and specific scene prompts; The food feature information is enhanced based on the scene prompt words to obtain enhanced food feature information; Food identification is performed based on enhanced food feature information.

4. The method for suppressing hallucinations based on refrigerator food identification results according to claim 1, characterized in that, The food ingredient recognition is based on a trained food ingredient text recognition model; the training steps for the food ingredient text recognition model include: Acquire training data; the training data includes food feature samples; The teacher model is trained based on the training data to obtain the teacher model parameters; Transfer the teacher model parameters to the student model; The student model is adjusted so that the difference between the parameters of the student model and the parameters of the teacher model is constrained to a range of differences; The adjusted student model was used as the food ingredient text recognition model.

5. The method for suppressing hallucinations based on refrigerator food identification results according to claim 1, characterized in that, The process of performing illusion verification on the initial food identification result based on the matching relationship between the initial food identification result and the food image identification result includes: Determine the initial identification vector of the food ingredient based on the initial identification result; Determine the food image recognition vector of the food image recognition result; Obtain the similarity between the initial recognition vector of the food ingredient and the image recognition vector of the food ingredient; The initial identification results of the ingredients are verified based on the similarity.

6. The method for suppressing hallucinations based on refrigerator food identification results according to claim 1, characterized in that, The step of correcting the initial identification result of the food ingredient based on the food ingredient image recognition result includes: Replace the initial food identification result with the food image identification result; or The food image recognition result is fused with the initial food recognition result to obtain a fused recognition result, and the initial food recognition result is replaced with the fused recognition result.

7. The method for suppressing hallucinations based on refrigerator food identification results according to claim 1, characterized in that, The method further includes: The task to be executed is determined based on user interaction actions; the task to be executed includes at least one of the following: ingredient management task, recipe generation task, and ingredient health management task. The execution result of the task to be executed is determined based on the target food identification result, and the food display information in the refrigerator display interface is updated based on the execution result.

8. The method for suppressing hallucinations based on refrigerator food identification results according to claim 7, characterized in that, When the task to be performed is a food ingredient management task, determining the execution result of the task based on the target food ingredient identification result includes: Based on the target ingredient identification results, the ingredient location identification results and ingredient quantity identification results are obtained; The food location identification results are matched with the preset food storage areas inside the refrigerator to obtain food storage information; The food storage information is updated based on the food quantity identification result; The execution result of the food management task is determined based on the updated food storage information.

9. The method for suppressing hallucinations based on refrigerator food identification results according to claim 7 or 8, characterized in that, Determining the execution result of the food management task based on the updated food storage information includes: If the updated food storage information indicates insufficient food inventory in the refrigerator, a replenishment list is generated based on the food quantity identification result and the target food identification result, and the replenishment list is used as the execution result of the food management task; and / or If the updated food storage information indicates that the food in the refrigerator meets the expiration reminder conditions, a list of food items with expiration dates is generated, and the list of food items with expiration dates is used as the execution result of the food management task.

10. The method for suppressing hallucinations based on refrigerator food identification results according to claim 7, characterized in that, When the task to be performed is a recipe generation task, determining the execution result of the task based on the target ingredient identification result includes: Recipe information is generated based on the user's health data and the target ingredient identification results; The recipe information is used as the result of the recipe generation task.

11. The method for suppressing hallucinations based on refrigerator food identification results according to claim 7, characterized in that, When the task to be performed is a food health management task, determining the execution result of the task based on the target food identification result includes: The ingredients selected by the user are determined based on the user's interaction actions and the target recognition results; Based on the user's health data and the ingredients the user consumes, generate health alert information; The health alert information will be used as the result of the food health management task.

12. A refrigerator, characterized in that, include: A controller configured to perform the illusion suppression method based on the refrigerator food identification results as described in any one of claims 1-11.