Refrigerator food recognition model training method, refrigerator, and storage medium

Abstract

The application discloses a refrigerator food material recognition model training method, a refrigerator and a storage medium, and belongs to the technical field of home appliances. The refrigerator food material recognition model training method comprises the following steps: obtaining multi-modal sample data of a food material to be recognized, wherein the multi-modal sample data at least comprises voice sample data and video sample data; extracting food material visual features from the video sample data and food material text features from the voice sample data; fusing the food material associated features, the food material visual features and the food material text features to obtain target food material recognition features; inputting the target food material recognition features into a preset food material recognition model to obtain a food material prediction result; obtaining an identification loss value based on the difference between the food material prediction result and food material label information; and updating network parameters of the preset food material recognition model based on the identification loss value to obtain a food material recognition model used for managing food materials in the refrigerator. The application improves the recognition accuracy of food materials in the refrigerator.

Description

Technical Field

[0001] This application belongs to the field of home appliance technology, and in particular relates to a training method for a refrigerator food recognition model, a refrigerator, and a storage medium. Background Technology

[0002] With the development of smart home appliance technology, large language models and multimodal models are widely used in the fields of food identification and dietary health management.

[0003] In training refrigerator food recognition models, related technologies typically rely on short-term data from a single interaction or data from a single modality to update the model, in order to achieve rapid recognition of food inside the refrigerator and iterative updates of model parameters.

[0004] However, this approach is difficult to handle complex interactive scenarios where multiple dynamic information is intertwined during the user's food storage and retrieval process. Furthermore, single-modal data or short-term data from a single interaction are insufficient to capture the inherent relationships between the aforementioned dynamic information. This results in low recognition accuracy of the trained refrigerator food recognition model, which is prone to misidentification or omission, thus affecting the user experience. Summary of the Invention

[0005] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, this invention proposes a training method for a refrigerator food identification model, a refrigerator, and a storage medium to improve the accuracy of food identification within the refrigerator.

[0006] Firstly, this application provides a training method for a refrigerator food recognition model, the method comprising: Acquire multimodal sample data of the food to be identified. The multimodal sample data includes at least voice sample data and video sample data. The video sample data is used to obtain the food association features between user interaction operations and the food to be identified. Visual features of food ingredients are extracted from video sample data, and textual features of food ingredients are extracted from audio sample data. By fusing food association features, food visual features, and food text features, the target food identification features are obtained; Input the target ingredient identification features into the preset ingredient identification model to obtain the ingredient prediction result; The identification loss value is obtained based on the difference between the food ingredient prediction results and the food ingredient labeling information; the food ingredient labeling information is obtained by labeling multimodal sample data. The network parameters of the preset food identification model are updated based on the recognition loss value until the training termination condition is met, thus obtaining a food identification model for managing food in the refrigerator.

[0007] According to one embodiment of this application, the fusion of food-related features, food visual features, and food text features specifically includes: obtaining feature samples from the user's historical interaction operations; extracting discrimination data associated with food-related features, food visual features, and food text features from the feature samples respectively; and performing weighted feature fusion on the food-related features, food visual features, and food text features based on the discrimination data.

[0008] According to one embodiment of this application, the fusion of food-related features, food visual features, and food text features specifically includes: obtaining preset food feature expansion rules; generating food expansion features based on the food feature expansion rules to supplement food-related features, food visual features, and food text features; and fusing the food expansion features, food-related features, food visual features, and food text features.

[0009] According to one embodiment of this application, the fusion of food-related features, food visual features, and food text features specifically includes: obtaining the user's historical interaction operations and determining the interaction scenarios corresponding to the historical interaction operations; obtaining the historical features of food in the interaction scenarios; and fusing the historical features of food with the food-related features, food visual features, and food text features.

[0010] According to one embodiment of this application, the preset food ingredient recognition model includes a memory model, and the method further includes: acquiring the user's historical interaction operations; matching the scene data representing the interaction scenario in the user's historical interaction operations with the food ingredient prediction results to obtain interaction events, and acquiring an interaction event sequence composed of multiple interaction events; associating the target food ingredient recognition features with the interaction event sequence to obtain a target interaction event sequence; and updating the parameters of the memory model based on the target interaction event sequence.

[0011] According to one embodiment of this application, the memory model includes a short-term memory model; updating the memory model based on a target interaction event sequence specifically includes: determining the maximum sequence length that the short-term memory model can store; if the length of the target interaction event sequence exceeds the maximum sequence length, filtering the interaction events in the target interaction event sequence based on time order and / or event similarity to make the length of the target interaction event sequence match the maximum sequence length; updating the value function of the short-term memory model based on the filtered target interaction event sequence to update the short-term memory model.

[0012] According to one embodiment of this application, the memory model includes a long-term memory model; updating the memory model based on a target interaction event sequence specifically includes: acquiring multiple historical target interaction event sequences; determining at least one baseline interaction event sequence from the multiple historical target interaction event sequences according to a preset sequence rule; and updating the value function of the long-term memory model based on the baseline interaction event sequence to update the long-term memory model.

[0013] According to one embodiment of this application, the memory model includes a pseudo-experience replay model; updating the memory model based on the target interaction event sequence specifically includes: obtaining correct execution samples from the user's historical interaction operations to obtain demonstration sample data; obtaining counterfactual interaction data based on the target interaction event sequence; constructing a reward function based on the demonstration data and / or counterfactual interaction data; iteratively optimizing the reward function; and updating the pseudo-experience replay model based on the optimized reward function.

[0014] According to one embodiment of this application, the food ingredient prediction result includes a predicted category result, a predicted location result, and a predicted quantity result; the food ingredient labeling information includes a labeled category result, a labeled location result, and a labeled quantity result; based on the difference between the food ingredient prediction result and the food ingredient labeling information, an identification loss value is obtained, specifically including: determining a category loss value based on the difference between the predicted category result and the labeled category result; determining a location loss value based on the difference between the location result and the labeled location result; determining a quantity loss value based on the difference between the predicted quantity result and the labeled quantity result; and obtaining an identification loss value based on the category loss value, the location loss value, and the quantity loss value.

[0015] According to one embodiment of this application, after obtaining a food identification model for managing food in a refrigerator, the method further includes: identifying the food in the refrigerator based on the food identification model to obtain food identification results; generating food management information based on the food identification results; and updating the food display information in the refrigerator display interface based on the food management information.

[0016] Secondly, this application provides a refrigerator including a processor configured to implement a training method for a refrigerator food recognition model as described in the first aspect above.

[0017] Thirdly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the training method for the refrigerator food recognition model as described in the first aspect above.

[0018] Fourthly, this application provides a training device for a refrigerator food recognition model, the device comprising: The acquisition module is used to acquire multimodal sample data of the food to be identified. The multimodal sample data includes at least voice sample data and video sample data. The video sample data is used to acquire the food association features between user interaction operations and the food to be identified. The extraction module is used to extract visual features of food ingredients from video sample data and textual features of food ingredients from audio sample data. The fusion module is used to fuse food association features, food visual features, and food text features to obtain target food recognition features; The input module is used to input the target ingredient recognition features into the preset ingredient recognition model to obtain the ingredient prediction result; The loss module is used to obtain the recognition loss value based on the difference between the food ingredient prediction results and the food ingredient labeling information; the food ingredient labeling information is obtained by labeling multimodal sample data. The update module is used to update the network parameters of the preset food identification model based on the recognition loss value until the training termination condition is met, so as to obtain a food identification model for managing food in the refrigerator.

[0019] Fifthly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the training method for the refrigerator food recognition model as described in the first aspect above.

[0020] Fifthly, this application provides a chip, which includes a processor and a communication interface, the communication interface being coupled to the processor, and the processor being used to run programs or instructions to implement the training method for the refrigerator food recognition model as described in the first aspect.

[0021] Sixthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the training method for the refrigerator food recognition model as described in the first aspect above.

[0022] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects: By acquiring multimodal sample data of the ingredients to be identified, as well as the ingredient association features between user interaction operations and the ingredients, both static attribute information and dynamic interaction information of the ingredients can be obtained simultaneously. This provides a rich data foundation containing semantic information of operational behavior for subsequent feature extraction and fusion, effectively avoiding the problem of insufficient information in ingredient identification from single-modal data. Furthermore, by extracting visual and textual features of the ingredients, and converting video and audio sample data into feature representations respectively, the ingredient identification model can perceive ingredient attributes from both visual and textual dimensions, effectively improving the model's ability to perceive multidimensional features of ingredients. Subsequently, the obtained ingredient association features, visual features, and textual features are fused to obtain the target ingredient identification features, making the input features more discriminative and anti-interference, effectively alleviating the problem of unstable single-dimensional feature recognition caused by visual occlusion, lighting changes, similar appearance, or differences in viewing angle, thereby effectively improving the model's ability to identify easily confused ingredients. The system first assesses the distinguishing ability of the target food. Then, it inputs the target food identification features into a preset food identification model, which outputs the food prediction result. Based on the difference between the food prediction result and the food label information, it obtains the identification loss value. By calculating the difference between the prediction result and the true label, it provides clear error feedback for the directional optimization of the model parameters, enabling the model to iteratively update in the direction of error reduction, effectively ensuring the convergence and stability of the training process. Finally, it updates the network parameters of the preset food identification model based on the identification loss value until the training termination condition is met, resulting in a food identification model for managing food in the refrigerator. By driving the continuous iterative optimization of the model parameters through the identification loss value, the final food identification model can effectively adapt to the diverse food category identification needs in the refrigerator scenario, achieving efficient identification and health management of food in the refrigerator, and effectively improving the identification accuracy and user experience of the refrigerator in the food management scenario.

[0023] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0024] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart illustrating the training method of the refrigerator food recognition model provided in some embodiments of this application; Figure 2 This is a schematic diagram of the prompt chain strategy provided in some embodiments of this application; Figure 3 This is a schematic diagram of the short-term memory model structure provided in some embodiments of this application; Figure 4 This is a schematic diagram of the long-term memory model structure provided in some embodiments of this application; Figure 5 This is a schematic diagram of the pseudo-experience replay model structure provided in some embodiments of this application; Figure 6 This is a schematic diagram of the multimodal model structure provided in some embodiments of this application; Figure 7 This is a schematic diagram of the training device structure for a refrigerator food recognition model provided in some embodiments of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in some embodiments of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0026] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0027] In refrigerator food identification, Visual Language Models (VLM) or Multi-Modal Large Language Models (M-LLM) are typically used to process image and audio data to identify information such as food type, quantity, and location. However, these models commonly suffer from "illusion" phenomena in refrigerator food identification scenarios, meaning the model outputs results that do not match reality. For example, they might identify packaged "crab sticks" as "beef," "salmon" as "fish oil," packaged "goji berries" as incorrect text with extra or missing characters, misidentify mixed Chinese and English labels on alcoholic beverages, misidentify "celery" as "leeks," and identify frozen "steak" as "lamb." Furthermore, for food images with blurred feature boundaries (such as different brands of mineral water bottles or similarly colored beverage cans), multiple identifications often yield inconsistent results, indicating a lack of robustness and stability in the model. These illusion phenomena and inconsistencies severely impact the accuracy of food identification, causing the refrigerator to display incorrect food information and degrading the user experience.

[0028] In related technologies, methods for suppressing hallucinations mainly include cue word engineering, supervised fine-tuning, and reward-based reinforcement learning. However, relying solely on cue word methods is insufficient to effectively suppress hallucinations. Furthermore, when faced with easily confused food items and image inputs with blurred feature boundaries, a single cue word can easily lead to conflicting content, resulting in a high error rate and hindering the improvement of accuracy. Supervised fine-tuning, on the other hand, is prone to cognitive biases, causing the model to catastrophically forget old categories when learning new food categories. Insufficient scale and diversity of training data also lead to overfitting of the model to the predicted data. Reinforcement learning based on reward models only provides scores without providing the scoring criteria or reasoning process, and the scores obtained from the reward function are limited to isolated intervals, making it difficult to fundamentally solve the problems of catastrophic forgetting and hallucinations.

[0029] In the construction of refrigerator food recognition models, relevant solutions usually rely solely on short-term memory from a single interaction to utilize experience, resulting in low experience utilization. This makes it difficult to compensate for the distribution differences between training data and prediction targets. Consequently, when faced with diverse food categories, random placement, and complex lighting conditions in real-world scenarios, the food recognition model struggles to effectively utilize historical interaction experience for self-optimization, ultimately affecting the accuracy and stability of food recognition results and reducing user experience.

[0030] In view of this, this application provides a training method for a refrigerator food identification model, which aims to solve the problem of low food identification accuracy caused by model illusion and the limitation of single-modal information. By acquiring multimodal sample data, extracting food visual features, food text features, and food association features generated by user interaction and fusing them, the model parameters are updated based on the difference between the fused features and the food label information, thereby obtaining a food identification model that can effectively suppress illusion and improve the ability to distinguish easily confused and similar-looking foods, so as to improve the identification accuracy of food in the refrigerator.

[0031] The following description, in conjunction with the accompanying drawings, details the training method of the refrigerator food recognition model provided in this application, as well as the refrigerator and storage medium, through specific embodiments and application scenarios.

[0032] The refrigerator food recognition model training method provided in this application can be executed by an electronic device or a functional module or entity within an electronic device capable of implementing the training method for the refrigerator food recognition model. The electronic device mentioned in this application includes, but is not limited to, a terminal or a server.

[0033] Among them, the terminals include, but are not limited to, one or more of the following: refrigerators, smart home terminals, smartphones, tablets, computers, etc.; refrigerators include, but are not limited to, refrigerators, freezers, and vehicle refrigerators in household refrigeration equipment, display freezers, large commercial refrigerators, ice makers, cold storage, etc. in commercial refrigeration equipment, and refrigerated containers in industrial refrigeration equipment.

[0034] A server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0035] In the embodiments of this application, the training method of the refrigerator food recognition model can be executed by the local controller of the refrigerator alone, or by a cloud server that communicates with the refrigerator, or by both in combination. This application does not limit this.

[0036] The following describes the training method of the refrigerator food recognition model provided in this application embodiment, taking the controller as the execution subject as an example. Exemplarily, the controller may be one or more of the following: a central processing unit (CPU), a microcontroller unit (MCU), or a processing chip or module with data processing and instruction execution capabilities.

[0037] Figure 1 This is a flowchart illustrating the training method of a refrigerator food recognition model provided in some embodiments of this application. For example... Figure 1 As shown, the training method for the refrigerator food recognition model includes steps 110 to 160.

[0038] Step 110: Obtain multimodal sample data of the food to be identified. The multimodal sample data includes at least voice sample data and video sample data. The video sample data is used to obtain the food association features between user interaction operations and the food to be identified.

[0039] Multimodal sample data refers to data containing two or more different modalities of the food items to be identified inside the refrigerator. In some implementations, multimodal sample data includes, but is not limited to, at least two of the following: voice sample data, video sample data, image sample data, text sample data, and audio-visual sample data.

[0040] Furthermore, voice sample data refers to audio signal data containing user voice commands or voice descriptions of food information; video sample data refers to a sequence of continuous image frames containing the interior scene of the refrigerator, the food placement status, and the user's operation process.

[0041] Food association features refer to feature information used to characterize the relationship between user interaction operations and the food to be identified, and to characterize the spatial positional relationship and temporal change relationship between the interaction operation and the food to be identified. For example, the interaction operation includes, but is not limited to, one or more of the following: food storage and retrieval operations, hand-held food movement operations, and food position adjustment operations within the refrigerator; food association features include, but are not limited to, one or more of the following: hand posture features generated during the interaction operation, relative positional features between the hand and the food, positional change features of the food within the refrigerator space, and motion trajectory features of the food during hand-held or movement.

[0042] In some embodiments, the food association features between user interaction operations and the food to be identified can be obtained through video sample data. For example, the controller performs target detection and tracking on a continuous sequence of image frames in the video sample data to obtain the spatial positional relationship and temporal changes between the user's hand and the food during the interaction operation as food association features.

[0043] In some implementations, voice and video sample data can be acquired using a data acquisition device installed on the refrigerator. This acquisition device includes, but is not limited to, one or more combinations of a camera, microphone, sensor, microphone, and radar signal collector. In other implementations, voice and video sample data can be acquired using an external device communicatively connected to the refrigerator. This external device includes, but is not limited to, one or more mobile phones, tablets, smart wearable devices, and mobile applications. In still other implementations, voice and video sample data can be acquired from a remote server or cloud storage via wired or wireless communication methods. These communication methods include, but are not limited to, one or more of WiFi, Bluetooth, cellular networks, and Transmission Control Protocol / Internet Protocol (TCP / IP).

[0044] The controller collects multimodal sample data of the food to be identified by calling the data acquisition device configured on the refrigerator and external devices. This data is used to extract visual features, textual features, and association features of the food. The multimodal sample data includes at least audio and video sample data. Following this, the controller uses the video sample data to obtain the association features between user interactions and the food to be identified, which serve as feature information reflecting the semantic dimension of user behavior in subsequent feature fusion.

[0045] For example, when a user opens the refrigerator door and takes food (such as an apple) out of the refrigerator, a camera deployed on the refrigerator captures continuous video data including the user's hand and the food, while a microphone simultaneously captures the user's voice data (such as "take out an apple"). The controller performs object detection and tracking algorithms on the continuous image frame sequence in the video data to identify and track the hand area and the food area, obtaining the relative positional relationship between the hand and the food, as well as the motion trajectory of the food as the hand moves, as food association features.

[0046] Through the above steps, the controller can acquire multimodal sample data containing visual appearance information, speech semantic information, and user operation behavior information, providing a data foundation for subsequent multidimensional feature extraction and fusion.

[0047] Step 120: Extract visual features of ingredients from video sample data and extract text features of ingredients from audio sample data.

[0048] Food visual features refer to the feature information extracted from video sample data used to characterize the visual appearance attributes of the food to be identified. Food visual features include, but are not limited to, one or more of the following: color features, shape features, texture features, size features, surface pattern features, and packaging marking features. In some implementations, food visual features can be obtained by inputting keyframe images from the video sample data into a visual feature extraction model.

[0049] Food ingredient text features refer to the feature information extracted from speech sample data that characterizes the semantic descriptive attributes of the food ingredient to be identified. Food ingredient text features include, but are not limited to, one or more of the following: food ingredient name features, food ingredient category features, food ingredient quantity features, and food ingredient state description features. In some implementations, food ingredient text features can be obtained by transcribing the speech sample data into text data and then semantically encoding the text data.

[0050] In some implementations, the controller performs frame sampling on a continuous sequence of image frames in the video sample data, and inputs the sampled image frames into a pre-trained model, such as a Contrastive Language-Image Pre-Training (CLIP) model, to obtain the semantic embedding vector of each image frame. Subsequently, the controller calculates the semantic similarity based on the semantic embedding vector of each image frame, identifies image frames with semantic similarity below a preset threshold as redundant frames and removes them, identifies the remaining image frames as key frames, and inputs the key frames into a visual feature extraction model to obtain the visual features of the food ingredients.

[0051] In other implementations, the controller performs voice activity or endpoint detection on the voice sample data, detects valid voice segments from the voice sample data, and inputs the valid voice segments into a speech recognition-to-transcription device to obtain the corresponding text data. Afterward, the controller inputs the text data into a language model for semantic encoding to obtain the text features of the food ingredient. For example, the speech recognition-to-transcription device can be implemented using a speech recognition model based on a deep neural network.

[0052] In feasible implementations, the controller may preprocess the video sample data before extracting the visual features of the food ingredients. Exemplarily, preprocessing includes, but is not limited to, one or more of video decoding, frame rate normalization, image scaling, image enhancement, and denoising. In other feasible implementations, the controller may also preprocess the speech sample data before extracting the textual features of the food ingredients. Exemplarily, preprocessing includes, but is not limited to, one or more of denoising, echo cancellation, and beamforming.

[0053] Through the above steps, the controller can extract visual and textual information about ingredients from multimodal sample data, so that the subsequent fused recognition features have multiple semantic information.

[0054] Step 130: Fuse the food association features, food visual features, and food text features to obtain the target food identification features.

[0055] In this context, fusion refers to the process of integrating feature vectors from different modalities or multiple information channels according to preset rules to form a unified feature representation. Feature fusion includes, but is not limited to, one or more of the following fusion methods: feature concatenation, attention-based weighted fusion, feature addition, and feature multiplication. Target ingredient recognition features refer to the comprehensive feature representation obtained by fusing ingredient association features, ingredient visual features, and ingredient text features, which is used as input to a preset ingredient recognition model for ingredient prediction. In other words, target ingredient recognition features integrate feature information from three dimensions: the visual appearance information, linguistic semantic information, and user interaction behavior information of the ingredient.

[0056] After acquiring the food association features, food visual features, and food text features respectively, the controller fuses the three to obtain the target food recognition features.

[0057] In some implementations, the controller can concatenate food association features, food visual features, and food text features along the feature dimension to obtain a multi-dimensional fused feature as the target food identification feature. For example, if the food visual features are vector dimension d1, the food text features are vector dimension d2, and the food association features are vector dimension d3, then the concatenated target food identification feature is vector dimension d1+d2+d3.

[0058] In other implementations, the controller can calculate the importance weight of each feature for the food identification task, and then perform a weighted sum of the food association features, food visual features, and food text features based on their respective importance weights to obtain the target food identification features. For example, the controller can automatically learn the importance weights of each feature using an attention mechanism network or a gating network.

[0059] In some other implementations, the controller may first perform cross-modal attention fusion of the visual features and textual features of the ingredients to obtain visual-language fusion features; then, it may further fuse the visual-language fusion features with the ingredients-related features to obtain the target ingredients recognition features.

[0060] In feasible implementations, since the vector dimensions of each feature may differ, the controller can unify the dimensions of each feature before fusion. For example, the controller can map the food association features, food visual features, and food text features to the same feature dimension using a fully connected layer or a linear mapping layer before performing the fusion process.

[0061] Through the above steps, the controller obtains target food identification features that integrate three dimensions: visual appearance information, linguistic semantic information, and user interaction behavior information. This provides richer input for the subsequent food identification model to predict food, thereby improving the accuracy of food identification.

[0062] Step 140: Input the target ingredient recognition features into the preset ingredient recognition model to obtain the ingredient prediction result.

[0063] Among them, the pre-built ingredient recognition model refers to a deep learning model that is pre-constructed and used to output ingredient recognition results based on input features. The pre-built ingredient recognition model includes, but is not limited to, one or more combinations of the following: Vision Language Model (VLM), Multi-Modal Large Language Model (M-LLM), Large Language Model (LLM), Transformer-based deep network model, Convolutional Neural Network (CNN)-based deep network model, and diffusion-based deep network model.

[0064] The food ingredient prediction result refers to the prediction result output by the preset food ingredient recognition model after reasoning about the input target food ingredient recognition features. For example, the food ingredient prediction result includes, but is not limited to, one or more of the following: food ingredient category information, food ingredient name information, food ingredient quantity information, food ingredient location information, and food ingredient confidence information.

[0065] The controller inputs the target food identification features obtained in step 130 into the preset food identification model, which then performs identification and reasoning on the target food identification features and outputs the food prediction result.

[0066] In some implementations, the preset food identification model may include, for example, an encoder and a decoder. The controller inputs the target food identification features into the encoder, which performs deep semantic encoding on the target food identification features to obtain encoded features; the controller inputs the encoded features into the decoder, which performs decoding inference on the encoded features and outputs the food prediction result.

[0067] In some implementations, the preset food identification model can also employ a decoder based on a deep neural network (Transformer) architecture with a self-attention mechanism. The input features are transformed layer by layer through a multi-head self-attention mechanism and a feedforward neural network, and the food prediction results are finally output through a classification head or a linear output layer.

[0068] In feasible implementations, the food ingredient prediction results may also include a prediction confidence score, which is used to characterize the degree of certainty of the preset food ingredient recognition model regarding the current prediction result. The higher the confidence score, the more certain the preset food ingredient recognition model is regarding the prediction result, and the lower the confidence score, the more uncertain the preset food ingredient recognition model is regarding the prediction result.

[0069] In other feasible implementations, before the preset food identification model outputs the food prediction result, the controller can also perform feature mapping on the target food identification features through the preset food identification model to generate the probability distribution corresponding to each preset food category, so that the controller can determine the food category corresponding to the maximum probability value based on the probability distribution, as the food category information in the food prediction result.

[0070] Through the above steps, the controller obtains the food prediction results output by the preset food recognition model for the target food recognition features, providing a predictive basis for calculating the recognition loss value in subsequent steps.

[0071] Step 150: Based on the difference between the food ingredient prediction results and the food ingredient labeling information, obtain the identification loss value; the food ingredient labeling information is obtained based on the labeling of multimodal sample data.

[0072] In this context, food labeling information refers to the tag data obtained by annotating the true category, true quantity, and true location of the food to be identified in the multimodal sample data. Food labeling information serves as a supervisory signal, used to measure the difference between the food prediction results output by the preset food identification model and the actual situation. In feasible implementations, food labeling information includes, but is not limited to, one or more of the following: food category labeling, food name labeling, food quantity labeling, and food location labeling.

[0073] The recognition loss value is a quantitative indicator used to characterize the degree of difference between the predicted result and the labeled information of the food ingredient. Generally, the larger the recognition loss value, the greater the deviation between the prediction result of the preset food ingredient recognition model and the true label; the smaller the recognition loss value, the closer the prediction result is to the true label. The recognition loss value is used to drive the network parameter update of the preset food ingredient recognition model during the training phase.

[0074] After obtaining the food prediction results output by the preset food recognition model, the controller compares the food prediction results with the food labeling information, calculates the difference between the two, and generates a recognition loss value.

[0075] In some implementations, the controller can use a cross-entropy loss function to calculate the recognition loss value. Specifically, the controller performs cross-entropy calculation between the predicted category probability distribution in the food ingredient prediction results and the true category label in the food ingredient labeling information to obtain the recognition loss value.

[0076] In other implementations, the controller can use a mean squared error loss function to calculate the recognition loss value. Specifically, the controller calculates the mean squared error between the predicted values ​​(such as predicted quantity and predicted location coordinates) in the food prediction results and the actual values ​​in the food labeling information to obtain the recognition loss value.

[0077] In some other implementations, the controller may also use a joint loss function to calculate the recognition loss value, that is, to simultaneously calculate the weighted sum of the category prediction loss, the quantity prediction loss and the location prediction loss as the final recognition loss value.

[0078] In feasible implementations, food ingredient labeling information can be obtained through annotation. For example, during the training data preparation phase, the food ingredient categories, names, quantities, and locations in the multimodal sample data are labeled one by one to generate food ingredient labeling information. In other feasible implementations, food ingredient labeling information can also be obtained by semi-automatic annotation tools or through model pre-annotation. For example, the annotation tool automatically performs preliminary annotation on the food ingredient regions, followed by verification and correction, to generate the final food ingredient labeling information.

[0079] In other feasible implementations, after calculating the recognition loss value, the controller can also generate gradient information based on the recognition loss value and use the gradient information to update the network parameters of the preset food recognition model.

[0080] Step 160: Update the network parameters of the preset food identification model based on the recognition loss value until the training termination condition is met, and obtain the food identification model for managing food in the refrigerator.

[0081] Here, network parameters refer to the learnable parameters such as weights and biases of each network layer in the preset food identification model. During training, the controller continuously adjusts the network parameters so that the prediction results of the preset food identification model gradually approach the true labeled information.

[0082] Training termination conditions refer to the criteria used to determine whether the preset food recognition model has completed training. For example, training termination conditions include, but are not limited to, one or more of the following: the recognition loss value is less than or equal to a preset loss threshold, the number of training iterations reaches a preset maximum number of iterations, the accuracy of the validation set reaches a preset accuracy threshold, and the recognition loss value no longer decreases for a preset number of consecutive rounds.

[0083] A food identification model is the final model obtained after training, used to identify food items in the refrigerator during the inference phase. The food identification model can receive multimodal data of the food items to be identified and output the identification results for managing the food items in the refrigerator.

[0084] In feasible implementations, the management of food in the refrigerator includes, but is not limited to, responding to user queries by outputting information on the category, name, quantity, and location of food in the refrigerator; pushing reminders when food expires or the quantity falls below a preset threshold; and generating food purchase suggestions or replenishment lists based on the stock information of food in the refrigerator.

[0085] After obtaining the recognition loss value, the controller first calculates the gradient of each network parameter in the preset food recognition model based on the recognition loss value, and then uses an optimization algorithm to update each network parameter along the gradient direction to complete one iteration of training. After this, the controller returns to step 110 and repeats the process until the training termination condition is met, thus obtaining a food recognition model for managing food in the refrigerator.

[0086] In some implementations, the controller can use a backpropagation algorithm to calculate the gradient of each network parameter. Specifically, the controller propagates the recognition loss value back through the output layer to the input layer of the preset food recognition model, calculates the gradient value corresponding to each parameter in each network layer, and then updates the network parameters based on the gradient values.

[0087] In other implementations, the controller may employ optimization algorithms to update network parameters. Exemplary optimization algorithms include, but are not limited to, one or more of Stochastic Gradient Descent (SGD), Momentary Stochastic Gradient Descent, Adaptive Moment Estimation (Adam), and Root Mean Square Propagation (RMSProp).

[0088] In some other implementations, during each training iteration, the controller randomly samples a batch of samples from the multimodal sample data and inputs them into the preset food recognition model. It then calculates the average recognition loss value corresponding to that batch of samples and updates the network parameters until all training samples have been traversed, completing one iteration. After multiple iterations, when the training termination condition is met, the network parameters of the current preset food recognition model are saved as the final parameters of the food recognition model.

[0089] According to the training method of the refrigerator food recognition model provided in this application embodiment, by acquiring multimodal sample data of the food to be recognized and the food association features between user interaction operations and the food to be recognized, the static attribute information and dynamic interaction information of the food to be recognized can be obtained simultaneously. This provides a rich data foundation containing the semantics of operation behavior for subsequent feature extraction and fusion, effectively avoiding the problem of insufficient information in food recognition from single-modal data. Furthermore, by extracting the visual features and text features of the food, the video sample data and voice sample data are converted into feature representations respectively, enabling the food recognition model to perceive food attributes from both visual and textual dimensions, effectively improving the model's ability to perceive multidimensional features of the food. Subsequently, the obtained food association features, visual features, and text features are fused to obtain the target food recognition features, making the input features more discriminative and anti-interference, effectively alleviating the problem of unstable single-dimensional feature recognition caused by visual occlusion, changes in lighting, similar appearance, or differences in viewing angle. This process effectively enhances the model's ability to distinguish easily confused ingredients. Then, the target ingredient's identification features are input into a pre-defined ingredient identification model, which outputs a predicted ingredient result. Based on the difference between the predicted result and the ingredient's labeling information, a recognition loss value is obtained. Calculating the difference between the predicted result and the actual labeling provides clear error feedback for targeted optimization of the model parameters, allowing the model to iteratively update in the direction of error reduction, effectively ensuring the convergence and stability of the training process. Finally, the network parameters of the pre-defined ingredient identification model are updated based on the recognition loss value until the training termination condition is met, resulting in an ingredient identification model for managing ingredients in the refrigerator. By driving continuous iterative optimization of the model parameters through the recognition loss value, the final ingredient identification model can effectively adapt to the diverse ingredient category recognition needs in refrigerator scenarios, achieving efficient identification and health management of ingredients in the refrigerator, effectively improving the refrigerator's recognition accuracy and user experience in ingredient management scenarios.

[0090] In some embodiments, the controller acquires multimodal sample data of the food to be identified by invoking the acquisition device configured on the refrigerator and / or external devices. This data is used to extract visual features, textual features, and association features of the food during subsequent training. In feasible implementations, the voice sample data includes voice data with and without a wake-up word, while the video sample data includes a sequence of continuous image frames depicting the interior scene of the refrigerator, the placement of the food, and the user's operation. In other feasible implementations, the multimodal sample data may further include expert knowledge data, including action data and behavioral data related to interaction with the environment.

[0091] In other embodiments, after acquiring the multimodal sample data, the controller can also perform data aggregation processing on the multimodal sample data. Exemplarily, the data aggregation processing includes, but is not limited to, one or more of the following: voice data processing, video data processing, and keyframe extraction processing.

[0092] In some implementations, the controller preprocesses the noisy speech data to obtain pure speech data; the controller performs endpoint detection (Voice Activity Detection, VAD) on the pure speech data, inputs the detected valid speech segments to the speech recognition transcriber, and the speech recognition transcriber transcribes the valid speech segments into text data; the controller repairs or corrects the recognition errors generated during the speech recognition transcriber at the decoder level, and outputs the corrected text content for speech data processing.

[0093] In other implementations, the controller performs audio-video separation processing on the audio and video data to obtain separated audio and video data; the controller performs video feature sampling on the video data, detects whether video features have been extracted, extracts frame information and saves frame data for video data processing.

[0094] In some other implementations, the controller uses the CLIP model to extract keyframes from video data, stores the acquired keyframes in a cache, and constructs a cache chain mechanism: First, the controller obtains keyframes with similar temporal sequences and semantics through the CLIP model; second, the controller calculates the semantic score of each keyframe through semantic association or similarity, and removes redundant keyframes with semantic scores below a preset threshold; third, the controller stores the retained keyframes in the cache and compares the semantic information of the current keyframe with that of the nearest keyframe in the cache, removing the keyframe with the farthest semantic distance, so that the keyframes with the optimal semantic information are retained in the cache; finally, the controller constructs the optimal semantic chain of keyframes through the caching mechanism, and outputs the optimal keyframe to the next step for keyframe extraction processing.

[0095] In related technologies, when fusing food association features, food visual features, and food text features, direct concatenation or fixed-weight fusion methods are commonly used. However, in actual refrigerator food recognition scenarios, the contributions of different modal features to the recognition results are not constant. When the quality of food visual features deteriorates due to insufficient lighting or occlusion, the contribution of visual features should decrease; when the user's voice description is unclear or there is environmental noise, the contribution of text features should decrease; similarly, when the association between user interaction and food is ambiguous, the contribution of association features should also decrease. If a fixed-weight fusion method is used, it is difficult to adapt to the dynamic changes in modal quality, resulting in the fused target food recognition features failing to fully reflect the true contribution of each modal feature in the actual scenario, thus affecting the accuracy of food recognition.

[0096] Based on this, in some embodiments, the food-related features, food visual features, and food text features are fused, specifically including: obtaining feature samples from the user's historical interaction operations; extracting discrimination data associated with the food-related features, food visual features, and food text features from the feature samples respectively; and performing weighted feature fusion on the food-related features, food visual features, and food text features based on the discrimination data.

[0097] Among them, the feature samples in the user's historical interaction operations refer to the set of multimodal feature data collected and recorded when the user interacts with the refrigerator within a preset historical time period. For example, historical interaction operations include, but are not limited to, one or more of the following: the user's past operations of opening the refrigerator to access food, querying information about food in the refrigerator via voice or touch, and viewing or managing food through a terminal device.

[0098] Furthermore, the feature samples include historical food association features, historical food visual features, historical food text features collected during historical interaction operations, and the correctness markers of the recognition results corresponding to each historical feature.

[0099] Discriminant data refers to quantitative indicators used to evaluate the contribution of each feature to food identification and to determine the weight allocation of each feature in the fusion process. For example, discriminant data includes, but is not limited to, one or more of the following: confidence scores of each modality feature, signal-to-noise ratio of each modality feature, and consistency scores between each modality feature and food labeling information.

[0100] The controller first acquires feature samples from the user's historical interactions and extracts discrimination data associated with food-related features, visual features, and textual features from these samples. In some implementations, the controller can calculate the ratio of the number of correct recognition results for each modality feature in the historical interactions to the total number of recognitions, using this ratio as the confidence score for that modality feature. In other implementations, the controller can also calculate the semantic similarity between each modality feature in the feature samples and the food labeling information, as a consistency score.

[0101] Following this, the controller performs weighted feature fusion on the food association features, food visual features, and food text features based on the discrimination data. In some implementations, the controller uses the confidence score of each modality feature as a weighting coefficient to perform a weighted sum of the food association features, food visual features, and food text features to obtain the target food identification features. Specifically, modality features with higher confidence scores are assigned higher weights during fusion, while modality features with lower confidence scores are assigned lower weights.

[0102] In other implementations, the controller may employ a dynamic weight allocation strategy during feature fusion. Specifically, the controller inputs the discrimination data of each modal feature into a preset weight generation network, which outputs the fusion weights corresponding to each modal feature. The controller then uses these fusion weights to perform weighted fusion of each modal feature to obtain the target food identification features.

[0103] In some other implementations, the discrimination data may also include quality assessment scores for each modality feature. For example, the controller may calculate an image sharpness score for the visual features of the food, a speech recognition confidence score for the textual features of the food, and a target tracking stability score for the associated features of the food, and use the quality assessment scores as a reference for weight allocation.

[0104] In the above embodiments, by acquiring feature samples from users' historical interaction operations and extracting discrimination data, and then performing weighted fusion of the three modal features based on the discrimination data, the weight allocation of feature fusion is no longer fixed, but can be adaptively adjusted according to the actual quality and historical contribution of each modal feature, thereby improving the quality and robustness of the target food identification features.

[0105] In some embodiments, the fusion of food association features, food visual features, and food text features specifically includes: obtaining preset food feature expansion rules; generating food expansion features based on the food feature expansion rules to supplement food association features, food visual features, and food text features; and fusing the food expansion features, food association features, food visual features, and food text features.

[0106] Among them, the preset food feature expansion rules refer to a set of rules that are pre-defined to generate supplementary feature information based on existing food features. For example, the preset food feature expansion rules include, but are not limited to, one or more of the following: food category hierarchy expansion rules, food visual similarity expansion rules, and food association expansion rules.

[0107] Furthermore, food ingredient extension features refer to supplementary feature information generated according to preset food ingredient feature extension rules and associated with the food ingredient to be identified. These features are used to supplement the information of the currently extracted food ingredient association features, food ingredient visual features, and food ingredient text features.

[0108] The controller first obtains preset food feature expansion rules. In some implementations, the preset food feature expansion rules can be pre-stored in the controller or on a cloud server. For example, the preset food feature expansion rules can include one or more of the following: semantic hierarchy relationships between various food categories (e.g., the "vegetables" category includes "leafy vegetables" and "root vegetables", and "leafy vegetables" includes "spinach", "rapeseed", "leeks", etc.), visual similarity relationships (e.g., "apple" and "pear" have a higher appearance similarity than "apple" and "banana"), and food co-occurrence relationships in a refrigerator scenario (e.g., "milk" is often stored in the refrigerator at the same time as "yogurt", and "eggs" are often placed in the same storage area as "vegetables").

[0109] Based on food feature extension rules, the controller generates extended food features to supplement food association features, food visual features, and food text features. For example, the controller performs semantic parsing on the food names contained in the food text features, matches the parsed food names against a rule base, and obtains the category hierarchy information, visually similar food information, and co-occurring related food information of the food as extended food features.

[0110] Following this, the controller fuses the extended food features, associated food features, visual food features, and textual food features. For example, the controller can concatenate the extended food features with the three original modal features to obtain target food recognition features that integrate explicit modal information and implicit association information.

[0111] In other implementations, the preset food feature expansion rules may include food category hierarchy expansion rules. For example, when the controller identifies the food name as "Fuji apple" from the food text features, the controller obtains the category hierarchy information of "Fuji apple → apple → Rosaceae fruit → fruit" according to the category hierarchy expansion rules, encodes this hierarchy information as an expanded feature, and then fuses it with the original feature, so that the preset food recognition model can simultaneously perceive fine-grained category information and coarse-grained category information.

[0112] In some other implementations, the preset food feature extension rules may also include food visual similarity extension rules. For example, when the controller identifies from the food visual features that the current food is highly similar to the visual features of "cola", the controller generates a prompt message containing "similarity to visual features of cola ≥ 90%" as a food extension feature according to the visual similarity extension rules, which is used to prompt the preset food recognition model that the current food may be confused with cola in appearance.

[0113] In a feasible implementation, the controller can also dynamically adjust the food expansion features based on information about other food items currently in the refrigerator. For example, the controller obtains a list of food items currently in the refrigerator and, based on preset food feature expansion rules, generates one or more of the following: association features or difference features between the current food item and other food items in the refrigerator, as food expansion features.

[0114] In other feasible implementations, the preset ingredient feature expansion rules may also include contextual expansion rules. For example, the controller obtains current time information (such as morning, noon, evening) or seasonal information (such as summer, winter), and generates corresponding ingredient preference features as ingredient expansion features based on the time or seasonal information.

[0115] In the above embodiments, by obtaining preset food feature expansion rules, food expansion features are generated to supplement the current three modal features. The food expansion features are then fused with the original three modal features. This allows the target food recognition features to not only include the explicit information already present in the current multimodal sample data, but also to integrate food category hierarchy information, visual similarity information, and scene association information mined through prior rules. This effectively improves the generalization ability of the preset food recognition model for food categories not appearing in the training samples and cross-category recognition scenarios, thereby improving the accuracy of refrigerator food recognition.

[0116] In some embodiments, the fusion of food-related features, food visual features, and food text features specifically includes: obtaining the user's historical interaction operations and determining the interaction scenarios corresponding to the historical interaction operations; obtaining the historical features of food in the interaction scenarios; and fusing the historical features of food with the food-related features, food visual features, and food text features.

[0117] The user's historical interaction operations refer to all interaction behaviors related to food management that occurred between the user and the refrigerator within a preset historical time period before the current moment. In feasible implementations, the preset historical time period can be flexibly set according to the actual application scenario, such as the interaction operations of the past week, the past month, the past preset number of times (such as 50 times), or all interaction records since the refrigerator was first turned on.

[0118] It should be noted that the difference between a user's historical interaction operations and interaction operations is that interaction operations refer to the operation data collected in real time during the current single interaction, which is used to extract the food association features of the current batch; while historical interaction operations refer to the historical interaction data that has occurred and been recorded before the current moment, which is used to extract the user's historical interaction behavior patterns to assist in the current identification decision.

[0119] An interaction scenario refers to the specific operational context in which a user interacts with the refrigerator. For example, interaction scenarios include, but are not limited to, one or more of the following: food storage and retrieval scenarios, food query scenarios, food organization scenarios, and food restocking reminder scenarios. In feasible implementations, interaction scenarios are characterized by scenario identification information. For example, scenario identification information includes, but is not limited to, one or more of the following: current interaction time, interaction operation type, refrigerator door open / closed status, and sensor trigger signals inside the refrigerator.

[0120] Furthermore, the historical features of ingredients refer to the historical multimodal feature information extracted and recorded by the user for ingredients during historical interaction operations. For example, the historical features of ingredients include, but are not limited to, one or more of the following: historical ingredient visual features, historical ingredient text features, historical ingredient association features, historical recognition results, and historical recognition result correctness markers.

[0121] The controller acquires the user's historical interaction operations and determines the interaction scenario corresponding to each historical interaction operation. For example, the controller acquires all interaction records between the user and the refrigerator within a preset historical time period (such as the past 7 days), including the timestamp of each interaction, the type of interaction operation (access operation, query operation, organization operation), the refrigerator door opening and closing status, and sensor trigger signals, etc. Then, based on the above information, the controller classifies each historical interaction operation into a scenario, thereby determining the interaction scenario to which each historical interaction operation belongs.

[0122] Subsequently, the controller acquires the historical features of the ingredients in the interaction scenario. In some implementations, the controller uses the interaction scenario to which the current interaction belongs as the target scenario, filters out historical interaction operations belonging to the target scenario from the historical interaction records, and acquires the historical features of the ingredients corresponding to each historical interaction operation in that scenario. For example, when the current interaction belongs to the "access scenario," the controller acquires all the visual features and text features of the ingredients in the historical access scenarios, as well as the corresponding recognition results.

[0123] Following this, the controller performs feature fusion with the historical features of the ingredient, its associated features, its visual features, and its textual features. For example, the controller concatenates or weights the historical features of the ingredient with the three currently extracted modal features along the feature dimension to obtain the target ingredient recognition features.

[0124] In a feasible implementation, before acquiring the historical features of ingredients in the interactive scene, the controller can also determine whether the number of historical interactive operations corresponding to the target scene has reached a preset threshold. If the preset threshold is not reached, it indicates that the historical data of the current scene is insufficient, and the controller executes a preset default fusion strategy (such as only fusing the current three modal features and not adding historical features) to avoid fusion noise caused by the sparsity of historical data.

[0125] In other feasible implementations, the controller can perform statistical processing on multiple historical features of the food ingredients to generate scene prior features for the corresponding interaction scenario. For example, the controller performs average pooling or clustering processing on all historical food ingredient visual features in the target scenario to obtain representative food ingredient visual features in the scenario as scene prior features, and then fuses the scene prior features with the current three modal features.

[0126] In some other feasible implementations, the controller can also determine the weight of each historical interaction operation based on the time interval between the current interaction operation and the historical interaction operations after determining the interaction scenario corresponding to the historical interaction operation. For example, historical interaction operations with shorter time intervals from the current interaction operation are assigned higher weights, and historical interaction operations with longer time intervals are assigned lower weights, so that the fused features pay more attention to recent interaction history and improve the timeliness of features.

[0127] In the above embodiments, by acquiring the user's historical interaction operations and determining the corresponding interaction scenario, the historical features of the ingredients in the interaction scenario are obtained, and the historical features of the ingredients are fused with the current three modal features. This makes the target ingredient recognition features not only include the real-time multimodal information of the current interaction, but also incorporate the historical prior information under the same interaction scenario. This enables the preset ingredient recognition model to better utilize historical experience to assist in recognition decision-making based on the current interaction scenario, effectively improving the model's recognition adaptability and generalization ability under different interaction scenarios.

[0128] In some implementations, the controller can also optimize the food features based on a cue chain strategy to obtain optimized food features. Figure 2 This is a schematic diagram of the prompt chain strategy provided in some embodiments of this application. For example... Figure 2As shown, the cue chain strategy comprises three modules: cue distillation, cue inference, and cue caching. The controller acquires text or image tokens and constructs a cue distillation task based on these tokens. The cue distillation task builds a teacher model using pre-collected expert cue knowledge, resulting in an expert-level teacher model. This teacher model then transfers soft-label parameters and knowledge to the student model, enabling it to effectively recognize cue content at the token level. For example, the controller inputs text content transcribed from speech sample data into the student model, which outputs distilled token-level cue features. The cue content generated during this distillation process, along with corresponding state and action data, forms the basic input for the simulated experience playback tuple. Subsequently, after obtaining the token-level cue, the controller performs inference, which includes using causal inference, parallel inference, or logical inference techniques to calculate cue hit rates. For example, when the input prompt is a red can of Coca-Cola, the controller determines the attribute association between "red can" and "Coca-Cola" through causal reasoning, calculates the hit matching degree between the prompt and existing prompts in the preset prompt library, and outputs the reasoning hit result. The hit result output by this prompt reasoning process is used to determine whether the current prompt is a valid prompt. Prompts with high hit rates correspond to experience tuples with high reward evaluation values ​​and are retained in the short-term memory cache using a first-in-first-out (FIFO) strategy or a nearest neighbor strategy. Finally, the controller inputs the reasoned prompt into the prompt cache for hit calculation. The hit calculation includes calculating the hit matching degree between the current prompt and historical prompts in the cache using a cache pipeline or a nearest neighbor cache mechanism. For example, when a hit record for "red, can, Coca-Cola" already exists in the cache, the controller calculates the similarity between the current prompt and the cached record. If the similarity is higher than a preset threshold, a hit is determined, and the hit result and corresponding cached prompt information are output. The historical hit records accumulated during this prompt caching process form knowledge deposits in long-term memory.

[0129] The controller will use the optimized cue features obtained after cue distillation, cue reasoning and cue caching as food extended features or food history features, and fuse them with food association features and food visual features to obtain target food recognition features.

[0130] In some embodiments, the preset food identification model includes a memory model. The controller acquires the user's historical interaction operations; matches the scene data representing the interaction scenario in the user's historical interaction operations with the food prediction results to obtain interaction events, and acquires an interaction event sequence composed of multiple interaction events; associates the target food identification features with the interaction event sequence to obtain a target interaction event sequence; and updates the parameters of the memory model based on the target interaction event sequence.

[0131] The memory model refers to the network module in the pre-defined food identification model used to store and utilize historical interaction experience information. In feasible implementations, the memory model includes, but is not limited to, one or more of the following: short-term memory model, long-term memory model, and pseudo-experience replay model. Typically, the short-term memory model is used to store interaction event information within the current or recent interaction cycle; the long-term memory model is used to store important interaction event information within historical interaction cycles; and the pseudo-experience replay model is used to sample and replay the stored interaction events to assist in model training.

[0132] An interaction event refers to an event record with complete semantic information consisting of a single user interaction and its corresponding recognition result. In feasible implementations, an interaction event includes one or more of the following: interaction scenario information, interaction operation information, food ingredient prediction result information, and prediction result correctness marker information. For example, a complete interaction event can be represented as "the user took out a tomato in a storage scenario, the model recognized the tomato, and the recognition result is correct." Multiple interaction events arranged in chronological order constitute an interaction event sequence.

[0133] The controller acquires the user's historical interaction operations and matches the scene data representing the interaction scenario in the user's historical interaction operations with the food prediction results to obtain interaction events. In a feasible implementation, the controller extracts scene data for each interaction from the historical interaction operations. For example, scene data includes, but is not limited to, one or more of the following: interaction timestamp (e.g., morning, noon, evening), interaction operation type (access operation, query operation, organization operation), refrigerator door open / close status, and sensor trigger signals inside the refrigerator. The controller matches the scene data with the food prediction results corresponding to the interaction to generate an interaction event with a scene label. For example, the controller matches the scene data of "7:30 AM, opening the door to retrieve an item, access scenario" with the food prediction result of "model identifies it as milk" to obtain the interaction event "Access scenario - milk, recognition result correct".

[0134] Subsequently, the controller acquires an interaction event sequence consisting of multiple interaction events. For example, the controller arranges all interaction events within a preset historical time period (e.g., the past 7 days) in chronological order to generate the interaction event sequence. In other words, the interaction event sequence records the complete trajectory of the user's interaction behavior within a specific time window.

[0135] The controller associates the target ingredient identification features with the interaction event sequence to obtain the target interaction event sequence. In a feasible implementation, the controller obtains the target ingredient identification features corresponding to the current interaction, matches the ingredient category information in the target ingredient identification features with the ingredient prediction results of each interaction event in the interaction event sequence, and filters out historical interaction events associated with the current ingredient to form the target interaction event sequence. For example, if the current target ingredient identification features indicate that the ingredient is "tomato", the controller filters out all historical interaction events related to "tomato" from the interaction event sequence (such as "access scenario - tomato, identification result correct" and "query scenario - tomato, identification result correct") to form the target interaction event sequence associated with the current ingredient.

[0136] Following this, the controller updates the parameters of the memory model based on the target interaction event sequence. In one feasible implementation, the controller can encode the temporal and scene information in the target interaction event sequence into feature vectors, input the feature vectors into the memory model, and update the network parameters of the memory model through a backpropagation algorithm. In other feasible implementations, the controller can also use the target interaction event sequence as training samples for the memory model, and train and update the memory model using supervised learning or reinforcement learning methods.

[0137] In some implementations, before associating the target food identification features with the interaction event sequence, the controller can also calculate the semantic similarity between the target food identification features and the food prediction results of each interaction event in the interaction event sequence. When the semantic similarity is greater than a preset threshold, the interaction event is identified as an associated interaction event, and the target interaction event sequence is composed of multiple associated interaction events.

[0138] In other implementations, the memory model includes a short-term memory model and a long-term memory model. The controller stores the earlier-order interactive events in the target interactive event sequence into the short-term memory model, and stores the interactive events in the target interactive event sequence that meet a preset importance condition into the long-term memory model. For example, the controller stores interactive events that are closer to the current time into the short-term memory model, and stores interactive events that are correctly identified and have a high recurrence frequency into the long-term memory model.

[0139] In other implementations, after updating the parameters of the memory model based on the target interaction event sequence, the controller also adjusts the parameters of at least one network module of the preset food ingredient recognition model based on the updated memory model. For example, the controller uses the updated memory model parameters as prior information and adjusts the parameters of the backbone network of the preset food ingredient recognition model through feature concatenation or attention mechanisms.

[0140] In the above embodiments, by acquiring the user's historical interaction operations, matching the scene data in the historical interaction operations with the food prediction results to obtain interaction events and constructing an interaction event sequence, the memory model is updated based on the target interaction event sequence. This fully utilizes the temporal scene information in the user's historical interaction operations for parameter optimization, realizing memory enhancement based on cross-time interaction scene association. This effectively improves the temporal modeling and memory capabilities of the preset food recognition model in complex interaction scenarios, further enhancing the accuracy and stability of refrigerator food recognition.

[0141] In some embodiments, the memory model includes a short-term memory model; updating the memory model based on the target interaction event sequence specifically includes: determining the maximum sequence length that the short-term memory model can store; if the length of the target interaction event sequence exceeds the maximum sequence length, filtering the interaction events in the target interaction event sequence based on time order and / or event similarity to make the length of the target interaction event sequence match the maximum sequence length; updating the value function of the short-term memory model based on the filtered target interaction event sequence to update the short-term memory model.

[0142] The maximum sequence length of the short-term memory model refers to the maximum number of interactive events that the model can accommodate within a single storage cycle. In feasible implementations, the maximum sequence length is determined by the network structure parameters and computational resource limitations of the short-term memory model. For example, the maximum sequence length can be a preset value such as 10, 20, or 50.

[0143] A value function is a function used to evaluate the contribution of an interactive event to the parameter updates of a short-term memory model. The value function outputs a value assessment score; a higher score indicates greater training value for the short-term memory model. In feasible implementations, the value function is constructed based on one or more factors selected from the interactive event's temporal information, semantic information, contextual information, and historical hit information.

[0144] The controller first determines the maximum sequence length that the short-term memory model can store. For example, the controller reads the configuration parameters of the short-term memory model to obtain the maximum sequence length. For instance, the maximum sequence length can be preset to 20, meaning the short-term memory model can store a maximum of 20 interaction events.

[0145] Subsequently, the controller determines whether the length of the target interaction event sequence exceeds the maximum sequence length. If the length of the target interaction event sequence is less than or equal to the maximum sequence length, the controller does not need to perform filtering and stores all target interaction event sequences in the short-term memory model; if the length of the target interaction event sequence is greater than the maximum sequence length, the controller filters the interaction events in the target interaction event sequence based on time order and / or event similarity to ensure that the number of filtered interaction events matches the maximum sequence length.

[0146] In some implementations, the controller filters based on time sequence. Specifically, the controller calculates the time interval between each interaction event in the target interaction event sequence and the current time, prioritizing the retention of interaction events with shorter time intervals (i.e., more recent events) and discarding interaction events with longer time intervals (i.e., older events), until the number of retained interaction events equals the maximum sequence length.

[0147] In other implementations, the controller filters based on event similarity. Specifically, the controller calculates the similarity between each interaction event in the target interaction event sequence and the current interaction event. For example, similarity may include one or more of the following: scene similarity, food category similarity, and interaction operation type similarity. The controller prioritizes retaining interaction events with higher similarity and discards interaction events with lower similarity until the number of retained interaction events equals the maximum sequence length.

[0148] Furthermore, similarity can be determined, for example, by calculating the cosine similarity or Euclidean distance between the embedded feature vectors corresponding to the interactive events. For obtaining embedded feature vectors, taking scene similarity as an example, the controller pre-constructs a scene encoding table, mapping each interactive scene (such as access scene, query scene, and organization scene) to its corresponding scene embedding vector. For food category similarity, the controller maps food categories to category embedding vectors using preset food category codes, and calculates the cosine similarity between the food category embedding vector in each interactive event and the current interactive food category embedding vector, using this as a quantitative indicator of food category similarity. For interactive operation type similarity, the controller maps interactive operation types such as access operation, query operation, and organization operation to their corresponding operation embedding vectors, and determines the interactive operation type similarity by calculating the cosine similarity between the operation embedding vector in the interactive event and the current interactive operation embedding vector.

[0149] In some other implementations, the controller performs filtering based on a combination of time sequence and event similarity. Specifically, the controller first sorts the target interaction event sequence according to time sequence, assigning a first time weight to the interaction events that are ranked first; simultaneously, the controller calculates the similarity between each interaction event and the current interaction event, assigning a second time weight to interaction events with higher similarity; the controller determines a comprehensive score for each interaction event based on the weighted sum of the first time weight and / or the second time weight, and retains interaction events in descending order of comprehensive score until the number of retained interaction events equals the maximum sequence length.

[0150] Subsequently, the controller updates the value function of the short-term memory model based on the filtered target interaction event sequence to update the short-term memory model. For example, the controller inputs the filtered target interaction event sequence as training samples into the short-term memory model, calculates the value assessment value corresponding to each interaction event, and updates the value function parameters of the short-term memory model using algorithms such as temporal difference learning, thereby updating the parameters of the short-term memory model.

[0151] In feasible implementations, when filtering based on time sequence and / or event similarity, the controller can also combine the confidence level of the identification results for each interaction event. For example, the controller may prioritize retaining interaction events with higher confidence levels and discard interaction events with lower confidence levels.

[0152] In other feasible implementations, after updating the value function of the short-term memory model based on the filtered target interaction event sequence, the controller can also reorder the importance of each interaction event in the target interaction event sequence according to the updated value function for use in memory model update decisions.

[0153] Figure 3 This is a schematic diagram of the short-term memory model structure provided in some embodiments of this application. For example... Figure 3As shown, the controller acquires the streaming state token and action token for the current interaction cycle, and constructs a streaming state-action token sequence based on the streaming state token and action token, forming a pseudo-experience replay tuple. Here, the streaming state token is a vectorized representation obtained by encoding the target food identification features and / or the sequence of interaction events within the current interaction cycle; the action token is a vectorized representation obtained by encoding the food prediction result output by the preset food identification model and / or the corresponding model parameter update direction. The controller inputs the pseudo-experience replay tuple into the reward function module of the short-term memory model, which generates a set of token reward function values ​​or a sequence value to represent the immediate reward evaluation value of the token for the current state-action. The controller uses the set of token reward function values ​​or the sequence value as the reward token, and inputs it along with the streaming state token and action token into the replay cache module of the short-term memory model.

[0154] In some embodiments, the controller updates the cached content in the replay cache using a preset update strategy. In feasible implementations, the preset update strategy includes one or more of the following: a time-order-based first-in-first-out (FIFO) strategy and / or a data similarity-based nearest neighbor strategy, a value-based strategy, and / or a context-based strategy, to update the network parameters of the short-term memory model. For example, the controller can use a FIFO strategy where, when the number of pseudo-experience replay tuples stored in the replay cache reaches a preset maximum sequence length, the earliest-stored pseudo-experience replay tuple is removed from the cache, and the pseudo-experience replay tuple for the current interaction period is stored at the end of the cache. If the controller uses a nearest neighbor strategy, it calculates the feature distance between the current pseudo-experience replay tuple and each historical pseudo-experience replay tuple in the cache. When the feature distance is less than a preset threshold, the current pseudo-experience replay tuple replaces the historical pseudo-experience replay tuple with the closest feature distance, thus updating the cache and ensuring that the number of pseudo-experience replay tuples in the cache matches the preset maximum sequence length.

[0155] In other feasible implementations, the replay cache can also be used to store pseudo-experience replay tuples and provide these tuples for updating the network parameters of the short-term memory model during each training iteration. Specifically, the controller samples a set of pseudo-experience replay tuples from the replay cache, updates the value function of the short-term memory model based on the sampled tuples, and updates the network parameters of the short-term memory model by minimizing the temporal difference error corresponding to the value function, thereby training and updating the short-term memory model. Through the above steps, the controller achieves efficient caching and replay utilization of recent interaction experiences by the short-term memory model, providing timely historical experience support for the preset food recognition model.

[0156] In the above embodiments, by determining the maximum sequence length of the short-term memory model, when the target interaction event sequence length exceeds the limit, filtering is performed based on time order and / or event similarity to match the maximum sequence length. The value function of the short-term memory model is then updated based on the filtered sequence. This allows the short-term memory model to prioritize the retention of interaction events that are highly relevant to the current recognition task, have strong timeliness, or high semantic similarity within limited storage capacity. This avoids the problem of low-value events occupying storage resources and high-value events being discarded in a single time order filtering strategy. It effectively improves the storage efficiency of the short-term memory model and the accuracy of the value function update, further enhancing the contribution of the short-term memory model to the recognition capability of the preset food ingredient recognition model.

[0157] In some embodiments, the memory model includes a long-term memory model; updating the memory model based on the target interaction event sequence specifically includes: acquiring multiple historical target interaction event sequences; determining at least one benchmark interaction event sequence from the multiple historical target interaction event sequences according to a preset sequence rule; and updating the value function of the long-term memory model based on the benchmark interaction event sequence to update the long-term memory model.

[0158] Among them, the long-term memory model refers to the network module used to store and utilize information about important interaction events within the historical interaction cycle.

[0159] It should be noted that long-term memory models and short-term memory models differ in their storage time span and storage capacity. Specifically, short-term memory models are used to store interaction event information within the current or recent preset time window (such as the past few hours or days), and their storage capacity is limited. Long-term memory models are used to store important interaction event information over a longer time span (such as the past weeks, months, or longer), and their storage capacity is usually larger than that of short-term memory models and can continuously accumulate. In other implementations, the difference between long-term memory models and short-term memory models lies in the dimension of interaction event value assessment. Long-term memory models store interaction events determined to have long-term importance (such as frequently occurring ingredients or recurring interaction scenarios), while short-term memory models store interaction events with immediate timeliness.

[0160] A historical target interaction event sequence refers to a sequence of interaction events recorded within a historical time period that are associated with the identification characteristics of the target food ingredient, arranged in chronological order. For example, a historical target interaction event sequence includes a sequence of all interaction events related to "milk" in the past week, arranged in chronological order, or a sequence of all interaction events related to "vegetables" in the past month, arranged in chronological order.

[0161] Preset sequence rules refer to a set of pre-defined rules used to select a baseline sequence of interactive events from multiple historical sequences of target interactive events. For example, preset sequence rules include, but are not limited to, one or more of the following: sampling rules based on time span, sampling rules based on event frequency, sampling rules based on event diversity, and sampling rules based on event value assessment.

[0162] A baseline interaction event sequence refers to one or more representative interaction event sequences selected from multiple historical target interaction event sequences according to preset sequence rules, used as sample data for updating the value function of the long-term memory model.

[0163] The controller acquires multiple historical target interaction event sequences. For example, the controller reads all interaction event records within a preset historical time period (e.g., the past 30 days) from long-term memory storage, groups the interaction events according to different food categories in the target food identification features, and arranges the interaction events corresponding to the same food category in chronological order to form multiple historical target interaction event sequences. For instance, the controller acquires historical target interaction event sequences corresponding to "milk," "apple," and "tomato," etc.

[0164] The controller determines at least one baseline interaction event sequence from multiple historical target interaction event sequences according to preset sequence rules. In some embodiments, the preset sequence rules may include sampling rules based on event diversity, i.e., the controller selects the sequence containing the most diverse types of interaction events from all historical target interaction event sequences as the baseline interaction event sequence. In other embodiments, the preset sequence rules may include sampling rules based on time span, i.e., the controller selects the sequence with the longest time span from all historical target interaction event sequences as the baseline interaction event sequence. In still other embodiments, the preset sequence rules may also include sampling rules based on event frequency: the controller selects the sequence with the highest frequency of interaction events from all historical target interaction event sequences as the baseline interaction event sequence.

[0165] For example, the controller uses a combined sampling strategy. First, it selects the top K candidate sequences with the highest diversity scores from all historical target interaction event sequences based on event diversity. Then, it selects the sequence with the longest time span from the K candidate sequences as the benchmark interaction event sequence.

[0166] Following this, the controller updates the value function of the long-term memory model based on the benchmark interaction event sequence to update the long-term memory model. In a feasible implementation, the controller inputs the benchmark interaction event sequence into the value function network of the long-term memory model, calculates the long-term value assessment value corresponding to each interaction event in the benchmark interaction event sequence, and updates the value function network parameters of the long-term memory model by minimizing the temporal difference error. Specifically, the controller divides the benchmark interaction event sequence into multiple time steps according to time order, calculates the temporal difference error between the current state value and the state value of the next time step for each time step, and backpropagates the temporal difference error to the value function network to update the network parameters. In this way, the controller can use the benchmark interaction event sequence to train and update the long-term memory model.

[0167] In some implementations, before determining at least one baseline interaction event sequence from multiple historical target interaction event sequences according to preset sequence rules, the controller may preprocess each historical target interaction event sequence. This preprocessing may include one or more of the following: removing sequences with incomplete timing, removing sequences with fewer than a preset threshold number of interaction events, and removing sequences with a higher proportion of incorrect identification results than a preset threshold.

[0168] In other implementations, after updating the value function of the long-term memory model based on the baseline interaction event sequence, the controller can also re-evaluate the importance of the interaction events stored in the short-term memory model according to the updated value function of the long-term memory model, and transfer the interaction events that are determined to be important in the long term from the short-term memory model to the long-term memory model.

[0169] In feasible implementations, the value function updates of the long-term memory model and the short-term memory model can share some network parameters. Specifically, the value function networks of the long-term memory model and the short-term memory model share parameters at the bottom feature extraction layer, but operate independently at the top output layer, thereby enabling shared utilization of bottom-level features by both short-term and long-term memory.

[0170] Figure 4 This is a schematic diagram of a long-term memory model structure provided in some embodiments of this application. For example... Figure 4As shown, the controller acquires the state token and action token corresponding to the interaction event, constructs a state-action token sequence based on the state token and action token, and stores the state-action token sequence as interaction experience in the long-term memory model. Here, a state token is a vectorized representation obtained by encoding the scene data of the target food identification features and / or interaction event; an action token is a vectorized representation obtained by encoding the food prediction result and / or the corresponding model parameter update direction. The interaction experience includes one or more interaction events from the historical target interaction event sequence. The controller arranges the state tokens and action tokens corresponding to each interaction event in chronological order to form an interaction experience sequence and stores it in the storage unit of the long-term memory model. The controller samples at least one experience tuple from the long-term memory model. The experience tuple includes state tokens and action tokens. Based on the state tokens and action tokens in the sampled experience tuple, the controller updates the value function parameters of the long-term memory model through the value function network of the long-term memory model. Simultaneously, based on the state tokens and action tokens in the sampled experience tuple, the controller updates the policy parameters of the long-term memory model through the policy network.

[0171] In some implementations, the controller inputs the sampled state token into the value function network to obtain the current value corresponding to the current state; then it inputs the sampled action token and the corresponding reward evaluation value into the value function network to obtain the target value; subsequently, the controller calculates the temporal difference error between the current value and the target value, and backpropagates the temporal difference error to the value function network to update the network parameters.

[0172] In other implementations, the controller inputs the sampled state tokens into the policy network to obtain the probability distribution of each action under the current policy; the controller calculates the policy gradient based on the probability distribution and the sampled action tokens, and backpropagates the policy gradient to the policy network to update the network parameters.

[0173] In feasible implementations, the value function network and the policy network share some network parameters to improve the training efficiency and parameter update coordination of the long-term memory model.

[0174] In the above embodiments, by acquiring multiple historical target interaction event sequences, determining at least one benchmark interaction event sequence according to preset sequence rules, and updating the value function of the long-term memory model based on the benchmark interaction event sequence, the parameter update of the long-term memory model no longer relies on redundant calculations of all historical interaction events. Instead, it selects representative benchmark sequences from a large number of historical sequences for efficient updating through preset sequence rules. This effectively improves the update efficiency and accuracy of the long-term memory model's value function, enabling the long-term memory model to extract high-value temporal patterns and experiential knowledge from long-term historical interactions. This provides cross-timescale historical experience support for the preset food identification model, further enhancing the identification stability and adaptability of the refrigerator food identification model during long-term use.

[0175] In some embodiments, the memory model includes a pseudo-experience replay model; updating the memory model based on the target interaction event sequence specifically includes: obtaining correct execution samples from the user's historical interaction operations to obtain demonstration sample data; obtaining counterfactual interaction data based on the target interaction event sequence; constructing a reward function based on the demonstration data and / or counterfactual interaction data; iteratively optimizing the reward function; and updating the pseudo-experience replay model based on the optimized reward function.

[0176] Among them, the pseudo-experience replay model refers to a network model used to store interaction events and sample and replay the stored interaction events to assist in the training of a pre-set food identification model. Typically, the pseudo-experience replay model stores historical interaction experiences and randomly or according to a pre-set sampling strategy extracts experience samples from the storage for replay during the training process to break the temporal correlation between training samples and improve the stability of model training and the efficiency of sample utilization.

[0177] Demonstration sample data refers to interaction samples with correct identification results selected from users' historical interaction operations, used as positive experience to guide the construction of the reward function. For example, demonstration sample data includes interaction events in historical interactions where the identification result was correct and the user feedback was "correct".

[0178] Counterfactual interaction data refers to hypothetical interaction data generated by counterfactual reasoning based on actual interaction events in a target interaction event sequence. In some implementations, counterfactual reasoning involves changing one or more conditions in the actual interaction events (such as changing the interaction scenario, changing the food category, or changing the recognition result) to generate interaction data that did not actually occur but is logically plausible. Counterfactual interaction data is used to simulate recognition errors or biases to expand the coverage of training samples for pseudo-experience replay models.

[0179] The controller acquires correct execution samples from the user's historical interaction operations to obtain demonstration sample data. In some implementations, the controller extracts interaction events from historical interaction records where the recognition result is correct and the user feedback is "correct," and uses these interaction events as demonstration sample data. For example, after each user interaction, the refrigerator displays the recognition result and asks "Is the recognition result correct?" The user clicks "correct" or "incorrect" to provide feedback, and the controller filters the interaction events with feedback of "correct" as correct execution samples.

[0180] The controller generates counterfactual interaction data based on the target interaction event sequence. In some implementations, the controller selects actual interaction events from the target interaction event sequence and performs counterfactual reasoning on these actual interaction events. For example, the controller replaces the recognition result "apple" in the actual interaction event with another category, "pear," which has similar visual features to an apple, to generate counterfactual interaction data; or the controller replaces the interaction scenario in the actual interaction event from "access scenario" to "query scenario" to generate counterfactual interaction data. Through the above counterfactual reasoning, the controller generates a preset number of counterfactual interaction data to expand the coverage of the training samples.

[0181] In a feasible implementation, the controller can construct a reward function based on the demonstration sample data. The controller uses the state-action pairs in the demonstration sample data as positive samples and the correct recognition results corresponding to the demonstration sample data as target labels. It trains an initial reward function network using supervised learning, enabling the reward function network to output high reward scores for the state-action pairs in the demonstration sample data. For example, the controller extracts the joint feature vector of the state token and action token from the demonstration sample data, inputs the joint feature vector into the initial reward function network, and updates the parameters of the reward function network by minimizing the mean squared error between the reward score output by the reward function network and the preset high reward target value, thus obtaining the reward function constructed based on the demonstration sample data. The preset high reward target value is a pre-set reward score representing correct recognition. For example, the preset high reward target value can be set to a positive number such as 1 or 10, and the specific value can be flexibly set according to the actual application scenario.

[0182] In other feasible implementations, the controller can construct a reward function based on counterfactual interaction data. The controller uses state-action pairs from the counterfactual interaction data as negative samples and the corresponding hypothetical error identification results as auxiliary labels. It trains an initial reward function network using supervised learning or contrastive learning, causing the reward function network to output low reward scores for state-action pairs in the counterfactual interaction data. For example, the controller extracts the joint feature vector of state tokens and action tokens from the counterfactual interaction data, inputs the joint feature vector into the initial reward function network, and updates the parameters of the reward function network by minimizing the mean squared error between the reward score output by the reward function network and a preset low reward target value, thus obtaining the reward function constructed based on the counterfactual interaction data. The preset low reward target value is a pre-set reward score representing identification errors or biases. For example, the preset low reward target value can be set to 0 or -1, and the specific value can be flexibly set according to the actual application scenario.

[0183] In other feasible implementations, the controller constructs a reward function based on the demonstration sample data and the counterfactual interaction data. For example, the controller constructs a reward function that outputs a higher reward score for the demonstration sample data and a lower reward score for the counterfactual interaction data. Specifically, the controller uses the demonstration sample data as positive samples and the counterfactual interaction data as negative samples, employing a binary classifier or regression model to construct the reward function, enabling the reward function to distinguish between positive and counterfactual interaction samples.

[0184] Following this, the controller iteratively optimizes the reward function and updates the pseudo-experience replay model based on the optimized reward function. In some implementations, the controller uses adversarial training to iteratively optimize the reward function: the interactive events generated by the pseudo-experience replay model are input into the reward function, which outputs a reward score; the controller adjusts the parameters of the pseudo-experience replay model based on the reward score, making the model generate interactive events that more closely resemble the demonstration sample data. Simultaneously, the controller adjusts the discriminative boundary of the reward function based on the interactive events generated by the pseudo-experience replay model, forming an adversarial iterative optimization process. In this process, the reward function acts as a discriminator to distinguish between the interactive events generated by the pseudo-experience replay model and the demonstration sample data, while the pseudo-experience replay model acts as a generator to generate interactive events that can obtain high reward scores. The controller maximizes the reward function, i.e., optimizes its discriminative ability; simultaneously, the controller optimizes the pseudo-experience replay model, making the interactive events generated by the model more closely resemble the distribution of the demonstration sample data. Through this adversarial training, the controller iteratively optimizes the reward function until it can no longer effectively distinguish between the interactive events generated by the pseudo-experience replay model and the demonstration sample data, at which point the reward function reaches its optimal state. The controller updates the network parameters of the pseudo-experience replay model based on the optimized reward function.

[0185] In some implementations, when the controller obtains counterfactual interaction data based on the target interaction event sequence, it can also determine the diversity constraints of the counterfactual interaction data according to preset counterfactual generation rules. These diversity constraints are designed to ensure that the generated counterfactual interaction data differs from the actual interaction events in the target interaction event sequence in preset feature dimensions, thereby preventing the generated samples from being too similar to positive experiences and thus failing to provide effective negative training signals.

[0186] In other implementations, the reward function may include multiple reward sub-functions, each corresponding to a different evaluation dimension. For example, the reward function may include, but is not limited to, one or more of the following: category recognition reward sub-function, quantity recognition reward sub-function, and location recognition reward sub-function. The controller performs a weighted summation of the outputs of each reward sub-function to obtain the final reward score.

[0187] In a feasible implementation, after constructing the reward function based on the demonstration sample data and / or counterfactual interaction data, the controller can also re-label the experience tuples in the pseudo-experience replay model based on the optimized reward function. For example, the controller recalculates the reward score for each experience tuple using the optimized reward function and stores the updated reward score as new labeling information for that experience tuple in the pseudo-experience replay model.

[0188] In other feasible implementations, the controller further outputs the optimized reward function to the backbone network of the preset food identification model to assist in the parameter adjustment of the backbone network. Specifically, the controller uses the optimized reward function to score the output of the preset food identification model, and uses the score result as an auxiliary loss signal to jointly calculate with the identification loss value to update the parameters of the preset food identification model.

[0189] Figure 5 This is a schematic diagram of the pseudo-experience playback model structure provided in some embodiments of this application. For example... Figure 5 As shown, the controller acquires state-action pair data and constructs implicit or explicit rules based on this data. These rules map the state-action pair data to an accumulated reward function, which is then used as the generator's output. The accumulated reward function refers to the intermediate reward function output by the generator during adversarial training, before it has undergone balance adjustment. The controller inputs this accumulated reward function to a reward function discriminator, which then discriminates against it. Based on the discriminator's results, the controller optimizes the accumulated reward function through iterative adversarial training between the generator and the discriminator. The controller then maps the optimized accumulated reward function to a reward space. This reward space, representing the reward space of the quasi-experience replay model, is used to store and manage the optimized reward function. Within this reward space, the controller performs balance and diversity adjustments on the reward sequence of the optimized reward function, ensuring that the distribution of the reward sequence within the reward space meets preset balance and diversity conditions. This prevents the reward sequence from concentrating on isolated intervals, achieving both balance and diversity in the reward sequence.

[0190] Among these, balance adjustment refers to controlling the evenness of the distribution of reward values ​​in the reward space, avoiding concentration of reward values ​​in specific intervals; diversity adjustment refers to controlling the reward function to output reward values ​​that are discriminative for different state-action pairs, avoiding different state-action pairs receiving the same or overly similar reward values. The controller optimizes the reward function in the reward space to maximize its ability to distinguish between demonstration sample data and counterfactual interaction data, obtaining an optimized reward function. Finally, the controller updates the pseudo-experience replay model based on the optimized reward function.

[0191] It is easy to understand that without balance adjustment, the reward function may only output high reward values ​​for specific state-action pairs, while outputting low reward values ​​for other state-action pairs. This causes the reward values ​​to be concentrated in isolated ranges between high-scoring intervals or low-scoring intervals in the reward space, resulting in reward collapse. Without diversity adjustment, different state-action pairs may receive the same or too similar reward values, causing the reward function to be unable to effectively distinguish the quality differences of different interaction events.

[0192] In the above embodiments, demonstration sample data is obtained by acquiring correct execution samples from users' historical interactions, counterfactual interaction data is obtained based on the target interaction event sequence, a reward function is constructed based on the demonstration sample data and / or counterfactual interaction data, and the pseudo-experience replay model is updated after iterative optimization of the reward function. This allows the construction of the reward function to simultaneously integrate positive demonstration experience and counterfactual reasoning experience, enabling the reward function to more comprehensively cover various situations in the refrigerator food recognition scenario. This effectively improves the efficiency of the pseudo-experience replay model in utilizing positive experience and its adaptive correction ability for recognition bias, further enhancing the contribution of the pseudo-experience replay model to the robustness of the preset food recognition model. As a result, the generalization ability and fault tolerance of the refrigerator food recognition model in complex scenarios are improved.

[0193] In some embodiments, the food prediction results include predicted category results, predicted location results, and predicted quantity results; the food labeling information includes labeled category results, labeled location results, and labeled quantity results; based on the difference between the food prediction results and the food labeling information, an identification loss value is obtained, specifically including: determining a category loss value based on the difference between the predicted category results and the labeled category results; determining a location loss value based on the difference between the predicted location results and the labeled location results; determining a quantity loss value based on the difference between the predicted quantity results and the labeled quantity results; and obtaining an identification loss value based on the category loss value, the location loss value, and the quantity loss value.

[0194] The predicted category result refers to the predicted category of the food item output by the preset food item recognition model. For example, the predicted category result may include the model's predicted probability distribution for each preset food item category, such as "apple: 0.85, pear: 0.10, others: 0.05," etc. The predicted location result refers to the predicted location of the food item in the image or refrigerator space output by the preset food item recognition model. For example, the predicted location result may include the coordinates of the detection box in the area where the food item is located (such as the coordinates of the upper left and lower right corners of the detection box) or the coordinates and size information of the center point of the food item, etc. The predicted quantity result refers to the predicted quantity of the food item output by the preset food item recognition model. For example, the predicted quantity result may include the number of similar food items predicted by the model, such as "apples: 3," etc.

[0195] The label category result, label location result, and label quantity result refer to the actual category, actual location, and actual quantity of the food item corresponding to the label information. For example, the label category result is "apple", the label location result could be the coordinates of the food item area detection box, and the label quantity result is "3".

[0196] Furthermore, the category loss value, location loss value, and quantity loss value refer to the loss values ​​calculated based on the differences between the prediction results and the labeling information in the category dimension, location dimension, and quantity dimension, respectively, and are used to quantify the degree of prediction bias of the model in each task dimension.

[0197] The controller acquires the food prediction results output by the preset food recognition model. The food prediction results include the predicted category result, the predicted location result, and the predicted quantity result. In some implementations, the predicted category result is the probability distribution vector of each category output by the model, the predicted location result is the coordinate information of the food detection box output by the model, and the predicted quantity result is the numerical value of the food quantity output by the model.

[0198] The controller acquires food labeling information corresponding to the food prediction results. This labeling information includes label category results, label location results, and label quantity results. In some implementations, the label category result is a pre-labeled true food category label, the label location result is the pre-labeled true food detection box coordinate information, and the label quantity result is a pre-labeled true food quantity label.

[0199] The controller determines the class loss value based on the difference between the predicted class result and the labeled class result. In a feasible implementation, the controller uses the cross-entropy loss function to calculate the class loss value. In other feasible implementations, the controller may also use the focus loss function to calculate the class loss value to alleviate the class imbalance problem.

[0200] The controller determines the location loss value based on the difference between the predicted location and the marked location. In feasible implementations, the controller uses a smoothing loss function or a cross-union (CIoU) loss function to calculate the location loss value, substituting the coordinate differences (including center point coordinate differences, width differences, and height differences) between the predicted and marked detection boxes into the smoothing loss function to obtain the location loss value. In other feasible implementations, the controller can also use a complete intersection over union (CIoU) loss function to calculate the location loss value, to simultaneously consider the overlap area of ​​the detection boxes, the center point distance, and aspect ratio consistency.

[0201] The controller determines the quantity loss value based on the difference between the predicted quantity and the labeled quantity. In a feasible implementation, the controller uses the mean squared error loss function to calculate the quantity loss value. In other feasible implementations, the controller uses the mean absolute error loss function to calculate the quantity loss value, in order to reduce the impact of outliers on the gradient in quantity prediction.

[0202] Subsequently, the controller derives the recognition loss value based on the category loss value, location loss value, and quantity loss value. In some embodiments, the controller performs a weighted summation of the category loss value, location loss value, and quantity loss value to obtain the recognition loss value.

[0203] In other embodiments, before obtaining the identification loss value based on the category loss value, location loss value, and quantity loss value, the controller may also normalize each loss value to eliminate scale differences between the loss values ​​of different tasks. For example, the controller calculates the mean and standard deviation of the category loss value, location loss value, and quantity loss value in the current batch, normalizes each loss value to the same scale, and then performs a weighted summation.

[0204] In some other embodiments, the preset weight coefficients for the category loss, location loss, and quantity loss can be dynamically adjusted coefficients. The controller dynamically adjusts the magnitude of each weight coefficient based on the model's current performance metrics for each task on the validation set. For example, the controller calculates the category recognition accuracy, location detection accuracy, and quantity prediction accuracy on the validation set, assigning higher weight coefficients to dimensions with poor performance to guide the model to focus on those task dimensions during subsequent training.

[0205] In some implementations, the controller can adopt a multi-task learning network structure, which sets up category prediction branches, location prediction branches and quantity prediction branches on the basis of a shared backbone network. Each branch independently outputs the corresponding prediction result, calculates its corresponding loss value, and then performs joint backpropagation.

[0206] In the above embodiments, by splitting the food prediction results and food labeling information into three dimensions—category, location, and quantity—and calculating the category loss value, location loss value, and quantity loss value for each, and then obtaining the recognition loss value based on the combination of the three, the recognition loss value can reflect the prediction deviation of the model in the three different task dimensions of category discrimination, location positioning, and quantity prediction. This allows the model to focus on the optimization objectives of each task dimension during training, effectively improving the model's comprehensive recognition performance in the three task dimensions of category, location, and quantity, and further improving the recognition accuracy and reliability of the refrigerator food recognition model in practical application scenarios.

[0207] In some embodiments, after obtaining a food identification model for managing food in the refrigerator, the controller identifies the food in the refrigerator based on the food identification model to obtain the food identification result; generates food management information based on the food identification result, and updates the food display information in the refrigerator display interface based on the food management information.

[0208] The food identification result refers to the predicted information output after the trained food identification model is deployed on the refrigerator or cloud server and the multimodal data of the food to be identified is identified during the inference stage. The food identification result includes, but is not limited to, one or more of the following: category information, name information, quantity information, location information, freshness information, and shelf-life information of the food in the refrigerator.

[0209] Food management information refers to information generated based on food identification results and used to manage the food in the refrigerator. For example, food management information includes, but is not limited to, one or more of the following: food list update information, food inventory warning information, food shelf-life reminder information, food purchase suggestion information, and food replenishment list information.

[0210] Food display information refers to the visual information related to food shown to the user on the refrigerator's display interface. For example, food display information includes, but is not limited to, one or more of the following: food name, food quantity, food location, food icon, food status indicator, and food layout diagram.

[0211] The controller deploys the trained food recognition model on the refrigerator or a cloud server. During the inference phase, the controller acquires multimodal data of the food inside the refrigerator through the acquisition device configured on the refrigerator and / or external devices, inputs the multimodal data into the food recognition model, and the food recognition model identifies the food inside the refrigerator and outputs the food recognition result.

[0212] Subsequently, the controller generates food management information based on the food identification results. In some embodiments, the controller compares the food identification results with the existing food list currently stored in the refrigerator to determine whether there are any newly added foods, consumed foods, or foods whose locations have changed, and generates food list update information accordingly. In other embodiments, the controller determines whether the inventory is below a preset threshold based on the type and quantity of foods in the food identification results; if it is below the preset threshold, it generates food purchase suggestion information. In still other embodiments, the controller queries a preset food database based on the food name in the food identification results to obtain the shelf life and freshness information of the food; if the food is nearing its expiration date or has expired, it generates a food freshness reminder message.

[0213] In a feasible implementation, the controller updates the food display information on the refrigerator's display interface based on food management information. For example, the refrigerator includes a display screen located on the refrigerator door or inside the refrigerator, which displays visual information about the food inside the refrigerator to the user. The controller sends the generated food management information to the display driver module of the display screen, which then controls the display screen to update the currently displayed food list.

[0214] In the above embodiments, by deploying the trained food identification model on the refrigerator, the food inside the refrigerator is identified based on the food identification model and the food identification results are generated. Structured food management information is generated based on the food identification results, and the food display information on the refrigerator display interface is updated in real time based on the food management information. This realizes a complete closed loop from food identification to food management to information display, so that the food information on the refrigerator display interface can be updated in real time with the actual changes of the food inside the refrigerator. This avoids the problem of users getting outdated or incorrect information when querying food information, and effectively improves the real-time performance, accuracy and user interaction experience of smart refrigerator food management.

[0215] Figure 6 This is a schematic diagram of a multimodal model structure provided in some embodiments of this application. For example... Figure 6 As shown, the multimodal model provided in this application embodiment includes a data acquisition layer, an aggregation layer, a perception layer, and a multimodal fusion layer. The data acquisition layer acquires voice data supporting wake-up and wake-free modes, as well as video data containing images and audio. The aggregation layer receives the voice data and performs voice preprocessing; for existing voice features, it directly converts them into text; if the features are not existing, they are converted into text using a speech-to-text converter, and finally, the resulting text is input into a text prompter. Simultaneously, after receiving video data, it performs video feature sampling, detects existing features, extracts and stores frame information, and then processes it through a video extractor. The text and video features output by the aggregation layer enter the perception layer: the perception layer extracts keyframes based on temporal comparison learning of previous and subsequent frames and performs keyframe caching; the multimodal fusion layer receives the keyframe features output by the perception layer and the text output by the aggregation layer, which are processed by the language model LLM module, the image / object classification model, the video model, the food object detection model, and the packaged food ingredient recognition model, respectively. The processed results are verified by the detection result validity verification module, thereby completing the complete reasoning from multimodal input to target food ingredient recognition result, and suppressing the illusion of food ingredient recognition inside the refrigerator.

[0216] Through the above Figure 6 The multimodal model structure shown allows the controller to uniformly encode and cross-modal fuse speech, text, image, and video data from different modalities. Through the illusion suppression mechanism and the detection result validity verification mechanism at the decoder level, it effectively suppresses the illusion phenomenon of the multimodal model in the refrigerator food recognition scenario, thereby providing accurate and reliable recognition results for refrigerator food recognition.

[0217] The training method for a refrigerator food recognition model provided in this application can be executed by a training device for the refrigerator food recognition model. This application uses an example of a training device for the refrigerator food recognition model executing the training method to illustrate the training device for the refrigerator food recognition model provided in this application.

[0218] This application also provides a training device for a refrigerator food identification model. Figure 7 This is a schematic diagram of the training device structure for a refrigerator food recognition model provided in some embodiments of this application. For example... Figure 7 As shown, the training device for the refrigerator food recognition model includes: an acquisition module 701, an extraction module 702, a fusion module 703, an input module 704, a loss module 705, and an update module 706.

[0219] The acquisition module 701 is used to acquire multimodal sample data of the food to be identified. The multimodal sample data includes at least voice sample data and video sample data. The video sample data is used to acquire the food association features between user interaction operations and the food to be identified. Extraction module 702 is used to extract visual features of food ingredients from video sample data and text features of food ingredients from audio sample data. The fusion module 703 is used to fuse food association features, food visual features and food text features to obtain target food recognition features; Input module 704 is used to input the target food ingredient recognition features into a preset food ingredient recognition model to obtain food ingredient prediction results; The loss module 705 is used to obtain the recognition loss value based on the difference between the food ingredient prediction result and the food ingredient labeling information; the food ingredient labeling information is obtained by labeling multimodal sample data. The update module 706 is used to update the network parameters of the preset food identification model based on the identification loss value until the training termination condition is met, so as to obtain a food identification model for managing food in the refrigerator.

[0220] According to the training device for the refrigerator food recognition model provided in this application embodiment, by acquiring multimodal sample data of the food to be recognized and the food association features between user interaction operations and the food to be recognized, the static attribute information and dynamic interaction information of the food to be recognized can be obtained simultaneously. This provides a rich data foundation containing the semantics of operation behavior for subsequent feature extraction and fusion, effectively avoiding the problem of insufficient information in food recognition from single-modal data. Furthermore, by extracting the visual features and text features of the food, the video sample data and voice sample data are converted into feature representations respectively, enabling the food recognition model to perceive food attributes from both visual and textual dimensions, effectively improving the model's ability to perceive multidimensional features of the food. Subsequently, the obtained food association features, food visual features, and food text features are fused to obtain the target food recognition features, making the input features more discriminative and anti-interference, effectively alleviating the problem of unstable single-dimensional feature recognition caused by visual occlusion, changes in lighting, similar appearance, or differences in viewing angle. This process effectively enhances the model's ability to distinguish easily confused ingredients. Then, the target ingredient's identification features are input into a pre-defined ingredient identification model, which outputs a predicted ingredient result. Based on the difference between the predicted result and the ingredient's labeling information, a recognition loss value is obtained. Calculating the difference between the predicted result and the actual labeling provides clear error feedback for targeted optimization of the model parameters, allowing the model to iteratively update in the direction of error reduction, effectively ensuring the convergence and stability of the training process. Finally, the network parameters of the pre-defined ingredient identification model are updated based on the recognition loss value until the training termination condition is met, resulting in an ingredient identification model for managing ingredients in the refrigerator. By driving continuous iterative optimization of the model parameters through the recognition loss value, the final ingredient identification model can effectively adapt to the diverse ingredient category recognition needs in refrigerator scenarios, achieving efficient identification and health management of ingredients in the refrigerator, effectively improving the refrigerator's recognition accuracy and user experience in ingredient management scenarios.

[0221] In some embodiments, the fusion module is further configured to acquire feature samples from the user's historical interaction operations; extract discrimination data associated with food-related features, food visual features, and food text features from the feature samples respectively; and perform weighted feature fusion on the food-related features, food visual features, and food text features based on the discrimination data.

[0222] In some embodiments, the fusion module is further configured to obtain preset food feature extension rules; generate food extension features based on the food feature extension rules to supplement food association features, food visual features and food text features; and fuse the food extension features, food association features, food visual features and food text features.

[0223] In some embodiments, the fusion module is further configured to acquire the user's historical interaction operations and determine the interaction scenario corresponding to the historical interaction operations; acquire the historical features of ingredients in the interaction scenario; and fuse the historical features of ingredients with the associated features of ingredients, the visual features of ingredients, and the textual features of ingredients.

[0224] In some embodiments, the input module is further configured to acquire the user's historical interaction operations; match the scene data representing the interaction scenario in the user's historical interaction operations with the food ingredient prediction results to obtain interaction events, and acquire an interaction event sequence composed of multiple interaction events; associate the target food ingredient identification features with the interaction event sequence to obtain a target interaction event sequence; and update the parameters of the memory model based on the target interaction event sequence.

[0225] In some embodiments, the input module is further configured to determine the maximum sequence length that the short-term memory model can store; if the length of the target interaction event sequence exceeds the maximum sequence length, filter the interaction events in the target interaction event sequence based on time order and / or event similarity to make the length of the target interaction event sequence match the maximum sequence length; and update the value function of the short-term memory model based on the filtered target interaction event sequence to update the short-term memory model.

[0226] In some embodiments, the input module is further configured to acquire multiple historical target interaction event sequences; determine at least one benchmark interaction event sequence from the multiple historical target interaction event sequences according to a preset sequence rule; and update the value function of the long-term memory model based on the benchmark interaction event sequence to update the long-term memory model.

[0227] In some embodiments, the input module is further configured to obtain correct execution samples from the user's historical interaction operations to obtain demonstration sample data; obtain counterfactual interaction data based on the target interaction event sequence; construct a reward function based on the demonstration data and / or counterfactual interaction data; iteratively optimize the reward function; and update the pseudo-experience replay model based on the optimized reward function.

[0228] In some embodiments, the loss module is further configured to determine a category loss value based on the difference between the predicted category result and the labeled category result; determine a location loss value based on the difference between the predicted location result and the labeled location result; determine a quantity loss value based on the difference between the predicted quantity result and the labeled quantity result; and obtain an identification loss value based on the category loss value, the location loss value, and the quantity loss value.

[0229] In some embodiments, the update module is further configured to identify the food in the refrigerator based on the food identification model, obtain the food identification result, generate food management information based on the food identification result, and update the food display information in the refrigerator display interface based on the food management information.

[0230] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0231] The training device for the refrigerator food recognition model in this application embodiment can be an electronic device or a component of an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal, such as a server.

[0232] The training device for the refrigerator food recognition model provided in this application embodiment can realize the various processes implemented in the above-described refrigerator food recognition model training method embodiment. To avoid repetition, it will not be described again here.

[0233] Based on the same inventive concept, this application also provides a refrigerator, including a processor, which is configured to implement the various processes implemented in the above-described refrigerator food recognition model training method embodiment. To avoid repetition, these will not be described again here.

[0234] In some embodiments, Figure 8 This is a schematic diagram of the structure of an electronic device provided in some embodiments of this application. For example... Figure 8 As shown, this application embodiment also provides an electronic device 800, including a processor 801, a memory 802, and a computer program stored on the memory 802 and executable on the processor 801. When the program is executed by the processor 801, it implements the various processes of the above-described refrigerator food recognition model training method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0235] This application provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described training method embodiment for the refrigerator food recognition model and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0236] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable media, such as computer read-only memory (ROM), random-access memory (RAM), magnetic disks, or optical disks.

[0237] The computer-readable storage medium may include: read-only memory (ROM), random-access memory (RAM), magnetic disk or optical disk, etc.

[0238] This application provides a computer program product, including a computer program that, when executed by a processor, implements the training method for the refrigerator food recognition model described above.

[0239] This application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described training method embodiment for refrigerator food recognition model, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0240] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0241] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0242] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0243] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0244] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0245] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A training method for a refrigerator food recognition model, characterized in that, include: Acquire multimodal sample data of the food to be identified, wherein the multimodal sample data includes at least voice sample data and video sample data; The video sample data is used to obtain the food association features between user interaction operations and the food to be identified. Visual features of food ingredients are extracted from the video sample data, and textual features of food ingredients are extracted from the audio sample data; The food association features, food visual features, and food text features are fused to obtain the target food identification features; The target ingredient identification features are input into a preset ingredient identification model to obtain the ingredient prediction result; The identification loss value is obtained based on the difference between the food ingredient prediction result and the food ingredient labeling information; the food ingredient labeling information is obtained based on the labeling of the multimodal sample data. The network parameters of the preset food identification model are updated based on the identification loss value until the training termination condition is met, thus obtaining a food identification model for managing food in the refrigerator.

2. The training method for the refrigerator food recognition model according to claim 1, characterized in that, The fusion of the food ingredient association features, food ingredient visual features, and food ingredient text features includes: Obtain feature samples from the user's historical interaction operations; Discriminant data associated with the food ingredients' features, visual features, and textual features are extracted from the feature samples, respectively. The food ingredient association features, food ingredient visual features, and food ingredient text features are weighted feature fusion based on the discrimination data.

3. The training method for the refrigerator food recognition model according to claim 1, characterized in that, The fusion of the food ingredient association features, food ingredient visual features, and food ingredient text features includes: Obtain pre-defined rules for expanding ingredient characteristics; Based on the food feature extension rules, food extension features are generated to supplement the food association features, food visual features, and food text features. The extended features of the ingredients, the associated features of the ingredients, the visual features of the ingredients, and the textual features of the ingredients are fused together.

4. The training method for the refrigerator food recognition model according to claim 1, characterized in that, The fusion of the food ingredient association features, food ingredient visual features, and food ingredient text features includes: Obtain the user's historical interaction operations and determine the interaction scenarios corresponding to the historical interaction operations; Obtain the historical characteristics of the ingredients in the interactive scenario; The historical features of the ingredients, the associated features of the ingredients, the visual features of the ingredients, and the textual features of the ingredients are fused together.

5. The training method for the refrigerator food recognition model according to claim 1, characterized in that, The preset food ingredient recognition model includes a memory model, and the method further includes: Obtain the user's historical interaction actions; The scene data representing the interaction scenario in the user's historical interaction operations are matched with the food prediction results to obtain interaction events, and an interaction event sequence composed of multiple interaction events is obtained. The target food ingredient identification features are associated with the interaction event sequence to obtain the target interaction event sequence; The memory model is updated based on the target interaction event sequence.

6. The training method for the refrigerator food recognition model according to claim 5, characterized in that, The memory model includes a short-term memory model; updating the memory model based on the target interaction event sequence includes: Determine the maximum sequence length that the short-term memory model can store; If the length of the target interaction event sequence exceeds the maximum sequence length, the interaction events in the target interaction event sequence will be filtered based on time order and / or event similarity so that the length of the target interaction event sequence matches the maximum sequence length. The value function of the short-term memory model is updated based on the filtered target interaction event sequence to update the short-term memory model.

7. The training method for the refrigerator food recognition model according to claim 5, characterized in that, The memory model includes a long-term memory model; updating the memory model based on the target interaction event sequence includes: Obtain multiple historical target interaction event sequences; At least one baseline interaction event sequence is determined from multiple historical target interaction event sequences according to a preset sequence rule; The value function of the long-term memory model is updated based on the benchmark interaction event sequence to update the long-term memory model.

8. The training method for the refrigerator food recognition model according to claim 5, characterized in that, The memory model includes a pseudo-experience replay model; updating the memory model based on the target interaction event sequence includes: Obtain correct execution samples from the user's historical interaction operations to obtain demonstration sample data; Counterfactual interaction data is obtained based on the target interaction event sequence; Construct a reward function based on the demonstration data and / or counterfactual interaction data; The reward function is iteratively optimized, and the pseudo-experience replay model is updated based on the optimized reward function.

9. The training method for the refrigerator food recognition model according to claim 1, characterized in that, The food ingredient prediction results include prediction category results, prediction location results, and prediction quantity results; the food ingredient labeling information includes label category results, label location results, and label quantity results. The identification loss value is obtained based on the difference between the food ingredient prediction result and the food ingredient labeling information, including: The category loss value is determined based on the difference between the predicted category result and the labeled category result; Based on the difference between the predicted location result and the marked location result, the location loss value is determined; Based on the difference between the predicted quantity result and the labeled quantity result, the quantity loss value is determined; The recognition loss value is obtained based on the category loss value, location loss value, and quantity loss value.

10. The training method for the refrigerator food recognition model according to claim 1, characterized in that, After obtaining the food identification model for managing food in the refrigerator, the method further includes: Based on the food identification model, the food in the refrigerator is identified to obtain the food identification result; Based on the food identification results, food management information is generated, and the food display information in the refrigerator display interface is updated based on the food management information.

11. A refrigerator, characterized in that, Includes a processor configured to perform a training method for a refrigerator food recognition model as described in any one of claims 1-10.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the refrigerator food recognition model as described in any one of claims 1-10.

Images