Food material size recognition method for intelligent electrical appliance and intelligent electrical appliance

Abstract

The application relates to a food material size identification method for a smart electric appliance and the smart electric appliance, and applies to the field of smart electric appliances, wherein the method comprises the following steps: using a target lightweight network to identify food materials in an obtained food material image of the smart electric appliance, so as to obtain a food material category of a to-be-identified food material in the food material image; the target lightweight network is obtained by knowledge distillation of a target convolutional neural network in a cloud connected with the smart electric appliance; the target convolutional neural network is used for guiding a preset lightweight network to identify food material sizes; based on the food material category, the network depth of the target lightweight network is adjusted to obtain an updated target lightweight network; according to the food material category, the geometric features of the to-be-identified food material in the food material image are extracted by using the updated target lightweight network; and the food material size of the to-be-identified food material is determined according to the geometric features of the to-be-identified food material. Through the application, the identification accuracy and efficiency of the food material size are improved.

Description

Technical Field

[0001] This application relates to the field of smart appliances, and in particular to a method for recognizing the size of food ingredients in smart appliances and the smart appliances themselves. Background Technology

[0002] With the improvement of people's living standards and the promotion and popularization of technologies such as the Internet, big data, artificial intelligence, and voice interaction, more and more traditional lifestyles are gradually changing, and the use of home appliances is gradually moving towards intelligence. While bringing more convenience to users, the functions of various home appliances are also becoming more diversified. In current intelligent cooking equipment, it is necessary to recognize the size of ingredients in order to match the corresponding cooking program according to the size of the ingredients, thereby improving the user's cooking experience.

[0003] However, current methods for identifying food size mainly rely on measuring the size using contact sensors or manually inputting the food category and size parameters. Contact sensor measurement may damage the food and cannot identify irregular shapes, resulting in low accuracy. Manual input depends on human experience, but since the size of the same food varies and errors can occur, repeated input is necessary, making it difficult to balance accuracy and efficiency in food size identification.

[0004] There is currently no effective solution to the problem of balancing accuracy and efficiency in recognizing the size of food items in smart appliances in related technologies. Summary of the Invention

[0005] This embodiment provides a method for recognizing the size of food ingredients in smart appliances and a smart appliance, in order to solve the problem in related technologies that it is difficult to balance the accuracy and efficiency of recognizing the size of food ingredients in smart appliances.

[0006] In a first aspect, this embodiment provides a method for recognizing the size of food items in smart appliances, the method comprising:

[0007] Using a target lightweight network, food identification is performed on food images in smart appliances to obtain the food category of the food to be identified in the food images; the target lightweight network is obtained by knowledge distillation of a target convolutional neural network in the cloud connected to the smart appliance; the target convolutional neural network is trained on a preset target convolutional neural network based on a preset set of food images, and is used to guide the preset lightweight network to identify food size.

[0008] Based on the food category, the network depth of the target lightweight network is adjusted to obtain an updated target lightweight network; using the updated target lightweight network, the geometric features of the food to be identified in the food image are extracted according to the food category.

[0009] The size of the food ingredient to be identified is determined based on its geometric features.

[0010] Through the above steps, the capabilities of the cloud-based large model are transferred to the target lightweight network on the smart appliance side using knowledge distillation technology. Guided by the cloud-based large model, high-precision size recognition is achieved with limited computing resources on the device side. At the same time, the network depth of the target lightweight network is updated according to the food categories identified by the target lightweight network, so as to perform size recognition of the food to be identified in a targeted manner, avoiding the occupation of unnecessary computing resources, thereby improving the efficiency of food size recognition.

[0011] In some embodiments, the method further includes: scoring the contribution of multiple output channels in a preset lightweight network based on a preset importance scoring strategy, and obtaining multiple scoring results;

[0012] Based on the multiple scoring results, identify redundant channels among the multiple output channels;

[0013] The redundant channels are removed from the preset lightweight network to obtain the target lightweight network.

[0014] By quantifying the contribution of each output channel to the model's output accuracy through the above steps, output channels with low contribution are eliminated, thereby reducing computational complexity and model size.

[0015] In some embodiments, the method further includes:

[0016] Obtain the model parameters of the target convolutional neural network;

[0017] Based on the model parameters, the preset lightweight network is updated to obtain the target lightweight network.

[0018] Through the above steps, the parameters of the lightweight network are updated according to the model parameters of the target convolutional neural network, so that the updated target lightweight network learns the target convolutional neural network, thereby improving the accuracy of the lightweight network in recognizing food size.

[0019] In some embodiments, the method further includes: modifying the data type of the target lightweight network according to a preset quantization strategy based on hardware instructions in the smart appliance.

[0020] By using the above steps and hardware acceleration instructions, the problem of hardware "execution efficiency" is solved, thereby improving the efficiency of food size recognition.

[0021] In some embodiments, the step of using a target lightweight network to perform food identification on the acquired food images of smart appliances to obtain the food category of the food to be identified in the food images includes:

[0022] The acquired food ingredient images are preprocessed.

[0023] Based on a preset lightweight network, food classification prediction is performed on the food images after image preprocessing to obtain the food category probability distribution of the food to be identified.

[0024] The food category of the food to be identified is determined based on the probability distribution of the food categories.

[0025] Through the above steps, a lightweight network is used to classify and predict the ingredients to be identified in the preprocessed food images to obtain the probability distribution of the food category corresponding to the ingredients to be identified. By using this probability distribution, the corresponding food category can be determined, which helps to improve the accuracy of identifying the size of food in smart appliances.

[0026] In some embodiments, the step of using an updated target lightweight network to extract geometric features of the food to be identified from the food image according to the food category includes:

[0027] Based on the food category, determine the size key points corresponding to the food to be identified;

[0028] Based on the aforementioned size key points and the food bounding box of the food to be identified determined by the preset lightweight network, the geometric features of the food to be identified are extracted.

[0029] By taking the steps described above, we can determine the key dimensions of different food categories and, in conjunction with the food bounding boxes, identify the geometric features of the food to be identified. This helps to achieve targeted identification of different food categories and thus improves the accuracy of food size identification.

[0030] In some embodiments, determining the size of the ingredient to be identified based on its geometric features includes:

[0031] The size calculation strategy for the food to be identified is determined based on the food category of the food to be identified;

[0032] Based on the size calculation strategy, and combining the size key points in the geometric features with the food bounding box, the size of the food to be identified is determined.

[0033] By following the steps above, and based on the size calculation strategies corresponding to different food categories, combined with the size key points in the geometric features of the food to be identified and the food bounding box, the size of the food is determined, thus improving the accuracy of food size recognition.

[0034] In some embodiments, the method further includes:

[0035] Obtain the coordinate information of the food to be identified in a preset coordinate system; the preset coordinate system is established with the center of the placement plane in the smart appliance where the food to be identified is placed as the origin, the horizontal direction of the placement plane as the horizontal axis, and the vertical direction in the placement plane that is perpendicular to the horizontal direction as the vertical axis.

[0036] Based on the center of the placement plane, the coordinate information of the food to be identified is used for position verification to obtain the verification result;

[0037] When the verification result indicates that the food to be identified is within a preset range, the number of layers of the food to be identified is determined based on a preset depth estimation algorithm; the preset range is defined with the center of the placement plane as the origin.

[0038] Through the above steps, the coordinate information of the food to be identified is determined in the preset coordinate system on the placement plane of the smart appliance. When the coordinate information of the food to be identified indicates that the current position of the food to be identified is within the preset range, it means that the recognition accuracy of the food to be identified is the highest. Then, based on the depth estimation algorithm of the tongue, the depth of the food to be identified, i.e. the number of layers, is determined, thereby avoiding the situation where the recognition accuracy of the food to be identified is reduced due to the positional offset of the food to be identified.

[0039] Secondly, this embodiment provides a smart appliance that uses the food size recognition method for smart appliances as described in any one of the first aspects to recognize the image to be recognized in the food image; the smart appliance is one of a smart oven, a smart steamer, or a smart air fryer.

[0040] Thirdly, this embodiment provides a storage medium storing a computer program that, when executed by a processor, implements the food size recognition method for smart appliances as described in any of the first aspects above.

[0041] Compared with related technologies, the food size recognition method and smart appliance provided in this embodiment, through knowledge distillation technology, transfer the capabilities of a large cloud model to a target lightweight network on the smart appliance side. Based on the guidance of the large cloud model, high-precision size recognition is achieved under limited computing resources on the edge. At the same time, the network depth of the target lightweight network is updated according to the food category identified by the target lightweight network, so as to perform size recognition of the food to be identified in a targeted manner, avoiding the occupation of unnecessary computing resources, thereby improving the efficiency of food size recognition.

[0042] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0043] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0044] Figure 1 This is a hardware structure block diagram of the terminal for the food size recognition method for smart appliances provided in the embodiments of this application;

[0045] Figure 2 This is a flowchart of a method for recognizing food size in smart appliances provided in an embodiment of this application;

[0046] Figure 3 This is a flowchart of a method for determining food categories provided in an embodiment of this application;

[0047] Figure 4 This is a flowchart of the food ingredient location verification method provided in the embodiments of this application;

[0048] Figure 5 This is a schematic diagram of the food size recognition system in this specific embodiment. Detailed Implementation

[0049] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0050] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0051] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal for a food size recognition method for smart appliances provided in an embodiment of this application. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0052] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the food size recognition method for smart appliances in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0053] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0054] Currently, the following technical solutions are mainly used in smart appliances for recognizing food size:

[0055] 1. Mechanical sensor measurement: This method measures the size of food ingredients using contact sensors, but it may damage the ingredients and cannot identify irregular shapes.

[0056] 2. Simple Image Processing: Size estimation is performed based on traditional computer vision algorithms such as edge detection and contour extraction. Specifically, segmentation is first performed, and the food is accurately extracted from the background using color and edges. Then, a scale is established, and the relationship between pixels and the real world is established through reference objects or camera geometry. Finally, calculation is performed: depending on the accuracy requirements of the application scenario, the minimum bounding rectangle is selected to calculate the two-dimensional size, or multi-view calculation is used. Figure 3 3D reconstruction can estimate complex volumes. However, due to limited computing resources at the edge, high-precision dimension recognition cannot be achieved.

[0057] 3. Fixed parameter preset: Users manually input food category and size parameters, which relies on human experience, resulting in low accuracy and efficiency in size recognition.

[0058] Therefore, in order to improve the accuracy and efficiency of food size recognition, this embodiment provides a food size recognition method for smart appliances. Figure 2This is a flowchart of a method for recognizing food size in smart appliances provided in an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0059] Step S210: Using a target lightweight network, food identification is performed on the food images in the smart appliance to obtain the food category of the food to be identified in the food image; the target lightweight network is obtained by knowledge distillation of a target convolutional neural network in the cloud connected to the smart appliance; the target convolutional neural network is used to guide the preset lightweight network to identify the food size.

[0060] The target convolutional neural network is trained on a pre-defined set of food images. After placing the food to be identified in the smart appliance, images of the food in the smart appliance are first acquired using a camera device, such as a webcam; and then the food images undergo image preprocessing.

[0061] In some of these embodiments, Figure 3 This is a flowchart of a method for determining food ingredient categories provided in an embodiment of this application. (Refer to...) Figure 3 In step S210, a target lightweight network is used to perform food identification on the acquired food images from smart appliances to obtain the food category of the food to be identified in the food images, including:

[0062] Step S211: Perform image preprocessing on the acquired food images.

[0063] Step S212: Based on a preset lightweight network, perform food classification prediction on the food images after image preprocessing to obtain the food category probability distribution of the food to be identified.

[0064] Step S213: Determine the food category of the food to be identified based on the food category probability distribution.

[0065] Specifically, the food images undergo illumination compensation, geometric correction, and background segmentation. First, adaptive histogram equalization is used to eliminate uneven illumination. Then, lens distortion correction is performed based on the camera's calibration parameters. Finally, adaptive histogram equalization and U-Net background segmentation are used to improve image quality and target separation capabilities, resulting in pre-processed food images.

[0066] Subsequently, a target lightweight network is used to identify food items from images of smart appliances. This target lightweight network primarily integrates three branches: classification, detection, and key point recognition. By constructing a multi-task deep learning network and combining it with an adaptive extraction algorithm for food geometric features, it achieves accurate identification of key dimensions for various food items.

[0067] The target lightweight network includes three detection branches: a food classification branch, a bounding box regression branch, and a keypoint detection branch. When recognizing preprocessed food images, the food classification branch outputs the probability distribution of different food categories, the bounding box regression branch predicts the bounding rectangle of the food, and the keypoint detection branch defines the keypoints for different food items.

[0068] Through the above steps, a lightweight network is used to classify and predict the ingredients to be identified in the preprocessed food images to obtain the probability distribution of the food category corresponding to the ingredients to be identified. By using this probability distribution, the corresponding food category can be determined, which helps to improve the accuracy of identifying the size of food in smart appliances.

[0069] Meanwhile, the target lightweight network is mainly obtained by knowledge distillation through the target convolutional neural network trained in the cloud, so that the model capabilities of the target convolutional neural network in the cloud can be transferred to the target lightweight network to achieve an optimized balance between the accuracy and efficiency of food size recognition.

[0070] Step S220: Based on the food category, adjust the network depth of the target lightweight network to obtain the updated target lightweight network; using the updated target lightweight network, extract the geometric features of the food to be identified in the food image according to the food category.

[0071] In this process, the network depth of the target lightweight network is adjusted according to the type of food. For example, shallow features of the target lightweight network are used for simple food, while deep features of the target lightweight network are used for complex food, thereby reducing the inference latency of food geometric features under limited computing power.

[0072] In some embodiments, an updated target lightweight network is used to extract geometric features of the food to be identified from the food image based on the food category, including:

[0073] Based on the food category, determine the size key points corresponding to the food to be identified; based on the size key points and the food bounding box of the food to be identified determined by the preset lightweight network, extract the geometric features of the food to be identified.

[0074] Through the above steps, the food classification branch in the detection branch of the above model outputs the probability distribution of different food categories, predicts the bounding rectangle of the food based on the bounding box regression branch, and defines key points for different food categories based on the key point detection branch. By determining the size key points corresponding to different food categories and combining them with the food bounding boxes, the geometric features of the food to be identified are determined. This facilitates targeted identification of different food categories, thereby improving the accuracy of food size identification.

[0075] Step S230: Determine the size of the food to be identified based on its geometric features.

[0076] Specifically, based on the food category of the food to be identified, a size calculation strategy for the food to be identified is determined; based on the size calculation strategy, the size of the food to be identified is determined by combining the size key points in the geometric features and the food bounding box.

[0077] After acquiring the geometric features of the food to be identified, the food size is further determined based on its food category. By employing size calculation strategies corresponding to different food categories, and combining key size points in the corresponding geometric features with the food's bounding box, the food size is determined, improving the accuracy of food size recognition.

[0078] Through the above steps, the capabilities of the cloud-based large model are transferred to the target lightweight network on the smart appliance side using knowledge distillation technology. Guided by the cloud-based large model, high-precision size recognition is achieved with limited computing resources on the device side. At the same time, the network depth of the target lightweight network is updated according to the food categories identified by the target lightweight network, so as to perform size recognition of the food to be identified in a targeted manner, avoiding the occupation of unnecessary computing resources, thereby improving the efficiency of food size recognition.

[0079] In some embodiments, the method for recognizing food size in smart appliances, which processes a preset lightweight network to obtain a target lightweight network, further includes: scoring the contribution of multiple output channels in the preset lightweight network based on a preset importance scoring strategy to obtain multiple scoring results; identifying redundant channels among the multiple output channels based on the multiple scoring results; and removing redundant channels from the preset lightweight network to obtain the target lightweight network.

[0080] Since the target lightweight network has limited computational resources, channel pruning is necessary to reduce its computational load and parameter size, thereby lowering the recognition and inference latency at the corresponding endpoints. During channel pruning, a pre-defined importance scoring strategy quantifies the contribution of each channel (or filter) in the convolutional neural network to the output accuracy of the target lightweight network model. Channels with high scores are considered key feature extractors, while low-scoring channels are often redundant and their removal has little impact on accuracy, thus reducing computational complexity and model size while maintaining performance. The pre-defined importance scoring strategy is typically defined based on channel weights or activation statistics; its core idea is that channels with less impact on the model output receive lower importance scores.

[0081] For example, a predefined importance scoring strategy can be determined using scoring criteria corresponding to weight norm scoring, activation value scoring, scaling factor scoring, and gradient sensitivity scoring. The weight norm scoring calculates the L1 norm (sum of absolute values) or L2 norm (square root of the sum of squares) of the convolutional kernel weights corresponding to each channel. Channels with smaller weight norms tend to have weaker activation outputs and lower importance. The activation value scoring calculates statistics (such as mean absolute value or sparsity) of the channel output activation values ​​on the validation dataset. Channels with activation values ​​closer to zero contribute less and are less important. The scaling factor scoring assigns a trainable scaling factor to each channel in the batch normalization (BN) layer. . The size directly controls the activation intensity of this channel: The closer the value is to zero, the easier the channel is to suppress and the lower its importance. This method is widely used in lightweight networks (such as MobileNetV3) because Batch Normalization (BN) layers are standard components and γ has a clear physical meaning: adjusting the activation scale of channels. Gradient sensitivity scoring assesses importance by calculating the gradient magnitude of the loss function with respect to the channel weights. Channels with small gradient magnitudes have little impact on the training process and may be redundant.

[0082] In this embodiment, for edge devices (such as cameras equipped with NPUs in smart appliances), a scaling factor scoring method that is computationally efficient, easy to integrate, and can directly reflect the actual contribution of the channel during inference is preferred.

[0083] By quantifying the contribution of each output channel to the model's output accuracy and then eliminating output channels with low contribution, the target lightweight network on the edge is compressed, thereby reducing computational complexity and model size. This is beneficial for improving the efficiency of food size recognition through the target lightweight network.

[0084] In some embodiments, the method for recognizing food size in smart appliances further includes: obtaining model parameters of a target convolutional neural network; and updating a preset lightweight network based on the model parameters to obtain the target lightweight network.

[0085] This involves establishing a closed-loop optimization system between a cloud-based big data analysis platform and a small edge-side model, continuously improving recognition accuracy through ongoing learning. Specifically, images of food and geometric features and size data of the food to be identified are collected by cameras in edge devices (i.e., smart appliances). The data is then cleaned, feature aligned, and retrained by the cloud-based target convolutional network model. Data cleaning removes low-quality samples, feature alignment unifies the features collected from different edge devices to form a data flow architecture, and model retraining optimizes the target convolutional network model based on incremental data.

[0086] Subsequently, an error correction mechanism was set up to correct various data in the data flow architecture. The error correction mechanism includes outlier detection, consistency verification, and historical data comparison. Outlier detection is used to identify abnormal size data and provide feedback for re-labeling. Consistency verification is used for cross-validation of data from multiple camera perspectives. Historical data comparison is used to verify the distribution of similar food items against historical sizes.

[0087] Furthermore, the outlier detection here identifies anomalous size data primarily based on outlier detection of physical rationality and statistical distribution, consistency verification based on multi-source information, and dynamic comparison based on historical data and context.

[0088] Specifically, outlier detection based on physical plausibility and statistical distribution primarily screens raw data from a single measurement. It identifies outliers through absolute physical boundary filtering and statistical outlier detection methods. Absolute physical boundary filtering sets a very broad upper and lower physical limit (e.g., an object identified as a "potato" cannot be less than 1 cm or greater than 30 cm in length). This is mainly used to filter out obvious sensor malfunctions or identification errors (such as misidentifying a spoon as a potato). Statistical outlier detection uses statistical methods to identify outliers in data on similar food items collected over a short period (e.g., within a day).

[0089] For example, the interquartile range (IQR) of the diameter of all "Red Fuji apples" is calculated, and data points exceeding (Q3 + 1.5 × IQR) or falling below (Q1 - 1.5 × IQR) are marked as outliers to be investigated. This method is primarily based on the assumption that "most data points should cluster around a central trend." Under normal supply chains and varieties, the size distribution of an apple is relatively concentrated; a sudden appearance of a size far exceeding the norm is likely due to measurement or labeling errors.

[0090] Consistency verification based on multi-source information generally uses cross-validation to discover contradictions, which is the most powerful means of identifying anomalies. It is mainly based on multi-view geometric consistency, weight-volume correlation verification, and semantic consistency between the identification results and dimensions.

[0091] The recognition system corresponding to multi-view geometric consistency is equipped with multiple end-side cameras. If one camera captures a "carrot" that is 20cm long, but another camera calculates its length as only 10cm using 3D reconstruction or triangulation, then at least one of these two data points is an anomaly.

[0092] In weight-volume correlation verification, if the system integrates a weighing sensor, a simple density model can be established. For example, an object identified as a "steak" may have a large volume calculated from an image, but a very light weight, which constitutes a contradiction in physical properties, and the size data is likely abnormal.

[0093] In semantic consistency recognition of identification results and size, it is crucial to determine whether the category output by the identification model matches the estimated size in a matter of common sense. For example, if an object is identified as a "quail egg" but the size data shows a diameter of 5 centimeters (close to the size of a chicken egg), then at least one piece of information may be incorrect.

[0094] Based on dynamic comparison of historical data and context, identification is mainly based on historical distribution comparison and scene context analysis.

[0095] The historical distribution comparison involves comparing the current measured "green bell pepper" size with the historical green bell pepper size distribution of the same user or all users in the cloud database. Methods such as the KS test are used to determine whether the current data point significantly deviates from the historical distribution. If the user has previously purchased green bell peppers between 10-15cm in length, and a data point of 25cm suddenly appears, it will be flagged.

[0096] Contextual analysis focuses on analyzing and identifying data such as user habits and procurement batches. For example, if the system learns that user A always places fish diagonally on the plate, resulting in a larger length measurement, it will compensate for this. Only data deviating from the user's "personal baseline" will be considered suspicious. If all the "tomatoes" uploaded by a user this time are larger than the historical average, it may be a new batch or a new variety, not necessarily an anomaly. The system will observe this as a cluster rather than reporting errors individually.

[0097] The aforementioned continuous learning method helps reduce the recognition error rate of the target convolutional neural network. Subsequently, the model parameters of the target convolutional neural network are used to update the parameters of the lightweight network, thereby transferring the capabilities of the cloud-based convolutional neural network model to the edge-side target lightweight network. This helps reduce the error rate of the edge-side target lightweight network in recognizing food size.

[0098] Through the above steps, the parameters of the lightweight network are updated according to the model parameters of the target convolutional neural network, so that the updated target lightweight network learns the target convolutional neural network, thereby improving the accuracy of the lightweight network in recognizing food size.

[0099] In some embodiments, the food size recognition method for smart appliances further includes: modifying the data type of the target lightweight network according to a preset quantization strategy based on hardware instructions in the smart appliance.

[0100] Preferably, this embodiment employs an INT8 quantization strategy optimized for the characteristics of the edge NPU hardware instruction set. Quantizing an FP32 (32-bit floating-point) model into an INT8 (8-bit integer) model is a commonly used optimization technique in deep learning model deployment. It can significantly reduce model size, memory usage, and inference latency, while maintaining high accuracy in most cases.

[0101] Quantization acts as both an "enabler" and an "optimizer." It is thanks to mature compression techniques like INT8 quantization that relatively compact, lightweight target network models obtained through knowledge distillation, such as the MobileNetV3 model, can be further transformed into a form that can run in real-time (<100ms) on the edge NPU. Without quantization, edge inference latency might be difficult to meet.

[0102] This technical solution is a system-level optimization scheme that organically combines knowledge distillation, channel pruning, quantization strategies, and adaptive reasoning to address the unique challenges of "end-side food ingredient recognition" in this specific scenario.

[0103] Among these, knowledge distillation addresses the "capacity ceiling" problem of small models (by transferring knowledge from large models). Channel pruning addresses the "intrinsic redundancy" problem of models (through structured simplification). Quantization strategies address the "execution efficiency" problem of hardware (by utilizing hardware to accelerate instructions). Adaptive inference addresses the "dynamic allocation" problem of resources (by adjusting computational load as needed).

[0104] By using the above steps and hardware acceleration instructions, the problem of hardware "execution efficiency" is solved, thereby improving the efficiency of food size recognition.

[0105] In some of these embodiments, Figure 4 This is a flowchart of the food ingredient location verification method provided in the embodiments of this application, see reference. Figure 4 The food size recognition method for smart appliances also includes:

[0106] Step S410: Obtain the coordinate information of the food to be identified in the preset coordinate system; The preset coordinate system is established with the center of the placement plane on which the food to be identified is placed in the smart appliance as the origin, the horizontal direction of the placement plane as the horizontal axis, and the vertical direction perpendicular to the horizontal direction in the placement plane as the vertical axis.

[0107] Step S420: Based on the center of the placement plane, perform position verification on the coordinate information of the food to be identified to obtain the verification result.

[0108] Step S430: When the verification result indicates that the food to be identified is within a preset range, the number of layers of the food to be identified is determined based on a preset depth estimation algorithm; the preset range is defined with the center of the placement plane as the origin.

[0109] In this process, a planar coordinate system is established with the center of the plane on which the food to be identified is placed in the smart appliance as the origin, such as the center of the baking pan in the smart appliance as the origin. Then, the centroid coordinates of the food to be identified are calculated, which is the coordinate information of the food to be identified.

[0110] Furthermore, when multiple food items are placed in a smart appliance to be identified, the Euclidean distance between the food items is calculated in real time to ensure that the spacing between the multiple food items reaches a preset interval, such as no less than 1 cm. The distance between multiple food items to be identified is expressed by the formula:

[0111] ;

[0112] Where (x1, y1) and (x2, y2) represent the coordinate information of the two ingredients to be identified in the planar coordinate system.

[0113] Simultaneously, a central region constraint needs to be applied to the food to be identified. A circular area with radius R, centered on the center of the plane where the food is placed within the smart appliance, is defined as the preset area. This constraint is applied when the food's coordinate information (x, y) satisfies... In this case, it indicates that the ingredient to be identified is within a preset range.

[0114] Additionally, the preset range can be adaptively adjusted according to the size of the surface on which the food to be identified is placed.

[0115] After obtaining the coordinate information of the food to be identified, it is necessary to obtain the depth information of the food to be identified. Specifically, the depth value of the food to be identified is determined according to the preset depth estimation algorithm, and then the number of layers of the food to be identified is determined.

[0116] The preset depth estimation algorithms include those that use an RGB-D camera to directly acquire depth maps, as well as those based on monocular vision. No specific depth estimation algorithm is specified here.

[0117] Once the depth value d of the food to be identified is determined using a preset depth estimation algorithm, the number of layers of the food to be identified is determined based on a preset layer number judgment logic. For example, the depth value range corresponding to the first layer is [0, d1]; the depth value range corresponding to the second layer is [d1, d2], and the depth threshold of each layer is determined through calibration.

[0118] Through the above steps, the coordinate information of the food to be identified is determined in the preset coordinate system on the placement plane of the smart appliance. When the coordinate information of the food to be identified indicates that the current position of the food to be identified is within the preset range, it means that the recognition accuracy of the food to be identified is the highest. Then, based on the depth estimation algorithm of the tongue, the depth of the food to be identified, i.e. the number of layers, is determined, thereby avoiding the situation where the recognition accuracy of the food to be identified is reduced due to the positional offset of the food to be identified.

[0119] Secondly, this embodiment provides a smart appliance that uses the above-mentioned food size recognition method for smart appliances to recognize the image to be recognized in the food image; the smart appliance is one of a smart oven, a smart steamer, or a smart air fryer.

[0120] The present embodiment will be described and explained below through specific examples.

[0121] Figure 5 This is a schematic diagram of the food size recognition system according to a specific embodiment of the present invention, as shown below. Figure 5 As shown, the system includes a cloud platform and an edge device, which are connected. The cloud platform houses a target convolutional neural network, while the edge device houses a target lightweight network. Knowledge distillation is performed on the target lightweight network on the edge device using the cloud-based target convolutional neural network. This achieves collaborative optimization of food size recognition accuracy through big data from the cloud and small models on the edge, while simultaneously enabling high-precision real-time size recognition under limited computing resources on the edge.

[0122] In this specific embodiment, knowledge distillation technology combined with an adaptive inference mechanism is used to transfer the capabilities of a large cloud model to a small edge model. Combined with adaptive computing resource allocation, an optimized balance between accuracy and efficiency is achieved. The target lightweight network is a lightweight backbone network based on MobileNetV3, and the target convolutional neural network is ResNet-50. The ResNet-50 trained in the cloud is used as the teacher model to guide the training of the MobileNetV3 (lightweight network) student model, and redundant convolutional channels in MobileNetV3 are removed based on importance scores.

[0123] Meanwhile, the MobileNetV3 on the device receives optimized ResNet-50 model parameters pushed from the cloud. The cloud performs periodic model retraining and testing based on anonymized data from a massive number of devices, and pushes the new version of the model parameters with better performance after verification to the device as an update package for upgrade. This achieves a recognition accuracy of no less than 95% and inference latency within 100ms under limited computing power on the device.

[0124] Obtaining the importance score is a systematic process that combines model training and validation data. The specific steps are as follows (taking the scaling factor score as an example):

[0125] Baseline Model Training: The knowledge-distilled MobileNetV3 was used as the baseline model (student model) and trained until convergence on the food recognition dataset. During training, the scaling factor γ of the BN layer was automatically optimized, and the magnitude of γ naturally reflected the channel importance.

[0126] Score Calculation: Evaluate the importance score for each channel on the validation dataset. For each convolutional layer (followed by a BN layer), the importance score Si for channel i is defined as: .in, It is the scaling factor for the corresponding channel in the BN layer. The smaller its absolute value, the less important the channel is.

[0127] Furthermore, to enhance robustness, activation statistics from the validation set can be used to normalize the scores (e.g., layer-by-layer normalization). Sorting and pruning decisions: For each layer, all channels are sorted in ascending order based on the score Si (the channel with the lowest score is the most redundant). A pruning rate is set (e.g., removing the lowest 20% of channels in each layer), or removal is based on a global threshold (e.g., channels with scores below 0.01). The pruning rate can be dynamically adjusted according to the computing power constraints of the endpoint (e.g., a low pruning rate when the endpoint is in high-performance mode and a high pruning rate when in energy-saving mode).

[0128] Fine-tuning to restore accuracy: The accuracy of a pruned model may decrease due to structural changes, requiring light fine-tuning on the training dataset (usually training for a few epochs) to restore recognition accuracy. During fine-tuning, the pruned architecture is fixed, and only the remaining weights are updated.

[0129] Iterative optimization: To balance accuracy and efficiency, the above process can be iterated multiple times (with fine-tuning after pruning a small number of channels each time) until the edge latency requirement is met (e.g., inference time is less than 100ms).

[0130] In this food recognition scenario, channel pruning can reduce the size of the MobileNetV3 model by 30%-50% and improve inference speed by 20%-40%, while maintaining recognition accuracy of no less than 95% through fine-tuning. In conjunction with adaptive inference mechanisms (such as complexity awareness), the overall system achieves an average inference latency of less than 100ms on the edge NPU, meeting real-time requirements.

[0131] In the edge device, the backbone network includes a lightweight backbone network based on MobileNetV3, integrating three tasks: classification, detection, and keypoint recognition. The detection head includes a food classification branch that outputs the probability distribution of six types of food, a bounding box regression branch that predicts the bounding rectangle of the food, and a keypoint detection branch that defines keypoints for different food types.

[0132] For example, when the ingredient to be identified is a sweet potato, the key points are the two ends and the point of maximum thickness (3 key points); when the ingredient to be identified is a steamed bun, the key points are the top center point and the bottom outline point (5 key points); when the ingredient to be identified is steamed egg custard, the key points are the edge of the bowl and the center of the bottom of the bowl (8 key points); when the ingredient to be identified is a fish, the key points are the head, tail, and the widest points on both sides (4 key points); when the ingredient to be identified is a crab, the key points are the four corners of the shell (4 key points); when the ingredient to be identified is a molded cake, the key points are the top edge of the container and the bottom center point (6 key points).

[0133] Design specific key points for each type of food ingredient and calculate dimensional parameters based on geometric relationships:

[0134] Sweet potato: The maximum thickness is calculated using three key points, expressed by the following formula:

[0135] ;

[0136] Where thickness represents the thickness of the sweet potato, distance(mid_point, thickness_point) represents the distance between the midpoint and the point of maximum thickness of the sweet potato, and scale_factor represents the scaling factor. The midpoint of the sweet potato is determined by the two endpoints.

[0137] Steamed Bun: Fit an ellipse based on 5 key points to calculate height and diameter. Steamed Egg Custard: Fit a circle through the rim of the bowl to calculate diameter; calculate height using the distance from the bottom of the bowl to the rim plane. Fish: Calculate length using the head and tail points; calculate width using the widest points on both sides. Crab: Calculate the area of ​​the circumscribed rectangle using the 4 corner points. Mold Cake: Calculate container dimensions similar to those used for steamed egg custard.

[0138] A dedicated algorithm is designed for the geometric characteristics of each type of food, which significantly improves the accuracy of size recognition and thus effectively handles the diverse shape features of multiple types of food.

[0139] Meanwhile, when identifying food size using MobileNetV3, an adaptive inference mechanism is incorporated to adjust the network depth based on the food category. For simple food categories such as sweet potatoes and steamed buns, MobileNetV3 uses shallow features, resulting in an inference time of less than 50ms. For complex food categories such as crabs and fish, MobileNetV3 uses deep features, achieving an inference time of less than 100ms and greater than 50ms. An image pyramid is constructed for different food categories to detect food of different sizes at different scales, thereby improving the accuracy and efficiency of food size recognition.

[0140] It should be noted that the steps shown in the above process or in the flowcharts in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions.

[0141] This embodiment also provides an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0142] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0143] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0144] S1, using a target lightweight network, performs food identification on the food images in the acquired smart appliance to obtain the food category of the food to be identified in the food image; the target lightweight network is obtained by knowledge distillation of the target convolutional neural network in the cloud connected to the smart appliance; the target convolutional neural network is obtained by training a preset target convolutional neural network based on a preset food image set, and is used to guide the preset lightweight network to identify food size.

[0145] S2, based on the food category, adjust the network depth of the target lightweight network to obtain the updated target lightweight network; using the updated target lightweight network, extract the geometric features of the food to be identified in the food image according to the food category.

[0146] S3. Determine the size of the ingredient to be identified based on its geometric features.

[0147] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0148] Furthermore, in conjunction with the food size recognition method for smart appliances provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the food size recognition methods for smart appliances described in the above embodiments.

[0149] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0150] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0151] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0152] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for recognizing food size in smart appliances, characterized in that, The method includes: Using a target lightweight network, food identification is performed on food images in smart appliances to obtain the food category of the food to be identified in the food images; the target lightweight network is obtained by knowledge distillation of a target convolutional neural network in the cloud connected to the smart appliance; the target convolutional neural network is trained on a preset target convolutional neural network based on a preset set of food images, and is used to guide the preset lightweight network to identify food size. Based on the food category, the network depth of the target lightweight network is adjusted to obtain an updated target lightweight network; using the updated target lightweight network, the geometric features of the food to be identified in the food image are extracted according to the food category. The size of the food ingredient to be identified is determined based on its geometric features.

2. The method for food size recognition in smart appliances according to claim 1, characterized in that, The method further includes: Based on a preset importance scoring strategy, the contribution of multiple output channels in a preset lightweight network is scored, resulting in multiple scoring results. Based on the multiple scoring results, identify redundant channels among the multiple output channels; The redundant channels are removed from the preset lightweight network to obtain the target lightweight network.

3. The method for food size recognition in smart appliances according to claim 2, characterized in that, The method further includes: Obtain the model parameters of the target convolutional neural network; Based on the model parameters, the preset lightweight network is updated to obtain the target lightweight network.

4. The method for food size recognition in smart appliances according to claim 1, characterized in that, The method further includes: Based on the hardware instructions in the smart appliance, the data type of the target lightweight network is modified according to a preset quantization strategy.

5. The method for recognizing food size in smart appliances according to any one of claims 1 to 4, characterized in that, The process of using a target lightweight network to identify food ingredients in images of smart appliances, and determining the food category of the ingredients to be identified in the images, includes: The acquired food ingredient images are preprocessed. Based on a preset lightweight network, food classification prediction is performed on the food images after image preprocessing to obtain the food category probability distribution of the food to be identified. The food category of the food to be identified is determined based on the probability distribution of the food categories.

6. The method for recognizing food size in smart appliances according to claim 5, characterized in that, The step of using the updated target lightweight network to extract the geometric features of the food to be identified in the food image according to the food category includes: Based on the food category, determine the size key points corresponding to the food to be identified; Based on the aforementioned size key points and the food bounding box of the food to be identified determined by the preset lightweight network, the geometric features of the food to be identified are extracted.

7. The method for food size recognition in smart appliances according to claim 6, characterized in that, Determining the size of the ingredient to be identified based on its geometric features includes: The size calculation strategy for the food to be identified is determined based on the food category of the food to be identified; Based on the size calculation strategy, and combining the size key points in the geometric features with the food bounding box, the size of the food to be identified is determined.

8. The method for recognizing food size in smart appliances according to claim 5, characterized in that, The method further includes: Obtain the coordinate information of the food to be identified in a preset coordinate system; the preset coordinate system is established with the center of the placement plane in the smart appliance where the food to be identified is placed as the origin, the horizontal direction of the placement plane as the horizontal axis, and the vertical direction in the placement plane that is perpendicular to the horizontal direction as the vertical axis. Based on the center of the placement plane, the coordinate information of the food to be identified is used for position verification to obtain the verification result; When the verification result indicates that the food to be identified is within a preset range, the number of layers of the food to be identified is determined based on a preset depth estimation algorithm; the preset range is defined with the center of the placement plane as the origin.

9. A smart appliance, characterized in that, The food size recognition method for smart appliances as described in any one of claims 1 to 8 is used to identify the image to be recognized in the food image; the smart appliance is one of a smart oven, a smart steamer, or a smart air fryer.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the food size recognition method for smart appliances as described in any one of claims 1 to 8.

Images