Ingredient depth supplementation methods and cooking equipment

Abstract

This invention relates to the field of smart home appliance technology, specifically to a method for food depth completion, comprising the following steps: acquiring an RGB image of the food and its corresponding depth information image, and performing image preprocessing to generate a depth completion training set; constructing a food depth completion model with a foreground attention module and a decoder, and training the food depth completion model based on the depth completion training set; acquiring a depth map of the food to be completed, and inputting the depth map of the food to be completed into the trained food depth completion model; the food depth completion model using the foreground attention module to extract global features and multi-scale contour edge features of the input image, fusing them, and then processing them through the decoder layer to obtain the completed food depth map; obtaining the food completion depth value based on the food completion depth map, the food completion depth value being used to configure the cooking time and temperature of the cooking equipment.

Description

Technical Field

[0001] This invention relates to the field of smart home appliance technology, and more specifically, to a method for deep food ingredient supplementation and its cooking equipment. Background Technology

[0002] In intelligent cooking scenarios, obtaining the height of ingredients and thus their corresponding volume and weight is crucial. However, due to the irregular nature of ingredients, their shapes are often irregular, and some areas may be obscured when placed in cooking equipment, making it impossible to obtain complete depth information. Therefore, a depth completion model is needed to complete the depth information.

[0003] Current depth completion algorithms are mostly used in indoor and outdoor scenarios, primarily addressing the issue of missing pixels on the depth map caused by environmental factors (such as lighting, reflection, and occlusion) during depth estimation. However, in close-up scenarios, their depth completion accuracy is insufficient, and they struggle to handle large, continuous depth gaps in food-related depth completion scenarios.

[0004] Therefore, there is an urgent need to develop a method for deep food ingredient replenishment and its cooking equipment that overcomes the above-mentioned defects. Summary of the Invention

[0005] This application provides a method for estimating the depth of food ingredients and a cooking device thereof, in order to at least solve the problems of insufficient accuracy of food ingredient depth completion in close-range scenes and difficulty in handling large-area continuous missing parts in existing depth completion models.

[0006] In a first aspect, the present invention provides a method for deep food ingredient supplementation, comprising the following steps:

[0007] Obtain the RGB image of the ingredients and the corresponding depth information image, perform image preprocessing, and generate a depth completion training set;

[0008] Construct a food ingredient depth completion model with a foreground attention module and a decoder, and train the food ingredient depth completion model based on the depth completion training set;

[0009] Obtain the food depth map to be completed, and input the food depth map to be completed into the trained food depth completion model;

[0010] The food depth completion model uses the foreground attention module to extract global features and multi-scale contour edge features of the input image, fuses them, and then processes them through the decoder layer to obtain the completed food depth map.

[0011] The ingredient completion depth value is obtained based on the ingredient completion depth map, and the ingredient completion depth value is used to configure the cooking time and temperature of the cooking equipment.

[0012] In some embodiments, the food depth completion model includes:

[0013] Select a depth completion model;

[0014] A foreground attention module is constructed, which includes: a one-dimensional convolutional layer, a two-way parallel global feature extraction module, a boundary attention module, and a feature concatenation layer;

[0015] The foreground attention module is embedded into the output layer of each encoder module of the depth completion model to obtain the food depth completion model.

[0016] In some embodiments, the global feature extraction module includes multiple large convolutional units of different scales connected in sequence. Each large convolutional unit includes a large convolutional layer, a BN layer, and a ReLU activation function layer connected in sequence. These units are used to perform multi-scale feature extraction on the input feature map to obtain a global feature map, which is then input to the feature concatenation layer of the foreground attention module.

[0017] In some embodiments, the boundary attention module includes a max pooling unit, a first convolutional unit, an average pooling unit, and a second convolutional unit connected in sequence.

[0018] The max pooling unit is used to perform max pooling on the input feature map in the height and width directions respectively to generate a fused directional feature map.

[0019] The first convolutional unit uses N parallel convolutional paths of different scales to perform multi-scale feature extraction on the fused directional feature map, generating a first multi-scale local feature map;

[0020] The average pooling unit is used to perform average pooling processing on the first multi-scale local feature map in the height direction and the width direction respectively to generate a fused mean feature map.

[0021] The second convolutional unit uses N parallel convolutional paths of different scales to extract features from the fused mean feature map at multiple scales, generate multi-scale local feature maps, and input them into the feature splicing layer of the foreground attention module.

[0022] In some embodiments, the input feature map is convolved by a one-dimensional convolutional layer and then max-pooled by two parallel height max-pooling layers and width max-pooling layers. After feature extraction by fully connected layers, the feature map is fused with the feature map output by the one-dimensional convolutional layer to generate the fused direction feature map.

[0023] In some embodiments, the fused directional feature map is convolved by N parallel convolutional layers of different scales, and then the feature maps of the output channels of the downsampling layer are spliced ​​together to generate the first multi-scale local feature map.

[0024] In some embodiments, the first multi-scale local feature map is convolved by a one-dimensional convolutional layer and then averaged by two parallel height-average pooling layers and width-average pooling layers. After feature extraction by fully connected layers, the feature map is fused with the feature map output by the one-dimensional convolutional layer to generate the fused mean feature map.

[0025] In some embodiments, the fused mean feature map is convolved through N parallel convolutional layers of different scales, and then the feature maps of the output channels of the downsampling layer are spliced ​​together to generate the multi-scale local feature map.

[0026] On the other hand, the present invention provides a cooking apparatus, comprising:

[0027] A housing, which forms the external outline of the cooking equipment;

[0028] An inner liner is located inside the box body, and the inner liner forms a cooking cavity for placing and cooking food ingredients.

[0029] A heating element, which is disposed at least between the inner liner and the housing, is used to heat the cooking cavity;

[0030] A controller is configured to store at least a food depth completion model, the food depth completion model including a foreground attention module and a decoder; the food depth completion model is pre-trained based on a depth completion training set.

[0031] The controller is connected to the heating assembly;

[0032] The controller is configured to acquire the food depth map to be completed and input the food depth map to be completed into the trained food depth completion model;

[0033] The food depth completion model uses the foreground attention module to extract global features and multi-scale contour edge features of the input image, fuses them, and then processes them through the decoder layer to obtain the completed food depth map.

[0034] Based on the food completion depth map, the food completion depth value is obtained, and the food completion depth value is used to configure the cooking time and temperature of the cooking equipment.

[0035] In some embodiments, the food depth completion model includes:

[0036] Select a depth completion model;

[0037] A foreground attention module is constructed, which includes: a one-dimensional convolutional layer, a two-way parallel global feature extraction module, a boundary attention module, and a feature concatenation layer;

[0038] The foreground attention module is embedded into the output layer of each encoder module of the depth completion model to obtain the food depth completion model.

[0039] Compared with related technologies, the advantages and positive effects of the present invention are as follows:

[0040] This invention provides a method for food depth completion and a cooking device thereof. By constructing a food depth completion model with a foreground attention module, it can effectively integrate global features and multi-scale contour edge features to process complex information of food at different scales. This avoids information loss or detail neglect that may occur when using only single-scale features, thereby improving the ability to capture food depth information. It is suitable for food depth completion tasks with large-area continuous missing parts in close-range scenes, significantly improving the accuracy and robustness of food depth completion. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating a method for completing the depth of food ingredients according to an exemplary embodiment of the present invention.

[0042] Figure 2 This is a schematic diagram of the CompletionFormer deep completion model according to an exemplary embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of the structure of a foreground attention module according to an exemplary embodiment of the present invention;

[0044] Figure 4 A schematic diagram showing the embedding location of the foreground attention module according to an exemplary embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram of the structure of a global feature extraction module according to an exemplary embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of the boundary attention module according to an exemplary embodiment of the present invention;

[0047] Figure 7 This is a schematic diagram of the input and output depth maps of the food depth completion module according to an exemplary embodiment of the present invention;

[0048] Figure 8 This is a schematic diagram of the structure of a cooking device according to an exemplary embodiment of the present invention;

[0049] Figure 9 This is a schematic diagram of the structure of another cooking device according to an exemplary embodiment of the present invention;

[0050] Figure 10 This is a hardware configuration diagram of a controller according to an exemplary embodiment of the present invention.

[0051] In the above image:

[0052] 1. Cooking equipment; 2. Heating components; 3. Memory; 4. Controller; 80. Bus; 81. Processor; 83. Communication interface. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0054] In the description of this application, it should be understood that the terms "center," "upper," "lower," "front," "rear," "left," "right," "vertical," "level," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0055] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0056] In intelligent cooking scenarios, obtaining the height of ingredients and thus their corresponding volume and weight is crucial. However, due to the irregular shape of ingredients, and the fact that some areas may be obscured when placed in cooking equipment, complete depth information cannot be obtained. Currently, depth completion models are typically used to achieve this.

[0057] Current depth completion algorithms are mostly used in indoor and outdoor scenarios, primarily addressing the issue of missing pixels on the depth map caused by environmental factors (such as lighting, reflection, and occlusion) during depth estimation. They are not suitable for depth estimation of food items at close range and struggle to handle large areas of continuous food depth loss.

[0058] To address the above problems, this invention provides a method for in-depth food ingredient information supplementation. Figure 1 This is a flowchart illustrating a method for completing the depth of food ingredients, as an exemplary embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0059] Step S100: Obtain the RGB image of the ingredients and the corresponding depth information image, and perform image preprocessing to generate a depth completion training set;

[0060] In some embodiments, a camera device is used to capture RGB images of the food ingredients, and a depth camera is used to obtain a depth image corresponding to the RGB image.

[0061] In some embodiments, image preprocessing to generate a depth completion training set specifically includes:

[0062] The RGB image is preprocessed, including noise reduction and color correction. Image processing techniques are used to extract color information from the RGB image, including histogram equalization and color space conversion.

[0063] The depth images captured by the depth camera are preprocessed, including removing invalid values ​​and filtering the depth data. Based on the principle and calibration parameters of the depth camera, the depth images are converted into real-world depth values.

[0064] Linear interpolation is used to map the color information of RGB images onto depth images, thereby achieving the fusion of color information in RGB images with depth information acquired by depth cameras.

[0065] The fused data is converted into a depth image format to generate a depth completion training set, which serves as the pre-training dataset for the food depth completion model.

[0066] Step S200: Construct a food depth completion model with a foreground attention module and train the food depth completion model based on the depth completion training set. The food depth completion model uses the foreground attention module to extract global features and multi-scale contour edge features of the input image, and then fuses them. After processing by the decoder layer, the completed food depth map is obtained.

[0067] In some embodiments, constructing a food depth completion model with a foreground attention module specifically includes:

[0068] Step S201: Select a depth completion model;

[0069] like Figure 2 As shown, in some embodiments, the depth completion model is selected as the CompletionFormer depth completion model.

[0070] Figure 2 This is a schematic diagram of the CompletionFormer deep completion model according to an exemplary embodiment of the present invention.

[0071] The CompletionFormer deep completion model is a hybrid architecture model for deep completion tasks, combining the advantages of Convolutional Neural Networks (CNNs) and Vision Transformers. The following are its main structural features:

[0072] The CompletionFormer deep completion model adopts a single-branch U-Net structure and enhances its feature extraction capabilities by introducing joint convolutional attention and a Transformer module (JCAT module). While maintaining a computational cost comparable to traditional CNN methods, it significantly improves efficiency.

[0073] The CompletionFormer depth completion model takes a sparse depth map and its corresponding RGB image as input. The input RGB image and sparse depth map are encoded through convolutional layers, and then their features are fused.

[0074] In some disclosed embodiments, the RGB image is embedded into a 48-channel feature map through a 3×3 convolutional layer, while the sparse depth map is embedded into a 16-channel feature map through another 3×3 convolutional layer.

[0075] The encoder part of the CompletionFormer deep completion model consists of repeatedly stacked encoder modules, which employ joint convolutional attention and Transformer modules. ，This refers to the JCAT module. The encoder section, composed of the JCAT module, is the core unit of the CompletionFormer. It deeply couples the convolutional attention layer and the Vision Transformer for multi-scale feature extraction.

[0076] The decoder employs skip connections to upsample and fuse the output features from each layer of the encoder. The features of each layer are upsampled through deconvolutional layers and combined with a convolutional attention mechanism to better handle multi-scale features.

[0077] In the final depth prediction stage, the CompletionFormer depth completion model uses a Spatial Propagation Network (SPN) to refine the initial depth estimate.

[0078] The CompletionFormer depth completion model primarily uses indoor and outdoor scene datasets. For example, the NYU Depth V2 dataset is an indoor scene dataset, with a camera-to-object distance ranging from 1 to 5 meters. KITTI and Cityscapes, on the other hand, are outdoor datasets, with a greater camera-to-object distance, approximately 5 to 20 meters. For these indoor and outdoor scenarios, a few centimeters of error in the depth estimation model output has a negligible impact on the final calculation result.

[0079] However, in the specific task scenario of calculating the height of ingredients, in order to ensure that the ingredients are clearly visible in the photos, the distance between the camera and the ingredients is usually controlled at about 0.5 meters. Since most ingredients are relatively short, an error of a few centimeters can lead to a large deviation in the final result when calculating the height of ingredients.

[0080] Applying the CompletionFormer depth completion model directly to food depth completion scenarios results in low accuracy of the completed food depth values. Furthermore, it is unsuitable for large, continuous depth gaps in close-range food depth estimation scenarios. Therefore, the CompletionFormer depth completion model needs to be improved to make it suitable for close-range food depth completion scenarios.

[0081] Step S202: Build the foreground attention module. Figure 3 This is a schematic diagram of the structure of a foreground attention module according to an exemplary embodiment of the present invention.

[0082] In some embodiments, such as Figure 3 As shown, the Foreground Attention (BGAT) module includes a one-dimensional convolutional layer, which is used to reduce the feature dimension and complete the initial preprocessing of the input feature map.

[0083] In some embodiments, such as Figure 3 As shown, the foreground attention module includes a two-way parallel global feature extraction module, a boundary attention module, and a feature concatenation layer.

[0084] The boundary attention module is used to extract the edge features of the food outline, combined with the depth value of the edge baking pan.

[0085] The global feature extraction module is used to extract the overall global features of the ingredients and baking pan. Combined with the overall depth value features of the baking pan, the boundary features and global features are input into the feature stitching layer and stitched based on channels to improve the accuracy and precision of depth completion.

[0086] The feature stitching layer is used to stitch and fuse the features extracted by the boundary attention module and the global feature module, and to fuse the overall global features of the ingredients and baking pan as well as the edge contour features, so as to provide accurate and reliable information support for depth completion, thereby improving the accuracy and precision of depth completion.

[0087] Step S203: After embedding the foreground attention module into the output layer of each encoder module of the depth completion model, the food depth completion model is obtained.

[0088] Figure 4 This is a schematic diagram of the structure of a food depth completion model according to an exemplary embodiment of the present invention, as shown below. Figure 4 As shown, after embedding the foreground attention module into the output layer of each encoder module of the CompletionFormer depth completion model, the food depth completion model is obtained.

[0089] Specifically, the encoder module of the CompletionFormer deep completion model uses the JCAT module.

[0090] Specifically, the feature map output by the JCAT module is subjected to a 1x1 convolution to reduce the number of channels to half of its original number. That is, the original number of channels is c, and after the 1×1 convolution, the number of channels becomes c / 2. These channels are then input into the global feature extraction module and the boundary attention module, respectively, to extract boundary features and global features. The two feature maps with extracted features are then concatenated along the channel dimension. The number of channels in both feature maps after feature extraction is c / 2. After concatenation along the channel dimension, the number of channels becomes c, which is then used as the input to the next module.

[0091] Figure 5 This is a schematic diagram of the structure of a global feature extraction module according to an exemplary embodiment of the present invention, as shown below. Figure 5 As shown, the global feature extraction module includes at least two large convolutional units of different scales connected in sequence. These units are used to perform multi-scale feature extraction on the input feature map to obtain a global feature map, which is then input to the feature concatenation layer of the foreground attention module.

[0092] In some embodiments, a large convolutional unit includes a large convolutional layer. ， It is used to capture a wider range of contextual information in the input feature map and learn global features.

[0093] In some embodiments, the large convolutional unit includes a BN (Batch Normalization) layer connected to the large convolutional layer, which is used to normalize the output of the convolutional layer, making the data distribution more stable and accelerating model convergence.

[0094] In some embodiments, the large convolutional unit includes a ReLU (Rectified Linear Unit) activation function layer connected to the large convolutional layer, which is used to introduce nonlinear factors into the model and enhance the model's expressive power.

[0095] In some embodiments, the configuration of the global feature extraction module is exemplified, but not limited to, that the global feature extraction module sets up two large convolutional units of different scales.

[0096] The first large convolutional unit uses an N×N convolutional kernel, which can cover a large image area in one convolution operation, perform preliminary feature extraction and integration on the input feature map, and extract the macroscopic feature information of the image.

[0097] After being processed by this large convolutional layer, the data enters the BN layer for normalization, making the data distribution more stable and helpful for subsequent training and computation. Then, it goes through the ReLU activation function layer for nonlinear transformation, enhancing the model's expressive power and obtaining the pre-processed feature map.

[0098] The second large convolutional unit uses an M×M convolutional kernel, which is smaller than the kernel size of the first large convolutional unit. It performs further feature extraction on the feature map based on the first large convolutional unit.

[0099] Because the convolutional kernel size is relatively small, it can process features more finely, extract more detailed information, and supplement features that were missed or not fully extracted by the previous large convolutional units. Similarly, after processing by BN layers and ReLU activation function layers, the feature map is normalized and non-linearized to optimize the feature map.

[0100] Finally, the acquired global feature map is input into the feature concatenation layer of the foreground attention module, providing effective global feature information for subsequent depth completion operations.

[0101] The global feature extraction module not only focuses on the information around the position to be filled, but also takes into account the overall information of the depth map. After the global feature extraction module, the depth information features of the entire baking pan can be obtained, so as to better fill in the missing food positions.

[0102] In some embodiments, the global feature extraction module can also be configured as three large convolutional units of different scales to capture information at different levels in the image and extract richer global features.

[0103] Figure 6 This is a schematic diagram of the boundary attention module according to an exemplary embodiment of the present invention.

[0104] like Figure 6 As shown, in some embodiments, the boundary attention module includes a max pooling unit, which performs max pooling on the input feature map in the height and width directions respectively to generate a fused orientation feature map.

[0105] In some embodiments, the input feature map is convolved by a one-dimensional convolutional layer and then max-pooled by two parallel height max-pooling layers and width max-pooling layers. After feature extraction by fully connected layers, the feature map is fused with the feature map output by the one-dimensional convolutional layer to generate a fused direction feature map.

[0106] Max pooling units can focus on the contour features of food in different directional dimensions, such as the features of the top and bottom contours of the food. This helps the model focus on key information in the height dimension and the edge features of the food in the horizontal direction, which is very important for recognizing the range and contour boundaries of the food on the plane.

[0107] Specifically, let the input feature map be I, with dimensions H×W×C. in Where H represents height, W represents width, and C represents height. in Indicates the number of input channels.

[0108] The input feature map is convolved using a 1×1 convolution kernel to obtain the output feature map F. 1×1 Its formula is:

[0109]

[0110] Where i∈[0,H-1], j∈[0,W-1], c∈[0,C] out -1], b(c) is the bias term of the c-th output channel.

[0111] For the output feature map F 1×1 Max pooling is performed in both the x and y directions, as follows:

[0112] Let the pooling kernel size of the max pooling layer in the x-direction be (1, K). x ), with a step size of (1, S) x For the input feature map F 1×1 Perform max pooling in the x-direction to obtain The formula is:

[0113]

[0114] Where i∈[0,H-1], c∈[0,C out -1].

[0115] Let the pooling kernel size of the maximum pooling layer in the y-direction be (K). y ,1), step size is (S) y ,1), for the input feature map F 1×1 Perform max pooling in the y-direction to obtain The formula is:

[0116]

[0117] in, j∈[0,W-1], c∈[0,C] out -1].

[0118] Feature map and feature map Features are extracted through fully connected layers, as follows:

[0119] For the feature map after max pooling in the x-direction After the fully connected layer W x (size is) ) and bias term b x get The formula is:

[0120]

[0121] Where i∈[0,H-1], σ is an activation function (such as the ReLU function: σ(x) = max(0,x)).

[0122] For the feature map after max pooling in the y direction After the fully connected layer W y (size is) ) and bias term b y get The formula is:

[0123]

[0124] in, j∈[0,W-1],

[0125] After feature extraction via a fully connected layer, the feature map is... Feature map The feature map F output by the one-dimensional convolutional layer 1×1 Feature fusion is performed to generate a fused direction feature map, as follows:

[0126] Let the fusion direction feature map be F. f Its formula is:

[0127]

[0128] Where i∈[0,H-1], j∈[0,W-1], c∈[0,C] out -1].

[0129] like Figure 6 As shown, in some embodiments, the boundary attention module includes a first convolutional unit connected to a max-pooling unit. The first convolutional unit uses N parallel convolutional paths of different scales to perform multi-scale feature extraction on the fused directional feature map, generating a first multi-scale local feature map.

[0130] In some embodiments, the fused directional feature map is convolved through N parallel convolutional layers of different scales, and then the feature maps of the output channels of the downsampling layer are spliced ​​to generate a first multi-scale local feature map.

[0131] In some embodiments, the configuration of the first convolutional unit is exemplified, but not limited to: the first convolutional unit includes three parallel convolutional layers of different scales, namely a convolutional layer with a 1×1 kernel, a convolutional layer with a 3×3 kernel, and a convolutional layer with a 5×5 kernel.

[0132] The fused directional feature map is processed by three convolutional kernels of different scales: 1×1, 3×3, and 5×5 to extract multi-scale features. By extracting features at different scales, edge information can be obtained better. The feature maps after 3×3 and 5×5 convolution are upsampled to the same size as the 1×1 convolution and then concatenated with the feature map after 1×1 convolution based on the channel dimension to obtain the first multi-scale local feature map.

[0133] like Figure 6 As shown, in some embodiments, the boundary attention module includes an average pooling unit connected to the first convolutional unit. The average pooling unit is used to perform average pooling on the first multi-scale local feature map in both the height and width directions to generate a fused mean feature map.

[0134] In some embodiments, the first multi-scale local feature map is convolved by a one-dimensional convolutional layer and then averaged by two parallel height average pooling layers and width average pooling layers. After feature extraction by fully connected layers, the feature map is fused with the feature map output by the one-dimensional convolutional layer to generate a fused mean feature map.

[0135] Applying average pooling to the first multi-scale local feature map in both the height and width directions can smooth the local features in these directions, reducing the impact of local noise and subtle fluctuations. In food depth completion tasks, the food surface may contain some fine textures or noise, which may interfere with the accuracy of depth completion. Average pooling in both the height and width directions can reduce noise interference, thereby providing more stable and reliable feature information for depth completion.

[0136] Specifically, let the first multi-scale local feature map be F. l Its size is H l ×W l ×C l .

[0137] The first multi-scale local feature map F is processed by a 1×1 convolution kernel. l Perform a convolution operation to obtain the output feature map F. conv Its formula is:

[0138]

[0139] Where i∈[0,H l -1],j∈[0,W l -k] (when the convolution kernel performs convolution in the width direction) or i∈[0,H] l -k],j∈[0,W l -1] (when the convolution kernel performs convolution in the height direction), c out ∈[0,C out -1],b conv (c out ) is the cth out The bias term for each output channel.

[0140] For the output feature map F conv Average pooling is performed in both the x and y directions, as follows:

[0141] Let the pooling kernel size in the x-direction be (h) p ,1), step size is (s) hp ,1), where h p Let s be the size of the pooling kernel in the height direction. hp Let F be the step size in the height direction. conv Feature maps are obtained by performing high-average pooling. The formula is:

[0142]

[0143] in, j∈[0,W l -1],c∈[0,C out -1].

[0144] Let the pooling kernel size in the y-direction be (1, w). p ), with a step size of (1, s) wp ), where w p s represents the quantization size of the pooling kernel in the width direction. wp Let F be the step size in the width direction. conv The feature map is obtained by performing width-average pooling. The formula is:

[0145]

[0146] Where i∈[0,H l -1], c∈[0,C out -1].

[0147] Let W be the weight matrix of the fully connected layer. fh (For feature maps after high average pooling) and W fw (For the feature map after width-average pooling), its dimensions are N. h ×M h and N w ×M w The bias terms are b fh and b fw .

[0148] For the feature map after high average pooling The feature map is obtained after passing through a fully connected layer. The formula is:

[0149]

[0150] in, j∈[0,W l -1],m∈[0,M h -1], where σ is the activation function (e.g., the ReLU function: σ(x) = max(0,x)).

[0151] For the feature map after width average pooling The feature map is obtained after passing through a fully connected layer. The formula is:

[0152]

[0153] Where i∈[0,H l -1], m∈[0,M w -1].

[0154] The height and width average pooled feature maps processed by the fully connected layer are compared with the feature map F output by the one-dimensional convolutional layer. conv Feature fusion is performed to obtain the fused mean feature map F. fusion Its formula is:

[0155]

[0156] Where i∈[0,H l -1],j∈[0,W l -1],c∈[0,C out -1].

[0157] like Figure 6 As shown, in some embodiments, the boundary attention module includes a second convolutional unit connected to the average pooling unit. The second convolutional unit uses N parallel convolutional paths of different scales to perform multi-scale feature extraction on the fused mean feature map, generating a multi-scale local feature map, which is then input to the feature concatenation layer of the foreground attention module.

[0158] In some embodiments, the fused mean feature map is convolved through N parallel convolutional layers of different scales, and then the feature maps of the output channels of the downsampling layer are spliced ​​to generate a multi-scale local feature map.

[0159] In some embodiments, the configuration of the second convolutional unit is exemplified, but not limited to: the second convolutional unit includes three parallel convolutional layers of different scales, namely a convolutional layer with a 1×1 kernel, a convolutional layer with a 3×3 kernel, and a convolutional layer with a 5×5 kernel.

[0160] The fused mean feature map is processed by three convolutional kernels of different scales: 1×1, 3×3, and 5×5 to extract multi-scale features. By extracting features at different scales, edge information can be better obtained. The feature maps after 3×3 and 5×5 convolution are upsampled to the same size as the 1×1 convolution and then concatenated with the feature map after 1×1 convolution based on the channel dimension to obtain a multi-scale local feature map.

[0161] Step S300: Obtain the food depth map to be completed, input the food depth map to be completed into the trained food depth completion model to obtain the food depth map, and obtain the food completion depth value. The food completion depth value is used to configure the cooking time and temperature of the cooking equipment.

[0162] Figure 7 This is a schematic diagram of the input and output depth maps of the food depth completion module according to an exemplary embodiment of the present invention, as shown below. Figure 7 As shown, the left image is the food depth map to be completed, and the right image is the food depth map output after the food depth completion model.

[0163] In some embodiments, the depth map to be completed for the food ingredients needs to be obtained using a food ingredient depth estimation model and a segmentation algorithm.

[0164] Specifically, the current method for obtaining the height of food ingredients involves acquiring a depth information map of the ingredients based on an ingredient depth estimation algorithm, and then calculating the height of the ingredients at that point by subtracting the depth information of the corresponding cooking equipment component, such as the baking pan. However, due to the tilt of the depth camera and other equipment, the fact that the baking pan is not an ideal plane, and the fact that the shape of the ingredients is usually irregular and may obscure part of the baking pan below, complete depth information cannot be obtained, affecting the calculation of the ingredient height.

[0165] To address the aforementioned influencing factors, an image segmentation algorithm is used to separate the food region from the food depth information map. This separates the food region from the background and other cooking equipment components in the depth information map, and sets the depth information of the food region to 0, thus obtaining... Figure 7 The image in the middle left shows the depth map of the ingredients that needs to be completed.

[0166] Combination Figure 7 The food depth map to be completed is input into the trained food depth completion model to obtain the food completion depth map, such as... Figure 7 The right-middle figure shows the depth of the entire cooking equipment component. The height of the food at that point is obtained by subtracting the depth information of the food component corresponding to that point from the depth information of the food at that point.

[0167] This application also provides a cooking device. Figure 8 This is a schematic diagram of the structure of a cooking device according to an exemplary embodiment of the present invention.

[0168] In some embodiments, such as Figure 8 As shown, the cooking device 1 includes: a housing for forming the external outline of the cooking device; an inner liner disposed inside the housing, forming a cooking cavity for placing and cooking ingredients.

[0169] In some embodiments, such as Figure 8 As shown, the cooking device 1 includes a heating element 2 for changing the temperature of the cavity. The heating element 2 is located at least between the inner pot and the casing, and is used to heat the cooking cavity. The heating element 2 can raise the internal temperature of the cooking device 1.

[0170] In some embodiments, such as Figure 9As shown, the cooking device 1 includes a controller 4. The controller 4 includes a memory 3, which stores at least a food depth completion model, which is obtained by pre-training based on a depth completion training set.

[0171] Controller 4 is connected to heating component 2.

[0172] In some embodiments, the controller 4 is configured to acquire a depth map of the ingredients to be completed, input the depth map of the ingredients to be completed into a trained depth completion model to acquire a depth map of the ingredients to be completed, and obtain a depth value of the ingredients to be completed. The depth value of the ingredients to be completed is used to configure the cooking time and temperature of the cooking device.

[0173] In some embodiments, the controller is also configured to:

[0174] Constructing a food ingredient depth completion model with a foreground attention module includes:

[0175] Select a depth completion model;

[0176] A foreground attention module is constructed, which includes: a one-dimensional convolutional layer, a two-way parallel global feature extraction module, a boundary attention module, and a feature concatenation layer;

[0177] By embedding the foreground attention module into the output layer of each encoder module of the depth completion model, the food depth completion model is obtained.

[0178] In addition, combined Figure 1 The food ingredient depth completion method described in this application embodiment can be implemented by cooking equipment 1. Figure 10 This is a hardware configuration diagram of controller 4 according to an embodiment of this application.

[0179] The controller 4 of the cooking device 1 may include a processor 81 and a memory 3 storing computer program instructions.

[0180] Specifically, the processor 81 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0181] The memory 3 may include a large-capacity storage device for data or instructions. For example, and not limitingly, the memory 3 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 3 may include removable or non-removable (or fixed) media. Where appropriate, the memory 3 may be internal or external to the data processing device. In a particular embodiment, the memory 3 is non-volatile memory. In a particular embodiment, the memory 3 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0182] The memory 3 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 81.

[0183] The processor 81 reads and executes the computer program instructions stored in the memory 3 to implement the recipe cooking system in the above embodiment.

[0184] In some embodiments, the controller 4 of the cooking device 1 may further include a communication interface 83 and a bus 80. For example, Figure 10 As shown, the processor 81, memory 3, and communication interface 83 are connected through bus 80 and complete communication with each other.

[0185] The communication interface 83 is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of this application. The communication interface 83 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0186] Bus 80 includes hardware, software, or both, that couples the components of cooking appliance 1 together. Bus 80 includes, but is not limited to, at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. For example, and not as a limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 80 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0187] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0188] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for deep food ingredient completion, characterized in that, Includes the following steps: Obtain the RGB image of the ingredients and the corresponding depth information image, perform image preprocessing, and generate a depth completion training set; Construct a food ingredient depth completion model with a foreground attention module and a decoder, and train the food ingredient depth completion model based on the depth completion training set; Obtain the food depth map to be completed, and input the food depth map to be completed into the trained food depth completion model; The food depth completion model uses the foreground attention module to extract global features and multi-scale contour edge features of the input image, fuses them, and then processes them through the decoder layer to obtain the completed food depth map. The ingredient completion depth value is obtained based on the ingredient completion depth map, and the ingredient completion depth value is used to configure the cooking time and temperature of the cooking equipment.

2. The method for completing the food ingredient depth according to claim 1, characterized in that, The food ingredient depth completion model includes: Select a depth completion model; A foreground attention module is constructed, which includes: a one-dimensional convolutional layer, a two-way parallel global feature extraction module, a boundary attention module, and a feature concatenation layer; The foreground attention module is embedded into the output layer of each encoder module of the depth completion model to obtain the food depth completion model.

3. The method for completing the food ingredient depth according to claim 2, characterized in that, The global feature extraction module includes multiple large convolutional units of different scales connected in sequence. Each large convolutional unit includes a large convolutional layer, a BN layer, and a ReLU activation function layer connected in sequence. These units are used to perform multi-scale feature extraction on the input feature map to obtain a global feature map, which is then input to the feature concatenation layer of the foreground attention module.

4. The method for completing the food ingredient depth according to claim 2, characterized in that, The boundary attention module includes a max pooling unit, a first convolutional unit, an average pooling unit, and a second convolutional unit connected in sequence. The max pooling unit is used to perform max pooling on the input feature map in the height and width directions respectively to generate a fused directional feature map. The first convolutional unit uses N parallel convolutional paths of different scales to perform multi-scale feature extraction on the fused directional feature map, generating a first multi-scale local feature map; The average pooling unit is used to perform average pooling processing on the first multi-scale local feature map in the height direction and the width direction respectively to generate a fused mean feature map. The second convolutional unit uses N parallel convolutional paths of different scales to extract features from the fused mean feature map at multiple scales, generate multi-scale local feature maps, and input them into the feature splicing layer of the foreground attention module.

5. The method for completing the food ingredient depth according to claim 4, characterized in that, The input feature map is convolved by a one-dimensional convolutional layer, then max-pooled by two parallel height max-pooling layers and width max-pooling layers. After feature extraction by fully connected layers, the feature map is fused with the feature map output by the one-dimensional convolutional layer to generate the fused direction feature map.

6. The method for completing the food ingredient depth according to claim 4, characterized in that, The fused directional feature map is convolved through N parallel convolutional layers of different scales, and then the feature maps of the output channels of the downsampling layer are spliced ​​together to generate the first multi-scale local feature map.

7. The method for completing the food ingredient depth according to claim 4, characterized in that, The first multi-scale local feature map is convolved by a one-dimensional convolutional layer and then averaged by two parallel height-average pooling layers and width-average pooling layers. After feature extraction by fully connected layers, it is fused with the feature map output by the one-dimensional convolutional layer to generate the fused mean feature map.

8. The method for completing the food ingredient depth according to claim 4, characterized in that, The fused mean feature map is convolved through N parallel convolutional layers of different scales, and then the feature maps of the output channels of the downsampling layer are spliced ​​together to generate the multi-scale local feature map.

9. A cooking device, characterized in that, include: A housing, which forms the external outline of the cooking equipment; An inner liner is located inside the box body, and the inner liner forms a cooking cavity for placing and cooking food ingredients. A heating element, which is disposed at least between the inner liner and the housing, is used to heat the cooking cavity; A controller is configured to store at least a food depth completion model, the food depth completion model including a foreground attention module and a decoder; the food depth completion model is pre-trained based on a depth completion training set. The controller is connected to the heating assembly; The controller is configured to acquire the food depth map to be completed and input the food depth map to be completed into the trained food depth completion model; The food depth completion model uses the foreground attention module to extract global features and multi-scale contour edge features of the input image, fuses them, and then processes them through the decoder layer to obtain the completed food depth map. Based on the food completion depth map, the food completion depth value is obtained, and the food completion depth value is used to configure the cooking time and temperature of the cooking equipment.

10. The cooking apparatus according to claim 9, characterized in that, The food ingredient depth completion model includes: Select a depth completion model; A foreground attention module is constructed, which includes: a one-dimensional convolutional layer, a two-way parallel global feature extraction module, a boundary attention module, and a feature concatenation layer; The foreground attention module is embedded into the output layer of each encoder module of the depth completion model to obtain the food depth completion model.

Images