Refrigerator foodstuff self-adaptive scale recognition method, device and storage medium

Abstract

The present application relates to a kind of refrigerator foodstuff self-adapting scale identification method, equipment and storage medium, the method includes the steps of: constructing deconvolution regression network framework, which includes regression network and classification network;Regression network uses scale predictor to carry out self-adapting scale distribution to multiple target candidate frames, produce multiple multi-scale target candidate frames, and then carry out boundary frame regression processing, obtain multiple candidate frame regressions;Classification network extracts the feature map in the multiple multi-scale target candidate frames produced in regression network, and the multiple multi-scale feature maps produced are respectively spliced and fused with corresponding scale, to obtain final classification result;According to the above method, deconvolution regression network is obtained by training;The image to be identified is input into the network, to obtain target type and corresponding storage location.The separation of regression and classification improves the flexibility and accuracy of extracting features on feature maps of different resolutions to regress positions.

Description

Technical Field

[0001] This invention relates to the field of household appliances, and more particularly to a method, device, and storage medium for adaptive scale recognition of food ingredients in a refrigerator. Background Technology

[0002] In a refrigerator scenario, food items placed on different shelves typically have different sizes, and these different sizes often have different appearances and even different postures. Using a single feature description is far from sufficient. Especially for small-scale food items at the bottom of the refrigerator, a generic feature representation often leads to insufficient classification capabilities. Therefore, designing a scale-adaptive feature description is crucial for improving food detection and recognition performance.

[0003] Among existing object detectors based on convolutional neural networks, the two-stage Faster R-CNN and the single-stage YOLO are the most popular detection methods. Although these methods handle targets of different sizes by introducing multi-scale feature processing, for a refrigerator overhead shot, the same object has a huge scale difference between the top layer (closest to the camera) and the bottom layer (farthest from the camera). Traditional object detection methods often perform brute-force matching on feature maps based on the scale of such targets, leading to the loss or missed detection of target information during the matching process. This results in weak feature representation capabilities and is prone to misclassification of the target category. Summary of the Invention

[0004] The purpose of this invention is to provide a method, device and storage medium for adaptive scale recognition of food ingredients in a refrigerator, which combines multi-scale target regression and classification to enhance the target recognition effect.

[0005] To achieve the aforementioned objectives, this invention provides a method for adaptive scale recognition of food ingredients in a refrigerator. The method includes the following steps: acquiring food ingredient image information and creating a target training dataset; constructing a deconvolutional regression network framework, which includes a regression network and a classification network; the regression network uses a scale predictor to adaptively allocate the scale of multiple target candidate boxes, generating multiple multi-scale target candidate boxes, and then performs bounding box regression processing on the multiple multi-scale target candidate boxes according to their scale size to obtain multiple multi-scale target candidate box regressions; the classification network extracts feature maps from the multiple multi-scale target candidate boxes generated in the regression network, and concatenates and fuses the generated multiple multi-scale feature maps at their corresponding scales to obtain the final classification result; training the deconvolutional regression network according to the method for constructing the regression network and the classification network; and inputting the food ingredient image to be recognized into the trained deconvolutional regression network to obtain the target food ingredient type and its corresponding storage location.

[0006] As a further improvement of the present invention, the method further includes: the scale predictor performs adaptive scale allocation based on the size and feature distribution of the target candidate box.

[0007] As a further improvement of the present invention, the method further includes: the deconvolutional regression network framework uses a pre-trained convolutional neural network as the basic feature extractor.

[0008] As a further improvement of the present invention, the method further includes: the pre-trained convolutional neural network includes a VGG16 base network.

[0009] As a further improvement of the present invention, the step method further includes: during the training process of constructing the deconvolutional regression network, the extractor uses an upsampling method or an unpooling method to extract feature maps.

[0010] As a further improvement of the present invention, the step of "performing border regression processing on the multiple multi-scale target candidate boxes according to their scales" specifically includes: using non-maximum suppression to remove duplicate target candidate boxes from the target candidate boxes to obtain deduplicated target candidate boxes; establishing a regression equation between the deduplicated target candidate boxes and a preset target value; adjusting the coordinates of the deduplicated target candidate boxes to obtain the final position result.

[0011] As a further improvement of the present invention, the step of "extracting feature maps from multiple multi-scale target candidate boxes generated by the classification network and the regression network" specifically includes: concatenating and fusing the multiple multi-scale feature maps according to their scale size and channel dimension to obtain multiple fused feature maps; performing a convolution operation on the multiple fused feature maps to reduce the dimensionality to 1 channel, resulting in multiple dimensionality-reduced fused feature maps; and performing weighted fusion processing on the multiple dimensionality-reduced fused feature maps to obtain targets at multiple scales.

[0012] As a further improvement of the present invention, the step of "splitting and fusing the multiple multi-scale feature maps according to their scale size and channel dimension" is further included by performing ROI Align pooling operation on the multiple multi-scale feature maps to obtain multiple fixed-size feature maps.

[0013] The present invention also provides a refrigeration device, the refrigeration device including a memory and a processor, the memory storing a computer program that can run on the processor, the processor executing the program to implement the steps in the refrigerator food adaptive scale recognition method as described in any of the above.

[0014] The present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps in the refrigerator food adaptive scale recognition method described in any of the above claims.

[0015] Compared with existing technologies, this invention has the following advantages: This invention constructs a deconvolutional regression network based on a pre-trained convolutional neural network. This network separates target regression from target type recognition, improving robustness and flexibility. A scale predictor is introduced during the regression network construction process to adaptively allocate the scale of multiple extracted target candidate boxes. The target candidate boxes are then regressed according to their scale, enabling the prediction of targets at different scales on feature maps of different resolutions, thus improving the flexibility and accuracy of target location regression at different scales.

[0016] Meanwhile, during the construction of the classification network, feature maps are extracted from multiple multi-scale candidate boxes generated by the classification network using upsampling or unpooling methods. This results in high-resolution feature maps, providing a good data source for subsequent location regression and type recognition. The multiple multi-scale feature maps are then concatenated and fused according to their scale to generate multiple fused features of different scales. These fused features are then weighted and fused to form a multi-scale feature representation. This invention fully integrates features of different scales at different resolutions, reducing the loss or missed detection of target information during feature map extraction, thereby improving recognition accuracy and enhancing category recognition performance. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the refrigerator food adaptive scale recognition method in an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of the deconvolutional regression network framework in an embodiment of the present invention.

[0019] Figure 3 This is a schematic diagram of the candidate box regression processing during the construction of the regression network in an embodiment of the present invention.

[0020] Figure 4 This is a schematic diagram of the process of fusing extracted candidate box feature maps during the construction of the classification network in this embodiment of the invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0023] This application discloses an embodiment of a refrigerator food adaptive scale recognition method, although this application provides the following embodiments or processes. Figure 1 The method described herein involves steps that, while based on conventional or non-creative labor, involve steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the order provided in the embodiments of this application. For example... Figure 1 As shown in the figure, this invention provides a method for adaptive scale recognition of food ingredients in a refrigerator. The method includes the following steps, which are described below:

[0024] S1: Obtain food image information and create the target training dataset.

[0025] S2, construct a deconvolutional regression network framework, which includes a regression network and a classification network.

[0026] S3, the regression network uses a scale predictor to adaptively assign scales to multi-level, multi-scale candidate boxes, resulting in multiple scale candidate box regressions.

[0027] S4, the classification network extracts the feature maps of multiple multi-scale candidate boxes generated in the regression network, generates multiple multi-scale feature maps, and performs weighted fusion on them according to their respective scales to obtain the final classification result.

[0028] S5. Train the network according to the methods in steps S3 and S4 to obtain the deconvolutional regression network.

[0029] S6, Input the image of the food to be identified into the network to obtain the food type and storage location.

[0030] In one embodiment of the present invention, images of food items in a refrigerator can be acquired using a high-definition camera or an infrared camera. The collected images are then filtered to create a target training dataset. This target training dataset provides training samples for the subsequent construction of a deconvolutional regression network.

[0031] Using an arbitrary convolutional neural network pre-trained on ImageNet as the basic feature extractor, in a preferred embodiment of the present invention, a VGG16 network is used. The VGG16 network is a type of convolutional neural network. This network structure contains a total of 16 layers with parameters, namely 13 convolutional layers, 5 pooling layers or 5 convolutional layers, 3 fully connected layers, and a softmax output layer. The layers are separated by max-pooling, and the activation units of all hidden layers use the ReLU function.

[0032] For the VGG16 convolutional neural network, pooling layers serve as the dividing line. The VGG16 network has a total of 6 modules, each with the same number of channels. Because both convolutional and fully connected layers have weights, they are also called weighted layers. There are 13 convolutional layers, 3 fully connected layers, and pooling layers do not involve weights. Therefore, the 13 convolutional layers and 5 pooling layers in this network are responsible for feature extraction, while the final 3 fully connected layers and the softmax output layer are responsible for the classification task.

[0033] In one embodiment of the present invention, a deconvolutional regression network framework is constructed based on the VGG16 network. This framework includes a regression network and a classification network. The regression network uses an arbitrary convolutional neural network pre-trained on ImageNet as the basic feature extractor. Based on the target scale in the input image and using a selective search algorithm, it extracts multiple target candidate boxes of varying scales from each layer of the network. During the extraction of multiple target candidate boxes at various scales, a scale predictor is introduced to adaptively allocate the scales of these multiple target candidate boxes. The scale predictor performs adaptive scale allocation based on the scale and feature distribution of the target candidate boxes. Different scale target candidate boxes extracted from each layer, such as large-scale and small-scale target candidate boxes, are processed separately to generate multiple corresponding scale target candidate box regressions. The use of the scale predictor allows for separate processing of targets of different scales, improving the regression accuracy for targets of different scales. Then, candidate box regression processing is performed on these multiple corresponding scale target candidate boxes to obtain multiple multi-scale target candidate box regressions, such as large-scale candidate box regressions and small-scale candidate box regressions.

[0034] In one embodiment of the present invention, the classification network extracts feature maps from multiple target candidate boxes of various scales generated in the regression network, generating multiple feature maps of various scales. These feature maps are then concatenated and fused according to their respective scales to generate a multi-scale fused feature map. This fused feature map is then subjected to weighted fusion processing to obtain the final target classification result. Specifically, the convolutional neural network has multiple layers. Each layer uses a selective search algorithm to extract multiple target candidate boxes of different scales. A scale predictor is then introduced to classify these multiple target candidate boxes according to their scale, generating multiple target candidate boxes of various scales. These multiple multi-scale candidate boxes are then fed into the classification network. Feature extraction is performed on each of the multiple target candidate boxes of each scale (because each layer extracts target candidate boxes of the corresponding scale, multiple layers will generate multiple target candidate boxes), generating multiple feature maps. These feature maps are then concatenated and fused, thus avoiding the loss of target information during the feature map extraction process at each layer and improving the accuracy of subsequent image type recognition.

[0035] Based on the construction methods of the regression and classification networks described above, a deconvolutional regression network is trained using the target training dataset. The food image to be identified is then input into this network, which automatically identifies the type of targets at different scales in the food image and their corresponding locations in the refrigerator. The location of a target in the image is typically represented by (x, y, w, h), where x and y are the center point of the target, and w and h are the length and width of the target candidate box. The type of the target indicates what the target is specifically, such as an apple or a tomato.

[0036] Figure 2 The diagram shows a schematic of a deconvolutional regression network framework. In this embodiment of the invention, the pre-trained arbitrary convolutional neural network preferably uses VGG16. This network includes a multi-layer network feature map extraction structure. Before feature map extraction, each layer uses a selective search algorithm to extract target candidate boxes. Based on this algorithm, a scale predictor is introduced to adaptively allocate the scale of the generated target candidate boxes, generating target candidate boxes of various scales. This allows the positions of both large-scale and small-scale targets in the image to be detected. In other words, targets of different scales are predicted on feature maps of different resolutions. Large targets are predicted on low-resolution feature maps, medium-scale targets are predicted on medium-resolution feature maps, and small targets are predicted on high-resolution feature maps, thus realizing the regression of target candidate boxes of different scales, i.e., determining the position of targets of different scales.

[0037] On the other hand, such as Figure 2As shown, the feature maps extracted from each layer are also fused at the corresponding positions according to the target scale. That is, firstly, large-scale target candidate boxes and small-scale target candidate boxes are extracted from each layer of the network. Then, feature maps of large-scale target candidate boxes at corresponding positions in the multi-layer network are extracted. The resulting multiple large-scale target feature maps are spliced ​​and fused to produce the fused feature map of the large-scale target. Similarly, the same fusion operation is performed on the small-scale target candidate boxes to produce the fused feature map of the small-scale target. Finally, the fused feature maps of all different scales are combined and weighted to identify the type of target at each scale in the image.

[0038] Figure 3 This is a schematic diagram of the candidate box regression processing during the construction of the regression network in this embodiment of the invention, specifically including the following steps:

[0039] S31, use non-maximum suppression to remove duplicate candidate boxes, and obtain deduplicated candidate boxes.

[0040] S32. Establish a regression equation, adjust the coordinates of the deduplicated candidate boxes, and obtain the final position result.

[0041] This algorithm uses a selective search approach to obtain multi-level, multi-scale target candidate boxes. For example, an image may contain hundreds of target candidate boxes, which may be redundant. Non-Maximum Suppression (NMS) is used to select the candidate boxes with the highest scores while suppressing those with lower scores, resulting in deduplicated target candidate boxes. A regression equation is then established between the finally selected target candidate boxes and a preset target value. The algorithm seeks a regression candidate box that, after mapping the input target candidate box through this equation, more closely resembles the true target candidate box, thus improving prediction accuracy. In other words, by establishing a mapping relationship between the input target candidate box and the target value, the position of the input candidate box is adjusted to obtain the final target location result.

[0042] Figure 4 This is a schematic diagram of the process of fusing extracted candidate box feature maps during the construction of the classification network in this embodiment of the invention, specifically including the following steps:

[0043] S41 uses ROI Align pooling to obtain multiple candidate boxes of fixed size.

[0044] S42, extract multiple feature maps from multiple candidate boxes and stitch them together to generate multiple fused feature maps.

[0045] S43, weighted processing of multiple fused feature maps to obtain the target type.

[0046] The classification network obtains multiple target candidate boxes of various scales from the regression network. The size of these target candidate boxes is not fixed. The ROI Align (Region of Interest Align) pooling operation is performed on them to fix the size of the target candidate boxes, which provides better assistance for subsequent target type identification.

[0047] After performing ROI Align pooling on candidate boxes at each scale, feature maps from each layer of target candidate boxes are extracted and concatenated along the channel dimension to obtain multiple fused feature maps. These are then subjected to a convolution operation to reduce the dimensionality to one channel, ensuring that the resulting fused feature map is spatially similar to the input feature map. Figure 1 To achieve this, target candidate boxes at various scales are all fused using the same method, resulting in multiple fused feature maps. These fused feature maps are then subjected to weighted fusion processing to identify the target type. The weights selected during the weighted fusion process are related to the scene in which the target was captured. For example, if the target is close to the camera and the target scale is large, the larger-scale feature map will have a higher weight. This strengthens the target's feature information and improves the accuracy of target type identification. The result of the weighted fusion processing is passed through a fully connected layer of VGG16. The resulting feature map is unfolded into a vector, and a convolution operation is performed on this vector to reduce its dimensionality. This vector is then input into the output layer, i.e., the softmax layer, to obtain the confidence score of the target category at each scale in the image, thereby identifying the type of multi-scale targets.

[0048] This invention also provides a refrigeration device, which includes a memory and a processor. The memory stores instructions, and the processor calls the instructions in the memory to enable the refrigeration device to implement the refrigerator food adaptive scale recognition method as described above.

[0049] This invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the refrigerator food adaptive scale recognition method described in any of the above embodiments.

[0050] In summary, the refrigerator food adaptive scale recognition method, device, and storage medium provided by this invention effectively handle targets of different scales. First, based on a pre-trained basic network, the invention uses a scale predictor to adaptively extract features from feature maps of different resolutions to regress the location according to the size and feature distribution of the target candidate box. In contrast, traditional methods regress targets of all scales in the final convolutional layer. Therefore, this scale-adaptive regression can extract more accurate feature representations, thereby better representing the regression of targets of different scales.

[0051] On the other hand, in order to further enhance the judgment ability of the target type classifier, this invention also integrates multi-layer deep features to predict the type of the target. The fused feature map forms a multi-scale feature representation, which can not only reduce the loss of some information of the target object when performing ROI pooling operation, but also significantly enhance the target feature representation. Compared with the traditional method of only using the last feature for classification, this invention can effectively capture more refined information of the target and enhance the type recognition effect.

[0052] It should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0053] The detailed descriptions listed above are merely specific descriptions of feasible implementations of the present invention and are not intended to limit the scope of protection of the present invention. All equivalent implementations or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. An adaptive image recognition method for food ingredients, characterized in that, Includes the following steps: Acquire food image information and create a target training dataset; Construct a deconvolutional regression network framework, which includes a regression network and a classification network; The regression network uses a scale predictor to adaptively allocate the scale of multiple target candidate boxes, generating multiple multi-scale target candidate boxes. Then, the multiple multi-scale target candidate boxes are subjected to bounding box regression processing according to their scale size to obtain multiple multi-scale target candidate box regressions. The classification network extracts feature maps from multiple multi-scale target candidate boxes generated in the regression network, and then concatenates and fuses the multiple multi-scale feature maps at their respective scales to obtain the final classification result. Based on the methods for constructing regression and classification networks, a deconvolutional regression network is obtained through training. The deconvolutional regression network framework uses a pre-trained convolutional neural network as the basic feature extractor. During the training process of constructing the deconvolutional regression network, the basic feature extractor uses an upsampling method or an unpooling method to extract feature maps. The scale predictor implements adaptive scale allocation based on the size of the target candidate box and the feature distribution. The image of the food to be identified is input into a trained deconvolutional regression network to obtain the target food type and its corresponding storage location.

2. The adaptive food image recognition method according to claim 1, characterized in that... The method further includes: The pre-trained convolutional neural network includes the VGG16 base network.

3. The adaptive food image recognition method according to claim 1, characterized in that... The phrase "then perform bounding box regression processing on the multiple multi-scale target candidate boxes according to their scale" specifically includes: Non-maximum suppression is applied to the target candidate boxes to remove duplicate target candidate boxes, resulting in deduplicated target candidate boxes; A regression equation is established between the deduplicated target candidate boxes and the preset target values. The coordinates of the deduplicated target candidate boxes are then adjusted to obtain the final position result.

4. The adaptive food image recognition method according to claim 1, characterized in that, The phrase "the classification network extracts feature maps from multiple multi-scale target candidate boxes generated by the regression network" specifically includes: The multiple multi-scale feature maps are spliced ​​and fused according to their scale size and channel dimension to obtain multiple fused feature maps; The multiple fused feature maps are then subjected to a convolution operation to reduce the dimensionality to one channel, resulting in multiple dimensionality-reduced fused feature maps. The multiple dimensionality-reduced fusion feature maps are subjected to weighted fusion processing to obtain target recognition types at various scales.

5. The adaptive food image recognition method according to claim 4, characterized in that, The step of "splitting and fusing the multiple multi-scale feature maps according to their scale size and channel dimension" includes the following: ROI Align pooling is performed on the multiple multi-scale feature maps to obtain multiple fixed-size feature maps.

6. A refrigeration device, characterized in that, The refrigeration device includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements the food image adaptive recognition method as described in any one of claims 1-5.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps in the food image adaptive recognition method according to any one of claims 1-5.

Images