Oven food maturity grade detection method based on improved Mask R-CNN

Abstract

The invention belongs to the technical field of intelligent detection, and discloses an improved Mask R-CNN-based oven food maturity grade detection method, which comprises the steps of S1, collecting RGB images of each maturity stage in an oven food baking process, and dividing all the images into a training set, a verification set and a test set; s2, based on the collected image data set, labeling the data set by using label to obtain the category and mask information to which each target belongs, and converting the category and mask information into a data set format required by a model; s3, carrying out improvement based on an existing Mask R-CNN and carrying out model training; and S4, verifying the model training effect based on the test set in the S1, and testing the detection effect of the food maturity condition. By optimizing the network framework structure and the training parameters, the food of different maturity conditions in the same category can be accurately identified. The model can effectively identify the maturity level of food in the oven, and the intelligent detection capability of the oven is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent detection, and in particular relates to a method for detecting the maturity level of oven food based on an improved Mask R-CNN. Background Art

[0002] In recent years, with the improvement of living standards, the intelligent kitchen experience has brought many substantial benefits and pleasant experiences to people. For example, people can control kitchen equipment through verbal instructions, which greatly improves the convenience of kitchen operation, or they can connect to kitchen equipment through the Internet to achieve remote control of the baking process. These intelligent kitchen appliances allow people to easily make a variety of delicacies, but due to the difficulty in deciding the baking temperature and baking time, the expected delicacies may become burnt or unsatisfactory. Especially for devices such as ovens, where the baking process is in a confined space and is not easy to view, it is easy to cause unsatisfactory food baking by relying solely on its pre-set baking program. Although some ovens are equipped with corresponding temperature probes to detect the temperature inside the food to avoid over- or under-baking, this method is too complicated and overly dependent on manual operation. Long-term consumption of over-mature or under-mature food can cause considerable harm to human health. Therefore, it is crucial to realize the intelligent perception of the maturity level of oven food to ensure the quality of oven food baking.

[0003] Machine vision is a technology that simulates the human visual system. With its high adaptability, high stability and high convenience, it is widely used in various detection and recognition fields, and its application scenarios are also very wide. For example, Lara et al. used machine vision to systematically evaluate the changes in quality parameters and surface characteristics of corn biscuits in the oven at different temperatures. Fan et al. designed an oven food image acquisition device to ensure that the image of baked food is collected under the same light source, and used image features to segment the food background. The state and quality of baked food were evaluated based on the proposed indicators, which helped to accurately control the temperature inside the oven. The development of deep learning in recent years has brought revolutionary changes to the field of machine learning. Due to its high generalization performance, deep learning can directly analyze a large amount of data, and automatically and fully extract features to establish various recognition and detection models. While achieving good results, it simplifies the steps of data preprocessing and feature extraction, and is widely used in image recognition and classification tasks. For example, Xiao et al. used Swin Transformer to classify the maturity of apples and achieved good classification results. Shuprajhaa et al. used CNN (Convolution Neural Network, CNN) to detect and classify the ripening stage of bananas, and the results were better than other linear classification methods. Although these methods have achieved quite good results in the field of food recognition and detection, it is difficult to accurately locate each target in the image during application, so they have certain limitations in practical applications.

[0004] Mask R-CNN is a deep learning model that integrates target detection and real object segmentation. When multiple targets appear in an image, it can still detect the exact location and category of each target in the image. In addition, it can generate a mask for each target to facilitate subsequent processing in the application. At present, it is more used in multiple different categories of detection and recognition fields such as autonomous driving and medical image analysis, while it is not widely used in multi-level detection and recognition under the same category. Wang et al. integrated Swin Transformer into the structure of Mask R-CNN and realized the maturity level detection of strawberries based on the red ratio, achieving good results. However, the data set required is too large, which is very time-consuming and labor-intensive. In addition, the ripening process of strawberries is very complicated. It is often difficult to accurately determine its maturity level based on the red ratio alone.

[0005] In summary, the problems existing in the prior art are:

[0006] (1) Manually judging the maturity level of oven food is too time-consuming, labor-intensive and subjective. Existing pre-set baking programs often make it difficult to accurately identify the maturity level of oven food.

[0007] (2) Food detection methods based on deep learning such as Swin Transformer and CNN are not easy to accurately locate multiple targets and are not convenient for actual production applications. They also have certain difficulties in detecting targets of different levels under the same category.

[0008] Therefore, in terms of judging the maturity level of oven food, there is an urgent need for an efficient and accurate intelligent detection method to achieve accurate identification of the maturity of oven food during the production process and promote the further development of intelligent oven baking. Summary of the invention

[0009] The object of the present invention is to provide a method for detecting the maturity level of oven food based on an improved Mask R-CNN to solve the above-mentioned technical problems.

[0010] In order to solve the above technical problems, a specific technical solution of a method for detecting the maturity level of oven food based on an improved Mask R-CNN of the present invention is as follows:

[0011] A method for detecting the maturity level of oven food based on an improved Mask R-CNN comprises the following steps:

[0012] S1: Collect RGB images of various maturity stages of food baking in the oven, and divide all images into training set, validation set and test set;

[0013] S2: Based on the collected image dataset, labelme is used to annotate the dataset, obtain the category and mask information of each target, and convert it into the dataset format required by the model;

[0014] S3: Improve the existing Mask R-CNN and train the model; the improvement is to optimize the network framework structure and hyperparameters to adapt to the distribution and characteristics of the data set;

[0015] S4: Verify the model training effect based on the test set in S1 and test the detection effect of food maturity.

[0016] Furthermore, the RGB images in S1 are collected in a controllable scene using an industrial camera from directly above the food. A total of 300 images are collected, all with a size of 3072*2048, and each image contains multiple food samples.

[0017] Furthermore, the maturity stage in S1 refers to five different maturity levels in the baking process, namely, cake_raw, cake_lightly, cake_partially, cake_well, and cake_fully; the division of the training set, validation set, and test set is random, and the validation set is randomly selected from the training set during the training process; the number of images in the training set, validation set, and test set are 270, 100, and 30, respectively, and the number of food at each maturity level is uniform.

[0018] Furthermore, the dataset format required by the model in S2 is the COCO dataset format, including original images, mask images, category labels, and mask position coordinates.

[0019] Furthermore, the improved Mask R-CNN model of S3 has three levels: feature extraction, candidate region generation, and result prediction; the feature extraction network uses the ACFP-Net structure as the feature extractor of the image to fuse features of different scales; the ResNet101 in the feature extraction consists of 5 convolutional layers; the input of the first convolutional layer is the original image after size transformation, the convolution kernel size is 7*7, the step size is 2, and the data dimension is transformed from the original 1536*1536*3 to 384*384*64; after the second layer, the data dimension is 384*384*256; after the third layer, the data dimension becomes 192*192*512; after the fourth layer, it becomes 96*96*1024; after the fifth layer, it becomes 48*48*2048; the convolution kernel size is 3*3 in the last four convolutional layers;

[0020] The activation function used in the convolution process is Relu; its expression is as follows:

[0021] R(x)=max(0,x) (1)

[0022] In the above formula, x is the convolution output of the last layer of the discriminant model;

[0023] The ACFP-Net further processes the feature map generated by ResNet101, specifically, performing 1*1 convolution on the feature map outputted by the current layer and the layer above it, with the number of channels unified as 256. After that, the feature map of the current layer is upsampled to keep the same size as the feature map of the previous layer, and the two feature maps are fused pixel by pixel and subjected to 3*3 convolution to obtain new fused feature maps of each layer. The final feature image F ij Obtained through the following weighted calculation:

[0024]

[0025] In the formula, Fij is the final feature image, represents the fused feature image, represents the weight of each fused feature image, and the sum of each weight is 1, which is determined by formula (3), for example

[0026]

[0027] in is the control parameter of each weight, which is obtained by taking the average value of the convolution kernel in each convolution layer.

[0028] The ACFP-Net achieves cross-layer fusion of feature maps by upsampling the convolutional feature maps, and can generate richer feature representations to maximize the integration of shallow detail features such as the edge and texture of the target, and deep semantic features such as the overall structure of the target, the upper and lower relationships, and other information; and introduces weights by calculating the mean of the convolution kernel parameters. In order to characterize the relative importance of each convolutional layer and perform weighted calculation on the fused feature map according to these weights, the weight distribution of the convolutional layer can be effectively balanced, and the effective features can be utilized to the greatest extent without increasing the amount of network calculation, thereby improving the detection ability of different categories of cakes.

[0029] Furthermore, the S3 introduces weights in Mask R-CNN as well as The features at different levels are processed and fused. The candidate region generation adopts RPN to generate the candidate region of the cake on the feature map, and judge the probability of the candidate region being the foreground and background; the size of the candidate region generated depends on the size of the anchor frame setting;

[0030] The loss function of RPN includes regression loss and classification loss, and the calculation formula is as follows:

[0031]

[0032] Where L is the total loss, L c and L r are classification loss function and regression loss function respectively, N c and N r is the normalization parameter, λ is the coefficient for balancing classification loss and regression loss, i is the i-th anchor box, and p i is the true value of the target in the i-th anchor box, is the probability that there is a target in the predicted i-th anchor box, t i is the offset of the true value relative to the i-th anchor box, is the offset of the predicted bounding box relative to the i-th anchor box.

[0033] Furthermore, the result prediction in S3 has two branches, one is the fully connected layer that classifies the foreground and background and outputs the bounding box of the cake; the other outputs the mask image of the cake; the hyperparameters of the optimized Mask R-CNN model are: the input image is adjusted to 1536*1536, the batch size is 270, the learning rate is 0.001, and the number of iterations is 20; the anchor box sizes are 140, 145, 150, 155, and 160, the anchor box scaling factors are 0.8, 1, and 1.2, and the anchor box step size is 1; the batch size is the number of samples selected for one training; the anchor box is used to generate candidate areas of different sizes pixel by pixel in the RPN process; in order to save the training time of the model, the pre-trained weight file is used as initialization.

[0034] Furthermore, the weight of each maturity level of S3 during the training process is 1.

[0035] Furthermore, S3 enhances the distinction between cakes of different categories by fusing the extracted features.

[0036] Furthermore, the images used in the verification process of S4 are all images that have not participated in the training, and the AP values ​​of all categories are counted as the model performance index; the AP value represents the average of the precision values ​​corresponding to each recall value, and the calculation formula of the recall value and the precision value is as follows:

[0037]

[0038] In the above formula, TP, FP, and FN are the correctly predicted positive samples, the incorrectly predicted positive samples, and the incorrectly predicted negative samples, respectively;

[0039] The AP value of all categories represents the average accuracy of each category detection, which is within the interval [0,1] and is calculated as follows:

[0040]

[0041] In the formula, r i represents the i-th recall value, P i Indicates that i The precision value at ; mAP represents the average AP of all categories, and k represents the number of categories;

[0042] For the 270 images in the training set, the features of different cake samples are first extracted based on the Resnet101 convolutional neural network, and then fused with the different scale features generated by the ACFP-Net network structure. They are input into the RPN network to generate candidate areas for the cake samples. Finally, classification and regression are performed in the candidate areas with smaller error loss, and the corresponding mask images are predicted. The training and testing of the model are both implemented based on the tensorflow module of python.

[0043] The oven food maturity level detection method based on improved Mask R-CNN of the present invention has the following advantages:

[0044] (1) The present invention aims to solve the problems that manual identification of the maturity of oven food is too time-consuming, labor-intensive and subjective, and that the existing pre-set baking programs are difficult to accurately identify the maturity level of oven food. The present invention proposes an intelligent detection method for the maturity level of oven food based on machine vision and learning models.

[0045] (2) The improved Mask R-CNN model of the present invention can accurately identify and locate different targets in the same category. In the classification detection task of cakes with different image granularity levels, it can accurately classify and identify different maturity levels without the need for a very large data set.

[0046] (3) The improved Mask R-CNN model provided by the present invention can provide new ideas and technical assistance for other fine-grained image recognition and classification tasks in the field of intelligent detection technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Flowchart of the improved Mask R-CNN method for detecting the maturity level of oven food;

[0048] Figure 2 The collected cake images of each maturity level;

[0049] Figure 3 Schematic diagram of the process of using labelme to annotate a dataset;

[0050] Figure 4 This is a schematic diagram of the basic framework structure of the improved Mask R-CNN model;

[0051] Figure 5 It is the structural diagram of ACFP-Net;

[0052] Figure 6 This is a schematic diagram of the test results of the maturity level of oven cakes;

[0053] FIG7( a ) is a schematic diagram of the AP curve of the cake maturity level detection result before improvement;

[0054] FIG7( b ) is a schematic diagram of the AP curve of the improved cake maturity level detection result;

[0055] FIG8( a ) is a schematic diagram showing the test results of the maturity level of cakes in the oven before improvement;

[0056] Figure 8(b) is a schematic diagram of the detection results of the maturity level of the cake in the improved oven. DETAILED DESCRIPTION

[0057] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a method for detecting the maturity level of oven food based on an improved Mask R-CNN in conjunction with the accompanying drawings.

[0058] The present invention combines the deep learning convolutional network and the feature pyramid network framework, and in view of the problem that the maturity level of oven food is difficult to accurately and efficiently identify, the Mask R-CNN model is adaptively improved, and a method for oven food maturity level identification and positioning based on image feature fusion is provided. By optimizing the network framework structure and training parameters, it can accurately identify foods of different maturity levels under the same category. The model can effectively identify the maturity level of food in the oven and significantly improve the intelligent detection capability of the oven.

[0059] A method for detecting the maturity level of oven food based on improved Mask R-CNN is proposed. First, RGB images of food of different maturity levels are collected and divided into training set, validation set and test set. Then, based on the divided data set, Labelme is used to annotate different data and the annotated data is converted into COCO format. Then, the existing Mask R-CNN model is improved to optimize the network structure and hyperparameters, and the model is trained using the training set data. Finally, the maturity level of oven food is tested based on the models before and after the improvement to evaluate their performance.

[0060] The present invention provides an oven food maturity level detection method based on improved Mask R-CNN, the flow chart is as follows: Figure 1 As shown, the specific steps include:

[0061] S1: Collect RGB images of various maturity stages of food baking in the oven, and divide all images into training set, validation set and test set;

[0062] The RGB image is collected in a controllable scene using an industrial camera from directly above the food. A total of 300 images are collected, and the image size dimensions are all 3072*2048. The food sample in this embodiment is cake, and each image contains several cake samples. The specific example images collected are shown in FIG. Figure 2 ;

[0063] Each production generates about 16 to 18 cake samples; the maturity stage refers to 5 different maturity levels in the baking process, which are specifically divided by several professionals into five maturity levels: cake_raw, cake_lightly, cake_partially, cake_well, and cake_fully; the division of the training set, the validation set, and the test set is all performed randomly, wherein the validation set is randomly selected from the training set during the training process; the number of images in the training set, the validation set, and the test set are 270, 100, and 30, respectively, and the number of cakes at each maturity level is uniform;

[0064] S2: Based on the collected image dataset, labelme is used to annotate the dataset, obtain the category and mask information of each target, and convert it into the dataset format required by the model;

[0065] Labelme is a graphical image annotation software. The mask information mainly contains the position and size information of each object in the image. The annotation process is shown in the following figure. Figure 3 As shown;

[0066] The dataset format required by the model is the COCO dataset format, which mainly includes original images, mask images, category labels, and mask position coordinates;

[0067] S3: Improve the existing Mask R-CNN and train the model; the improvement is mainly to optimize the network framework structure and hyperparameters to adapt to the distribution and characteristics of the data set;

[0068] The improved Mask R-CNN model has three main layers: feature extraction, candidate region generation, and result prediction. Figure 4 The figure shows the basic framework structure diagram of the improved Mask R-CNN model;

[0069] The feature extraction network uses the ACFP-Net (Adaptive Cake Feature Proposal Network, ACFP-Net) structure as the image feature extractor to fuse features of different scales;

[0070] The ResNet101 used in feature extraction consists of 5 convolutional layers;

[0071] The input of the first convolutional layer is the original image after resizing, the convolution kernel size is 7*7, the step size is 2, and the data dimension is transformed from the original 1536*1536*3 to 384*384*64; after the second layer, the data dimension is 384*384*256; after the third layer, the data dimension becomes 192*192*512; after the fourth layer, it becomes 96*96*1024; after the fifth layer, it becomes 48*48*2048; the convolution kernel size is 3*3 in the last four convolutional layers;

[0072] The activation function used in the convolution process is Relu; its expression is as follows:

[0073] R(x)=max(0,x) (1)

[0074] In the above formula, x is the convolution output of the last layer of the discriminant model.

[0075] The ACFP-Net Figure 5 , which further processes the feature map generated by ResNet101, specifically, performing 1*1 convolution on the feature map output by the current layer and the previous layer respectively, with the number of channels unified to 256, and then upsampling the feature map of the current layer to keep the same size as the feature map of the previous layer, and fusion the two feature maps pixel by pixel and perform 3*3 convolution to obtain the new fused feature map of each layer The final feature image F ij Obtained through the following weighted calculation:

[0076]

[0077] In the formula, F ij is the final feature image, represents the fused feature image, represents the weight of each fused feature image, and the sum of each weight is 1, which is determined by formula (3), for example

[0078]

[0079] in is the control parameter of each weight, which is obtained by taking the average value of the convolution kernel in each convolution layer.

[0080] The ACFP-Net achieves cross-layer fusion of feature maps by upsampling the convolutional feature maps, and can generate richer feature representations to maximize the integration of shallow detail features such as the edge and texture of the target, and deep semantic features such as the overall structure of the target, the upper and lower relationships, and other information; and introduces weights by calculating the mean of the convolution kernel parameters. as well as In order to characterize the relative importance of each convolutional layer and perform weighted calculation on the fused feature map according to these weights, the weight distribution of the convolutional layer can be effectively balanced, and the effective features can be utilized to the greatest extent without increasing the amount of network calculation, thereby improving the detection ability of different categories of cakes.

[0081] Since cakes of different maturity levels have different degrees of difference, and the content of feature maps at different levels is different, for categories with smaller differences, the feature images generated by the traditional Mask R-CNN cannot fully contain the detailed features of the cake target. For example, some shallow edge information and deep semantic information cannot be fully utilized. Therefore, the weights are introduced here. as well as Features at different levels are processed and fused to maximize the integration and utilization of shallow detail features and deep semantic features, thereby improving the detection capabilities of cakes of different categories.

[0082] The candidate region generation uses RPN (Region Proposal Network, RPN) to generate the candidate region of the cake on the feature map and determine the probability of the candidate region being the foreground and background; the size of the candidate region generated depends on the size of the anchor box setting;

[0083] The loss function of RPN includes regression loss and classification loss, and the calculation formula is as follows:

[0084]

[0085] Where L is the total loss, L c and L r are classification loss function and regression loss function respectively, N c and N r is the normalization parameter, λ is the coefficient for balancing classification loss and regression loss, i is the i-th anchor box, and p i is the true value of the target in the i-th anchor box, is the probability that there is a target in the predicted i-th anchor box, t i is the offset of the true value relative to the i-th anchor box, is the offset of the predicted bounding box relative to the i-th anchor box.

[0086] The result prediction has two branches, one is the fully connected layer that classifies the foreground and background and outputs the bounding box of the cake; the other outputs the mask image of the cake;

[0087] The hyperparameters of the optimized Mask R-CNN model are as follows: the input image is adjusted to 1536*1536, the batch size is 270, the learning rate is 0.001, and the number of iterations is 20; the anchor box sizes are 140, 145, 150, 155, and 160, the anchor box scaling factors are 0.8, 1, and 1.2, and the anchor box step size is 1; the batch size is the number of samples selected for one training; the anchor box is used to generate candidate areas of different sizes pixel by pixel in the RPN process; in order to save the training time of the model, the pre-trained weight file is used as initialization;

[0088] Preferably, the data set S3 is distributed more evenly, the number of cakes at the five maturity levels is roughly the same, and the weight of each maturity level is 1 during the training process.

[0089] Since the main differences between the five types of cakes with different maturity levels are the texture and color features of the cake surface, although the convolutional neural network can extract its features to a certain extent, it is not possible to distinguish cakes with subtle differences in surface features. Therefore, the extracted features are fused here to enhance the distinction between cakes of different categories and detect cakes of different maturity levels more accurately.

[0090] S4: Verify the model training effect based on the test set in S1 and test the detection effect of food maturity.

[0091] The images used in the verification process are all images that have not participated in the training. The AP (Average Precision, AP) values ​​of all categories are counted as the model performance indicator; the AP value represents the average of the precision values ​​corresponding to each recall value. The calculation formulas for the recall value and the precision value are as follows:

[0092]

[0093] In the above formula, TP, FP, and FN are the correctly predicted positive samples, the incorrectly predicted positive samples, and the incorrectly predicted negative samples, respectively;

[0094] The AP value of all categories represents the average accuracy of each category detection, which is within the interval [0,1] and is calculated as follows:

[0095]

[0096] In the formula, r irepresents the i-th recall value, P i Indicates that i The precision value at ; mAP represents the average AP of all categories, and k represents the number of categories.

[0097] For the 270 images in the training set, the features of different cake samples are first extracted based on the Resnet101 convolutional neural network, and then fused with the different scale features generated by the ACFP-Net network structure. They are input into the RPN network to generate candidate areas of the cake samples, and finally classification and regression are performed in the candidate areas with smaller error loss, and the corresponding mask images are predicted.

[0098] The training and testing of the model are both implemented based on the tensorflow module of python. The final test result diagram is as follows Figure 6 As shown, all cakes in the image can be accurately detected with a high confidence score. The mAP values ​​of all test set images are counted, and it is found that the improved model has better performance, with an mAP value of 0.86, which is about 10% higher than the 0.75 before the improvement. The improved method provided by the present invention can also provide new ideas and technical assistance for other fine-grained image recognition and classification tasks in the field of intelligent detection technology.

[0099] Comparative Example 1

[0100] Randomly select images from the test set for detection, and draw their precision-recall curves, as shown in Figure 7(a) and Figure 7(b). The horizontal axis is the recall value, and the vertical axis is the precision value, both of which are in the range of [0,1]. For the target detection task, the larger the area enclosed by the curve and the coordinate axis, the better the prediction result, which means that under various recall values, the detection accuracy of the model is relatively high. Therefore, combined with Figure 7(a) and Figure 7(b), it can be seen that the improved model Figure 7(b) has a high level of accuracy under various recall values, which is far better than the detection result of Figure 8(a) before the improvement. After multiple random tests, it was found that the performance of the improved model is better than before the improvement.

[0101] Comparative Example 2

[0102] The detection results of the Mask R-CNN model before and after the improvement are shown in Figure 8(a) and Figure 8(b). The images of the test set were randomly detected using the model of Figure 8(a) before the improvement, and it was found that there were cases where the images could not be recognized, and the overall confidence score was much lower than the prediction result of the Mask R-CNN model of Figure 8(b) after the improvement. The mAP value of the overall test set of the model before the improvement was about 0.75, while the mAP of the overall test set of the improved model reached 0.86, an increase of about 10 percentage points. The performance of the model before the improvement is significantly inferior to that of the improved model. Therefore, the Mask R-CNN model with a feature fusion network structure adopted in the present invention is more effective.

[0103] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A method for detecting the maturity level of oven food based on improved Mask R-CNN, characterized in that: The following steps are involved: S1: Collect RGB images of various maturity stages of food baking in the oven, and divide all images into training set, validation set and test set; S2: Based on the collected image dataset, labelme is used to annotate the dataset, obtain the category and mask information of each target, and convert it into the dataset format required by the model; S3: Improve the existing Mask R-CNN and train the model; the improvement is to optimize the network framework structure and hyperparameters to adapt to the distribution and characteristics of the data set; S4: Verify the model training effect based on the test set in S1 and test the detection effect of food maturity.

2. The oven food maturity level detection method based on improved Mask R-CNN according to claim 1, characterized in that: The RGB images in S1 are collected in a controllable scene using an industrial camera from directly above the food. A total of 300 images are collected, all with a size of 3072*2048. Each image contains multiple food samples.

3. The oven food maturity level detection method based on improved Mask R-CNN according to claim 1, characterized in that: The maturity stage in S1 refers to five different maturity levels in the baking process, namely cake_raw, cake_lightly, cake_partially, cake_well, and cake_fully; the division of the training set, validation set, and test set is random, and the validation set is randomly selected from the training set during the training process; the number of images in the training set, validation set, and test set are 270, 100, and 30, respectively, and the number of food at each maturity level is uniform.

4. The oven food maturity level detection method based on improved Mask R-CNN according to claim 1, characterized in that: The dataset format required by the model in S2 is the COCO dataset format, including original images, mask images, category labels, and mask position coordinates.

5. The oven food maturity level detection method based on improved Mask R-CNN according to claim 1, characterized in that: The improved Mask R-CNN model of S3 has three levels: feature extraction, candidate region generation, and result prediction; the feature extraction network uses the ACFP-Net structure as the feature extractor of the image to fuse features of different scales; the ResNet101 in feature extraction consists of 5 convolutional layers; the input of the first convolutional layer is the original image after size transformation, the convolution kernel size is 7*7, the step size is 2, and the data dimension is transformed from the original 1536*1536*3 to 384*384*64; after the second layer, the data dimension is 384*384*256; after the third layer, the data dimension becomes 192*192*512; after the fourth layer, it becomes 96*96*1024; after the fifth layer, it becomes 48*48*2048; the convolution kernel size is 3*3 in the last four convolutional layers; The activation function used in the convolution process is Relu; its expression is as follows: R(x)=max(0,x) (1) In the above formula, x is the convolution output of the last layer of the discriminant model; The ACFP-Net further processes the feature map generated by ResNet101, specifically, performing 1*1 convolution on the feature map outputted by the current layer and the layer above it, with the number of channels unified as 256. After that, the feature map of the current layer is upsampled to keep the same size as the feature map of the previous layer, and the two feature maps are fused pixel by pixel and subjected to 3*3 convolution to obtain new fused feature maps of each layer. The final feature image F ij Obtained through the following weighted calculation: In the formula, F ij is the final feature image, represents the fused feature image, represents the weight of each fused feature image, and the sum of each weight is 1, which is determined by formula (3), for example in is the control parameter of each weight, which is obtained by taking the average value of the convolution kernel in each convolution layer.

6. The oven food maturity level detection method based on improved Mask R-CNN according to claim 5, characterized in that: The S3 introduces weights in Mask R-CNN as well as The features at different levels are processed and fused. The candidate region generation adopts RPN to generate the candidate region of the cake on the feature map, and judge the probability of the candidate region being the foreground and background; the size of the candidate region generated depends on the size of the anchor frame setting; The loss function of RPN includes regression loss and classification loss, and the calculation formula is as follows: Where L is the total loss, L c and L r are classification loss function and regression loss function respectively, N c and N r is the normalization parameter, λ is the coefficient for balancing classification loss and regression loss, i is the i-th anchor box, and p i is the true value of the target in the i-th anchor box, is the probability that there is a target in the predicted i-th anchor box, t i is the offset of the true value relative to the i-th anchor box, is the offset of the predicted bounding box relative to the i-th anchor box.

7. The oven food maturity level detection method based on improved Mask R-CNN according to claim 5, characterized in that: The result prediction in S3 has two branches, one is the fully connected layer that classifies the foreground and background and outputs the bounding box of the cake; the other outputs the mask image of the cake; the hyperparameters of the optimized Mask R-CNN model are: the input image is adjusted to 1536*1536, the batch size is 270, the learning rate is 0.001, and the number of iterations is 20; the anchor box sizes are 140, 145, 150, 155, and 160, the anchor box scaling factors are 0.8, 1, and 1.2, and the anchor box step size is 1; the batch size is the number of samples selected for one training; the anchor box is used to generate candidate areas of different sizes pixel by pixel in the RPN process; in order to save the training time of the model, the pre-trained weight file is used as initialization.

8. The method for detecting the maturity level of oven food based on improved Mask R-CNN according to claim 5, characterized in that: In the training process of S3, the weight of each maturity level is 1.

9. The method for detecting the maturity level of oven food based on improved Mask R-CNN according to claim 5, characterized in that: The S3 enhances the distinction between cakes of different categories by fusing the extracted features.

10. The oven food maturity level detection method based on improved Mask R-CNN according to claim 5, characterized in that: The images used in the verification process of S4 are all images that have not participated in the training, and the AP values ​​of all categories are counted as the model performance index; the AP value represents the average of the precision values ​​corresponding to each recall value, and the calculation formula of the recall value and the precision value is as follows: In the above formula, TP, FP, and FN are the correctly predicted positive samples, the incorrectly predicted positive samples, and the incorrectly predicted negative samples, respectively; The AP value of all categories represents the average accuracy of each category detection, which is within the interval [0,1] and is calculated as follows: In the formula, r i represents the i-th recall value, P i Indicates that i The precision value at mAP represents the average AP of all categories, and k represents the number of categories; For the 270 images in the training set, the features of different cake samples are first extracted based on the Resnet101 convolutional neural network, and then fused with the different scale features generated by the ACFP-Net network structure. They are input into the RPN network to generate candidate areas for the cake samples. Finally, classification and regression are performed in the candidate areas with smaller error loss, and the corresponding mask images are predicted. The training and testing of the model are both implemented based on the tensorflow module of python.

Images