A food detection system, a model training method and a food detection method

Abstract

The application provides a food detection system, a model training method and a food detection method. The system comprises a feature extractor and a detection head constructed based on a neural network. The feature extractor is used for performing feature extraction on an input image according to multiple processing layers to obtain image features. At least part of the processing layers is a dynamically deformable convolution layer. The dynamically deformable convolution layer comprises multiple deformable shared blocks. Each deformable shared block comprises a deformation unit used for determining deformation parameters adapted to the input image and multiple dynamic convolution branches used for adjusting a convolution mode according to the deformation parameters of the deformation unit. The detection head is used for performing detection on the input image according to the image features, determining a position region of food in the input image and a food category corresponding to each position region. The application can improve the detection performance of a food detection model.

Description

Technical Field

[0001] This invention relates to the field of neural network technology, specifically to the field of food image processing, and more specifically, to a food detection system, a model training method, and a food detection method. Background Technology

[0002] Food is the material foundation supporting people's lives, and diet is one of the factors most closely related to nutrition and health in daily life. Good eating habits can help people prevent various chronic diseases such as diabetes. Food testing is a typical testing application and a basic task in the field of food computing. Based on the results of food testing, upstream tasks such as automatic recording and settlement, dietary assessment, and sustainable diet monitoring can be achieved. This is of great significance for individuals to establish good eating habits and sustainable diets that take into account both nutrition and the environment.

[0003] Existing food testing technologies mainly follow two development paths:

[0004] (1) Food detection using general detection models. For example, researchers previously developed a YOLOv2-based food detection model on a dataset that combined UECFood-100 and UECFood-256. This model achieved an average detection performance of nearly 80% on the test set.

[0005] (2) Starting from instance segmentation methods. Previous researchers trained an instance segmentation model on 1620 food images to achieve food detection. Other studies have trained a fully convolutional neural network on food images collected from the internet to implement a network-supervised method for food proposal segmentation.

[0006] However, compared to ordinary target detection, food detection presents certain challenges:

[0007] (1) Foods typically have fine-grained characteristics, which can lead to differences in the appearance of the same type of food in different scenarios. Different foods using similar ingredients and seasonings may have similar appearances. When using a general detection model for food detection, misclassification and omission are likely to occur. This may be because the general detection model is not designed with the fine-grained characteristics of food in mind, thus affecting the accuracy of food detection.

[0008] (2) In real life, the influencing factors in the detection scenario (such as the container holding the food, the viewing angle of the image acquisition device, and the lighting of the detection scenario) are often rich and varied, which can cause phenomena such as appearance deformation and uneven lighting, thus leading to unsatisfactory detection performance. This may still be because the general detection model has not been designed with the model structure in mind for the fine-grained characteristics of food, thus affecting the detection performance.

[0009] (3) If a segmentation model is trained using pixel-level fine annotation, the pixel-level fine annotation can bring better training, but the actual annotation cost is relatively high, so generally only a relatively small amount of data is used to complete the training. Therefore, it may not be able to adapt to various scene influencing factors, and thus cannot achieve ideal performance in actual application.

[0010] It can be seen that when the existing target detection model is used, the model structure is not designed for the characteristics of food fine granularity, which affects the accuracy of food detection. Therefore, it is necessary to improve the prior art. SUMMARY

[0011] Therefore, the purpose of the present application is to overcome the defects of the prior art and provide a food detection system, a model training method and a food detection method.

[0012] The purpose of the present application is achieved by the following technical solutions:

[0013] According to a first aspect of the present application, a food detection system is provided, which comprises a feature extractor based on a neural network and a detection head, wherein the feature extractor is configured to extract features of an input image according to a plurality of processing layers to obtain image features, and at least part of the processing layers are dynamic deformable convolution layers, the dynamic deformable convolution layers comprise a plurality of deformable shared blocks, each deformable shared block comprises a deformation unit for determining deformation parameters adapted to the input image and a plurality of dynamic convolution branches for adjusting the convolution mode according to the deformation parameters of the deformation unit; the detection head is configured to detect the input image according to the image features to determine the position region of the food in the input image and the food category corresponding to each position region.

[0014] In some embodiments of the present application, the deformation unit is a convolution kernel which is convolved according to the feature map of the deformable shared block in which it is located to determine the offset of each weight parameter of the convolution kernel of the dynamic convolution branch in the corresponding dynamic convolution branch at each convolution position in the width and height directions.

[0015] In some embodiments of the present application, each deformable shared block comprises a perception unit configured to determine the weighting parameters corresponding to each dynamic convolution branch of the deformable shared block according to the feature map currently input into the deformable shared block, wherein the output of each dynamic convolution branch is the convolution weight obtained by multiplying the weighting parameters and the weight parameters of the corresponding convolution kernel, and the result of the convolution operation between the convolution weight and the feature map input into the dynamic convolution branch.

[0016] In some embodiments of the present application, each dynamic deformable convolution layer allocates the feature map input to the layer to each dynamic convolution branch in all deformable shared blocks contained in the layer for processing according to a predetermined rule, and the channels allocated to different dynamic convolution branches are not repeated.

[0017] In some embodiments of the present application, the perception unit of each deformable shared block is configured to determine a corresponding weighting parameter for each weight parameter of the convolution kernel of each dynamic convolution branch for convolution on the corresponding feature map on each convolution channel, and determine the convolution weight actually used for convolution operation according to the weighting parameter and the weight parameter.

[0018] In some embodiments of the present application, the dynamic deformable convolution layer further comprises a channel adjustment unit configured to adjust the number of channels of the feature map input to the dynamic deformable convolution layer based on convolution, and the adjusted feature map is allocated to each deformable shared block for processing; and a channel restoration unit configured to restore the channels of the feature map input based on convolution, so that the number of channels output by the dynamic deformable convolution layer is the same as the number of channels input thereto.

[0019] In some embodiments of the present application, the dynamic deformable convolution layer further comprises an attention unit configured to determine an attention value to be allocated to each deformable shared block, and the output of the dynamic deformable convolution layer is weighted based at least on the attention value allocated to each deformable shared block and the output of the corresponding deformable shared block of the dynamic deformable convolution layer.

[0020] According to the second aspect of the present application, a training method of a food detection system is provided, comprising: obtaining a training set, wherein the training set comprises a plurality of images used for training and a label corresponding to each image, the label indicating each position region in the corresponding image, whether the image contains food, and the food category of each position region; and training the food detection system of the first aspect using the training set, wherein the parameters of the feature extractor and the detection head are updated according to the total loss determined based on the output of the food detection system on the image sample and the corresponding label during training, and the total loss is related to the loss corresponding to the probability of the image containing food, the loss corresponding to the position region of the food, and the loss corresponding to the confidence of the food category of each position region.

[0021] According to a third aspect of the present invention, in some embodiments of the present invention, the detection head of the first aspect is replaced with an improved detection head, the improved detection head comprising: a basic detection head for detecting the probability of food being present in a corresponding image, the location region of the food, and the food category confidence of each location region based on image features; a focusing detection head for obtaining regions of interest in each location region based on the probability of food being present in the corresponding image, the location region of the food, and the food category confidence of each location region, and performing food classification and location deviation estimation on the regions of interest to detect the food category confidence and location region deviation of the regions of interest; and an output unit for determining the food detection result based on the location region of the food obtained by the basic detection head, the food category confidence of each location region, and the food category confidence and location region deviation of the regions of interest obtained by the focusing detection head, wherein the food detection result includes the finally obtained location region of the food and the food category confidence of each location region.

[0022] According to a fourth aspect of the present invention, a training method for a food detection system is provided, comprising: acquiring a training set, including multiple sample images and labels corresponding to each sample image, wherein the labels indicate the location regions of food in the sample images and the food categories corresponding to each location region; and training the food detection system described in the third aspect using the training set, wherein during training, the parameters of the feature extractor and the detection head are updated based on the total loss determined by the output of the food detection system to the image samples and the corresponding labels, wherein the total loss is obtained by updating the parameters of the feature extractor and the detection head with the total loss corresponding to the probability of food being present in the sample image, the loss corresponding to the location regions of the food, the loss corresponding to the confidence level of the food category in each location region, the loss corresponding to the confidence level of the food category in the region of interest, and the loss corresponding to the location region deviation.

[0023] According to a fifth aspect of the present invention, a food detection method is provided, comprising: acquiring an image to be detected; performing food detection on the image to be detected using a food detection system trained by the training method of the second or third aspect; and determining the location regions of the food therein and the confidence level of the food category in each location region.

[0024] According to a sixth aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to implement the steps of the method of the second, third, or fifth aspect of the present invention by executing the executable instructions.

[0025] Compared with the prior art, the advantages of the present invention are as follows:

[0026] The application is based on a neural network to construct a feature extractor and a detection head. The feature extractor is used to extract features of an input image according to at least one or more dynamic deformable convolutional layers to obtain image features. The dynamic deformable convolutional layer includes a plurality of deformable shared blocks, each of which includes a deformation unit for determining deformation parameters adapted to the input image and a plurality of dynamic convolution branches. The convolution mode of the convolution kernel in each dynamic convolution branch is adjusted according to the deformation parameters of the deformation unit of the deformable shared block where it is located. With the improvement of this model structure, on the one hand, the feature extractor can reasonably determine the deformation parameters adapted to each input image, thereby adjusting the convolution mode of the corresponding convolution kernel in the dynamic convolution branch. Combined with the characteristics of the morphologically changed convolution kernel and the dynamic convolution, the model can pay more attention to the feature extraction of the region related to food detection. On the other hand, through the structural setting of a dynamic deformable convolutional layer including a plurality of deformable shared blocks, each of which includes a plurality of dynamic convolution branches, the model can extract the features required for food detection from different dimensional levels. Thus, the detection performance of the food detection model is improved. BRIEF DESCRIPTION OF DRAWINGS

[0027] The embodiments of the application will be further described below with reference to the accompanying drawings, in which:

[0028] Figure 1 A structural schematic diagram of a dynamic deformable convolutional layer according to an embodiment of the application;

[0029] Figure 2 A structural schematic diagram of a food detection system according to an embodiment of the application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0031] As mentioned in the background section, when using existing target detection models, the accuracy of food detection is affected because the model structure is not designed for the fine-grained characteristics of food. During the research on fine-grained characteristics in food detection, the inventors found that the poor distinguishing ability of similar foods in the prior art is because the fine-grained characteristics of food are not considered, only the general characteristics of the target are considered, and no special design is made. This may be because the general detection model is mainly optimized and designed for small target detection and dense target detection in order to detect as many targets as possible with different sizes and categories, without considering the fine-grained characteristics. Through research on fine-grained related technologies, the inventors found that this defect can be solved by introducing a fine-grained feature extraction structure. During the research on the problem of variable scenes in food detection, the inventors found that the reason for the poor generalization ability of the prior art to diversified scenes is that the general convolution has limited generalization ability and can only adapt to limited scene factors. This may be because the scenes and situations in previous detection tasks are not rich enough, and the use of conventional convolution weights and sampling points can adapt to limited scene factors. Through research on deformable convolution and dynamic convolution, the inventors found that this defect can be solved by applying multi-branch deformable convolution to the backbone network and adding independently variable dynamic sampling weights to each branch. To this end, the present application constructs a feature extractor and a detection head based on a neural network. The feature extractor is used to extract features from an input image based on one or more dynamic deformable convolution layers to obtain image features. The dynamic deformable convolution layer includes a plurality of deformable shared blocks, each deformable shared block includes a deformation unit for determining deformation parameters adapted to the input image and a plurality of dynamic convolution branches, and the convolution form of the convolution kernel in each dynamic convolution branch is adjusted according to the deformation parameters of the deformation unit of the deformable shared block where it is located. After the model structure is improved, on the one hand, the feature extractor can reasonably determine the deformation parameters adapted to each input image, thereby adjusting the convolution form of the corresponding convolution kernel in the dynamic convolution branch. Combined with the characteristics of the convolution kernel with morphological changes and dynamic convolution, the model can pay more attention to the feature extraction of the region related to food detection. On the other hand, through the structure setting of one dynamic deformable convolution layer including a plurality of deformable shared blocks, each deformable shared block including a plurality of dynamic convolution branches, the model can extract the features required for food detection from different dimensional levels. Thus, the detection performance of the food detection model is improved.

[0032] In order to better illustrate the technical solutions of the present application, the following describes the model structure, training samples, training process, and application scenarios in detail.

[0033] I. Model structure

[0034] According to one embodiment of the present application, a food detection system (or a food detection model, a neural network) comprises a feature extractor configured to extract features of an input image, and a detection head configured to determine, based on the features of the input image, a location area of a food in the input image and a food category corresponding to each location area; wherein the feature extractor comprises a plurality of processing layers, the input image is processed by the plurality of processing layers to obtain the features of the input image, and at least some of the plurality of processing layers are dynamically deformable convolution layers. The dynamically deformable convolution layer is a convolution operation that combines deformable convolution and dynamic convolution, and has dynamic convolution weights and a deformable convolution shape. Alternatively, the feature extractor comprises a plurality of processing layers, the processing layers are divided into convolution layers and pooling layers configured to reduce the size of feature maps, and at least some of the convolution layers are dynamically deformable convolution layers. Preferably, the feature extractor comprises a plurality of processing layers, at least some of the convolution layers are dynamically deformable convolution layers, and at least some of the convolution layers are 3x3 convolution layers with a stride greater than 1 to reduce the size of the feature maps while performing convolution. The technical solution of this embodiment can achieve at least the following beneficial technical effects: adjusting the convolution shape of the corresponding convolution kernel in the dynamic convolution branch, combining the convolution kernel with a variable shape and the characteristics of dynamic convolution, and allowing the model to pay more attention to feature extraction in the region related to food detection, which helps to improve the performance of the model.

[0035] In order to better perform fine-grained feature extraction on the input image to improve the accuracy of food detection, according to one embodiment of the present application, the dynamic deformable convolutional layer includes a plurality of deformable shared blocks, each of which includes a deformation unit for determining deformation parameters adapted to the input image and a plurality of dynamic convolution branches, the convolution pattern of the convolution kernel in each dynamic convolution branch being adjusted according to the deformation parameters of the deformation unit of the deformable shared block in which it is located; each deformable shared block includes a perception unit for determining the weighting parameters corresponding to each dynamic convolution branch of the deformable shared block according to the feature map currently input into the deformable shared block, wherein the output of each dynamic convolution branch is the result of the convolution operation of the convolution weight obtained by multiplying the weighting parameters and the weight parameters of the corresponding convolution kernel on the feature map input into the dynamic convolution branch. Preferably, the deformation unit is a convolution kernel that performs convolution on the feature map input into the deformable shared block in which it is located to determine the offset of each weight parameter of the convolution kernel of the dynamic convolution branch in each convolution position in the width and height directions. Preferably, the perception unit of each deformable shared block is used to determine the corresponding weighting parameters of each weight parameter of the convolution kernel of each dynamic convolution branch on the corresponding feature map, and to determine the convolution weight actually used for convolution according to the weighting parameters and the weight parameters. In the dynamic deformable convolutional layer of the present embodiment, the convolution pattern of the convolution kernel is adapted by the deformation unit according to the input image, which can adjust the convolution pattern according to the input image to extract fine-grained features useful for food detection. At the same time, in the dynamic deformable convolutional layer of the present application, the convolution kernel is not directly used to perform convolution operation with the input feature map using the original weight parameters, but the weighting parameters corresponding to each channel of each dynamic convolution branch of the deformable shared block are determined according to the feature map currently input into the deformable shared block, and the convolution operation is performed on the input feature map by multiplying the convolution weight obtained by multiplying the weighting parameters and the weight parameters. Thus, more attention can be paid to the positions useful for food detection (the values in the corresponding convolution weight may be higher), and thus the deformation convolution and dynamic convolution are used in one convolution layer to improve the performance of food detection.

[0036] To better utilize the information of different deformable shared blocks, in the different deformable shared blocks of the dynamic deformable convolution layer, the features of which can obtain different features, the features of which are related to the analysis of food detection can be set to assign corresponding attention to the feature maps extracted by different deformable shared blocks. According to an embodiment of the present application, the dynamic deformable convolution layer further comprises an attention unit for determining the attention value that needs to be assigned to each deformable shared block, and the output of the dynamic deformable convolution layer is at least based on the feature map weighted by the output of all deformable shared blocks of the dynamic deformable convolution layer and the attention value assigned to each deformable shared block. The attention unit can adopt an existing structure, and the present application will not repeat it.

[0037] It should be understood that the attention unit of the above-mentioned embodiment is still feasible in principle without being set, and according to an embodiment of the present application, the output of the dynamic deformable convolution layer is a feature map stacked by the outputs of all deformable shared blocks contained therein.

[0038] In order to avoid the explosive growth of the calculation amount brought by multiple dynamic convolution branches, according to an embodiment of the present application, each dynamic deformable convolution layer divides the feature map input into the layer by channel as the division unit and assigns it to each dynamic convolution branch in all deformable shared blocks contained therein for processing according to a predetermined rule, and the channels assigned by different dynamic convolution branches are not repeated. In this way, the calculation amount can be reduced to ensure the calculation efficiency of the model.

[0039] It should be understood that if the influence of the calculation amount is not considered, the dynamic deformable convolution layer can also adopt other embodiments, and according to an embodiment of the present application, each dynamic deformable convolution layer inputs the feature map input into the layer to each dynamic convolution branch in all deformable shared blocks contained therein for processing.

[0040] Further, in order to make the dynamic deformable convolutional layer of the present application be able to be efficiently replaced or inserted into an existing model, thereby reducing the implementation difficulty and improving the efficiency, the dynamic deformable convolutional layer can be further improved. According to an embodiment of the present application, the dynamic deformable convolutional layer further comprises a channel adjustment unit for adjusting the number of channels of the feature map input into the dynamic deformable convolutional layer based on convolution, the adjusted feature map being distributed to each deformable shared block for processing in the channel; and a channel restoration unit for performing channel restoration on the feature map input into the channel restoration unit based on convolution, so that the number of channels output by the dynamic deformable convolutional layer is the same as the number of channels input. Preferably, the channel adjustment unit and the channel restoration unit can adopt 1x1 convolution or 3x3 convolution. If the channel adjustment unit and the channel restoration unit are not set, when the dynamic deformable convolutional layer is replaced or inserted into an existing model, due to the difference in the number of channels of the feature maps of different processing layers, more time is needed to design the number of channels allocated to each branch, and the number of channels allocated to different branches can also be uneven, resulting in that the structure of the dynamic deformable convolutional layer is too complex. When the structure of the present embodiment is adopted, the technical solution of the present embodiment can at least achieve the following beneficial technical effects: the dynamic deformable convolutional layer of the present embodiment can be efficiently replaced or inserted into an existing model; for example, for an ordinary convolutional layer in an existing model in the field which does not change the scale of the feature map, the dynamic deformable convolutional layer of the present embodiment can be directly used to replace it, and only the number of input and output channels of the channel adjustment unit and the channel restoration unit needs to be designed according to the number of channels of the input feature map and the data of the dynamic convolution branch, so that they can be evenly distributed to each dynamic convolution branch; the principle of inserting the dynamic deformable convolutional layer of the present embodiment between two processing layers of an existing model is similar, which will not be described here.

[0041] In order to more intuitively show the schematic structure of a dynamic deformable convolutional layer, the following will be described in combination with Figure 1 According to an embodiment of the present application, the dynamic deformable convolutional layer (which can be referred to as DynamiX convolution) comprises a channel adjustment unit, a plurality of deformable shared blocks (D in the figure) and an attention unit B and a channel restoration unit, each deformable shared block comprising a deformation unit and a plurality of dynamic convolution branches (K in the figure). Figure 1 Figure 1 ​The input feature map of the dynamic deformable convolution layer is first adjusted in channel number by a channel adjustment unit and then distributed to different dynamic convolution branches of different deformable shared blocks for processing, wherein the dynamic convolution branches under the same deformable shared block can share a deformation parameter determined by a deformation unit, the feature maps processed by all dynamic convolution branches of the dynamic deformable convolution layer are stacked and weighted by the attention value output by the attention unit B, and then processed by a channel restoration unit to obtain the output of the dynamic deformable convolution layer. The structural design of convolution in the dynamic deformable convolution layer adopts the structural principle of "adjusting-splitting-processing-merging-restoring", and includes the following processing procedures:

[0042] (1) The input feature map (assuming that the original channel number is C in ) is adjusted by a channel adjustment unit (for example, using 1x1 convolution) to obtain a feature map with DxGxJ channels;

[0043] (2) The feature map with DxGxJ channels is distributed to D deformable shared blocks, and each deformable shared block can distribute a feature map with GxJ channels;

[0044] (3) Each deformable shared block uses its deformation unit to determine a deformation parameter (i.e., an offset) based on the feature map with GxJ channels distributed to it;

[0045] (4) Each dynamic convolution branch of each deformable shared block obtains the deformation parameter determined by the deformation unit of the deformable shared block and a feature map with J channels, the dynamic convolution branch determines a weighting parameter based on the feature map with J channels, and performs dynamic deformable convolution according to the deformation parameter and the weighting parameter to obtain a feature map with J channels processed by the dynamic convolution branch; for example, assuming that the original weight parameters are W1, W2, …, W k , the weighting parameters determined for them are α1, α2, …, α k , and the convolution weights are α1*W1, α2*22, …, α k *W kand the actual sampling positions of the convolution weights determined based on the deformation parameters. It should be understood that if some convolution weights need to be multiplied and summed to obtain a value in the output feature map at the same point of the feature map, the corresponding weighted convolution weights can be summed to obtain a combined weight W (for example, W = a1w1+a2w2+…+akwk), and the result of the dynamic deformable convolution is calculated using the combined weight. Because convolution is a linear operation of digital multiplication, it satisfies the associative law, and the calculation of the convolution results of multiple convolution kernels on the same graph and then summing them should be numerically equal to the result of convolving the feature map (the original image or the feature map corresponding to the original image) after summing multiple convolution kernels, so that the calculation amount is smaller (basically equivalent to the calculation amount of general convolution).

[0046] (5) The attention unit B determines the attention value of the feature map output by each deformable shared block according to the D×G×J channel number feature map output by the channel adjustment unit, and the attention value allocated by each deformable shared block and the output of the corresponding deformable shared block of the dynamic deformable convolution layer are weighted to obtain the output of the dynamic deformable convolution layer, and the outputs of all deformable shared blocks are stacked after being weighted by the attention value, to obtain a feature map with Z channel numbers (Z can be D×G×J or other numerical values) after attention adjustment;

[0047] (6) The channel reduction unit is used to process the feature map with Z channel numbers after attention adjustment to obtain the output feature map (the channel number of the output feature map is equal to C in ).

[0048] For the detection head of the food detection system, the structure of the detection head can adopt the structure of the existing target detection model (such as the Deformable DETR model detection head, the YOLO model detection head) that performs target detection. If it is the Deformable DETR model detection head, the YOLO model detection head, the detection head determines the position area of the food in the input image and the food category corresponding to each position area from the image features composed of multi-scale feature maps; or its structure can also adopt an improved detection head, for example, preferably, the improved detection head includes: a basic detection head for detecting the probability of containing food in the corresponding image, the position area of the food, and the food category confidence of each position area according to the image features; a focused detection head for obtaining the region of interest in each position area according to the probability of containing food in the corresponding image, the position area of the food, and the food category confidence of each position area, and performing food classification and position deviation estimation on the region of interest to detect the food category confidence and the position area deviation of the region of interest; an output unit for determining the food detection result according to the position area of the food obtained by the basic detection head, the food category confidence of each position area, and the food category confidence and the position area deviation of the region of interest obtained by the focused detection head, wherein the food detection result includes the final position area of the food and the food category confidence of each position area. The final position area of the food is obtained by correcting the corresponding position area of the food obtained by the basic detection head using the position area deviation obtained by the focused detection head. For example, the final position area of the food is equal to the sum of the corresponding position area of the food obtained by the basic detection head and the position area deviation obtained by the focused detection head. Alternatively, the final position area of the food is equal to the sum of the corresponding position area of the food obtained by the basic detection head and the position area deviation obtained by the focused detection head multiplied by a predetermined weighting coefficient. In other words, the present embodiment proposes an improved detection head, which can determine the position area deviation of the detected food by additional category classification and bounding box regression with relatively low overhead, and improve the accuracy of detection. Specifically, the focused detection head (or food focusing head, FFH for short) is introduced after the network architecture of the existing detection network calculates the preliminary detection result and the region, and predicts the probability of containing food (Foodness, food probability value) in the image.The value of the focused detection head after the normalized activation function (for example, Sigmoid activation function or tanh) processing of the probability of food is multiplied by the food category confidence of each position area to obtain the confidence of each prediction (a score between 0 and 1), and the region of interest containing the potential food is obtained by setting an empirical threshold. The region of interest corresponding to the region of interest is extracted from the image features by region of interest alignment (region of interest alignment is known to those skilled in the art, such as a general ROI extractor can be used, and details are not described herein). The focused detection head classifies the food in the region of interest and estimates the position deviation (regression) according to the region of interest to obtain the food category confidence and the position deviation of the region of interest, thereby fine-tuning the original prediction. The food classification in the focused detection head can be realized by using convolutional layers and fully connected layers, and similarly, the position deviation estimation can also be realized by using convolutional layers and fully connected layers.

[0049] It should be understood that those skilled in the art can also improve some structures based on existing models to obtain the structure of the food detection system of the present application. According to one embodiment of the present application, the food detection system can be improved based on the Deformable DETR model. Referring to Figure 2 For example, in the food detection system improved based on the Deformable DETR model, the 3x3 convolutional layer in the backbone network (Resnet network) of the Deformable DETR model is replaced with the dynamic deformable convolutional layer of the present application, and the rest is not improved (for example: the detection head uses the structure of the original detection head of the Deformable DETR model). Alternatively, in the food detection system improved based on the Deformable DETR model, the 3x3 convolutional layer in the backbone network (Resnet network) of the Deformable DETR model is replaced with the dynamic deformable convolutional layer of the present application, and a focused detection head and an output unit are added in addition to the original detection head (corresponding to the basic detection head) in the Deformable DETR model. The output unit obtains the food detection result according to the output of the original detection head and the focused detection head. The basic detection head, the focused detection head, and the output unit can refer to the implementation details of the previous embodiments, and details are not described herein. In addition, as known to those skilled in the art, before obtaining the food detection result, due to many position areas to be detected, it is necessary to filter low-confidence predictions, and generally, non-maximum suppression and other filtering algorithms are used. If the existing detection head is used, the filtering algorithm is included in the detection head; if the improved detection head is used, the basic detection head can not be filtered, and the filtering algorithm is used in the output unit to obtain the food detection result.

[0050] It should be understood that the implementers of the art can also customize the structure of the food detection system as needed, according to one embodiment of the present application, the feature extractor of the food detection system includes stacked dynamic deformable convolutional layers and pooling layers (i.e.: the feature extractor is in the form of a structure of dynamic deformable convolutional layers, pooling layers, … dynamic deformable convolutional layers, pooling layers, or the feature extractor is in the form of a structure of dynamic deformable convolutional layers, pooling layers, … dynamic deformable convolutional layers), and the image features composed of multi-scale (size) feature maps are obtained from different dynamic deformable convolutional layers or pooling layers; the detection head uses an existing detection head or an improved detection head.

[0051] II. Training samples

[0052] According to one embodiment of the present application, the training set is obtained from a food detection dataset, such as: a subset of the food detection dataset is extracted to form the training set. The food detection dataset can be self-made by the implementer. The food detection dataset includes a plurality of sample images and a label corresponding to each sample image, the label indicating a position region (often indicated in the form of a bounding box) in the corresponding image, whether the image contains food, and the food category of each position region. For example, the applicant has made a FoodDet100K dataset, which is a dataset of about 100,000 pictures, spanning 10 restaurant scenes, containing multiple food categories, including: seafood (fish, shrimp, etc.), soup, fruit, staple food (bread, steamed dumplings, noodles, pizza, rice, etc.), meat (fried meat, grilled meat, stewed meat, etc.), vegetarian (tofu, mushrooms, green leaves, etc.), mixed dishes (common dishes containing eggs, vegetables, and meat), beverages (juice, beverages, etc.). Accordingly, a group meal detection dataset containing 491 food categories (491 food categories are set in the label) is made, including stewed tofu, pickled beans, pickled radish, braised noodles, orange juice dragon fish, Assam milk tea, Orleans chicken wings, eight treasure dish, eight treasure porridge, cabbage, cabbage tofu, cabbage vermicelli, white porridge, cabbage vermicelli, cabbage rice, flat bean, ice red tea, pineapple pizza, vegetable bag, chicken block with colored peppers, colored pepper fried bamboo shoots, colored pepper bacon, vegetable soup, tea tree mushroom, tea egg, fried bamboo shoots, fried cabbage, fried cauliflower, fried tea tree mushroom, fried winter melon, fried beans, fried bean sprouts, fried bean sprouts, fried rice, fried mushroom slices, fried cauliflower, fried clam, fried peanuts, fried ham, fried ham slices, fried chicken gizzard, fried chicken gizzard, fried bitter gourd, fried lotus root, fried soybean, fried bean sprouts, fried noodles, fried mushrooms, fried rice cake, fried lotus root slices, fried spare ribs, fried bean curd, fried eggplant, fried celery, fried green vegetables, fried green peppers, fried okra, fried seasonal vegetables, fried green beans, fried dried bamboo shoots, fried potatoes, fried potato strips, fried zucchini, fried broccoli, fried shiitake mushrooms, fried cherry tomatoes, fried duck blood, fried kidneys, fried squid, fried pork liver, wing tips, onion roast pork, onion roast beef, onion small yellow fish, onion belt fish, onion roll, onion noodles, coarse grain steamed cake, vinegar vermicelli, crispy radish, large pork chops, large chicken, belt fish, egg cake, egg custard, egg flower soup, egg dumplings, egg tart, egg soup, winter melon soup, Dongpo pork, dried tofu with celery, tofu brain, tofu bubble fried meat, dried bean curd with colored peppers, dried bean curd with malan, dried bean curd with eggs, soy milk, beans, beans with eggs, beans with eggplant, beans with meat, bean paste, bean sprouts, tomato egg flower soup, tomato soup, fillet steak, fillet steak noodles, dried beef, dried tofu, dried and fried flat beans, dry pot cauliflower, dry pot potato slices, dried beef, Gongbao chicken, seaweed, kelp, seafood fried rice, seafood rice, seafood stew, seafood noodles, seafood cuttlefish ball soup, seafood porridge, egg in the pocket, black pepper chicken, black rice porridge, red bean pie, red bean pizza, red bean porridge, red paste with crab, red-burned wing root, red-burned belt fish, red-burned winter melon, red-burned tofu fish head, red-burned dried bean curd, red-burned chicken tips, red-burned chicken legs, red-burned golden needle mushrooms, red-burned spare ribs, red-burned eggplant,Braised Pork, Braised Lion Head, Braised Bamboo Shoots, Braised Potatoes, Braised Dried Shiitake Mushrooms, Braised Small Yellow Croakers, Braised Ribs, Red Sugar Sponge Cake, Red Sugar Steamed Buns, Red Date Pumpkin, Red Date Yam Porridge, Spicy Soup, Fried Bean Curd with Carrots, Fried Cauliflower with Carrots, Fried Meatballs with Carrots, Cauliflower, Cauliflower Fried Golden Needle Mushrooms, Flower Clams, Flower Cow Milk, Plum and Peanut, Peanut Rice, Bean Sprouts Fried Celery, Yellow Bean Pork Knuckles, Cucumber Changed Eggs, Cucumber Fried Ham and Eggs, Cucumber Pickled Eggs, Cucumber Shreds, Fried Rice, Twice-cooked Pork, Dumplings, Fried Ham and Eggs, Ham Pizza, Steamed Ham and Egg, Stir-fried Potato Shreds, Eggs, Egg Pancakes, Fried Dumplings, Chicken Joints, Chicken Rice, Fried Chicken, Chicken Soup, Chicken Broth Noodles, Home-style Fried Meat, Home-style Tofu, Home-style Rice Bowl, Fried Dumplings, Spicy Pepper Beef, Spicy Pepper Chicken, Healthy Fried Dough Sticks, Soy Sauce Fried Eggs, Soy Sauce Cucumber, Soy Sauce Beef, Soy Sauce Duck, Soy Sauce Manchurian Bamboo Shoots, Spicy Salted Shrimp, Spicy Chicken Rice, Classic Pizza, Leek and Egg, Leek and Pork Dumplings, Orange, Curry Fish Balls, Roasted Wheat Bran, Roasted Chicken, Roasted Chicken Wings, Roasted Shrimp, Roasted Sausage, Roasted Duck, Roasted Duck Meat, Coca-Cola, Saliva Chicken, Sausage Fried Garlic, Sausage Slices, Spicy Pepper Fried Meat, Spicy Pepper Chicken Cubes, Cold Bean Skin, Cold Tofu, Cold Bean Pods, Cold Bean Skin, Cold Bean Vermicelli, Cold Seaweed, Cold Seaweed Shreds, Cold Black Fungus, Cold Cucumber, Cold Green Bean Sprouts, Cold Tendon, Cold Fungus, Cold Cucumber, Cold Mushroom, Cold Noodles, Durian Pizza, Lava Custard Buns, Chicken Feet in Broth, Braised Beef, Braised Duck Head, Pickled Radish, Russian Soup, Green Tea, Green Bean Soup, Green Bean Sprouts, Spicy Duck, Mapo Tofu, Spicy Balls, Spicy Tofu, Ants Climbing Tree, Pulse, Steamed Buns, Fried Soybeans with Mushrooms, Fried Soybeans with Shrimp, Spicy Hot Pot, Pork with Preserved Vegetables, Preserved Vegetables Roast Meat, American French Fries, Rice, Rice Wine Soup, Secret Chicken, Bread, Tendon, Noodles, Mushrooms Fried Okra, Fungus Fried Cauliflower, Fungus Fried Meat, Custard Buns, Creamy Mushroom Soup, Creamy Bacon Noodles, Pumpkin Pie, Pumpkin Soup, Pumpkin Puree, Pumpkin Porridge, Mud Snails, Milk, Beef Steak Noodles, Beef Steak, Beef Rice Bowl, Beef Rolls, Farmer's Spring, Lotus Root Slices, Pickles, Pickled Peppers, Pickled Pepper Chicken Feet, Century Eggs, Century Egg Tofu, Century Egg Lean Meat Porridge, Century Egg Porridge, Beer Duck, Pizza, Apple, Thousand-page Tofu, Celery Fried Bean Curd, Celery Fried Bean Sprouts, Celery Fried Yellow Bean Sprouts, Celery Fried Meat, Celery Bean Curd, Green Vegetable Tofu Soup, Green Vegetable Mushrooms, Green Vegetable Porridge, Stir-fried Lettuce and Okra, Stir-fried Winter Melon, Stir-fried Tofu, Mushrooms Fried Okra, Stir-fried Bitter Gourd, Stir-fried Eggplant, Stir-fried Seasonal Vegetables, Stir-fried Okra, Stir-fried Green Beans, Stir-fried Bamboo Shoots, Stir-fried Potato Shreds, Stir-fried Broccoli, Green Pepper Fried Chicken Breast, Green Pepper Fried Mushrooms, Green Pepper Fried Potatoes, Green Pepper Braised Tofu, Green Pepper Chicken, Green Pepper Beef, Green Pepper Beef Slices, Green Pepper Potatoes, Green Pepper Five Spice Pork, Clear Soup with Chinese Cabbage, Steamed White Crab, Steamed Small Yellow Croakers, Brick Fish, Autumn Mackerel, Japanese Tofu, Meat, Meat Buns, Meat Sauce Baked Rice, Meat Sauce Noodles, Meat Minced Vermicelli, Meat Minced Eggplant, Meat Minced Steamed Eggs, Fried Meat Shreds with Carrots, Meatballs, Steamed Eggs, Sarsar Pizza (a type of pizza named by a merchant in a group meal setting), Salad, Yam and Spinach, Colonel Chicken, Steamed Dumplings, Roast Duck, Raw Fried, Seasonal Vegetables, Seasonal Vegetables Fried Chicken Intestines, Lion Head, Lean Meat Broccoli, Sushi, Hand-torn Cabbage, Sushi Platter,potato salad, potato strips, fruit pizza, fruit salad, steamed buns, boiled eggs, boiled cauliflower, boiled pork slices, green beans with chicken, four-seasons chicken, roasted wheat bran, mock chicken, mock three fresh, sour cabbage, sour fish, spicy sour potato strips, sour radish, sour plum, garlic bread, garlic chive sausage, garlic meat, bamboo shoots, bamboo shoots with preserved vegetable, dried bamboo shoots with meat, Taiwanese sausage, Taiwanese rice roll, soup, sweet and sour chicken, sweet and sour chicken breast, sweet and sour fillet, sweet and sour spare ribs, pickled tomatoes, potato and beans, potato and green peppers, potato and sausage, potato roast spare ribs, mashed potatoes, potato beef, shredded potatoes, mother-in-law's vegetable, meatballs, meatball soup, fish bean curd, strong boy milk, lettuce and bamboo shoots, black chicken soup, tomatoes, tomatoes and eggs, tomatoes and scrambled eggs, zucchini, zucchini and meat, zucchini and green beans, dried shark skin and winter melon, Hawaiian pizza, Hawaiian noodles, salted vegetable squid, salted vegetable and green beans, salted vegetable and meat, salted vegetable and green beans, salted eggs, coriander and dried tofu, sausage, sausage roll, dried tofu and green peppers, mushroom roast, mushroom pizza, bananas, banana milk cake, spicy chicken wings, crispy chicken, crispy cod sticks, small knife cut, small cakes, small yellow croaker, steamed buns, millet porridge, crab congee, crab paste, New Orleans roast chicken, Sprite, preserved vegetable squid, smoked fish, toothpick meat, duck leg, salted green beans, onion and eggs, onion and meat, onion and potatoes, onion and potato slices, onion rings, coconut milkshake, tremella porridge, beverages, beverage 1, beverage 2, beverage 3, beverage 4, oil-fried eggplant, oil-fried shrimp, oil-fried tofu and green vegetables, oil-fried tofu and roast meat, oil-fried chicken, oil-fried dragon fish, oil-fried shrimp, fried dough sticks, oil-fried finfish, oil-fried ice cream, fried ribbonfish, corn, corn bread, corn dumplings, corn and shrimp meat, fish balls, fish-flavored shredded pork, mooncakes, wontons, breakfast rolls, preserved vegetable, preserved vegetable soup, fried chicken, fried chicken wings, fried chicken drumsticks, fried chicken, fried chicken legs, fried lotus root slices, fried potatoes, fried food combination 1, fried food combination 2, fried food combination 3, fried shrimp, steamed white crab, steamed coarse grains, steamed eggs, steamed dumplings, supreme pizza, porridge, boiled cabbage, boiled river shrimp, boiled pumpkin, purple cabbage, seaweed and egg flower soup, seaweed soup, zongzi. It should be understood that the above food categories are only illustrative, and according to specific application scenarios, adjustments can be made according to the needs of the implementer, such as combining food categories, for example, deleting some food categories, such as: fried food combination 1, fried food combination 2, fried food combination 3, potato salad, flower and flower milk, etc.; for example, adding some food categories, such as: crawfish,Halogen pig trotters. In other words, the food category can be all food categories preset by the corresponding application scenario (canteen scenario), and the corresponding image can also be collected and a data set is made from the specific application scenario. In order to guarantee the quality of the constructed data set, the following requirements can be followed when constructing the data set: (1) the image imaging should be as clear as possible, and the food taken should have basic recognition; (2) the image should be taken as much as possible with different light and shooting angles; the camera uses a standard lens, and the position during shooting is controlled at about 0.4-0.6m above the dish, and the viewing angle is allowed to vary within a range of 10 degrees around the vertical angle; (3) the dish is as much as possible located in the center part of the picture, and the proportion is at least more than 50%. In addition, the data set of food detection can use the existing data set. For example, UECFood-100, UECFood-256, UNIMIB2015, UNIMIB2016, EgocentricFood, School Lunch, Oktoberfest Food or Mixed Dish data set, these data sets have multiple sample images and corresponding labels in each sample image, the label indicates the position area of the food in the sample image and the food category corresponding to each position area, and a brief introduction is shown in Table 1:

[0053] Table 1

[0054] Dataset name Cuisine Number of classes Number of images Annotation form Release time UECFood-100 Japanese 100 14361 Bounding box 2012 UECFood-256 Japanese 256 25088 Bounding box 2014 UNIMIB2015 Western 15 2000 Polygon 2015 UNIMIB2016 Western 73 1027 Polygon 2016 Egocentric Food Western 9 5038 Bounding box 2016 School Lunch Japanese 21 4877 Bounding box 2017 Oktoberfest Food German 15 1110 Bounding box 2019 Mixed Dish Southeast Asian 164 9254 Bounding box 2019

[0055] III. Training process

[0056] For the training of the food detection system using the existing detection head, according to an embodiment of the present application, the food detection system of the foregoing embodiment is trained using the training set, wherein the parameters of the feature extractor and the detection head are updated according to the total loss determined by the output of the food detection system to the image sample and the corresponding label during training, wherein the total loss is related to the loss corresponding to the probability that the image contains food, the loss corresponding to the position area of the food, and the loss corresponding to the food category confidence of each position area. Correspondingly, the total loss function used to calculate the total loss can use the total loss function corresponding to the existing detection head, and this embodiment will not be described here.

[0057] For the training of a food detection system with a focused detection head, according to an embodiment of the present application, a training method of a food detection system, comprising: obtaining a training set, wherein a plurality of sample images and a label corresponding to each sample image are included, the label indicating the position area of the food in the sample image and the food category corresponding to each position area; using the training set to train the food detection system with the focused detection head in the foregoing embodiments, wherein the parameters of the feature extractor and the detection head are updated according to the total loss determined by the output of the food detection system to the image sample and the corresponding label during training, and the total loss is obtained by the loss corresponding to the probability that the sample image contains food, the loss corresponding to the position area of the food, the loss corresponding to the food category confidence of each position area, the loss corresponding to the food category confidence of the region of interest, and the loss corresponding to the position area deviation. The total loss updates the parameters of the feature extractor and the detection head. When performing backpropagation calculation, end-to-end training is performed, and all trainable parameters in the model participate in updating. Correspondingly, the total loss function for calculating the total loss needs to be adjusted, and the total loss function is defined as follows:

[0058] L = L f + L l + L s + L cls + L reg

[0059] Wherein, L f represents the loss corresponding to the probability that the image contains food, L l represents the loss corresponding to the position area of the food (since the position area is delimited by the bounding box, it is usually also called bounding box regression loss), L s represents the loss corresponding to the food category confidence of each position area (usually also called classification loss), L cls represents the food category confidence of the region of interest, L reg represents the loss corresponding to the position area deviation. L f is determined based on the deviation between the predicted probability of containing food and the indication in the label of whether the image contains food (such as 1 for food and 0 for no food). L l is determined based on the predicted position area of the food and the position area of the food indicated in the label, usually using L1 loss regression. L s is determined based on the food category confidence of the corresponding position area predicted by the basic detection head and the food category corresponding to the position area indicated in the label, usually using classification cross-entropy loss. L cls is determined based on the food category confidence of the corresponding position area (some regions of interest) predicted by the focused detection head and the food category corresponding to the position area indicated in the label, usually using classification cross-entropy loss. L regThe L1 loss can be obtained by determining the deviation between the estimated position region deviation and the deviation between the position region predicted by the base detection head and the actual position region specified in the label; for example, if the base detection head predicts the position of a target as (x0, y0, x1, y1), and the actual position of the target is then the L reg The L1 loss between the position region deviation of the focused detection head prediction and is calculated.

[0060] It should be understood that other modifications can also be made to the total loss function to obtain additional embodiments, such as adding a predetermined weighting coefficient to at least part of the loss terms in L f , L l , L s , L cls , L reg , and obtaining the corresponding total loss by weighted summation.

[0061] IV. Application scenarios

[0062] According to an embodiment of the present application, a food detection method is provided, comprising: obtaining an image to be detected, using the food detection system trained by the training method of the aforementioned embodiments to perform food detection on the image to be detected, and determining the position region of the food and the food category confidence of each position region.

[0063] In order to verify the effect of the present application, the applicant conducted a comparative experiment. In the experiment, three data sets were used, namely the UNIMIB2016, Oktoberfest two public food detection data sets, and the self-made FoodDet100K data set. The results of the comparative experiment are shown in Table 2:

[0064] Table 2 Comparative test results

[0065]

[0066] Note:

[0067] 1、"Model" column lists the basic model based on which the comparative experiment is carried out, which is three existing detection models: FCOS model (Fully Convolutional One-Stage Object Detector (FCOS), Dynamic R-CNN model, Deformable DETR model; the model name is followed by the main network used in the corresponding model. In the "Module" list, the specific module type is represented, wherein: "-" represents that the structure of the basic model is not adjusted, that is, the original basic model as the baseline model; "+DynamicX" represents that the original convolution layer in the main network of the basic model is replaced with the dynamic deformable convolution layer of the application; "+FFH" represents that the improved detection head is used in the basic model; "DyFoc" represents that the original convolution layer in the main network of the basic model is replaced with the dynamic deformable convolution layer of the application and the improved detection head is used in the basic model.

[0068] 2、12 models are experimented on 3 data sets (UNIMIB2016, Oktoberfest, FoodDet100K), each experiment provides 3 detection indicators (mAP, AP50, AP75), and the corresponding experimental results are collected. Among them, the meaning of AP indicator is average precision (Average Precision), and the AP value using the intersection ratio r is generally denoted as AP r , AP 50 , AP 75 , etc. The mAP average precision mean is to average a series of AP values of AP50, AP55, …, AP95 as a indicator, which is also a more comprehensive indicator to measure the performance of the model. From the results shown in Table 2, it can be seen that after using the dynamic deformable convolution layer and / or the improved detection head of the application, the performance of the obtained model is better than that of the baseline model.

[0069] Overall, the purpose of the present application is to overcome the limitations of fine-grained detection recognition performance and the problem of unsatisfactory performance in food detection scenarios. The present application proposes a food detection system using deformable dynamic convolution weights and focusing on food regions, called Dynamic and Focusing Detector (DyFoc), to improve the accuracy of food detection. Among them, multi-branch dynamic deformable convolution (corresponding to dynamic deformable convolution layer) is proposed to replace the conventional convolution weight, which can enhance the effect of the network on fine-grained detection and the performance of dealing with appearance differences in different detection scenarios. Specifically, multi-branch dynamic deformable convolution (DynamiX convolution) is proposed, which works by splitting the network input into several independent deformable convolution groups (corresponding to deformable shared blocks), each with independent sampling offsets; Each convolution group contains several channel branches (corresponding to dynamic convolution branches), each branch has independent weights. The commonly used deformable convolution can increase the adaptability of the convolutional neural network to the appearance deformation of the object, but when facing more food categories and more influencing factors, its weights and sampling points may still be insufficient to better and accurately realize feature extraction and help classification and positioning. While the multi-branch dynamic deformable convolution adds adaptability to multiple targets and influencing factors through multiple groups of sampling points and weights, it can ultimately help the detection network achieve more accurate detection.

[0070] It should be noted that although the above describes each step in a specific order, it does not mean that each step must be performed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a changed order, as long as the desired function can be achieved.

[0071] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions embodied therewith, which instructions are used to program computers to implement the various aspects of the present application.

[0072] A computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves.

[0073] Having described above several embodiments of the present application, any modifications and variations that fall within the scope of the described embodiments can be apparent to those skilled in the art. The foregoing description of various embodiments of the present application is exemplary and explanatory only and is not intended to be limiting. Various modifications and variations of the described embodiments of the application will be apparent to those skilled in the art from the foregoing description and teachings. It is intended that the scope of the application should be limited only by the broadest interpretation of the appended claims to be supported by this description, including the full range of equivalents to which such claims are entitled. The terminology used or introduced herein is for the purpose of describing particular embodiments only and is not intended to be limiting.

Claims

1. A food detection system, characterized in that, It includes a feature extractor and a detection head built on a neural network, wherein, A feature extractor is used to extract features from an input image based on multiple processing layers to obtain image features. At least some of the processing layers are dynamically deformable convolutional layers. The dynamically deformable convolutional layers include multiple deformable shared blocks. Each deformable shared block includes a deformation unit for determining deformation parameters adapted to the input image and multiple dynamic convolutional branches for adjusting the convolution shape according to the deformation parameters of the deformation unit. Each deformable shared block includes a perception unit for determining the weighting parameters corresponding to each dynamic convolutional branch of the deformable shared block based on the feature map currently input to the deformable shared block. The output of each dynamic convolutional branch is the result of convolution operation between the weighting parameters and the weight parameters of the corresponding convolution kernel, which is the result of convolution operation with the feature map input to the dynamic convolutional branch. The detection head is used to detect the input image based on image features, and to determine the location regions of food in the input image and the food categories corresponding to each location region.

2. The food detection system according to claim 1, characterized in that, The deformable unit is a convolution kernel. The convolution kernel performs convolution based on the feature map of the deformable shared block in which it is located, so as to determine the offset of each weight parameter in the width and height directions of the convolution kernel of the corresponding dynamic convolution branch that performs convolution on the feature map at each convolution position.

3. The food detection system according to claim 2, characterized in that, Each dynamically deformable convolutional layer distributes the input feature map into each of its dynamic convolutional branches according to a predetermined rule, using channels as the unit of division. The channels assigned to different dynamic convolutional branches are not duplicated.

4. The food detection system according to claim 3, characterized in that, The perceptual unit of each deformable shared block is used to determine the corresponding weight parameters of the convolution kernel for each dynamic convolution branch to perform convolution on the corresponding feature map. The weight parameters of each convolution kernel are determined on each convolution channel, and the actual convolution weights used for the convolution operation are determined according to the weight parameters and weight parameters.

5. The food detection system according to claim 1, characterized in that, The dynamic deformable convolutional layer also includes a channel adjustment unit for adjusting the number of channels of the feature map input to the dynamic deformable convolutional layer based on convolution. The feature map adjusted by the channel adjustment unit is distributed to each deformable shared block for processing according to the channels. And a channel reduction unit, which is used to perform channel reduction on the feature map in the input based on convolution, so that the number of channels output by the dynamically deformable convolutional layer is the same as the number of channels in its input.

6. The food detection system according to any one of claims 1-5, characterized in that, The dynamically deformable convolutional layer further includes an attention unit, which is used to determine the attention value that needs to be assigned to each deformable shared block. The output of the dynamically deformable convolutional layer is obtained by weighting the attention value assigned to each deformable shared block and the output of the corresponding deformable shared block of the dynamically deformable convolutional layer.

7. The food detection system according to claim 6, characterized in that, The detection head includes: A detection head used to detect the probability of food being present in a corresponding image, the location of the food, and the confidence level of the food category in each location based on image features; A detection head for focusing on the region of interest (ROI) based on the probability of food in the corresponding image, the location of the food, the confidence level of the food category in each location, the food category confidence level in each location, the food category confidence level in each location, the food category confidence level in each location, the food category confidence level in each location, the location bias estimation in each ROI, and the food category confidence level and location bias in each ROI. An output unit for determining food detection results based on the location region of the food obtained from the basic detection head, the food category confidence of each location region, the food category confidence of the region of interest obtained from the basic detection head, and the location region deviation, wherein the food detection results include the final obtained location region of the food and the food category confidence of each location region.

8. A training method for a food detection system, characterized in that, include: Obtain the training set, which includes multiple images for training and a label for each image. The label indicates the location region in the corresponding image, whether the image contains food, and the food category of each location region. The food detection system according to any one of claims 1-6 is trained using this training set, wherein during training, the parameters of the feature extractor and the detection head are updated according to the total loss determined by the output of the food detection system to the image samples and the corresponding labels, wherein the total loss is related to the loss corresponding to the probability that the image contains food, the loss corresponding to the location region of the food, and the loss corresponding to the confidence of the food category in each location region.

9. A training method for a food detection system, characterized in that, include: Obtain the training set, which includes multiple sample images and a label for each sample image. The label indicates the location region of food in the sample image and the food category corresponding to each location region. The food detection system of claim 7 is trained using this training set. During training, the parameters of the feature extractor and the detection head are updated based on the total loss determined by the output of the food detection system to the image samples and the corresponding labels. The total loss is obtained by combining the loss corresponding to the probability of food in the sample image, the loss corresponding to the location region of the food, the loss corresponding to the confidence of the food category in each location region, the loss corresponding to the confidence of the food category in the region of interest, and the loss corresponding to the location region deviation.

10. A food testing method, characterized in that, include: The image to be detected is acquired, and the food detection system trained by the training method of claim 8 or 9 is used to detect food in the image to be detected, and the location regions of the food and the confidence level of the food category in each location region are determined.

11. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 8 to 10.

12. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 8 to 10 by executing the executable instructions.

Images