A fruit detection method and electronic device

Abstract

Embodiments of the present application provide a system for fruit detection model training, a fruit detection method and an electronic device. The system comprises a fruit detection model, which comprises a backbone network and a fruit detection network. The backbone network comprises a front-end network and a back-end network. The front-end network is configured to perform convolution processing and down-sampling on an input image to obtain a first feature map. The back-end network is configured to perform dilated convolution processing and up-sampling processing on the first feature map to obtain a second feature map. The fruit detection network is configured to perform anchor-free object detection on the second feature map with fruits as targets. The system further comprises a density map prediction network configured to perform density map prediction based on the second feature map. The fruit detection model and the density map prediction network are trained using input training samples.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neural networks, in particular to the field of fruit positioning and recognition using neural networks, and more particularly to a fruit detection method and an electronic device. BACKGROUND

[0002] The orchard scene in a natural environment is very complex, and apples on the fruit trees are distributed in clusters, which can easily form a situation of mutual occlusion between fruits and fruit leaves. When an image is collected for the whole fruit tree to count the number of fruits, the fruit targets in the image are densely distributed and can be occluded. In addition, the light conditions in the orchard are complex, and under poor light conditions, the colors of fruits and fruit leaves are very similar, which seriously affects the accuracy of fruit recognition. The existing methods for recognizing densely distributed and occluded targets in the orchard mostly use additional information such as depth information, apple features, and label subdivision to improve the recognition accuracy. The improvements of the target recognition model mostly use re-clustering of the predicted box size, multi-scale detection, multi-scale feature fusion, and increasing the attention mechanism to improve the target detection accuracy. When detecting densely distributed and occluded targets, the labeling method of the anchor box is easily affected by occlusion, which can introduce a large amount of noise. The use of scale feature fusion and increasing the attention mechanism has certain improvement, but there is no targeted improvement for densely distributed and occluded targets.

[0003] In summary, the fruits on the fruit trees are characterized by clustered distribution, and the corresponding fruits can be occluded by other fruits or leaves, which affects the detection accuracy of the model for fruit targets, and the accuracy of fruit detection is relatively low. Therefore, it is necessary to improve the existing technology. SUMMARY

[0004] Therefore, the purpose of the present application is to overcome the defects of the above-mentioned prior art and provide a fruit detection method and an electronic device.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] According to a first aspect of the present application, a system for fruit detection model training is provided, comprising: a fruit detection model comprising a backbone network and a fruit detection network, wherein the backbone network comprises a front-end network and a back-end network, the front-end network is used for convolution processing and down-sampling of an input image to obtain a first feature map, the back-end network is used for dilated convolution processing and up-sampling processing of the first feature map to obtain a second feature map, and the fruit detection network is used for anchor-free target detection with fruits as targets based on the second feature map; a density map prediction network for density map prediction based on the second feature map; wherein the fruit detection model and the density map prediction network are trained using input training samples.

[0007] In some embodiments of the present application, the front-end network comprises a plurality of convolution blocks and pooling layers arranged in a stack, each convolution block comprising one or more convolution layers, and the number of channels of the output of each convolution block in the front-end network is greater than the number of channels of the input; and / or the back-end network comprises a plurality of dilated convolution blocks and up-sampling layers connected in sequence, and each dilated convolution block maintains the size of the output unchanged relative to the size of the input through a padding operation.

[0008] In some embodiments of the present application, a training method of a fruit detection model comprises: obtaining a plurality of training samples, each training sample comprising a sample image, a fruit annotation label, and a density map label; and training the system of the first aspect by using the plurality of training samples through a first training stage and a second training stage, wherein in the first training stage, the backbone network and the density map prediction network are trained multiple times using the training samples while the parameters of the fruit detection network are frozen, and in the second training stage, the fruit detection model and the density map prediction network are jointly trained multiple times.

[0009] In some embodiments of the present application, the density map label is calculated in the following manner: a full zero matrix of a preset size is constructed, the position of the center point of the corresponding fruit in the full zero matrix is set to 1 according to the fruit annotation label corresponding to the sample image, and a non-full zero matrix is obtained; and the non-full zero matrix is convoluted using a Gaussian kernel function to obtain the density map label, wherein the Gaussian kernel function is expressed as:

[0010]

[0011] wherein c x represents the horizontal coordinate of the center point of the fruit, c y represents the vertical coordinate of the center point of the fruit, and σ represents a control parameter for controlling the radial range of the Gaussian kernel function, and e represents the base of the natural logarithm.

[0012] In some embodiments of the present application, the first training stage comprises: inputting the sample image into the system and sequentially processing it through the backbone network and the density map prediction network to obtain a density map prediction result; calculating a density map loss according to the density map prediction result corresponding to the sample image and the density map label, and updating the parameters of the backbone network and the density map prediction network according to the density map loss.

[0013] In some embodiments of the present application, the second training stage comprises: inputting a sample image into the system to obtain target detection results and density map prediction results output by the fruit detection network and the density map prediction network, calculating a target detection loss according to the target detection results corresponding to the sample image and fruit annotation labels, calculating a density map loss according to the density map prediction results corresponding to the sample image and density map labels, calculating a total loss according to the target detection loss and the density map loss, and updating parameters of the backbone network, the fruit detection network and the density map prediction network according to the total loss.

[0014] In some embodiments of the present application, the total loss is calculated in the following manner:

[0015] L det =L heatmap +λ size L size +L off +L map；

[0016] wherein L heatmap +λ size L size +L off represents the target detection loss, which comprises a center point prediction loss L heatmap , a width-height prediction loss L size and a center point offset loss L off , λ size represents a weighting coefficient of L size , L map represents the density map loss, L heatmap adopts a focal loss function, L size and L off adopt a L1 loss function, and L map adopts a L2 loss function.

[0017] According to a third aspect of the present application, there is provided a fruit detection method, which comprises: obtaining a photographed fruit tree image as an input image, processing the input image by using a fruit detection model trained by the method of the second aspect to obtain target detection results.

[0018] According to a fourth aspect of the present application, there is provided an electronic device, which comprises: one or more processors; and a memory, wherein the memory is used to store executable instructions, and the one or more processors are configured to implement the steps of the method of the second aspect or the third aspect by executing the executable instructions. BRIEF DESCRIPTION OF DRAWINGS

[0019] The embodiments of the present application are further described below with reference to the accompanying drawings, in which:

[0020] Figure 1A module schematic diagram of a system for fruit detection model training according to an embodiment of the present application;

[0021] Figure 2 A structural schematic diagram of a system for fruit detection model training according to an embodiment of the present application;

[0022] Figure 3 A schematic diagram of fruit annotation labels and density map labels after label annotation according to an embodiment of the present application;

[0023] Figure 4 A schematic diagram of total loss calculation in a second training stage according to an embodiment of the present application;

[0024] Figure 5 Validation set MAE results and training loss of training a density map prediction network according to an embodiment of the present application;

[0025] Figure 6 A schematic diagram of detection effect comparison between a Yolov4 model and a fruit detection model trained according to an embodiment of the present application;

[0026] Figure 7 A schematic diagram of detection effect comparison between a CenterNet model and a fruit detection model trained according to an embodiment of the present application;

[0027] Figure 8 A schematic diagram of a general flow when implementing the present application according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0029] As mentioned in the background section, the fruits on the fruit trees have the feature of clustered distribution, and the corresponding fruits will be blocked by other fruits or leaves, which affects the detection accuracy of the model on the fruit target, and the accuracy of fruit detection is relatively low. In order to solve the problem of low detection accuracy of dense target detection and blocked target detection in fruit recognition, the present application introduces the density map information (i.e. density map label) of the fruit to assist the training process of the fruit detection model with dilated convolution, so as to improve the detection accuracy of the fruit detection model.

[0030] Before specifically introducing the embodiments of the present application, some terms used therein are explained as follows:

[0031] Dilated Convolutions, also known as dilated or atrous convolution, is a convolution with holes injected into the standard (normal) convolution kernel. Compared with the standard convolution, the dilated convolution has an additional hyperparameter called dilated rate.

[0032] Down-sampling, refers to the process of reducing the size of a graph (including input image, feature map) by pooling (such as max pooling, average pooling) processing.

[0033] Up-sampling, refers to the process of enlarging the size of a graph by interpolation processing.

[0034] Stride, refers to the step length of the convolution kernel in the corresponding direction of the graph. If the stride is 1, it means moving one pixel at a time in the corresponding direction on the graph. If the stride = 2, it means moving two pixels at a time in the corresponding direction on the graph.

[0035] In the scenario of fruit tree results, there are characteristics such as clustering of fruits, mutual occlusion of fruits, or occlusion of fruits by leaves. Although the general fruit annotation label also annotates the position of some occluded fruits, due to the change in external appearance after occlusion, the model does not pay enough attention to the corresponding area, resulting in the existing technology being prone to missed detection. In order to improve the detection accuracy, the present application constructs a system for training a fruit detection model, wherein the backbone network of the fruit detection model uses multiple down-sampling to obtain high-level semantic information and then enhances the receptive field through multiple dilated convolutions; in addition, when training the fruit detection model, an additional density map prediction network is also added to predict the density map based on the output of the backbone network to guide the training of the fruit detection model. Since the density map label can give the density information of the distribution of each fruit target (such as non-occluded target, occluded target, densely distributed target), during joint training, the density map loss can be used to guide the fruit detection model to pay attention to the area where the fruits are occluded and / or the fruits are clustered, thereby using the density information to guide the weight parameter update of the backbone network and the fruit detection network in the fruit detection model to improve the fruit detection accuracy of the fruit detection model and reduce the missed detection. After the guided training is completed, the density map prediction network can be removed, and the fruit detection model can be directly used for prediction.

[0036] According to an embodiment of the present application, referring to Figure 1The application provides a system for training a fruit detection model, comprising a fruit detection model and a density map prediction network. The fruit detection model comprises a backbone network and a fruit detection network, wherein the backbone network comprises a front-end network and a back-end network, the front-end network is used for performing convolution processing and down-sampling on an input image to obtain a first feature map, the back-end network is used for performing dilated convolution processing and up-sampling processing on the first feature map to obtain a second feature map, and the fruit detection network is used for performing anchor-free target detection on fruits as targets according to the second feature map. Preferably, the density map prediction network is used for performing density map prediction based on the second feature map. The fruit detection model and the density map prediction network are trained by using input training samples. Preferably, the training of the fruit detection model and the density map prediction network by using the input training samples comprises: calculating a density map loss based on a density map label calculated according to a fruit annotation label, and guiding the training of the fruit detection model by using the density map loss. The technical scheme of the embodiment can at least achieve the following beneficial technical effects: the application fuses density map information into the training of the fruit detection model in view of the dense distribution of fruit targets in an orchard and the occlusion of fruit trees and fruit leaves, uses the density map information to provide accurate position information of targets for the anchor-free target detection network, thereby guiding the fruit detection model to improve detection precision and reduce missed detection.

[0037] Preferably, the front-end network comprises a plurality of convolution blocks and pooling layers arranged in stacks, each convolution block comprises one or more convolution layers, and the number of channels of the output of each convolution block in the front-end network is greater than the number of channels of the input. Preferably, the back-end network comprises a plurality of dilated convolution blocks and an up-sampling layer connected in sequence, each dilated convolution block maintains the size of the output unchanged relative to the size of the input through a padding operation, and the dilated convolution block comprises one or more dilated convolution layers.

[0038] According to an embodiment of the application, the prediction accuracy of the network for generating the density map will decrease when the down-sampling multiple is too large, and the value range of the down-sampling multiple of all the pooling layers in the front-end network is an integer in (2, 8]. That is, the down-sampling multiple is an integer 4, 6 or 8.

[0039] According to an embodiment of the application, in order to enhance the receptive field of feature extraction, the value range of the number of layers of the dilated convolution layer in the back-end network is [2, 6]. For example, the back-end network is provided with 2, 4 or 6 layers of dilated convolution layers, so that the accuracy of model detection is improved through the operation of enhancing the receptive field multiple times.

[0040] According to an embodiment of the application, the implementation structure of the system for training the fruit detection model is described in detail in the description of the fruit detection model Figure 2 .

[0041] The information in the form of (Channel, Height, Width) represents the size (channel number, height, width) of the input or output feature map, for example, the input image information (3, 1280, 720) represents that the input image has 3 channels, the height is 1280 pixels, and the width is 720 pixels. The corresponding (64, 320, 180) at the upsampling position represents that the channel number of the output feature map after upsampling is 64 channels, the height is 320 pixels, and the width is 180 pixels.

[0042] The structure of the backbone network includes a front-end network and a back-end network:

[0043] In the front-end network, each box represents a convolutional block or a pooling layer, and the information in the form of Conv2d (out_channels, kernel_size, stride, padding) x M represents the structure in the convolutional block, that is, there are M Conv2d (out_channels, kernel_size, stride, padding) convolutional layers in the convolutional block, out_channels represents the output channel number of the convolutional layer, kernel_size represents the convolution kernel size, stride represents the step length, and padding represents the padding number. For example, Conv2d (64, 3, 1, 1) represents that the output of the convolutional layer is 64 channels, the convolution kernel size is 3x3, the step length is 1, and the input is filled with 1 circle 0 or 1 (i.e., a circle of 0 or 1 is supplemented around the input feature map of the convolutional layer).

[0044] In the front-end network, the information in the form of MaxPool2d (2, 2, 0) represents the structure of the pooling layer, MaxPool2d represents two-dimensional maximum pooling, the first 2 in the parentheses represents that the pooling kernel size is 2x2, the second 2 represents the step length is 2, and 0 represents the padding number, i.e., no padding.

[0045] In the front-end network, the (Channel, Height, Width) in the lower row of information represents the size of the feature map output by the layer, i.e., (channel number, height, width).

[0046] In the backend network, due to the use of dilated convolution, the information in the form of Conv2d (out_channels, kernel_size, stride, padding, dilated rate) has an additional dilated rate in the backend network than in the frontend network, which is used to control the degree of dilation. For example, Conv2d (512, 3, 1, 2, 2) x 3 indicates that the output of the convolutional layer is 512 channels, the original size of the convolution kernel is 3x3, the step is 1, the input is filled with 2 circles of 0 or 1 around the input, and the dilated rate of the convolution kernel is 2. The dilated convolution block is stacked by three dilated convolution layers with the structure of Conv2d (512, 3, 1, 2, 2).

[0047] In the prediction part, from top to bottom, it corresponds to the density map prediction network, and the center point prediction network, the width and height prediction network, and the center point offset prediction network in the anchor-free fruit detection network. Taking the center point prediction network as an example, in the center point prediction, the output of the backend network is sequentially processed by the convolutional layer with the structure of Conv2d (64, 3, 1, 1) and the processing with the structure of Conv2d (1, 3, 1, 1), to obtain the center point prediction result with the size of (1, 320, 180), that is, a feature map with 1 channel, a height of 320, and a width of 180, wherein the numerical value of 0-1 represents the probability size of a certain point being a center point. In the width and height prediction result (2, 320, 180), 2 channels, a height of 320, and a width of 180 are recorded, wherein one channel records the predicted width, and the other channel records the predicted height. Similarly, in the center point offset prediction result, one channel records the predicted width direction offset, and the other channel records the predicted height direction offset.

[0048] Based on the system structure of Figure 2 From the perspective of the change of the image size in the processing process, the size of the input image is (3, 1280, 720), there are 4 convolution blocks and 3 pooling layers cross-stacked in the frontend network, the convolution block increases the number of channels while maintaining the output relative to the output width and height size unchanged, so as to discover different levels of semantic information from more channels, each of the 3 pooling layers reduces the image to one half of the original size, and the frontend network outputs a feature map with the size of (512, 160, 90). It can be seen that the frontend network reduces the width and height size of the output feature map to one eighth of the input image; the backend network includes 4 stacked dilated convolution blocks and one up-sampling layer (2 times up-sampling), and the backend network outputs a feature map with the size of (64, 320, 180). It can be seen that the backend network adjusts the size of the output feature map to one fourth of the input image.

[0049] It should be understood that the implementation of the backbone network is not limited to Figure 2The front-end network can be composed of a commonly used feature extraction network such as VGG, ResNet, etc. For example, the first 13 layers of the VGG16 network are used, and the structure after the first 13 layers is removed, as the front-end network Figure 2 An alternative to the front-end network is shown, thereby obtaining another embodiment of the backbone network. For another example, the front-end network can also be replaced according to the structure of the ResNet50 network, mainly using the first 3 stages of the backbone network of the ResNet50 network, in order to prevent the problem of low resolution of the feature map caused by excessive down-sampling, the stride of the last stage is set to 1, so that the size of the output feature map does not change when the last stage extracts the feature, only the number of channels of the feature map changes. In addition, the back-end network can also be adjusted, for example, the dilated convolution block in the back-end network shown in Figure 2 is increased or decreased; or the number of dilated convolution layers in the corresponding dilated convolution block is adjusted; or the up-sampling layer is arranged between two dilated convolution blocks.

[0050] When detecting densely distributed and occluded targets, the labeling method of the anchor frame is easy to introduce a large amount of noise due to occlusion, so the anchor-free target detection algorithm is selected in the present application. The system structure during training is shown in Figure 2 According to an embodiment of the present application, in the system, after the input image is processed by the front-end network and the back-end network, anchor-free target detection is performed based on the output of the back-end network using the fruit detection network, including center point prediction, width and height prediction, and center point offset prediction; in addition, density map prediction is also performed based on the output of the back-end network; the total loss is calculated based on the loss function according to the results of the density map prediction, the center point prediction, the width and height prediction, and the center point offset prediction and the corresponding labels, and the gradient is solved according to the total loss and the parameters of the density map prediction network, the fruit detection network, the back-end network, and the front-end network are updated by back propagation.

[0051] Before training, a data set containing training samples needs to be prepared. According to an embodiment of the present application, the training samples include sample images, fruit annotation labels, and density map labels.

[0052] According to an embodiment of the present application, the sample image includes a photographed fruit tree image. For example, in the detected fruit tree plantation, a camera is used to collect images of the whole fruit tree, the image pixel is 1280x720, and a schematic sample image is shown in Figure 3 a, Figure 3a is an image of an apple tree. It should be understood that the fruit tree can be an apple, orange, orange or grapefruit tree. However, since the external characteristics of the fruits of different types of fruit trees are different, the data set and the model should be trained separately, for example, a data set of apples is collected separately for apple trees, and then a fruit detection model for detecting apples is trained using the data set of apples. If other fruits need to be detected, a corresponding data set is additionally prepared and a corresponding fruit detection model is trained.

[0053] According to an embodiment of the present application, the fruit annotation label is obtained by using label processing software to annotate the fruit target with a shielding area less than or equal to a predetermined shielding ratio on the sample image. For example, after image acquisition, the fruit target is annotated using Labelimg software, and the fruit with a shielding ratio less than or equal to 0.7 is annotated. That is, the apple with a leaf shielding and a fruit overlapping area ratio greater than 0.7 is not annotated. When the fruit is annotated, only the unshielded part of the fruit is annotated, that is, the part of the shielded fruit that is as visible as possible is framed with the smallest annotation box. The annotation box is a rectangular box, and the width and height of the fruit target are indicated by the coordinates of two corner points. After the annotation is completed, the target center point is calculated according to the coordinates of the two corner points of the rectangular box. That is, after the annotation is completed, the fruit annotation label includes the information of the width and height of the fruit target and the information of the target center point. For example, the rectangular box is composed of the position coordinates of the upper left corner and the lower right corner of the target in the image, and the coordinates of the upper left corner and the lower right corner of the target can calculate the width and height and the center point coordinates of the target. For Figure 3 The local image of the fruit annotation label of a local area of a is shown in Figure 3 b.

[0054] According to an embodiment of the present application, the density map label is calculated based on the fruit annotation label. When the density map label is calculated based on the center point coordinates obtained from the fruit annotation label, a full zero matrix with the same size as the sample image is constructed, and all target center point coordinates corresponding positions in the full zero matrix are set to 1 to obtain a non-full zero matrix; the non-full zero matrix is convolved using a Gaussian kernel function to obtain the density map label.

[0055] According to an embodiment of the present application, the Gaussian kernel function is represented as:

[0056]

[0057] wherein c x represents the horizontal coordinate of the fruit center point, c y represents the vertical coordinate of the fruit center point, and σ represents a control parameter for controlling the radial range of the Gaussian kernel function, and e represents the base of the natural logarithm.

[0058] In the crowded scene, the sum of distances between the target and its adjacent k targets is used as the control parameter σ of the Gaussian kernel function, which can better represent the size information of the target features. The control parameter σ is calculated in the following manner using the sum of distances between the k nearest neighbors multiplied by a weight adjustment factor:

[0059]

[0060] where τ represents the weight adjustment factor, k represents the preset threshold of the nearest neighbor fruit, and represents the sum of distances between the nearest k neighbors when there are multiple nearest neighbor fruits near a center point. That is, the distance between the k nearest neighbors of each fruit is Preferably, the threshold of the nearest neighbor fruit is less than or equal to 6. The weight adjustment factor τ is set in the range of 0.1 to 0.5. The weight adjustment factor can be manually set, and the basis is that the distance between the k nearest neighbors is inversely proportional. In the experiment, the inventors tried values near 1 / k. For example, when the value of k is 4, 0.1, 0.2, and 0.3 are tried, and it is found that these three values have little negative impact on the final recognition result and are beneficial to improve the detection accuracy.

[0061] Preferably, when generating the density map, the corner point information of the upper left corner and the lower right corner of the label box is used to calculate the center point coordinates of each sample image, and the center point coordinates are used to generate the density map information (i.e., a non-full zero matrix) of the sample image. For each fruit, the distance between the nearest four neighbors is Calculate the Gaussian kernel parameter where the weight adjustment factor τ is set to 0.25, the density map information is convolved using the Gaussian kernel function, and the density map label is obtained. For Figure 3 a local region, the local map of the density map label is shown in Figure 3 c.

[0062] The present application uses the sum of distances between the nearest neighbor fruits to determine the Gaussian kernel parameter σ, and in the implementation process, the weight adjustment factor τ can also be adjusted according to the training results and experience on site to balance the quality of the density map generated in different numbers of fruit images, and a more accurate density map information can be generated in different complex environments.

[0063] According to one embodiment of the present application, the training set is used to train the system for fruit detection model training. Among them, all the sample images labeled are randomly divided into a training set, a validation set and a test set according to the proportions of 80%, 10% and 10%. The training set is used for training, the validation set is used for verification, and the test set is used for final effect test.

[0064] The entire training process is divided into two stages, and the system of the present application is trained by the first training stage and the second training stage using the plurality of training samples.

[0065] In the first training stage, the backbone network and the density map prediction network are trained and other network model parameters (i.e., the fruit detection network) are frozen. The first training stage includes: inputting a sample image into the system, sequentially processing the sample image through the backbone network and the density map prediction network, and obtaining a density map prediction result; calculating a density map loss according to the density map prediction result corresponding to the sample image and a density map label, and updating parameters of the backbone network and the density map prediction network according to the density map loss. The setting of the hyperparameters is, for example: the optimizer adopts a stochastic gradient descent, a momentum factor (Momentum) is set to 0.95, a weight decay is set to 5x10 -4 , the batch size of the image is set to 1, a fixed learning rate is used when training the density map prediction network, the learning rate is set to 1x10 -7 , and the iteration is performed for 400 times. The technical scheme of the embodiment can at least achieve the following beneficial technical effects: the purpose of the present application is to use the density map prediction result of the density map prediction network to guide the training of the fruit detection model, and in order to achieve better guiding effect, the present application first freezes the parameters of the fruit detection network, only trains the backbone network and the density map prediction network, first trains the backbone network and the density map prediction network for density map prediction, and achieves better density map prediction accuracy, so as to better play the role of guiding (or assisting) the training of the fruit detection model.

[0066] After the first training stage is completed, the parameters of the fruit detection network are unfrozen, and the second training stage is performed to train the entire system. In the second training stage, the density map loss is calculated based on the density map label calculated according to the fruit annotation label, and the fruit detection model is guided by the density map loss. Thus, the detection accuracy of the fruit detection model on the fruit trees is improved, and the probability of missing detection is reduced. The second training stage includes: inputting a sample image into the system, outputting a target detection result and a density map prediction result, calculating a target detection loss according to the target detection result corresponding to the sample image and the fruit annotation label, calculating a density map loss according to the density map prediction result corresponding to the sample image and the density map label, calculating a total loss according to the target detection loss and the density map loss, and updating parameters of the backbone network, the fruit detection network and the density map prediction network according to the total loss. The setting of the hyperparameters is, for example: the Adaptive Momentum Estimation optimizer is used, the learning rate is set to 1x10 -4 , and the weight decay is set to 5x10 -4In the training process, when the validation set accuracy stops updating, the learning rate is adjusted using the ReduceLROnPlateau method when the specified indicator does not change, for example, the validation set accuracy or the training set loss; the learning rate adjustment factor (Factor) is set to 0.5, the patience value (Patience) is set to 2, and the iteration number is set to 20. It should be understood that the learning rate adjustment method can be set according to training experience, for example, the learning rate can also be adjusted in the following ways: StepLR: adjust the learning rate at equal intervals according to a fixed step; MultiStepLR: adjust the learning rate according to different step lengths set; ExponentialLR: adjust the learning rate according to exponential decay; and CosineAnnealingLR: adjust the learning rate in a cosine cycle.

[0067] In the second training stage, the total loss is calculated as follows:

[0068] L det =L heatmap +λ size L size +L off +L map ;

[0069] wherein L heatmap +λ size L size +L off represents the target detection loss, which includes a center point prediction loss L heatmap , a width-height prediction loss L size , and a center point offset loss L off , λ size represents a weighting coefficient of L size , L map represents a density map loss, L heatmap adopts a focal loss function, L size and L off adopt an L1 loss function, and L map adopts an L2 loss function.

[0070] The target detection result corresponding to the anchor-free target detection model includes a center point prediction result, a width-height prediction result, and a center point offset prediction result. Therefore, referring to Figure 4 , the total loss is calculated according to the sub-losses corresponding to the center point prediction, the width-height prediction, the center point offset prediction, and the density map prediction.

[0071] According to an embodiment of the present application, each part of the sub-loss is represented as:

[0072] The center point prediction loss (also referred to as a heat map loss) is represented as:

[0073]

[0074] where Y i represents a label value indicating whether a point i is a center point, represents a corresponding center point prediction result W and H represent the original width and height of an image, R is a down-sampling factor, a is a first adjustment hyper-parameter, β is a second adjustment hyper-parameter, N represents the total number of center points, and M represents the total number of pixel points in a density map prediction result. In experiments, α = 2 and β = 4 are set.

[0075] The width and height prediction loss is represented as:

[0076]

[0077] where N represents the total number of target center points, is a predicted width and height value at a point k, S k is an actual width and height value at the point, and R is a down-sampling factor.

[0078] The center point offset loss is represented as:

[0079]

[0080] where N represents the total number of target center points, represents a predicted offset value, p k is a coordinate of a target center point k, R is a down-sampling factor, is an approximate integer coordinate of the target center point after scaling, represents an offset value of the center point after scaling. The actual label frame does not contain the center point offset, and the center point offset is calculated based on the difference between the center point of the actual target and the center point of the predicted target, and is mainly used to correct the predicted value of the center point position (i.e., the center point prediction result).

[0081] The density map loss is represented as:

[0082]

[0083] where represents a value corresponding to each point in a density map predicted by a network, D i represents a value corresponding to each point in an actual density map, and M represents the number of pixel points in the entire density map.

[0084] According to one embodiment of the present application, a total loss function is a weighted sum of the above four partial losses, where L size is a weight of the center point prediction loss, size= 0.1.

[0085] The present application is aimed at the problem of dense distribution of fruit targets and the shelter of fruit trees and leaves in orchards. The density map information is fused into the target detection network through the above trained model structure and training process, and the density map information is used to provide the accurate position information of the target for the anchor-free target detection network, thereby improving the detection accuracy of the fruit detection model.

[0086] The density map prediction only guides the fruit detection model to extract features and / or detect more accurately during the training of the model, and the density map prediction part is removed after the training is completed. According to an embodiment of the present application, a fruit detection method is provided, which comprises: acquiring a photographed fruit tree image as an input image, processing the input image by using the fruit detection model trained by the foregoing method, and obtaining a target detection result. The target detection result obtained during actual prediction includes a center point prediction result, a width-height prediction result, and a center point offset prediction result. If it is necessary to display the target detection result on the input image, the following processing is required: determining the peak points in the center point prediction result, wherein the peak point refers to a point whose value is greater than the values of all surrounding pixel points (8 points), and the total number of peak points does not exceed a preset upper threshold (for example, the peak point upper limit number topK is set to 150); obtaining the corresponding center position of the target according to the peak points The center position of the target box corresponding to the input image is calculated according to the predicted center position offset The predicted width and height and the down-sampling factor R. and the width and height of the target box: wherein, represents the predicted value of the center point k in the width direction, represents the predicted value of the center point k in the height direction, represents the predicted offset of the center point k in the width direction, represents the predicted offset of the center point k in the height direction, represents the predicted width value of the target of the center point k, represents the predicted height value of the target of the center point k.

[0087] The target detection model of the present application can also be used in the application scenario of fruit counting. According to an embodiment of the present application, a fruit counting method is provided, comprising: taking one picture of a tree from two opposite directions as two input images, respectively, and using the fruit detection model trained by the foregoing method to process the two input images respectively to obtain the corresponding target detection results; and using the target detection results corresponding to the two input images to calculate the number of fruits respectively and sum them up to obtain the number of fruits on the fruit tree. For example, a plurality of pictures can also be taken continuously around the fruit tree, and the number of fruits on the fruit tree can be obtained by removing the repeatedly appearing fruits in adjacent pictures through the motion trajectories of the fruits in adjacent pictures and summing up the number of fruits.

[0088] For the first training stage, the density map prediction network training loss and the mean absolute error (MAE) result of the validation set are as shown in FIG. 2, wherein the horizontal coordinate is the training epoch, and Figure 5 Figure 5 a is the change of the mean absolute error in the training process, Figure 5 b is the change of the training loss in the training process. The specific formula of the validation set MAE is:

[0089]

[0090] wherein N represents the number of images in the test set, D represents the density map predicted by the network, and D i represents the density map label.

[0091] The comparison results of the fruit detection model trained by the second training stage of the present application with other prior art on the test set are shown in Table 1, wherein the precision evaluation of target detection is measured by the average precision (AP) with an Intersection over Union (Iou) threshold of 0.5 between the target frame and the detection frame, and the model performance is measured by the parameter amount.

[0092] Table 1 Comparison of target detection model results

[0093]

[0094] According to the effect of the fruit detection model on the test set, the fruit detection model with the best precision is selected for fruit (apple) target recognition in the orchard. The comparison of the recognition effect of the final fruit detection model with the recognition effect of the existing model is shown in FIG. 3 and FIG. 4. Figure 6 and 7 wherein Figure 6 a, Figure 6 b is the detection result of the Yolov4 model; Figure 7 a,​Figure 7 b is the detection result of the CenterNet model, Figure 6 c, Figure 6 d, Figure 7 c, Figure 7 d is the detection result of the fruit detection model of the application (based on the structure shown in the figure). Wherein Figure 2 a, Figure 6 c, Figure 6 a, Figure 7 c represents the detected whole image result, Figure 7 b, Figure 6 d, Figure 6 b, Figure 7 d represents the local enlarged image result. It can be found through comparison that the detection results of the Yolov4 model and the CenterNet model have serious missing detection problems in the dense target area, the occluded target and the background interference area, so that part of the target is not detected. Under the same detection threshold, the fruit detection model trained by the application can accurately locate the dense target and the occluded target, so that the dense area and the occluded target can be better detected. Figure 7 According to the general flow of one embodiment of the application, see

[0095] , including S1, sample image acquisition and fruit label annotation; S2, generating a density map label according to the fruit label; S3, using the density map prediction network to guide (or assist) the training of the fruit detection model in the system of the application according to the sample image, the fruit label and the density map label; S4, using the trained fruit detection model for target detection. It should be noted that the application is obviously different from the utilization of the density map in some prior art, which directly trains the model to detect the target according to the density map, while the density map of the application only plays a guiding role in the training process, and after the training is completed or the density map prediction network is removed, only the trained fruit detection model can be used to realize the detection of the fruit. Figure 8 It should be noted that although the above describes each step in a specific order, it does not mean that each step must be executed in the above specific order, in fact, some of these steps can be executed concurrently, or even the order is changed, as long as the required function can be realized.

[0096] The application can be a system, a method and / or a computer program product. The computer program product can include a computer readable storage medium on which is loaded a computer readable program instruction for causing a processor to implement various aspects of the application.

[0097]

[0098] ​A computer readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, semiconductor, or any other suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves.

[0099] Having described above several embodiments, now will be described a number of modifications and alternatives. Such description is included to provide for a more complete understanding of the various embodiments and is not intended in any way to limit the scope of possibilities. Many modifications in addition to those described above can be made by one skilled in the relevant art without departing from the scope and spirit of the described embodiments. The scope of the various embodiments disclosed herein cover all technical solutions falling within the principles and their practical applications or improvements to the art, as well as enabling other ordinary skilled in the art to understand the various embodiments disclosed herein.

Claims

1. A training method for a fruit detection model, characterized in that, include: Acquire multiple training samples, each training sample including sample image, fruit annotation label and density map label; A system for training a fruit detection model is trained using the multiple training samples through a first training phase and a second training phase. The system includes: a fruit detection model comprising a backbone network and a fruit detection network; the backbone network comprising a front-end network and a back-end network; the front-end network performing convolution and downsampling on the input image to obtain a first feature map; the back-end network performing dilated convolution and upsampling on the first feature map to obtain a second feature map; the fruit detection network performing anchor-free target detection based on the second feature map, with fruit as the target; and a density map prediction network performing density map prediction based on the second feature map. In the first training phase, while freezing the parameters of the fruit detection network, the backbone network and density map prediction network are trained multiple times using training samples. This includes: inputting sample images into the system, processing them sequentially through the backbone network and density map prediction network to obtain density map prediction results; calculating density map loss based on the density map prediction results and density map labels corresponding to the sample images; and updating the parameters of the backbone network and density map prediction network based on the density map loss. In the second training phase, the fruit detection model and the density map prediction network are jointly trained multiple times, including: inputting sample images into the system to obtain the target detection results output by the fruit detection network and the density map prediction results output by the density map prediction network; calculating the target detection loss based on the target detection results and fruit labels corresponding to the sample images; calculating the density map loss based on the density map prediction results and density map labels corresponding to the sample images; calculating the total loss based on the target detection loss and density map loss; and updating the parameters of the backbone network, fruit detection network, and density map prediction network based on the total loss.

2. The training method for the fruit detection model according to claim 1, characterized in that, The front-end network includes multiple stacked convolutional blocks and pooling layers. Each convolutional block includes one or more convolutional layers. The number of output channels of each convolutional block in the front-end network is greater than the number of input channels; and / or The backend network includes multiple dilated convolutional blocks and upsampling layers connected in sequence. Each dilated convolutional block maintains the output size relative to the input size through a padding operation.

3. The training method for the fruit detection model according to claim 1 or 2, characterized in that, The density map labels are calculated as follows: Construct a zero matrix of a preset size, and set the position of the center point of the corresponding fruit in the zero matrix to 1 according to the fruit label of the corresponding sample image, to obtain a non-zero matrix; The density map labels are obtained by convolving a non-zero matrix with a Gaussian kernel function, where the Gaussian kernel function is expressed as: ， in, The x-coordinate of the center point of the fruit. The ordinate represents the center point of the fruit. This represents the control parameter used to control the radial range of the Gaussian kernel function. It represents the base of the natural logarithm.

4. The training method for the fruit detection model according to claim 1 or 2, characterized in that, The total loss is calculated as follows: ； in, This represents the target detection loss, which includes the center point prediction loss. Width and height prediction loss and center point offset loss , express The weighting coefficients, This represents density map loss. Employing the focus loss function, and Using the L1 loss function, The L2 loss function is used.

5. A method for detecting fruit, characterized in that, The method includes: The fruit tree image is captured as the input image, and the fruit detection model trained by the method described in any one of claims 1-4 is used to process the input image to obtain the target detection result.

6. A computer-readable storage medium, characterized in that, It contains a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1 to 5.

7. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store one or more executable instructions; The one or more processors are configured to implement the steps of the method according to any one of claims 1 to 5 by executing the one or more executable instructions.

Images