A method for detecting apple maturity

Abstract

The embodiment of the present invention provides an apple maturity detection method, which includes: obtaining an RGB image of apples to be picked, inputting it into an apple maturity detection system, and obtaining the position and maturity of the apples to be picked. The apple maturity detection system constructed by the present invention can judge the input image based on the RGB image, and after processing and recognition by an image enhancement module, a deep residual network, a feature pyramid network, a region generation network, a region assignment module, a SVM module and a decision tree module connected in sequence, and an image segmentation network connected to the region assignment module, the accuracy of outdoor apple maturity classification can be improved.

Description

Technical Field

[0001] The present invention relates to the field of neural networks, in particular to the field of target detection, and more particularly to a method for detecting the maturity of apples. Background Art

[0002] The current automatic apple picking robot uses existing detection methods to determine the location of the target fruit and whether the target fruit is ripe, so as to detect the picking target and realize automatic picking, wherein the data collected by the sensor installed on the picking device or the mobile platform is processed to detect the location of the picking target and the maturity of the target. Current technology uses traditional color sensors, such as infrared spectrometers, color recognition sensors, etc., to identify and extract red, so as to determine whether the apple is ripe. However, in the actual use of outdoor orchards, due to the low recognition accuracy of color recognition sensors such as infrared spectrometers, the working efficiency of the picking device is not high. Therefore, it is necessary to improve the existing technology. Summary of the invention

[0003] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method for constructing an apple maturity detection system and an apple maturity detection method.

[0004] The objective of the present invention is achieved through the following technical solutions:

[0005] According to a first aspect of the present invention, a method for constructing an apple maturity detection system is provided, comprising: S1, obtaining an apple data set consisting of RGB images, and preprocessing the apple data set to obtain a standard image; S2, marking the position and maturity of apples as recognition targets on the standard image to obtain an input image, and dividing the obtained part of the input image into a training set; S3, constructing an initial apple maturity detection system, comprising an image enhancement module, a deep residual network, a feature pyramid network, a region generation network, a region assignment module, a SVM module and a decision tree module connected in sequence, and an image segmentation network connected to the region assignment module; S4, performing multiple rounds of training on the initial apple maturity detection system using the input images of the training set to obtain a final apple maturity detection system.

[0006] In some embodiments of the present invention, step S1 includes: S11, removing blurred images and images without identifiable targets in the apple dataset; S12, cropping the RGB images in the apple dataset into standard images of preset specifications.

[0007] In some embodiments of the present invention, the image enhancement module adopts a fully connected neural network, which uses a convolution kernel with an input channel of 3, an output channel of 1, a size of 1×1, and a stride of 1 to combine the three-channel features of the input image into one channel for image enhancement.

[0008] In some embodiments of the present invention, the process of each round of training in step S4 includes: S41, setting an operable pixel area according to the operating range of the picking robot, and using an image enhancement module to combine the three-channel features of the operable pixel area of ​​the input image into one channel for image enhancement to obtain an enhanced image; S42, using a deep residual network to extract features of the enhanced image layer by layer to obtain a multi-layer feature map of the enhanced image; S43, using a feature pyramid network to extract a final feature map set from a preset partial layer feature map of the multi-layer feature map; S44, based on the final feature map set, generating multiple regions of interest through a region generation network; S45, using a region assignment module to calculate the feature value of each region of interest; S46, based on the feature value of each region of interest, using an SVM module to perform binary classification on the recognition target in each region of interest to determine whether it is an apple, using a decision tree module to perform multi-classification on the recognition target that is an apple to determine the maturity, and using an image segmentation network to determine the position of the recognition target in the input image; S47, calculating the loss value based on the maturity classification error and the regression error and updating the parameters of the apple maturity detection system through back propagation.

[0009] In some embodiments of the present invention, step S42 includes: using the deep residual network ResNet101 trained with the MS COCO dataset as the skeleton network, performing convolution and ReLU function activation pooling operations on the input image, and extracting the first layer feature map, the second layer feature map, the third layer feature map, the fourth layer feature map, and the fifth layer feature map of the enhanced image.

[0010] In some embodiments of the present invention, step S43 includes: S431, taking the second layer feature map, the third layer feature map, the fourth layer feature map, and the fifth layer feature map in the multi-layer feature map as the input of the feature pyramid network; S432, the feature pyramid network performs a 1×1 convolution operation for reducing the number of channels on each layer of the input second layer feature map, the third layer feature map, the fourth layer feature map, and the fifth layer feature map, unifies the number of output channels corresponding to each layer to 256 channels, and obtains a temporary feature map corresponding to each layer; S433, the feature pyramid network performs a 1×1 convolution operation for reducing the number of channels on each layer of the input second layer feature map, the third layer feature map, the fourth layer feature map, and the fifth layer feature map, unifies the number of output channels corresponding to each layer to 256 channels, and obtains a temporary feature map corresponding to each layer; S434, the feature pyramid network performs a 1×1 convolution operation for reducing the number of channels on each layer of the input second layer feature map, the third layer feature map, the fourth layer feature map, and the fifth layer feature map, unifies the number of output channels corresponding to each layer to 256 channels, and obtains a temporary feature map corresponding to each layer; S435 Each layer of the feature map, the fourth-layer feature map, and the fifth-layer feature map is upsampled in a top-down manner so that the size of the sampled feature map is consistent with that of the feature map of the next layer; S434, the temporary feature map corresponding to each layer is fused with the feature map obtained by upsampling the previous layer in an additive manner to obtain the fused feature map corresponding to each layer; S435, the fused feature map corresponding to each layer is processed with a convolution kernel of size 3×3 to obtain a final feature map set, which includes the second-layer map, the third-layer map, the fourth-layer map, and the fifth-layer map.

[0011] In some embodiments of the present invention, the step S44 includes: S441, selectively processing the second layer mapping map, the third layer mapping map, and the fourth layer mapping map in the final feature mapping set through a region generation network, and generating multiple regions of interest defined according to three preset area sizes and three preset ratios on each layer of the mapping map; S442, using a non-maximum suppression algorithm to remove duplicate regions of interest among the multiple regions of interest.

[0012] In some embodiments of the present invention, the three preset area sizes are 120×120, 160×160, and 200×200, and the three preset ratios are 1:1, 1:1.2, and 1.2:1.

[0013] According to a second aspect of the present invention, there is provided an apple maturity detection system constructed according to the method described in the first aspect, comprising an image enhancement module, a deep residual network, a feature pyramid network, a region generation network, a region assignment module, an SVM module and a decision tree module connected in sequence, and an image segmentation network connected to the region assignment module, wherein the apple maturity detection system is configured to detect the maturity of apples in the following manner: using the image enhancement module to combine the three-channel features of the operable pixel area of ​​the input image into one channel for image enhancement to obtain an enhanced image; using the deep residual network to perform layer-by-layer feature extraction on the enhanced image to obtain a multi-layer feature map of the enhanced image; using the feature pyramid network to extract a final feature map set from a preset part of the layer feature maps in the multi-layer feature map; based on the final feature map set, generating multiple regions of interest through a region generation network; using the region assignment module to calculate the feature value of each region of interest; based on the feature value of each region of interest, using the SVM module to perform binary classification on the recognition target in each region of interest to determine whether it is an apple, using the decision tree module to perform multi-classification on the recognition target that is an apple to determine the maturity, and using the image segmentation network to determine the position of the recognition target in the input image; and outputting the position and maturity of the recognition target in the input image.

[0014] According to a third aspect of the present invention, there is provided an apple maturity detection method implemented by using the above-mentioned apple maturity detection system, the method comprising: acquiring an RGB image of apples to be picked, inputting the image into the apple maturity detection system, and obtaining the position and maturity of the apples to be picked.

[0015] According to a fourth aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory, wherein the memory is used to store one or more executable instructions; the one or more processors are configured to implement the steps of the method described in the first aspect or the third aspect by executing the one or more executable instructions.

[0016] Compared with the prior art, the advantages of the present invention are:

[0017] The apple maturity detection system constructed based on the method of the present invention can judge the input image based on the RGB image, and after being processed and recognized by the image enhancement module, deep residual network, feature pyramid network, region generation network, region assignment module, SVM module and decision tree module connected in sequence, and the image segmentation network connected to the region assignment module, the accuracy of outdoor apple maturity classification can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0019] Figure 1A schematic diagram of a process for constructing an apple maturity detection system according to an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of the principle of a system constructed according to the method for constructing an apple maturity detection system according to an embodiment of the present invention;

[0021] Figure 3 Schematic diagram of the appearance of apples of four different degrees of maturity according to an exemplary maturity definition of an embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of an anchor frame according to an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of changes in the prediction accuracy of the system when training the system according to the method for constructing an apple maturity detection system according to an embodiment of the present invention;

[0024] Figure 6 This is a schematic diagram of the result of detecting an image using the apple maturity detection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0026] As mentioned in the background technology section, during actual use in outdoor orchards, color recognition sensors such as infrared spectrometers have low recognition accuracy, resulting in low working efficiency of the picking device. The apple maturity detection system constructed according to the method of constructing an apple maturity detection system (also referred to as an apple maturity detection model) of the present invention can judge the input image based on the RGB image, and output the position of the recognition target (picking target) in the input image and the maturity of the target. In this process, the three-channel features of the operable pixel area of ​​the input image are combined into one channel through the image enhancement module for image enhancement, so as to extract feature information on different channels for subsequent analysis to obtain an enhanced image; then, a deep residual network is used to extract features of the enhanced image layer by layer, and the features of the recognition target can be obtained at different resolution sizes to obtain a multi-layer feature map of the enhanced image; then, a feature pyramid network is used to extract the final feature mapping set from the preset partial layer feature maps in the multi-layer feature map, and artificial settings are canceled to cancel the influence of accuracy. Extracting a mapping map from a small part of the layer feature map can improve the recognition efficiency while ensuring the accuracy; then, based on the final feature mapping set, multiple regions of interest are generated through the region generation network and assigned values ​​to the regions of interest; based on the feature value of each region of interest, the SVM module is used to perform binary classification on the recognition target in each region of interest to determine whether it is an apple, and the decision tree module is used to perform multi-classification on the recognition target that is an apple to determine the maturity, and the image segmentation network is used to determine the position of the recognition target in the input image, which is equivalent to forming a combined classifier by the SVM module and the decision tree module. The SVM module first determines whether the target in the region of interest is an apple, and then the decision tree module further determines the maturity of the target that is an apple, thereby improving the accuracy of classification; thus, after the above processing, the accuracy and detection efficiency of outdoor apple detection are improved. When the detection results are output to the outdoor apple picking device for judging the picking target, it is helpful to improve the picking accuracy and efficiency of the outdoor apple picking device, and effectively help the outdoor apple picking device to achieve automation.

[0027] See also Figure 1 The present invention provides a method for constructing an apple maturity detection system, comprising: S1, S2, S3, and S4. Figure 2 This is a schematic diagram of the principle of constructing an apple maturity detection system according to an embodiment of the present invention. For a better understanding of the present invention, refer to Figure 1 and Figure 2 Each step is described in detail in conjunction with specific embodiments.

[0028] Step S1: Obtain an apple dataset consisting of RGB images, and preprocess the apple dataset to obtain a standard image.

[0029] According to one embodiment of the present invention, step S1 includes: S11, removing blurred images and images without recognition targets in the apple data set; S12, cropping the RGB images in the apple data set into standard images of preset specifications. The results of the apple maturity detection of the present invention are mainly used as the basis for the outdoor picking robot operation, and the operation refers to the automatic picking of apples using a robotic arm. The standard image of the preset specifications here mainly considers the size of the pixel and the working radius of the robotic arm. The size of the preset specifications is related to the placement of the camera and the robotic arm, because the upper, lower, left and right directions that the robotic arm can reach are limited, and the edge position in the image obtained by the camera imaging is a position that the robotic arm cannot reach. Therefore, according to the actual picking range, the preset specifications are set. For example, according to the picking range, the preset specifications are set to a 400×400 pixel area size. According to one embodiment of the present invention, if the number of samples in the apple data set is insufficient, multiple cropping centers can be randomly selected on an RGB image, and photos of 400x400 pixels can be cropped according to each cropping center for image augmentation. The technical solution of this embodiment can at least achieve the following beneficial technical effects: RGB images are processed to obtain standard images of required standard specifications so as to train the subsequent apple maturity detection system; image augmentation can crop multiple standard images from one RGB image when the number of RGB images is too small to enrich the samples used for training. Preferably, the RGB images in the apple data set include images of apples on apple trees and images of picked apples. This helps train the apple maturity detection system to learn the characteristics of the recognition target in the occluded and unoccluded states.

[0030] Step S2: Mark the location and maturity of the apple as the recognition target on the standard image to obtain an input image, and divide the obtained part of the input image into a training set.

[0031] According to one embodiment of the present invention, an anchor box is used to mark the area where the recognition target is located and its corresponding maturity on the standard image of the apple dataset. The maturity is marked in the form of a label. Figure 3 , the labels of maturity include: unripe, ripe for picking, edible ripe, overripe (in order of maturity from low to high). For example, if there is an unripe apple on an annotated image, first mark the apple with an anchor frame, and then mark the anchor frame with the unripe label to obtain an input image marked with the location and maturity of the recognition target. After the annotation is completed, the training set and the test set are divided into the required ratio in the annotated apple dataset. The training set and the test set are divided into the annotated apple dataset in a ratio of 8:2. For example, during a training, the inventor collected a total of 2,000 images as an apple dataset, of which 1,600 images were used as a training set and 400 images were used as a test set.

[0032] Step S3: construct an initial apple maturity detection system, including an image enhancement module, a deep residual network, a feature pyramid network, a region generation network, a region assignment module, a SVM module and a decision tree module connected in sequence, and an image segmentation network connected to the region assignment module.

[0033] According to one embodiment of the present invention, the image enhancement module adopts a three-channel fully connected neural network, including a fully connected layer. The deep residual network is Deep residual network, referred to as ResNet. The feature pyramid network is Feature Pyramid Network, referred to as FPN. The region generation network is Region Proposal Network, referred to as RPN. The region assignment module is used to assign values ​​to the region of interest generated by the region generation network. The image segmentation network adopts FCN segmentation network, which is a fully convolutional neural network (Fully Convolutional Networks). The SVM module is a support vector machine (Support Vector Machines). The decision tree module is constructed using a decision tree algorithm, for example, using the C4.5 algorithm, ID3 algorithm, and CART algorithm (Classification and Regression Tree).

[0034] Step S4: Perform multiple rounds of training on the initial apple maturity detection system using the input images of the training set to obtain the final apple maturity detection system.

[0035] According to one embodiment of the present invention, each round of training process includes: S41, S42, S43, S44, S46, S47, and each sub-step is as follows:

[0036] Step S41, setting an operable pixel area according to the operating range of the picking robot, and using an image enhancement module to combine the three-channel features of the operable pixel area of ​​the input image into one channel for image enhancement to obtain an enhanced image.

[0037] Preferably, the image enhancement module uses a fully connected neural network, which uses a convolution kernel with an input channel of 3, an output channel of 1, a size of 1×1 and a stride of 1 to combine the three-channel features of the input image into one channel for image enhancement.

[0038] Step S42: Use a deep residual network to perform layer-by-layer feature extraction on the enhanced image to obtain a multi-layer feature map of the enhanced image.

[0039] Preferably, step S42 includes: using the deep residual network ResNet101 trained with the MS COCO dataset as the skeleton network, performing convolution and ReLU function activation pooling operations on the input image, and extracting the first layer feature map, the second layer feature map, the third layer feature map, the fourth layer feature map, and the fifth layer feature map of the enhanced image. Figure 2 As shown in the figure, ResNet101 (Deep residual network, ResNet) trained with the MSCOCO dataset is used as the skeleton network. The input image is convolved, and the ReLU function (Rectified Linear Unit) is activated and pooled to extract image features and output feature maps. The feature maps are divided into five stages according to the size of the feature maps. They are stage 1, stage 2, stage 3, stage 4, and stage 5. The convolution layers are conv1, conv2, conv3, conv4, and conv5. The output of each stage corresponds to the first layer feature map C1, the second layer feature map C2, the third layer feature map C3, the fourth layer feature map C4, and the fifth layer feature map C5. The stride of each layer feature map relative to the original image is 2, 4, 8, 16, and 32 respectively. Figure 2 In the figure, 1x1 means the convolution operation is performed with a 1x1 convolution kernel, and 3x3 means the convolution operation is performed with a 3x3 convolution kernel. 0.5x means the image is reduced to half of its original size. 2x means the image is enlarged to twice its original size.

[0040] Step S43: extracting a final feature mapping set from a preset partial layer feature map in the multi-layer feature map using a feature pyramid network.

[0041] Preferably, in order to reduce the amount of calculation and improve the efficiency of detecting and identifying targets, the inventors only select some of the feature maps as the input of the feature pyramid network. The inventors select the feature maps of the remaining layers except the first layer as the preset partial layer feature maps. In this case, step S43 includes: S431, using the second layer feature map, the third layer feature map, the fourth layer feature map, and the fifth layer feature map in the multi-layer feature map as the input of the feature pyramid network; S432, the feature pyramid network performs a 1×1 convolution operation to reduce the number of channels on each layer of the input second layer feature map, the third layer feature map, the fourth layer feature map, and the fifth layer feature map, and unifies the number of output channels corresponding to each layer to 256 channels to obtain temporary feature maps corresponding to each layer; S433, the feature pyramid network performs a 1×1 convolution operation on the input second layer feature map, the third layer feature map, the fourth layer feature map, and the fifth layer feature map. Each layer of the fourth-layer feature map and the fifth-layer feature map is upsampled in a top-down manner so that the size of the sampled feature map is consistent with that of the feature map of the next layer; S434, the temporary feature map corresponding to each layer is fused with the feature map obtained by upsampling the previous layer in an additive manner to obtain the fused feature map corresponding to each layer; S435, the fused feature map corresponding to each layer is processed with a convolution kernel of size 3×3 to obtain the final feature mapping set, which includes the second-layer mapping map, the third-layer mapping map, the fourth-layer mapping map, and the fifth-layer mapping map. After the fusion, the present invention uses a 3*3 convolution kernel to process the fused features to eliminate the aliasing effect of upsampling. See Figure 2 , the final feature map set is {P2, P3, P4, P5}, P2 represents the second layer map, P3 represents the third layer map, P4 represents the fourth layer map, and P5 represents the fifth layer map. The elements in the final feature map set have the same number of channels. The present invention abandons the first layer feature map with a large amount of data as the input of the feature pyramid network, which can reduce the amount of calculation for obtaining the feature map set and improve the efficiency of target detection.

[0042] Step S44: Generate multiple regions of interest through a region generation network according to the final feature map set, and use a non-maximum suppression algorithm to reduce repeated regions of interest.

[0043] Optionally, step S44 includes: S441', processing the second layer map, the third layer map, the fourth layer map, and the fifth layer map through a region generation network, generating multiple regions of interest defined according to three preset area sizes and three preset ratios on each layer map; S442', using a non-maximum suppression algorithm to remove duplicate regions of interest in the multiple regions of interest. In this optional implementation, each layer of feature maps in the final feature map set is processed, but the recognition speed can be further improved.

[0044] Preferably, step S44 includes: S441, selectively processing the second layer mapping map, the third layer mapping map, and the fourth layer mapping map in the final feature mapping set through a region generation network, and generating multiple regions of interest defined according to three preset area sizes and three preset ratios on each layer of the mapping map; S442, using a non-maximum suppression algorithm to remove duplicate regions of interest among the multiple regions of interest. Figure 2 The region generation network shown in is a process configured to process only the second layer map P2, the third layer map P3, and the fourth layer map P4. When conducting experiments, the inventors found that not processing the fifth layer map and only selecting the second, third, and fourth layer maps can not only achieve high accuracy in detecting the maturity of apples, but also reduce the amount of calculation and improve detection efficiency. Because the volume of apples is much smaller than that of animals, people, vehicles, etc., the feature pixels of apples on the low-resolution fifth layer map are small and difficult to distinguish.

[0045] When generating the region of interest, if the anchor box is too small or too large, the speed and accuracy of apple maturity detection will be affected. Preferably, the three preset area sizes are 120×120, 160×160, and 200×200, and the three preset ratios are 1:1, 1:1.2, and 1.2:1. Figure 4 As shown, nine sizes of anchor frames are generated for each target. The inventors have found through continuous debugging and configuration that setting the size of the anchor frame according to the preset area size and preset ratio can achieve a better detection speed and a higher recognition accuracy when detecting the maturity of apples.

[0046] The following example illustrates how step S44 uses the region generation network to generate multiple regions of interest (ROIs, referred to as candidate regions in some literature) based on the final feature map set {P2, P3, P4, P5}. Due to the operating area limitation of the picking device, only apples with a pixel area larger than 64×64 in the original image can be picked. If the area is too small, it means that the corresponding apple is too far away and cannot be picked. Here, the size of ROI needs to be calculated according to the color sensor used and the actual working radius of the picking device. If a suitable ROI cannot be found on the low-resolution feature map, the low-resolution P5 can be discarded, and only the three mapping maps P2-P4 are used to input RPN. Taking P3 as an example, first a 3x 3 convolution with a step size of 1 is performed to obtain a 256 x 80 x 80 feature map, which can also be regarded as 80 x 80 256-dimensional feature vectors. Then, after two 1 x 1 convolutions, a 18 x 80 x 80 feature map and a 36 x 80 x80 feature map are obtained, that is, 80 x 80 x 9 results, each of which contains 2 scores and 4 coordinates. Among them, the 18x80x80 feature map is used to classify the generated anchor box, and the 36x80x80 feature map is used to regress the generated anchor box. Each anchor box has two categories, positive and negative. The anchor box with the background category is usually called the negative anchor box, and the rest is called the positive anchor box. 128 positive anchor boxes and 128 negative anchor boxes are randomly selected for training and passed to the following ROI pooling layer. The ROI pooling layer will extract the corresponding area in the feature map obtained in step 4 of the anchor box, and identify these areas (apple or not apple) and regress the anchor box. During inference, the non-maximum suppression algorithm (NMS) will be used to deduplicate the obtained prediction box. Generally, the threshold of NMS is set to 0.5.

[0047] Step S45: Use the region assignment module to calculate the feature value of each region of interest.

[0048] Preferably, the region assignment module obtains a plurality of cells by segmenting the region of interest, and each cell is segmented into four small squares, and bilinear interpolation is performed with the center of each small square as a sampling point, and the four sampling points in each cell are pooled at the maximum value to obtain the eigenvalue of the cell, and the eigenvalues ​​of all cells constitute the eigenvalue of the region of interest. Because the boundary of the region of interest (ROI boundary) generated in step S44 is obtained by calculation, it is usually not one-to-one corresponding to the feature map. In order to avoid its influence on the detection accuracy, the present invention adopts this method to calculate the eigenvalue of the region of interest. Since the coordinates of the sampling points are usually floating point numbers, it is necessary to perform bilinear interpolation on the sampling point pixels, and then perform maximum pooling on the four sampling points in each cell to obtain the eigenvalue of the cell.

[0049] S46. Based on the feature values ​​of each region of interest, use the SVM module to perform binary classification on the recognition targets in each region of interest to determine whether they are apples, use the decision tree module to perform multi-classification on the recognition targets that are apples to determine the maturity, and use the image segmentation network to determine the position of the recognition targets in the input image.

[0050] Preferably, the present invention uses FCN (Fully Convolutional Networks) segmentation network to locate the detected apple area more precisely and segment the accurate position of the apple. At the same time, a classifier is used to classify the detected apple area to obtain the maturity level of the detected apple. The classifier uses a decision tree and an SVM module to train the classifier to connect the region of interest. First, an SVM module is created, and the SMO (sequential minimal optimization, referred to as SMO) algorithm is used to determine the classifier, find the minimum interval data point, and then maximize it and find the parameters. The separating hyperplane that maximizes the geometric interval and the corresponding classification decision function are found, which are used as the original optimization problem, and the Lagrangian duality is applied to obtain the optimal solution of the original problem by solving the dual problem. The advantages of this are that the dual problem is often easier to solve; the second is to naturally introduce the kernel function, which is then extended to nonlinear classification problems. The eigenvalues ​​of the region of interest are input into the SVM module, and the output is whether the image in the region of interest is an apple. After obtaining all the regions of interest that are apples, the feature values ​​and maturity labels of all the regions of interest that are apples are input into the decision tree module as training data to create a decision tree. The training decision tree module learns the correspondence between the feature values ​​of the regions of interest of apples and various maturity levels, and outputs the maturity level of apples according to the feature values ​​of the regions of interest of apples. The decision tree module determines the probability of the identified target in the region of interest being each maturity level based on the feature values ​​of the region of interest based on the constructed decision tree, and outputs the maturity label with the highest probability as the maturity level of the identified target in the region of interest.

[0051] S47. Calculate the loss value based on the maturity classification error and regression error and update the parameters of the apple maturity detection system through back propagation.

[0052] Preferably, the maturity classification error is calculated according to the maturity judged by the decision tree module during this round of training and the maturity of the mark, and the regression error is calculated according to the position of the recognition target in the input image determined by the image segmentation network during this round of training and the position of the recognition target marked. The loss value is the sum of the classification loss calculated according to the maturity classification error and the regression loss calculated according to the regression error. The parameters of the image enhancement module, the deep residual network, the feature pyramid network, the region generation network, the SVM module and the decision tree module are adjusted according to the loss value. The maturity classification error can use the cross entropy loss function. The regression loss can use the Smooth L1 loss function. When training the apple maturity detection system, the loss value is the sum of the classification loss and the regression loss. During training, confirm whether the apple maturity detection system has been trained to convergence in any of the following ways: Method 1: The number of training rounds reaches the custom upper limit number of rounds; Method 2: The absolute value of the change in the accuracy of the apple maturity detection system on the validation set after a certain round of training compared with the previous round of training is less than or equal to the preset amplitude threshold, that is, the accuracy has not been improved. Preferably, the preset amplitude threshold is set to 0.5%. Some hyperparameter settings during the training process can be adjusted on-site as needed. For example, the inventor set Batch_Size to 10, Epoch to 200 or 500, and the learning rate to 0.005 during a training session. The learning rate was set to decay exponentially with a decay rate of 0.9. Figure 5 , the horizontal axis is the number of Epochs trained, and the vertical axis is the multi-classification evaluation index mAP. After the training is completed, the apple maturity detection system has an accuracy rate of 0.815 in predicting apple maturity. In comparison, the accuracy rate of predicting apple maturity using the traditional sensor method is only 0.683. It can be seen that compared with the traditional method, the apple maturity detection system constructed by the method of the present invention has a greatly improved accuracy rate in predicting apple maturity.

[0053] The present invention also provides an apple maturity detection system constructed according to the method of the aforementioned embodiment, comprising an image enhancement module, a deep residual network, a feature pyramid network, a region generation network, a region assignment module, an image segmentation network, an SVM module and a decision tree module, wherein the apple maturity detection system is configured to detect the maturity of apples in the following manner: using the image enhancement module to combine the three-channel features of the operable pixel area of ​​the input image into one channel for image enhancement to obtain an enhanced image; using the deep residual network to perform layer-by-layer feature extraction on the enhanced image to obtain a multi-layer feature map of the enhanced image; using the feature pyramid network to extract a final feature map set from a preset portion of layer feature maps in the multi-layer feature map; based on the final feature map set, generating multiple regions of interest through a region generation network; using the region assignment module to calculate the feature value of each region of interest; based on the feature value of each region of interest, using the SVM module to perform binary classification on the recognition target in each region of interest to determine whether it is an apple, using the decision tree module to perform multi-classification on the recognition target that is an apple to determine the maturity, and using the image segmentation network to determine the position of the recognition target in the input image; outputting the position and maturity of the recognition target in the input image.

[0054] The present invention also provides an apple maturity detection method implemented by the above apple maturity detection system, the method comprising: obtaining an image of apples to be picked, inputting it into the apple maturity detection system, and obtaining the position and maturity of the apples to be picked. During training, the inventors used apple1, apple2, apple3, and apple4 to represent the labels of unripe, ripe for picking, edible ripe, and overripe, respectively. After training, when detecting the maturity of apples on an image, the following is obtained: Figure 6 The percentage after the label indicates the confidence level of the maturity. For example, the confidence level that apple 2 in the upper left corner is ripe and ready for picking is 99%.

[0055] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0056] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0057] A computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. Computer-readable storage media may include, for example, but are not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0058] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for constructing an apple maturity detection system, characterized in that: include: S1. Obtain an apple dataset consisting of RGB images, and preprocess the apple dataset to obtain a standard image; S2, marking the location and maturity of the apple as the recognition target on the standard image to obtain an input image, and dividing the obtained part of the input image into a training set; S3, constructing an initial apple maturity detection system, including an image enhancement module, a deep residual network, a feature pyramid network, a region generation network, a region assignment module, a SVM module and a decision tree module connected in sequence, and an image segmentation network connected to the region assignment module; S4, using the input images of the training set to perform multiple rounds of training on the initial apple maturity detection system to obtain the final apple maturity detection system; The process of each round of training in step S4 includes: S41, setting an operable pixel area according to the operating range of the picking robot, and using an image enhancement module to combine three-channel features of the operable pixel area of ​​the input image into one channel for image enhancement to obtain an enhanced image; S42, using a deep residual network to perform layer-by-layer feature extraction on the enhanced image to obtain a multi-layer feature map of the enhanced image; S43, extracting a final feature map set from a preset partial layer feature map of the multi-layer feature map using a feature pyramid network; S44, generating multiple regions of interest through a region generation network according to the final feature map set; S45, using a region assignment module to calculate the characteristic value of each region of interest; S46, according to the feature value of each region of interest, using the SVM module to perform binary classification on the recognition target in each region of interest to determine whether it is an apple, using the decision tree module to perform multi-classification on the recognition target that is an apple to determine the maturity, and using the image segmentation network to determine the position of the recognition target in the input image; S47. Calculate the loss value based on the maturity classification error and regression error and update the parameters of the apple maturity detection system through back propagation.

2. The method for constructing an apple maturity detection system according to claim 1, characterized in that: The step S1 comprises: S11, remove blurred images and images without identifiable objects in the apple dataset; S12. Crop the RGB images in the apple dataset into standard images with preset specifications.

3. The method for constructing an apple maturity detection system according to claim 1, characterized in that: The image enhancement module adopts a fully connected neural network, which uses a convolution kernel with an input channel of 3, an output channel of 1, a size of 1×1, and a stride of 1 to combine the three-channel features of the input image into one channel for image enhancement.

4. The method for constructing an apple maturity detection system according to any one of claims 1 to 3, characterized in that: Step S42 includes: The deep residual network ResNet101 trained with the MS COCO dataset is used as the skeleton network, and convolution and ReLU function activation pooling operations are performed on the input image to extract the first layer feature map, the second layer feature map, the third layer feature map, the fourth layer feature map, and the fifth layer feature map of the enhanced image.

5. The method for constructing an apple maturity detection system according to claim 4, characterized in that: Step S43 includes: S431, using the second layer feature map, the third layer feature map, the fourth layer feature map, and the fifth layer feature map in the multi-layer feature map as inputs of the feature pyramid network; S432, the feature pyramid network performs a 1×1 convolution operation for reducing the number of channels on each layer of the second-layer feature map, the third-layer feature map, the fourth-layer feature map, and the fifth-layer feature map of the input, unifies the number of output channels corresponding to each layer to 256 channels, and obtains a temporary feature map corresponding to each layer; S433, the feature pyramid network performs an upsampling operation on each layer of the input second-layer feature map, third-layer feature map, fourth-layer feature map, and fifth-layer feature map in a top-down manner so that the size of the sampled feature map is consistent with the size of the feature map of the next layer; S434, fusing the temporary feature map corresponding to each layer with the feature map of the previous layer obtained by upsampling operation in an additive manner to obtain a fused feature map corresponding to each layer; S435. Process the fused feature maps corresponding to each layer with a convolution kernel of size 3×3 to obtain a final feature mapping set, which includes a second layer mapping map, a third layer mapping map, a fourth layer mapping map, and a fifth layer mapping map.

6. The method for constructing an apple maturity detection system according to claim 5, characterized in that: The step S44 comprises: S441, selectively processing the second layer map, the third layer map, and the fourth layer map in the final feature map set through a region generation network, and generating multiple regions of interest defined according to three preset area sizes and three preset ratios on each layer map; S442: Use a non-maximum suppression algorithm to remove duplicate regions of interest from the multiple regions of interest.

7. The method for constructing an apple maturity detection system according to claim 6, characterized in that: The three preset area sizes are 120×120, 160×160, and 200×200, and the three preset ratios are 1:1, 1:1.2, and 1.2:

1.

8. An apple maturity detection system constructed according to the method according to any one of claims 1 to 7, characterized in that: The apple maturity detection system comprises an image enhancement module, a deep residual network, a feature pyramid network, a region generation network, a region assignment module, a SVM module and a decision tree module connected in sequence, and an image segmentation network connected to the region assignment module. The apple maturity detection system is configured to detect the apple maturity in the following manner: The image enhancement module is used to combine the three-channel features of the operable pixel area of ​​the input image into one channel for image enhancement to obtain an enhanced image; Use a deep residual network to extract features of the enhanced image layer by layer to obtain a multi-layer feature map of the enhanced image; Extracting a final feature map set from a preset partial layer feature map of the multi-layer feature map using a feature pyramid network; According to the final feature map set, multiple regions of interest are generated through a region generation network; Use the region assignment module to calculate the feature value of each region of interest; According to the feature value of each region of interest, the SVM module is used to perform binary classification on the recognition target in each region of interest to determine whether it is an apple, the decision tree module is used to perform multi-classification on the recognition target that is an apple to determine the maturity, and the image segmentation network is used to determine the position of the recognition target in the input image; Output the location and maturity of the recognized object in the input image.

9. A method for detecting apple maturity using the apple maturity detection system of claim 8, the method comprising: The RGB image of the apples to be picked is obtained and input into the apple maturity detection system to obtain the position and maturity of the apples to be picked.

10. A computer-readable storage medium, characterized in that: A computer program is included thereon, and the computer program can be executed by a processor to implement the steps of the method in any one of claims 1 to 7 and 9.

11. An electronic device, characterized in that: include: one or more processors; as well as A memory, wherein the memory is used to store one or more executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1 to 7 and 9 by executing the one or more executable instructions.

Images