A fruit growth morphology recognition method based on image rendering

Abstract

The present invention relates to the technical field of convolutional neural networks, and in particular to a method for identifying fruit growth morphology based on image rendering, comprising collecting fruit images; enhancing image data; constructing a backbone network, and fusing features of the backbone network's network layers through a top-down and bottom-up bidirectional fusion network, thereby extracting features from the image; constructing an RPN network; using point representations to perform image segmentation, first selecting a small number of true value points for prediction; secondly, extracting point-by-point feature representations for each selected point, and finally constructing an MLP prediction classification; and sending the test set image into a trained network model for forward propagation. The present invention builds a feature extraction network based on deep learning and realizes accurate identification of fruit growth morphology based on the idea of image rendering, enabling a picking robot to automatically identify the fruit growth morphology, laying the foundation for further selecting a corresponding picking mechanism for it.

Description

Technical Field

[0001] The present invention relates to the technical field of convolutional neural networks, and in particular to a fruit growth morphology recognition method based on image rendering. Background Art

[0002] The scale and output of fruit cultivation are increasing year by year worldwide. However, most fruit harvesting is currently done manually, which is time-consuming and labor-intensive. This can lead to intractable conflicts in the traditional fruit industry, such as high costs and poor quality. Therefore, the development of highly intelligent harvesting robot technologies has important practical significance and broad application prospects. As a crucial component of harvesting robots, the recognition accuracy and speed of vision systems significantly impact their efficiency. Despite years of research on fruit and vegetable vision systems, few harvesting robots have reached commercial maturity. This is primarily due to the fact that in actual orchard environments, detection of target fruit is easily affected by factors such as complex backgrounds, fruit posture, and color.

[0003] At present, deep learning methods have achieved a high detection rate and a fast detection speed in fruit detection tasks, but most studies on fruit detection classify all target fruits as one category, including fruits that are partially visible but blocked or overlapped by branches. However, it is difficult for current existing picking robot prototypes to pick fruits and vegetables that are blocked or overlapped by branches. If the picking robot forcibly picks fruits that are blocked or overlapped by branches, it may cause damage to the end effector and damage to the fruit, thereby causing unnecessary economic losses. In order to avoid this situation, the visual recognition system carried by the picking robot first needs to identify fruits with different growth forms, and then the robot can select the corresponding method to complete the smooth picking of fruits with different growth forms. In order to obtain refined fruit areas with different growth forms, the present invention uses an instance segmentation method based on image rendering to achieve accurate identification of fruit growth forms. Summary of the Invention

[0004] In response to the shortcomings of existing algorithms: This invention builds a feature extraction network based on Mask RCNN and uses the idea of image rendering to achieve accurate identification of fruit growth morphology, so that the picking robot can automatically identify the fruit growth morphology, laying the foundation for further selecting the corresponding picking mechanism.

[0005] The technical solution adopted by the present invention is: a fruit growth morphology identification method based on image rendering comprises the following steps:

[0006] S1. Image acquisition: A single-lens reflex camera was used to capture images of fruits of different morphologies in several orchards. The remaining images were then annotated using Labelme software. Fruit growth morphologies mainly fall into four categories: single fruit with no branches or stems blocking it, single fruit with branches or stems blocking it, overlapping fruits with no branches or stems blocking it, and overlapping fruits with branches or stems blocking it.

[0007] S2. Image enhancement: Data augmentation is performed on the annotated images to expand the dataset. Data augmentation methods include saturation adjustment, contrast adjustment, brightness adjustment, sharpness adjustment, and clarity adjustment. The images are randomly divided into training set, validation set, and test set in a ratio of 6:2:2.

[0008] S3. Image feature extraction: This part of the task mainly constructs two parts of the network to complete. First, the backbone network is built based on MaskRCNN. The backbone network consists of 49 convolutional layers and 1 maximum pooling downsampling layer. The input image is first subjected to a 7×7 convolution operation with a stride of 2 and a padding of 3. Then it is downsampled by a maximum pooling module to obtain the preprocessed feature map. At the same time, the remaining 48 convolutional layers are divided into four processing stages. Each stage has 3, 4, 6 and 3 groups of convolution modules respectively. Each group contains 3 convolutional layers, namely 1 1×1 convolution, 1 3×3 convolution and 1 1×1 convolution. Finally, a total of 48 convolutional layers are used for feature extraction. Then, an improved spatial pyramid pooling layer (SPPF) is added to the output of the backbone network to convert feature maps of any size into fixed-size feature vectors, so that spatial feature information of different sizes can be extracted, improving the robustness of the model to spatial layout and object deformation.

[0009] Furthermore, SPPF first adjusts the number of channels of the feature map through a 1×1 convolution to obtain the convolution feature map x, and then sends x to three parallel maximum pooling layers with a pooling kernel size of 5, a stride of 1, and a padding of 2. The three outputs obtained are channel-fused with the original x as the final output;

[0010] Then, a top-down and a bottom-up bidirectional fusion network are added to fuse the network layers of the backbone network, so that the network has both deep semantic information and shallow texture information.

[0011] Furthermore, the top-down method is mainly to upsample the small feature maps of the top layer to the same size as the 3-1 stage feature maps of the backbone network. Figure 1 The bottom-up approach mainly involves downsampling the feature map and then fusing it with the feature map of the same size on the top-down path. Finally, in order to enhance the expressiveness of the learned features of the fusion network, an attention mechanism is added after each upsampling and downsampling operation on the two fusion paths.

[0012] Furthermore, the attention mechanism divides the input feature map into two directions, width and height, and performs global average pooling on each direction. Then, the feature maps in the width and height directions of the global receptive field are spliced together and sent to the shared convolution module to adjust the number of channels. The shared convolution module includes a 1×1 convolution kernel, a batch normalization layer, and a nonlinear activation function. The obtained feature map is then further split into two directions, width and height. 1×1 convolution is performed on the original height and width to obtain a feature map with the same number of channels as the original. The attention weights on height and width are obtained through the Sigmoid activation function. Finally, the output is obtained by weighted multiplication calculation on the original feature map.

[0013] S4. Image region recommendation: After obtaining the enhanced features, a region recommendation network (RPN) is built to help the network recommend regions of interest. First, a 3×3 convolution operation is performed on the input feature map, the number of channels is adjusted to 256, and then two full connections are performed to obtain 18 classification results and 36 regression results.

[0014] S5. Image edge rendering: This step mainly draws on the idea of image rendering, and regards image segmentation as an occupancy map of the underlying continuous entity. The segmentation output is obtained based on the "rendering" of the entity. Therefore, point representation is used to solve the image segmentation problem. It mainly includes three parts: selecting a small number of true value points for prediction, extracting point-by-point feature representation for each selected true value point, and building a small neural network (MLP) for prediction and classification.

[0015] Furthermore, a small number of true values are selected separately for training and inference. During inference, we start from the layer with the smallest resolution, upsample by a factor of 2 each time, and then directly select N uncertain points. The point selection strategy during training is mainly divided into three steps. First, K*N points are randomly selected from a uniform distribution. Then, the selected K*N points are calculated by interpolation, and uncertainty estimation is performed. Then, βN points are selected from them. Finally, the remaining (1-β)N points are selected according to a uniform distribution.

[0016] S6. Fruit growth morphology recognition: The test set images are fed into the trained network model for forward propagation, and the regression parameters, category probabilities, and instance masks of the predicted boxes are returned. Finally, the prediction results are saved.

[0017] Beneficial effects of the present invention:

[0018] Based on the idea of image rendering, a fruit growth morphology recognition method is proposed. Starting from image edge detection, the SPPF network and attention mechanism are added to the feature extraction network to obtain more refined target fruit features. Resampling the fruit edge points in the detection module can improve the detection rate of the fruit edge. At the same time, the present invention also enriches the current research technology of agricultural intelligence in the field of fruit growth morphology recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of the fruit growth morphology identification method based on image rendering of the present invention;

[0020] Figure 2 is a model of the feature extraction network of the present invention;

[0021] Figure 3 It is a model of the SPPF structure of the present invention;

[0022] Figure 4 is the model of the attention mechanism of the present invention;

[0023] Figure 5 is a schematic diagram of image edge rendering of the present invention;

[0024] Figure 6 It is the present invention that no branch stem covers a single fruit;

[0025] Figure 7 It is the invention's non-branched stem-shielded overlapping fruit;

[0026] Figure 8 It is the branches and stems of the present invention that shield the single fruit;

[0027] Figure 9 The invention relates to branches and stems shielding overlapping fruits. DETAILED DESCRIPTION

[0028] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore only shows the components related to the present invention.

[0029] like Figure 1 As shown, a fruit growth morphology identification method based on image rendering includes the following steps:

[0030] S1. Image acquisition: Images of fruits of different shapes in several orchards were captured using a SONY CYBERSHOT and a Canon IXUS200IS. After preliminary screening, data with severe occlusion, objects too far away, and blurred images were deleted. The remaining images were then annotated using Labelme software. Samples with insufficient or unclear pixel areas were not labeled to prevent overfitting of the neural network. Targets with an area less than 15% of the image edge were not labeled near the edge of the image because their specific growth forms could not be determined. The final fruit growth forms mainly included four categories: single fruit without stem occlusion, single fruit with stem occlusion, overlapping fruit without stem occlusion, and overlapping fruit with stem occlusion.

[0031] S2. Image enhancement: Since the amount of data after preliminary screening is small and the subsequent neural network cannot learn enough features, data enhancement operations are performed on the labeled data. The main methods used include: saturation adjustment and contrast adjustment to change the color vividness and contrast of the light and dark areas of the image to enhance the generalization ability of the neural network. The saturation is increased by 50% and decreased by 50% respectively, and the contrast enhancement factors are set to 1.5 and 0.5; brightness adjustment and sharpening adjustment are used to enhance the richness of the experimental data. The brightness enhancement factors are set to 1.5 and 0.5 respectively, and the sharpening enhancement factor is set to 3; Gaussian noise and salt and pepper noise are added to imitate the unclear images taken by SLR cameras in real scenes to further enhance the robustness of the detection model. The salt and pepper noise adds 0.05×604×604 noise points, and the standard deviation of the Gaussian noise is set to 0.1. After data enhancement, the data set is expanded to 10,000, and the training set, validation set, and test set are randomly divided according to the ratio of 6:2:2.

[0032] S3, image feature extraction: The network model used in this step is as follows Figure 2 As shown in the figure, it is mainly divided into two parts. The first is the backbone network, which consists of 49 convolutional layers and 1 maximum pooling downsampling layer. The input image is first subjected to a 7×7 convolution operation with a step size of 2 and a padding of 3. Then it is downsampled through a maximum pooling module to obtain the preprocessed feature map. At the same time, the remaining 48 convolutional layers are divided into four processing stages. Each stage has 3, 4, 6 and 3 groups of convolution modules respectively. Each group contains 3 convolutional layers, namely 1 1×1 convolution, 1 3×3 convolution and 1 1×1 convolution. Finally, there are 48 convolutional layers for feature extraction.

[0033] Then, an improved spatial pyramid pooling layer (SPPF) is added to the output of the backbone network to convert feature maps of any size into feature vectors of fixed size, thereby extracting spatial feature information of different sizes and improving the model's robustness to spatial layout and object deformation. The specific structure is as follows: Figure 3 As shown in the figure, the feature map is first subjected to a 1×1 convolution to adjust the number of channels to obtain x, and then x is sent to three parallel maximum pooling layers with a pooling kernel size of 5, a stride of 1, and a padding of 2. The three outputs obtained are fused with the original x as the final output;

[0034] Since the backbone network has been downsampling the feature maps, the low-level feature semantic information is less, but the target position is accurate, and the high-level feature semantic information is richer, but the target position is relatively rough, which will sharply reduce the detection performance of small targets. Therefore, a top-down fusion path is added to improve the detection performance of small targets. This part includes 3 upsampling modules (nearest neighbor upsampling) to adjust the size of the feature map, and 4 horizontal connections (1×1 convolution) to adjust the number of channels for feature fusion; the main operation is to adjust the number of channels through the horizontal connection from the output feature map of the 3rd, 2nd, and 1st stages of the backbone network, and at the same time upsample the feature map output by SPPF, and perform weighted fusion on the feature map of the same size as the horizontal connection output.

[0035] like Figure 2 As shown in Figure 2, for a top-down multi-scale fusion path, the path between high-level features and bottom-level features is long, which increases the difficulty of accessing accurate positioning information. Therefore, a bottom-up path enhancement is added on this basis. The main operation is to downsample the output of the last layer from the top to the bottom, including 3 downsampling and 3 feature fusions, which is consistent with the fusion strategy of the previous step. At the same time, in order to enhance the expression ability of the fusion network learning features, an attention mechanism is added to the two fusion paths (such as Figure 4 Attention mechanism model), added after upsampling and downsampling operations, the input feature map is divided into width and height directions, and global average pooling is performed respectively. Then, the feature maps of the width and height directions of the global receptive field are spliced together and sent to the shared convolution module to adjust the number of channels. The module includes a 1×1 convolution kernel, a batch normalization layer and a nonlinear activation function. Then, the obtained feature map is further split into width and height directions. According to the original height and width, 1×1 convolution is performed respectively to obtain the feature map with the same number of channels as the original. After the Sigmoid activation function, the attention weights on height and width are obtained. Finally, the output is obtained by weighted multiplication calculation on the original feature map. The specific structure is as follows Figure 4 shown.

[0036] S4. Image Region Recommendation: After obtaining the enhanced features, a Region Recommendation Network (RPN) is built to help the network recommend regions of interest. First, a 3×3 convolution operation is performed on the input feature map, adjusting the number of channels to 256. Then, two full connections are performed to obtain 18 classification results and 36 regression results.

[0037] S5. Image edge detection: This step mainly draws on the idea of image rendering, and regards image segmentation as an occupancy map of the underlying continuous entity. The segmentation output is obtained based on the "rendering" of the entity. The entity is regarded as a network characteristic map, and the interpolation method is used to evaluate it. The parameters are trained using this method, and predictions are made based on its characteristics. In computer graphics, these parameterized functions are equivalent to physics and geometry. Based on this, point representation is used to solve the image segmentation problem. It mainly includes three parts: selecting a small number of true value points for prediction, extracting point-by-point feature representation for each selected point, and building a small neural network for prediction and classification; a small number of true value points are operated separately in training and inference During inference, start from the layer with the smallest resolution, upsample by 2 times each time, and then directly select N uncertain points. The point selection strategy during training is mainly divided into three steps. First, randomly select K*N points from the uniform distribution, then calculate the selected K*N points by interpolation, perform uncertainty estimation, and then select βN points from them. Finally, the remaining (1-β)N points are selected according to the uniform distribution. Then, the features of the selected uncertain points are calculated. The features of each point include the fine-grained features extracted from the backbone network and a rough segmentation result calculated by the network. These two parts are spliced to form the features of a point, which are input to the subsequent neural network for prediction. The module architecture diagram of this part is as follows: Figure 5 shown.

[0038] S6. Fruit growth morphology recognition: Send the test set to the trained model for forward propagation, return the regression parameters, category probability and instance mask of the prediction box, and finally save the prediction results. The effect is as follows Figure 6-9 shown.

[0039] The present invention proposes a fruit growth morphology recognition method based on the idea of image rendering. The method can obtain more accurate recognition results and faster recognition speed. In addition, the present invention also starts from image edge detection to obtain more precise target positions, enriching the current research technology of agricultural intelligence in the field of fruit growth morphology recognition.

[0040] With the above-described preferred embodiments of the present invention as a guide, and with reference to the above description, relevant personnel are fully capable of making various changes and modifications without departing from the technical scope of this invention. The technical scope of this invention is not limited to the contents of the specification and must be determined according to the scope of the claims.

Claims

1. A fruit growth morphology identification method based on image rendering, characterized in that: The following steps are involved: S1, collect fruit images and mark them; S2, perform data enhancement on the annotated image by adjusting saturation, contrast, brightness, sharpness, and clarity; S3. Build a backbone network; and fuse the features of the backbone network layer through a top-down and bottom-up bidirectional fusion network to extract features from the image; From top to bottom, the top-level small feature maps are enlarged to the same size as the feature maps of the 3rd, 2nd, and 1st stages of the backbone network by upsampling, and then fused by addition; From top to bottom, it includes three upsampling modules to adjust the size of the feature map, and four 1×1 convolutional horizontal connections to adjust the number of channels for feature fusion. The feature maps output from the 3rd, 2nd, and 1st stages of the backbone network are horizontally connected to adjust the number of channels, and the feature maps output by SPPF are upsampled and weighted fused with the feature maps of the same size output by the horizontal connections. The bottom-up path is to downsample the feature map and then fuse it with the feature map of the same size in the top-down path; Bottom-up includes 3 downsampling and 3 feature fusions; At the same time, an attention mechanism is added after each upsampling and downsampling operation on the two fusion paths; S4. Build an RPN network to recommend regions of interest based on image features; S5. Use point representation to perform image segmentation. First, select a small number of true value points for prediction. During inference, start from the layer with the smallest resolution, upsample by 2 times each time, and then directly select N uncertain points. The strategy for selecting points during training is mainly divided into three steps: first, randomly select K*N points from a uniform distribution, then calculate the selected K*N points by interpolation, perform uncertainty estimation, and then select β N points, the remaining (1- β ) N points are selected in a uniformly distributed manner; Secondly, extract point-by-point feature representation for each selected point, and finally construct an MLP prediction classification; S6. Send the test set image to the trained network model and perform forward propagation, return the regression parameters, category probability and instance mask of the predicted box, and finally save the prediction result.

2. The fruit growth morphology identification method based on image rendering according to claim 1, wherein: The backbone network consists of 49 convolutional layers and 1 maximum pooling downsampling layer; First, the input image undergoes a 7×7 convolution operation with a stride of 2 and a padding of 3. Then, it is downsampled through a maximum pooling module to obtain the preprocessed feature map. Secondly, the remaining 48 convolutional layers are divided into four processing stages, each of which contains 3, 4, 6, and 3 groups of convolutional modules respectively. Each group contains 3 convolutional layers, namely 1 1×1 convolution, 1 3×3 convolution, and 1 1×1 convolution. Finally, add SPPF at the output of the backbone network.

3. The fruit growth morphology identification method based on image rendering according to claim 1, wherein: The SPPF is to adjust the feature map through a 1×1 convolution channel to obtain a convolution feature map; Then it is sent to three parallel maximum pooling layers with a pooling kernel size of 5, a stride of 1, and a padding of 2 for processing; the final three outputs are channel-fused with the convolution feature map as the final output.

4. The fruit growth morphology identification method based on image rendering according to claim 1, wherein: The attention mechanism divides the input feature map into two directions, width and height, and performs global average pooling on each direction. Then, the feature maps in the width and height directions of the global receptive field are concatenated and sent to the shared convolution module to adjust the number of channels. Then the obtained feature map is further split into width and height directions, and 1×1 convolution is performed according to the original height and width to obtain a feature map with the same number of channels as the original; The attention weights on height and width are then obtained through the Sigmoid activation function, and finally the output is obtained through multiplication weighted calculation on the original feature map.

5. The fruit growth morphology identification method based on image rendering according to claim 4, wherein: The shared convolution module includes a 1×1 convolution kernel, a batch normalization layer and a nonlinear activation function.

6. The fruit growth morphology identification method based on image rendering according to claim 1, wherein: The construction of the RPN network includes: First, a 3×3 convolution operation is performed on the input feature map, and the number of channels is adjusted to 256; Then, after two full connections, 18 classification results and 36 regression results were obtained.

Images