A fruit picking prediction method based on visual semantic segmentation

Abstract

The present application relates to a fruit picking prediction method based on semantic segmentation. The present application specifically includes fruit image preprocessing, image input main network prediction, accurate segmentation image, extraction of accurate segmentation prediction map fruit area block, identification module and fruit maturity prediction module; wherein, the fine adjustment module is based on two branches of convolution attention and Transformer-based attention, the feature map is processed to output the prediction feature map, and the prediction feature maps output by the two branches are fused, and then a layer of convolution layer, a layer of normalization and a layer of nonlinear activation layer are used, and finally a layer of convolution layer and an argmax function are used to output the accurate segmentation prediction map. The present application can not only accurately identify the fruit, but also classify the maturity of the fruit, improve the fruit picking efficiency, and also reduce the artificial picking cost.

Description

Technical Field

[0001] This invention belongs to the field of agricultural robot visual bionic technology, specifically relating to a fruit picking prediction method based on visual semantic segmentation. Background Technology

[0002] With the rise of smart agriculture, the application of smart agriculture prediction methods is increasing. Fruit picking robots are an emerging field in smart agriculture, and how to improve the efficiency and quality of fruit picking has always been a research focus in this field. Currently, there are two main picking methods for vision-based fruit picking robots: (1) Using a camera to continuously take pictures from multiple random directions until the fruit target is found, and then controlling the robot to pick it; however, this approach will increase the uncertainty of the robot in the environment, waste a lot of repetitive actions, and result in low efficiency. (2) Using a camera to take pictures from a distance, detecting all the fruit targets in the current scene at once, and then controlling the robot to pick them one by one; however, this approach raises a problem, namely, how to determine which fruit the robot picks first and which fruit to pick later. To address this problem, there are two main picking methods: one is to pick in the order from left to right and from top to bottom, but this does not take into account the distance between the fruit and the robot, and cannot ensure efficiency; the other is to plan the shortest path for the entire picking process to reduce the robot's consumption and improve the robot's picking efficiency. However, all of the above fruit harvesting methods neglect the quality of the harvested fruit, leading to inconsistent quality and increasing the burden of subsequent fruit sorting, resulting in high labor costs and directly impacting the overall income of fruit farmers. Therefore, designing a better fruit harvesting prediction method that simultaneously considers both the efficiency and quality of robotic harvesting is an urgent problem to be solved. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of existing technologies. This invention provides a fruit harvesting prediction method based on visual semantic segmentation, which accurately segments fruit images, then fuses them to output high-precision fruit classifications, and then classifies the fruit maturity levels to harvest the fruits with high maturity, thereby improving the quality and efficiency of fruit harvesting and reducing the cost of manual harvesting.

[0004] To achieve the above and other related objectives, the present invention provides the following technical solution: a fruit harvesting prediction method based on visual semantic segmentation, comprising: Step S1: Fruit image preprocessing: Read out the image, and during the training phase, randomly crop, flip, and shift the image to increase the amount of data. Then fix the image size to three dimensions: 256*128, 512*216, and 1024*512. Finally, scale the image to a resolution of 256*128. Step S2: Image input to backbone network processing: Input the above 256*128 resolution image into the HRNet+OCR backbone network and output a feature map with feature size (12, 256, 128); Step S3: Accurate image segmentation: The feature map from step S2 is segmented using a segmentation network based on convolutional attention mechanism and Transformer attention mechanism, and a predicted feature map is output. The predicted feature maps output from the two branches are then fused, passing through a convolutional layer, a normalization layer, and a non-linear activation layer. Finally, an accurate segmentation prediction map is output through a convolutional layer and an argmax function. Step S4: Extract the patch of the fruit region in the accurate segmentation prediction image: Generate the foreground rectangle corresponding to the fruit position based on the fruit segmentation image mask, and retain the position information in the box. Output the patch corresponding to the position of the fruit in the original image based on the position information in the box. Step S5: Recognition: Input the image patch into the fruit recognition module. The recognition algorithm performs strong classification of the fruit category to eliminate the prediction error of the segmentation algorithm for the fruit category. Based on the image patch of the fruit location, it is unified to a fixed size and input into a recognition network with five layers of convolutional layers and pooling layers. Then, it goes through two fully connected layers and a softmax layer to obtain the fruit category result. Step S6: Fruit maturity prediction: Based on the fruit category results obtained above, after passing through two fully connected layers, the fruit maturity prediction is output using a softmax layer; the fruit maturity prediction module has two fully connected layers and a softmax layer.

[0005] Furthermore, the attention-based branch involves inputting the coarse prediction map, which generates a channel-based attention feature vector (12, 1, 1). This vector is then multiplied by the coarse prediction map to obtain a new prediction feature map. The Transformer-based attention branch involves inputting the coarse prediction map, which is first divided into 16*16 (12, 16, 8) matrix blocks, then flattened into a (16*16, 16*8*12) matrix and input into the Transformer attention module. After passing through this module, the output is restored to the same size as the coarse prediction map, which is (12, 256, 128).

[0006] Furthermore, the recognition algorithm employs the BRISK algorithm and includes the following steps: Step S51: Obtain the original point cloud data of the fruit image to be processed; Step S52: Spatial division is performed by sequentially superimposing the original point cloud data using voxel meshes of various sizes. The point cloud layer formed by the previous spatial division is used as the point cloud layer to be divided in the next division, thus constructing a topological structure composed of multiple point cloud layers superimposed one by one. Step S53: Based on the first point cloud deep learning network and the topology, perform feature extraction on the original point cloud data to obtain fruit category features.

[0007] Furthermore, in step S6: maturity is divided into three categories; the first category: fruit maturity is less than 30%, and the fruit maturity is assigned the value "NotFruit"; the second category: fruit maturity is 30%-70%, and the fruit maturity is assigned the value "NotFruit"; the third category: fruit maturity is greater than 70%, and the fruit maturity is assigned the value "Fruit"; when the conditions are not met, the fruit maturity is assigned the value of an empty character.

[0008] Furthermore, based on the maturity classification, when the fruit maturity is "Fruit", it proceeds to the harvesting process; otherwise, it does not proceed.

[0009] Furthermore, the present invention also provides a system for a fruit picking prediction method based on visual semantic segmentation, including a picking robot, an end effector, and a cloud server. The picking robot is equipped with an end effector, and the cloud server is communicatively connected to the picking robot. The system is used to implement the aforementioned fruit picking prediction method based on visual semantic segmentation.

[0010] Furthermore, the present invention also provides a computer-readable storage medium storing one or more programs, wherein the computer-readable storage medium stores one or more computer programs, which, when executed by a processor, perform the aforementioned fruit picking prediction method based on visual semantic segmentation.

[0011] The present invention has the following beneficial effects: 1. This invention addresses the problem of fruit image segmentation in an orchard setting. The method proposed in this patent can effectively segment the fruit portion of the image, outperforming other algorithms and showing a significant improvement over traditional segmentation methods. 2. This invention proposes a novel segmentation fine-tuning module. This module is not only applicable to the HRNet+OCR backbone network, but can also be integrated into other backbone networks, as long as the backbone network can output feature maps containing image information. Simultaneously, this module applies two different forms of attention to the coarse prediction map, thereby obtaining a more accurate prediction map and improving the model's performance. 3. This invention proposes a segmentation-based target recognition method, which uses the segmentation network to extract accurate target foreground patches from the segmented image for recognition, which helps to improve the recognition accuracy and can use the recognition network to eliminate errors in the prediction results of the segmentation network; 4. This invention proposes a maturity prediction method based on convolutional networks. It utilizes the feature maps obtained from the convolutional network portion of the recognition network to access a fully connected network to predict fruit maturity. Furthermore, it can train a maturity module for specific fruits to obtain model parameters for those fruits, enabling it to achieve good results in predicting the maturity of different fruits.

[0012] 5. This invention proposes an efficient method for identifying specific fruits and predicting their maturity. In this patent, the fruit identification module and the fruit maturity prediction module are used in a serial manner. Maturity prediction is only performed on fruits that need to be picked, while maturity prediction is not performed on other fruits that do not need to be picked. This reduces the computational load of the model and improves the efficiency of operation.

[0013] 6. This invention can not only accurately identify fruits, but also classify the ripeness of fruits, thereby improving fruit picking efficiency and reducing manual picking costs. Attached Figure Description

[0014] Figure 1 This is a flowchart of a fruit picking prediction method based on visual semantic segmentation according to the present invention.

[0015] Figure 2 This is a schematic diagram of the operation of the fine-tuning module of the present invention. Detailed Implementation

[0016] The technical solutions in the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example: Please refer to Figure 1 A fruit picking prediction method based on visual semantic segmentation includes the following steps: Step S1: Fruit image preprocessing: Read out the image, and during the training phase, randomly crop, flip, and shift the image to increase the amount of data. Then fix the image size to three dimensions: 256*128, 512*216, and 1024*512. Finally, scale the image to a resolution of 256*128. Step S2: Image input to backbone network processing: Input the above 256*128 resolution image into the HRNet+OCR backbone network and output a feature map with feature size (12, 256, 128); Step S3: Accurate image segmentation: The feature map from step S2 is segmented using a segmentation network based on convolutional attention mechanism and Transformer attention mechanism, and a predicted feature map is output. The predicted feature maps output from the two branches are then fused, passing through a convolutional layer, a normalization layer, and a non-linear activation layer. Finally, an accurate segmentation prediction map is output through a convolutional layer and an argmax function. Step S4: Extract the patch of the fruit region in the accurate segmentation prediction image: Generate the foreground rectangle corresponding to the fruit position based on the fruit segmentation image mask, and retain the position information in the box. Output the patch corresponding to the position of the fruit in the original image based on the position information in the box. Step S5: Recognition: Input the image patch into the fruit recognition module. The recognition algorithm performs strong classification of the fruit category to eliminate the prediction error of the segmentation algorithm for the fruit category. Based on the image patch of the fruit location, it is unified to a fixed size and input into a recognition network with five layers of convolutional layers and pooling layers. Then, it goes through two fully connected layers and a softmax layer to obtain the fruit category result. Step S6: Fruit maturity prediction: Based on the fruit category results obtained above, after passing through two fully connected layers, the fruit maturity prediction is output using a softmax layer; the fruit maturity prediction module has two fully connected layers and a softmax layer.

[0018] Furthermore, the attention-based branch involves inputting the coarse prediction map, which generates a channel-based attention feature vector (12,1,1). This vector is then multiplied by the coarse prediction map to obtain a new prediction feature map. The Transformer-based attention branch involves inputting the coarse prediction map, which is first divided into 16*16 (12,16,8) matrix blocks, then flattened into a (16*16,16*8*12) matrix, and then input into the Transformer attention module. After passing through this module, the output is restored to the same size as the coarse prediction map, which is (12,256,128).

[0019] Furthermore, the recognition algorithm employs the BRISK algorithm and includes the following steps: Step S51: Obtain the original point cloud data of the fruit image to be processed; Step S52: Spatial division is performed by sequentially superimposing the original point cloud data using voxel meshes of various sizes. The point cloud layer formed by the previous spatial division is used as the point cloud layer to be divided in the next division, thus constructing a topological structure composed of multiple point cloud layers superimposed one by one. Step S53: Based on the first point cloud deep learning network and the topology, perform feature extraction on the original point cloud data to obtain fruit category features.

[0020] Furthermore, in step S6: maturity is divided into three categories; the first category: fruit maturity is less than 30%, and the fruit maturity is assigned the value "NotFruit"; the second category: fruit maturity is 30%-70%, and the fruit maturity is assigned the value "NotFruit"; the third category: fruit maturity is greater than 70%, and the fruit maturity is assigned the value "Fruit"; when the conditions are not met, the fruit maturity is assigned the value of an empty character.

[0021] Furthermore, based on the maturity classification, when the fruit maturity is "Fruit", it proceeds to the harvesting process; otherwise, it does not proceed.

[0022] Furthermore, the present invention also provides a system for a fruit picking prediction method based on visual semantic segmentation, including a picking robot, an end effector, and a cloud server. The picking robot is equipped with an end effector, and the cloud server is communicatively connected to the picking robot. The system is used to implement the aforementioned fruit picking prediction method based on visual semantic segmentation.

[0023] Furthermore, the present invention also provides a computer-readable storage medium storing one or more programs, wherein the computer-readable storage medium stores one or more computer programs, which, when executed by a processor, perform the aforementioned fruit picking prediction method based on visual semantic segmentation.

[0024] For details, please refer to Figure 2 First, image data was collected using network tools and cameras to capture images of fruit in the orchard background, which were then categorized and organized according to fruit type and maturity. Second, Labelimage was used to annotate the images, creating annotation files, and the dataset was split (7:3 ratio for training and testing). Next, the images and annotation files from the training set were input into a pre-designed network model for training. After training, the model's weight file was obtained. During the testing phase, by loading the weight file into the model, the model could segment, identify, and predict the maturity of images in the test set.

[0025] During model training, we used an HRNet+OCR model pre-trained on the ImageNet dataset. The network layer weights for our proposed fine-tuning module, as well as the recognition and maturity prediction modules, were randomly initialized using a Gaussian distribution. During training, we fixed the parameters of the backbone network model while training the fine-tuning module. The learning rate was set to 0.001 for the first 30 epochs of training, followed by 0.0001 for the next 20 epochs. All experiments were performed using the PyTorch framework, and our model ran on an Ubuntu 16.04 prediction method and a Tesla V100.

[0026] In summary, this invention can not only accurately identify fruits, but also classify their ripeness, thereby improving fruit harvesting efficiency and reducing manual harvesting costs.

[0027] The above description is merely an embodiment of the present invention, but the implementation of the present invention is not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention are equivalent substitutions and are included within the protection scope of the present invention.

Claims

1. A fruit harvesting prediction method based on visual semantic segmentation, characterized in that, include: Step S1: Fruit image preprocessing: Read out the image, and during the training phase, randomly crop, flip, and shift the image to increase the amount of data. Then fix the image size to three dimensions: 256*128, 512*216, and 1024*512. Finally, scale the image to a resolution of 256*128. Step S2: Image input to backbone network processing: Input the above 256*128 resolution image into the HRNet+OCR backbone network and output a feature map with feature size (12, 256, 128); Step S3: Accurate image segmentation: The feature map from step S2 is segmented using a segmentation network based on convolutional attention mechanism and Transformer attention mechanism, and a predicted feature map is output. The predicted feature maps output from the two branches are then fused, passing through a convolutional layer, a normalization layer, and a non-linear activation layer. Finally, an accurate segmentation prediction map is output through a convolutional layer and an argmax function. Step S4: Extract the patch of the fruit region in the accurate segmentation prediction image: Generate the foreground rectangle corresponding to the fruit position based on the fruit segmentation image mask, and retain the position information in the box. Output the patch corresponding to the position of the fruit in the original image based on the position information in the box. Step S5: Recognition: Input the image patch into the fruit recognition module. The recognition algorithm performs strong classification of the fruit category to eliminate the prediction error of the segmentation algorithm for the fruit category. Based on the image patch of the fruit location, it is unified to a fixed size and input into a recognition network with five layers of convolutional layers and pooling layers. Then, it goes through two fully connected layers and a softmax layer to obtain the fruit category result. Step S6: Fruit maturity prediction: Based on the fruit category results obtained above, after passing through two fully connected layers, the fruit maturity prediction is output using a softmax layer; the fruit maturity prediction module has two fully connected layers and a softmax layer.

2. The fruit picking prediction method based on visual semantic segmentation according to claim 1, characterized in that: The attention-based branch takes the coarse prediction map as input and generates a channel-based attention feature vector (12, 1, 1). This vector is then multiplied by the coarse prediction map to obtain a new prediction feature map. The Transformer-based attention branch takes the coarse prediction map as input, divides it into 16*16 (12, 16, 8) matrix blocks, flattens it into a (16*16, 16*8*12) matrix, and inputs it into the Transformer attention module. After passing through this module, the output is restored to the same size as the coarse prediction map, which is (12, 256, 128).

3. The fruit picking prediction method based on visual semantic segmentation according to claim 1, characterized in that: The recognition algorithm uses the BRISK algorithm and includes the following steps: Step S51: Obtain the original point cloud data of the fruit image to be processed; Step S52: Spatial division is performed by sequentially superimposing the original point cloud data using voxel meshes of various sizes. The point cloud layer formed by the previous spatial division is used as the point cloud layer to be divided in the next division, thus constructing a topological structure composed of multiple point cloud layers superimposed one by one. Step S53: Based on the first point cloud deep learning network and the topology, feature extraction is performed on the original point cloud data to obtain fruit category features.

4. The fruit harvesting prediction method based on visual semantic segmentation according to claim 1, characterized in that, In step S6: maturity is divided into three categories; the first category: fruit maturity is less than 30%, and the fruit maturity value is assigned to "NotFruit"; the second category: fruit maturity is 30%-70%, and the fruit maturity value is assigned to "NotFruit"; the third category: fruit maturity is greater than 70%, and the fruit maturity value is assigned to "Fruit"; when the conditions are not met, the fruit maturity value is assigned to an empty character.

5. The fruit harvesting prediction method based on visual semantic segmentation according to claim 4, characterized in that: Based on the maturity classification, when the fruit maturity is "Fruit", it proceeds to the harvesting process; otherwise, it does not proceed.

6. The fruit harvesting prediction method based on visual semantic segmentation according to claim 1, characterized in that: In step S6: the fruit identification module and the fruit maturity prediction module are used in a serial manner, and maturity prediction is only performed for fruits that need to be picked.

7. The fruit harvesting prediction method based on visual semantic segmentation according to claim 1, characterized in that: The segmentation network outputs an accurate segmentation prediction map using a segmentation fine-tuning module, which includes two branches: a convolutional attention mechanism and a Transformer-based attention mechanism.

8. A system for predicting fruit harvesting based on visual semantic segmentation as described in any one of claims 1-7, characterized in that: The system includes a harvesting robot, an end effector, and a cloud server. The harvesting robot is equipped with an end effector, and the cloud server is communicatively connected to the harvesting robot. The system is used to implement the fruit harvesting prediction method based on visual semantic segmentation as described in any one of claims 1-7.

9. A computer-readable storage medium for storing one or more programs, characterized in that, The computer-readable storage medium stores one or more computer programs, which, when executed by a processor, perform the fruit picking prediction method based on visual semantic segmentation as described in any one of claims 1 to 7.

Images