Agricultural robot arm grasping control method and system fusing deep learning vision

Abstract

The present application relates to the technical field of deep learning, in particular to an agricultural robot arm grabbing control method and system fusing deep learning vision, comprising: collecting a data set; based on the data set, obtaining a reserved occlusion image and its corresponding adjustment image; taking the reserved occlusion image and its corresponding adjustment image as an image control group, and training a self-encoding network by using the image control group to obtain a trained self-encoding network; freezing the network weight of the encoder, and fusing the feature map output by the encoder with the feature map of the corresponding level in the YOLO backbone network to obtain a fused network; training the fused network by using the data set to obtain a trained fruit recognition model; identifying the real-time fruit image collected by the robot arm based on the trained fruit recognition model, and controlling the robot arm to perform a grabbing operation. The present application can improve the recognition accuracy and robustness of fruits in a complex agricultural scene with serious branch and leaf occlusion.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and more specifically to an agricultural robotic arm grasping control method and system that integrates deep learning vision. Background Technology

[0002] In the process of fruit grasping using a deep learning-based vision-controlled robotic arm, accurate fruit identification is the primary step towards automated harvesting. Current technologies commonly employ object detection networks such as YOLO to identify fruits and output their bounding boxes. However, in real-world agricultural scenarios like orchards, fruit trees are densely covered with foliage, and fruits are often partially obscured by leaves and branches, revealing only a small portion. Existing YOLO networks are mostly trained on datasets collected under unobstructed or minimally obstructed conditions. This results in a significant drop in recognition performance when faced with highly occluded fruits in real-world scenarios, leading to missed detections and consequently affecting the completeness of the robotic arm's grasping process and operational efficiency. Summary of the Invention

[0003] This invention provides an agricultural robotic arm grasping control method and system that integrates deep learning vision to solve existing problems.

[0004] The agricultural robotic arm grasping control method and system integrating deep learning vision of the present invention adopts the following technical solution:

[0005] One embodiment of the present invention provides an agricultural robotic arm grasping control method integrating deep learning vision, the method comprising the following steps:

[0006] Collect a dataset; the dataset consists of original images containing the fruits of the target fruit tree variety.

[0007] Based on the dataset, obtain the preserved occluded image and its corresponding adjusted image;

[0008] The occluded image and its corresponding adjusted image are retained as an image control group, and the autoencoder network is trained using the image control group to obtain the trained autoencoder network.

[0009] The network weights of the encoder in the trained autoencoder network are frozen, and the feature map output by the encoder with frozen weights is fused with the feature map of the corresponding layer in the YOLO backbone network to obtain the fused network.

[0010] The fused network was trained using the dataset to obtain a trained fruit recognition model;

[0011] The robot arm recognizes real-time fruit images captured by the trained fruit recognition model and controls the robot arm to perform grasping operations.

[0012] Furthermore, the specific steps for obtaining the preserved occluded image and its corresponding adjusted image based on the dataset are as follows:

[0013] The original image is annotated to obtain the bounding box of each fruit in the original image;

[0014] Using the bounding box of each fruit in the original image, filter out the adjusted images from all the original images;

[0015] Based on the connected components of leaves that intersect with the bounding boxes of each fruit in the adjusted image, and the area ratio of each fruit, the occluded fruits in the adjusted image are obtained.

[0016] By adjusting the image and finding the occluded fruit, obtain the corresponding occluded image.

[0017] Filter all occluded images and select the ones that are retained.

[0018] Furthermore, the specific steps involved in selecting adjusted images from all original images using the bounding boxes of each fruit in the original images are as follows:

[0019] The ratio of the fruit area to the bounding box area of ​​each fruit in the original image is used to determine the area percentage of each fruit.

[0020] Fruits with an area ratio greater than a preset area threshold are identified as prominent fruits in the original image.

[0021] Calculate the ratio of the number of obvious fruits to the total number of fruits in the original image, and determine the original image with a ratio greater than a preset threshold as the adjusted image.

[0022] Furthermore, the specific steps for obtaining occluded fruits in the adjusted image based on the connected regions of leaves that intersect with the bounding boxes of each fruit in the adjusted image, and the area proportion of each fruit, are as follows:

[0023] The number of leaf connected components that intersect with the bounding box of each fruit in the adjusted image is obtained, and the number of leaf connected components is normalized to obtain the leaf connected component value of each fruit in the adjusted image.

[0024] Calculate the ratio of the leaf connected component value to the area ratio of each fruit in the adjusted image, and normalize the ratio to obtain the occlusion index of each fruit in the adjusted image.

[0025] Sort the occlusion indices of all fruits in the adjusted image in ascending order to obtain an ascending sequence;

[0026] The occlusion index threshold is obtained by thresholding the ascending sequence;

[0027] Fruits with an occlusion index greater than the occlusion index threshold are identified as occluded fruits in the adjusted image.

[0028] Furthermore, the specific steps for obtaining the occluded image corresponding to the adjusted image by utilizing the occluded fruit in the adjusted image are as follows:

[0029] The leaf closest to the occluded fruit in the adjusted image is taken as the target leaf, and the image of the target leaf is superimposed on the occluded fruit area to obtain the initial occlusion image corresponding to the adjusted image;

[0030] Image enhancement processing is performed on the initial occluded image corresponding to the adjusted image to obtain the occluded image corresponding to the adjusted image.

[0031] Furthermore, the specific steps for filtering all occluded images to obtain the retained occluded images are as follows:

[0032] The occluded image is input into the trained discrimination model to obtain the score corresponding to each occluded image;

[0033] The images with the highest scores and a predetermined percentage of occluded images are selected as the retained occluded images.

[0034] Furthermore, the trained discrimination model is obtained as follows:

[0035] Randomly select several original images and generate a composite image with the same number of original images;

[0036] Each original image is labeled with 1, and each synthetic image is labeled with 0. The labeled original images and synthetic images are used as the training dataset.

[0037] The CNN classification model is trained using the training dataset to obtain a well-trained discriminant model.

[0038] Furthermore, the specific steps involved in training the autoencoder network using an image control group to obtain a trained autoencoder network are as follows:

[0039] The occluded image in each image control group is used as input, and the mean square error between the adjusted image and the occluded image in the image control group is used as the loss function to train the autoencoder network, resulting in a trained autoencoder network.

[0040] Furthermore, the specific steps for fusing the feature map output by the encoder after freezing the weights with the feature map of the corresponding layer in the YOLO backbone network to obtain the fused network are as follows:

[0041] The feature map output by the encoder after freezing the weights is adjusted using a 1×1 convolution so that it has the same number of channels n as the feature map of the corresponding layer of the YOLO backbone network.

[0042] The adjusted encoder feature map is concatenated with the feature map of the corresponding layer of the YOLO backbone network in the channel dimension to obtain a fused feature map with 2n channels.

[0043] The number of channels in the fused feature map is compressed from 2n back to n using a 1×1 convolution to obtain an enhanced feature map, which is then input into subsequent layers of the YOLO backbone network to obtain the fused network.

[0044] One embodiment of the present invention provides an agricultural robotic arm grasping control system that integrates deep learning vision, the system comprising the following modules:

[0045] The acquisition module is used to acquire datasets; the datasets consist of original images containing the fruits of the target fruit tree variety.

[0046] The fusion module is used to obtain the preserved occluded image and its corresponding adjusted image based on the dataset;

[0047] The occluded image and its corresponding adjusted image are retained as an image control group, and the autoencoder network is trained using the image control group to obtain the trained autoencoder network.

[0048] The network weights of the encoder in the trained autoencoder network are frozen, and the feature map output by the encoder with frozen weights is fused with the feature map of the corresponding layer in the YOLO backbone network to obtain the fused network.

[0049] The fused network was trained using the dataset to obtain a trained fruit recognition model;

[0050] The recognition module is used to recognize real-time fruit images collected by the robotic arm based on a trained fruit recognition model, and to control the robotic arm to perform grasping operations.

[0051] The beneficial effects of the technical solution of the present invention are as follows: The embodiments of the present invention propose an agricultural robotic arm grasping control method and system that integrates deep learning vision. By first training an autoencoder network with high reconstruction capability for occluded fruits, and then fusing the robust features extracted by its encoder with the features of the YOLO network, the accuracy and robustness of fruit recognition in complex agricultural scenarios with severe occlusion of branches and leaves are significantly improved. This effectively overcomes the problem of missed detection that easily occurs when the fruit is partially occluded in the existing methods, and ensures the integrity and efficiency of the robotic arm grasping. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating the steps of the agricultural robotic arm grasping control method integrating deep learning vision according to the present invention.

[0054] Figure 2 This is a schematic diagram of the YOLO network structure in this invention;

[0055] Figure 3 This is a schematic diagram of the feature map fusion process in this invention;

[0056] Figure 4 This is a block diagram of the agricultural robotic arm grasping control system that integrates deep learning vision according to the present invention. Detailed Implementation

[0057] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of the agricultural robotic arm grasping control method and system integrating deep learning vision proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0059] The following description, in conjunction with the accompanying drawings, details the specific scheme of the agricultural robotic arm grasping control method and system that integrates deep learning vision provided by this invention.

[0060] Please see Figure 1 The diagram illustrates a flowchart of a method for grasping and controlling an agricultural robotic arm that integrates deep learning vision, according to an embodiment of the present invention. The method includes the following steps:

[0061] Step S001: Collect a dataset; wherein, the dataset consists of original images containing the fruit of the target fruit tree variety.

[0062] It should be noted that this embodiment relates to a visual recognition system for agricultural robotic arms. Its core is to train a deep learning detection network based on the YOLO architecture to achieve accurate recognition and positioning of fruit targets in real-time scene images captured by the robotic arm's camera.

[0063] To train the YOLO network, a high-quality fruit image dataset must first be constructed. Specifically, raw images containing fruit should be systematically collected in real agricultural settings (such as orchards and greenhouses) for the target fruit tree variety. The collection process should cover different lighting conditions, different fruit growth stages, and different shooting angles and distances to ensure the diversity and representativeness of the data.

[0064] The target fruit tree variety refers to the variety of fruit tree grasped by the robotic arm. For example: using a high-definition color camera or RGB-D camera mounted on the robotic arm or mobile platform to acquire images; during the acquisition process, covering different positions of the fruit tree along a preset path or random path to ensure that multi-view images of the fruit are obtained; repeatedly acquiring images under different weather conditions and time of day (such as morning, noon, and evening) to cover changes in light intensity.

[0065] Step S002: Based on the dataset, obtain the occluded image and its corresponding adjusted image.

[0066] It should be noted that this embodiment aims to improve the YOLO network's ability to recognize fruits occluded by branches and leaves. Here, occlusion refers to branches and leaves partially obscuring the surface of the fruit, not completely obscuring it. Directly training a YOLO network robust to occlusion would require a large number of diverse images of occluded fruits and their precise bounding box annotations. In agricultural scenarios, systematically collecting and annotating fruits with various occlusion conditions (such as occlusion of different degrees, shapes, and locations) is extremely costly and difficult to cover all real-world occlusion patterns. Directly trained YOLO networks tend to "memorize" occlusion patterns: if occlusion data is insufficient or lacks diversity, a directly trained YOLO network may simply memorize a limited number of occlusion patterns from the training set, failing to generalize to unseen occlusion situations. It learns "what a fruit with specific occlusion looks like," rather than "what the essential visual features of the fruit are."

[0067] Therefore, this embodiment first trains an autoencoder network specifically for fruit image reconstruction. Since the decoder part of the autoencoder network inherently possesses the ability to "repair" a complete image from low-dimensional features, its encoder part, in order to cooperate with the decoder to achieve this goal, is forced to learn to extract robust high-level features from occluded or incomplete input images that can characterize the essential structure, contour, and texture of the fruit. These features can effectively filter out occlusion interference and capture key local information about the fruit.

[0068] Based on this, this embodiment needs to incorporate the information obtained by the autoencoder network into the YOLO backbone network. To avoid the weight transfer problem caused by differences in network structure (where the existing autoencoder network gradually decreases the number of hidden layers from beginning to end, while the YOLO network gradually increases the number of hidden layers from beginning to end), and to more fully utilize the feature representation capabilities learned by the autoencoder network, this embodiment adopts an embedded feature enhancement strategy. Specifically, the encoder part of the fully trained autoencoder network is used as an independent feature extraction module, and its weights are frozen. The input of this module receives the same occluded scene image as the YOLO backbone network, and its output is adjusted in dimension by an adaptation layer (such as a 1×1 convolution) and then fused with the feature maps of the intermediate layers of the YOLO backbone network (e.g., through channel concatenation). This design is equivalent to introducing a parallel feature guidance path with the ability to "understand the essence of the fruit" into the YOLO network, enabling it to directly utilize the robust representations learned from the occluded data during training.

[0069] In this embodiment, an autoencoder network with a high recognition rate for occluded fruits is first trained. Then, when training the entire YOLO network on a standard fruit detection dataset (with bounding box annotations), the feature maps of the autoencoder network and the YOLO network are concatenated to improve the feature extraction capability robust to occlusion.

[0070] To train an autoencoder network capable of reconstructing a complete fruit even when partially occluded, it's necessary to create a contrast between occluded and unoccluded fruits. For example, in an image with three fruits A, B, and C, where fruit A is partially occluded, the autoencoder network needs to reconstruct the unoccluded portion of A to reconstruct it. However, directly collecting datasets makes it difficult to find identical images where only fruit A is unoccluded, or to specifically collect such images during manual photography—a time-consuming and laborious process. Therefore, this embodiment, based on the collected fruit image dataset, first selects images with very low occlusion levels for each fruit. Then, based on these images, random leaf occlusions are generated for some fruits to obtain contrast images, which are then used to train the autoencoder network.

[0071] Step S002 further includes steps S0021-S0025:

[0072] Step S0021: Annotate the original image to obtain the bounding box of each fruit in the original image.

[0073] Specifically, for each original image in the dataset, manual annotation is performed to obtain the bounding box of each fruit in the original image.

[0074] Step S0022: Using the bounding box of each fruit in the original image, filter out the adjusted images from all the original images.

[0075] Step S0022 further includes steps S0221-S0223:

[0076] Step S0221: Determine the area percentage of each fruit by the ratio of the fruit area to the bounding box area within the bounding box of each fruit in the original image.

[0077] Specifically, the number of pixels belonging to the fruit in each bounding box is counted and used as the fruit area; the total number of pixels in the bounding box is counted and used as the bounding box area; then the ratio of the two areas is calculated as the area ratio of the fruit region within each bounding box.

[0078] Step S0222: Identify fruits with an area ratio greater than a preset area threshold as prominent fruits in the original image.

[0079] It should be noted that the preset area threshold is set according to the specific situation, and is preferably 0.9 here.

[0080] Fruits with an area ratio greater than 0.9 are considered obvious fruits, meaning they are relatively obvious and easy to identify.

[0081] Step S0223: Calculate the ratio of the number of obvious fruits to the total number of fruits in the original image, and determine the original image with a ratio greater than a preset quantity threshold as the adjustment image.

[0082] It should be noted that the preset quantity threshold is set according to the specific situation, and 0.9 is preferred here.

[0083] For each original image, the total number of fruits in the image is counted and denoted as N. The number of obvious fruits in the image is counted and denoted as s. Images in which the ratio of s to N is greater than 0.9 are denoted as adjustment images. That is, obvious fruits need to be randomly selected on the adjustment images and leaf occlusion is generated. Then, a control image group of occluded images and adjustment images is obtained and used to train the autoencoder network.

[0084] Each adjusted image and its corresponding occluded image form a control image group.

[0085] The reason for selecting images with less occlusion of the fruit as adjustment images is that the adjustment images and the corresponding occluded images will be used as a control image group later. Therefore, it is necessary to minimize other variables. If the occlusion of the fruit in the image is more severe, it is not possible to directly obtain an unoccluded image and form a control image group.

[0086] Step S0023: Based on the connected regions of leaves that intersect with the bounding boxes of each fruit in the adjusted image, and the area ratio of each fruit, obtain the occluded fruits in the adjusted image.

[0087] Step S0023 further includes steps S0231-S0235:

[0088] Step S0231: Obtain the number of leaf connected components that intersect with the bounding box of each fruit in the adjusted image, and normalize the number of leaf connected components to obtain the leaf connected component value of each fruit in the adjusted image.

[0089] It should be noted that for each fruit in each adjusted image, the number of connected leaf regions that intersect with the bounding box of that fruit is obtained. The bounding box of each fruit in all the captured images generally intersects with the leaves, even if some fruits are not occluded at all, their bounding boxes often contain leaves.

[0090] Each fruit corresponds to a number of connected leaf components. The maximum number of connected leaf components is selected, and the ratio of the number of connected leaf components corresponding to each fruit to the maximum value is used as the normalized value of the number of connected leaf components corresponding to each fruit, denoted as n1. This is to adjust the value of the connected leaf components of each fruit in the image.

[0091] Step S0232: Calculate the ratio of the leaf connected component value to the area ratio of each fruit in the adjusted image, and normalize the ratio to obtain the occlusion index of each fruit in the adjusted image.

[0092] It should be noted that: the area percentage b of each fruit in the adjusted image is obtained.

[0093] For each fruit in each adjusted image, the larger n1 is and the smaller b is, the more the fruit is associated with more leaves, but it can be clearly identified on its own. The more naturally the fruit is occluded by the generated leaves, the more natural the corresponding occlusion image is generated.

[0094] For each fruit in each adjusted image, the ratio of n1 to b is calculated; then the maximum value of the ratio of n1 to b is selected, and the ratio of n1 to b corresponding to each fruit is divided by the maximum value of the ratio of n1 to b to normalize it. The normalized value is used as the occlusion index of the fruit.

[0095] Step S0233: Sort the occlusion indices of all fruits in the adjusted image in ascending order to obtain an ascending sequence.

[0096] Step S0234: Perform threshold segmentation on the ascending sequence to obtain the occlusion index threshold.

[0097] It should be noted that threshold segmentation is a well-known technique, and will not be elaborated on here.

[0098] The occlusion index threshold k is obtained by segmenting the ascending sequence using the Otsu threshold method.

[0099] Step S0235: Identify fruits with an occlusion index greater than the occlusion index threshold as occluded fruits in the adjusted image.

[0100] Step S0024: Using the occluded fruit in the adjusted image, obtain the occluded image corresponding to the adjusted image.

[0101] Specifically, it includes:

[0102] The leaf closest to the occluded fruit in the adjusted image is selected as the target leaf, and the image of the target leaf is superimposed on the occluded fruit area to obtain the initial occlusion image corresponding to the adjusted image.

[0103] It should be noted that the distance to the occluded fruit in the adjustment image is Euclidean distance. For each occluded fruit in the adjustment image, the target leaf is determined by calculating the distance. The size, rotation angle, and transparency of the target leaf are adjusted to make it look more natural. The target leaf image is then superimposed on the fruit area to form the occlusion, thus obtaining the occluded image corresponding to each adjustment image.

[0104] Size adjustment: Scale the target leaf according to the bounding box size of the obscured fruit (e.g., 120 pixels x 120 pixels) by a preset ratio (e.g., 0.5).

[0105] Rotation adjustment: Randomly generate a rotation angle (e.g., 15°) within the range of 0° to 360°, and perform corresponding rotation transformation on the target blade.

[0106] Transparency adjustment: Randomly generate an alpha channel value (e.g., 85%) within a preset transparency range (e.g., 70% to 90% opacity) to control the degree of blending when the blades are stacked.

[0107] The adjusted target leaf is then superimposed onto a designated area of ​​the corresponding occluded fruit in the adjusted image using an image fusion algorithm (a known technique), generating an initial occlusion image. During superposition, the leaf is preferentially placed over the visible portion of the fruit.

[0108] Image enhancement processing is performed on the initial occluded image corresponding to the adjusted image to obtain the occluded image corresponding to the adjusted image.

[0109] It should be noted that data augmentation of the initial occluded image, including random rotation, scaling, translation, color jittering, and noise addition, increases data diversity and improves the model's generalization ability. These data augmentation techniques are well-known in the field of image processing and will not be described in detail here.

[0110] Step S0025: Filter all occluded images and obtain the images to be retained.

[0111] It should be noted that since the occlusion images generated by the above method may not accurately reflect actual occlusion situations, further filtering is needed to obtain occlusion images that are realistic and approximate natural occlusion. This embodiment trains a discriminator to distinguish between real and generated images, retaining only the occlusion images that pass the discriminator's test.

[0112] Specifically, it includes:

[0113] The occluded image is input into the trained discrimination model to obtain the score corresponding to each occluded image.

[0114] Specifically, each occluded image is input into the trained discrimination model to obtain a score between 0 and 1. The closer the score is to 1, the closer the image is to the real image.

[0115] The images with the highest scores and a predetermined percentage of occluded images are selected as the retained occluded images.

[0116] It should be noted that the preset ratio is set according to the specific situation, and 30% is preferred here.

[0117] Specifically, the top 30% of occluded images in terms of score are selected and retained, as these images are considered to be close to naturally occluded images, and are used to participate in the training process of the autoencoder network.

[0118] The trained discriminant model is obtained as follows:

[0119] Randomly select several original images and generate a composite image with the same number of original images.

[0120] It should be noted that the number of randomly selected original images is preferably 2000.

[0121] Specifically, 2000 original images are randomly selected from the dataset. For these 2000 original images, an equal number of occlusion images are created using existing occlusion generation methods (such as methods that use GAN networks to generate occlusions) or the aforementioned occlusion image generation methods, and these are denoted as synthetic images.

[0122] Each original image is labeled with 1, and each synthetic image is labeled with 0. The labeled original and synthetic images are used as the training dataset.

[0123] Specifically, the data is labeled by assigning a 1 to each of the 2000 original images and a 0 to each of the 2000 synthesized images, resulting in 4000 labeled images as the training dataset.

[0124] The CNN classification model is trained using the training dataset to obtain a well-trained discriminant model.

[0125] Specifically, an existing CNN classification model (such as ResNet-18 network) is used as the network structure for the discriminator. The training dataset is divided into a training set and a test set in a ratio of 8:2. The cross-entropy loss function is used. The model is trained through multiple iterations until the verification accuracy of the model on real and synthetic images reaches more than 90%.

[0126] Step S003: Use the preserved occluded image and its corresponding adjusted image as an image control group, and use the image control group to train the autoencoder network to obtain the trained autoencoder network.

[0127] Specifically, the process involves using the preserved occluded image from each image control group as input, and using the mean square error between the adjusted image and the preserved occluded image in the image control group as the loss function to train the autoencoder network, thus obtaining the trained autoencoder network.

[0128] It should be noted that each occluded image and its corresponding adjusted image are retained as an image control group.

[0129] During the training of the autoencoder network, the occluded image in each image control group is used as the network input, and the MSE of the adjusted image and the occluded image in the image control group is used as the loss function to measure the difference between the reconstructed image and the original image. This makes the output of the autoencoder network closer and closer to the original image, i.e. the unoccluded image, thereby training the reconstruction ability of the autoencoder network.

[0130] Autoencoder networks employ a convolutional autoencoder structure: the left side is the encoder, the three nodes in the middle represent the compression space, i.e., the compressed representation of the original data, and the right side is the decoder. The input and output of an autoencoder network are the same; when an image is input, the autoencoder network will output an image that is close to the input image. Therefore, the structure of the autoencoder network itself helps to reconstruct the input image.

[0131] Step S004: Freeze the network weights of the encoder in the trained autoencoder network, and fuse the feature map output by the encoder after freezing the weights with the feature map of the corresponding layer in the YOLO backbone network to obtain the fused network.

[0132] It should be noted that: Figure 2 This is a schematic diagram of the YOLO network structure in this invention. The YOLO network used in this embodiment is the YOLOv3 network.

[0133] Given that the YOLO backbone network (such as Darknet-53 in YOLOv3) is a pure convolutional neural network consisting of standard convolutional layers, batch normalization layers, and the LeakyReLU activation function, and that the encoder part of the autoencoder network is also a feature extraction network composed of convolutional layers, pooling layers, etc., both are highly isomorphic in their underlying architecture, being feedforward convolutional stacked structures, a pre-trained autoencoder network is used as an auxiliary feature extractor, and its output is fused with features from a certain layer of the YOLO backbone network.

[0134] The encoder portion of the trained autoencoder network is "frozen" (its weights are not updated) and used as a forward inference module. The freezing operation means that after model training is complete, all parameters of the encoder portion are set to a non-updateable state, so that it only performs forward propagation to extract features during subsequent training, without participating in gradient calculation and weight optimization. The freezing operation is a well-known technique and will not be elaborated upon here.

[0135] The feature map output by the encoder is convolved to adjust the number of channels, and then concatenated or element-wise added with the feature map of a certain layer of YOLO in the channel dimension to provide supplementary information for the YOLO network.

[0136] In this embodiment, the feature map output by the encoder and the feature map of the YOLO network are spliced ​​together in the residual part. Specifically, the feature map obtained at positions 256, 512, and 1024 of the Darknet-53 part is spliced ​​together with the feature map of the encoder.

[0137] It should be noted that the structure of the autoencoder network is set to: 1024 channels, 512 channels, 256 channels, 128 channels, 64 channels, 32 channels, 16 channels, 8 channels, and 4 channels.

[0138] The 1024-channel, 512-channel, and 256-channel features of the autoencoder network use the same convolutional kernels as the corresponding number of channels in the Darknet YOLO3 network, thus ensuring consistent feature map sizes for direct concatenation. Specifically: the 1024-channel features of the autoencoder network use the same convolutional kernels as the 1024-channel features of YOLO3; the 512-channel features of the autoencoder network use the same convolutional kernels as the 512-channel features of YOLO3; and the 256-channel features of the autoencoder network use the same convolutional kernels as the 256-channel features of YOLO3.

[0139] The feature map output by the encoder after the weights are frozen is fused with the feature map of the corresponding layer in the YOLO backbone network to obtain the fused network, including steps S0041-S0043:

[0140] Step S0041: Use a 1×1 convolution to adjust the feature map output by the encoder after freezing the weights, so that it has the same number of channels n as the feature map of the corresponding layer of the YOLO backbone network.

[0141] Step S0042: Concatenate the adjusted encoder feature map with the feature map of the corresponding layer of the YOLO backbone network in the channel dimension to obtain a fused feature map with 2n channels.

[0142] Step S0043: Use 1×1 convolution to compress the number of channels of the fused feature map from 2n back to n to obtain the enhanced feature map, and input it into the subsequent layers of the YOLO backbone network to obtain the fused network.

[0143] Specifically, the process of enabling the encoder to participate in YOLO consists of the following three steps:

[0144] Feature alignment: A 1×1 convolution is used to adjust the number of channels in the feature map of the Lth layer of the encoder to the same number of channels n as the corresponding hidden layer in YOLOv3. By aligning and then compressing using 1×1 convolution, the consistency of input and output dimensions is ensured, allowing seamless insertion into any layer of YOLOv3 without disrupting the original structure.

[0145] Feature fusion: The adjusted encoder features are concatenated with the native YOLOv3 features along the channel dimension to obtain a fused feature with 2n channels. The concatenation operation preserves the "anti-occlusion prior knowledge" provided by the autoencoder network and the features of YOLOv3 itself to the greatest extent, with minimal information loss.

[0146] Feature compression: Using n 1×1 convolutional kernels, the 2n channels are compressed back to n channels, resulting in enhanced YOLOv3 features for use in subsequent layers. 1×1 convolution is a lightweight operation, and the increase in parameters and computational cost is controllable.

[0147] Step S005: Train the fused network using the dataset to obtain a trained fruit recognition model.

[0148] It should be noted that since the encoder is frozen, training the fused network is actually training the YOLO part of the fused network.

[0149] Specifically, the dataset is used as the training dataset for the YOLO network and participates in the training process of the YOLO network. The loss function uses standard object detection loss (such as: localization loss + classification loss + confidence loss). Through iterative training, a trained YOLO network is obtained, that is, a trained fruit recognition model is obtained.

[0150] Figure 3 This is a schematic diagram of the feature map fusion process in this invention. For example... Figure 3As shown, the input data is processed by a YOLO network to extract a 256-channel feature map, and simultaneously processed by an autoencoder network to extract another 256-channel feature map. These two sets of feature maps are concatenated along the channel dimension to obtain a 512-channel fused feature map. Subsequently, this 512-channel feature map is convolved using 256 convolutional kernels, ultimately outputting a 256-channel fused feature map that incorporates anti-occlusion information and YOLO's inherent detection capabilities.

[0151] Step S006: Based on the trained fruit recognition model, identify the real-time fruit images collected by the robotic arm and control the robotic arm to perform the grasping operation.

[0152] Specifically, image recognition: A camera on the agricultural robotic arm captures images of the fruit in front of it in real time. A pre-trained fruit recognition model is used to quickly analyze the image, locate all the fruits, and outline each fruit with a rectangular box, while also labeling the type and confidence level of each fruit.

[0153] Target selection: From all the selected fruits, the system determines the best target fruit to grab based on preset rules (such as selecting the ripest or the one closest to the top). This selected fruit frame becomes the "eye" for all actions.

[0154] Calculating position and size: First, find the gripping point: The system uses the center point of the selected box as a reference. Combining this with the distance measured by the depth camera, it calculates the specific spatial position of this center point relative to the robotic arm's base in the real world.

[0155] Then assess the fruit size: estimate the approximate volume of the fruit based on the width and height of the frame and the distance between the fruits.

[0156] Planning and executing the grasping action: The robotic arm control system receives information on the "grasping point location" and the "fruit size." The system plans a path that allows the robotic arm's end effector to safely move to the grasping point. The robotic arm moves to the target point, and the gripper adjusts its opening width based on the estimated size to steadily grasp the fruit.

[0157] Cyclic operation: The robotic arm places the fruit into the collection box, and then continues to identify and grab the next target until the task is completed.

[0158] Please see Figure 4 The diagram illustrates a block diagram of an agricultural robotic arm grasping control system integrating deep learning vision according to an embodiment of the present invention. The system includes the following modules:

[0159] The acquisition module 100 is used to acquire a dataset; wherein, the dataset consists of original images containing the fruits of the target fruit tree variety;

[0160] The fusion module 200 is used to obtain the preserved occluded image and its corresponding adjusted image based on the dataset;

[0161] The occluded image and its corresponding adjusted image are retained as an image control group, and the autoencoder network is trained using the image control group to obtain the trained autoencoder network.

[0162] The network weights of the encoder in the trained autoencoder network are frozen, and the feature map output by the encoder with frozen weights is fused with the feature map of the corresponding layer in the YOLO backbone network to obtain the fused network.

[0163] The fused network was trained using the dataset to obtain a trained fruit recognition model;

[0164] The recognition module 300 is used to recognize real-time fruit images collected by the robotic arm based on a trained fruit recognition model, and to control the robotic arm to perform grasping operations.

[0165] In summary, in this embodiment of the invention, by first training an autoencoder network with high reconstruction capability for occluded fruits, and then fusing the robust features extracted by its encoder with the features of the YOLO detection network, the accuracy and robustness of fruit recognition in complex agricultural scenarios with severe foliage occlusion are significantly improved. This effectively overcomes the problem of missed detection that easily occurs when the fruit is partially occluded in existing methods, ensuring the integrity of the robotic arm's grasping and operational efficiency.

[0166] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A grasping control method for agricultural robotic arms integrating deep learning vision, characterized in that, The method includes the following steps: Collect a dataset; the dataset consists of original images containing the fruits of the target fruit tree variety. Based on the dataset, obtain the preserved occluded image and its corresponding adjusted image; The occluded image and its corresponding adjusted image are retained as an image control group, and the autoencoder network is trained using the image control group to obtain the trained autoencoder network. The network weights of the encoder in the trained autoencoder network are frozen, and the feature map output by the encoder with frozen weights is fused with the feature map of the corresponding layer in the YOLO backbone network to obtain the fused network. The fused network was trained using the dataset to obtain a trained fruit recognition model; The robot arm recognizes real-time fruit images captured by the trained fruit recognition model and controls the robot arm to perform grasping operations.

2. The agricultural robotic arm grasping control method integrating deep learning vision according to claim 1, characterized in that, The specific steps involved in obtaining the preserved occluded image and its corresponding adjusted image based on the dataset are as follows: The original image is annotated to obtain the bounding box of each fruit in the original image; Using the bounding box of each fruit in the original image, filter out the adjusted images from all the original images; Based on the connected components of leaves that intersect with the bounding boxes of each fruit in the adjusted image, and the area ratio of each fruit, the occluded fruits in the adjusted image are obtained. By adjusting the image and finding the occluded fruit, obtain the corresponding occluded image. Filter all occluded images and select the ones that are retained.

3. The agricultural robotic arm grasping control method integrating deep learning vision according to claim 2, characterized in that, The specific steps involved in selecting adjusted images from all original images using the bounding boxes of each fruit in the original images are as follows: The ratio of the fruit area to the bounding box area of ​​each fruit in the original image is used to determine the area percentage of each fruit. Fruits with an area ratio greater than a preset area threshold are identified as prominent fruits in the original image. Calculate the ratio of the number of obvious fruits to the total number of fruits in the original image, and determine the original image with a ratio greater than a preset threshold as the adjusted image.

4. The agricultural robotic arm grasping control method integrating deep learning vision according to claim 2, characterized in that, The specific steps for obtaining occluded fruits in the adjusted image based on the connected components of leaves that intersect with the bounding boxes of each fruit in the adjusted image, and the area proportion of each fruit, are as follows: The number of leaf connected components that intersect with the bounding box of each fruit in the adjusted image is obtained, and the number of leaf connected components is normalized to obtain the leaf connected component value of each fruit in the adjusted image. Calculate the ratio of the leaf connected component value to the area ratio of each fruit in the adjusted image, and normalize the ratio to obtain the occlusion index of each fruit in the adjusted image. Sort the occlusion indices of all fruits in the adjusted image in ascending order to obtain an ascending sequence; The occlusion index threshold is obtained by thresholding the ascending sequence; Fruits with an occlusion index greater than the occlusion index threshold are identified as occluded fruits in the adjusted image.

5. The agricultural robotic arm grasping control method integrating deep learning vision according to claim 2, characterized in that, The specific steps for obtaining the occluded image corresponding to the adjusted image by utilizing the occluded fruit in the adjusted image are as follows: The leaf closest to the occluded fruit in the adjusted image is taken as the target leaf, and the image of the target leaf is superimposed on the occluded fruit area to obtain the initial occlusion image corresponding to the adjusted image; Image enhancement processing is performed on the initial occluded image corresponding to the adjusted image to obtain the occluded image corresponding to the adjusted image.

6. The agricultural robotic arm grasping control method integrating deep learning vision according to claim 2, characterized in that, The specific steps involved in filtering all occluded images to obtain the retained occluded images are as follows: The occluded image is input into the trained discrimination model to obtain the score corresponding to each occluded image; The images with the highest scores and a predetermined percentage of occluded images are selected as the retained occluded images.

7. The agricultural robotic arm grasping control method integrating deep learning vision according to claim 6, characterized in that, The trained discrimination model is obtained as follows: Randomly select several original images and generate a composite image with the same number of original images; Each original image is labeled with 1, and each synthetic image is labeled with 0. The labeled original images and synthetic images are used as the training dataset. The CNN classification model is trained using the training dataset to obtain a well-trained discriminant model.

8. The agricultural robotic arm grasping control method integrating deep learning vision according to claim 1, characterized in that, The specific steps involved in training the autoencoder network using an image control group to obtain a trained autoencoder network are as follows: The occluded image in each image control group is used as input, and the mean square error between the adjusted image and the occluded image in the image control group is used as the loss function to train the autoencoder network, resulting in a trained autoencoder network.

9. The agricultural robotic arm grasping control method integrating deep learning vision according to claim 1, characterized in that, The specific steps involved in fusing the feature map output by the encoder after freezing the weights with the feature map of the corresponding layer in the YOLO backbone network to obtain the fused network are as follows: The feature map output by the encoder after freezing the weights is adjusted using a 1×1 convolution so that it has the same number of channels n as the feature map of the corresponding layer of the YOLO backbone network. The adjusted encoder feature map is concatenated with the feature map of the corresponding layer of the YOLO backbone network in the channel dimension to obtain a fused feature map with 2n channels. The number of channels in the fused feature map is compressed from 2n back to n using a 1×1 convolution to obtain an enhanced feature map, which is then input into subsequent layers of the YOLO backbone network to obtain the fused network.

10. An agricultural robotic arm grasping control system integrating deep learning vision, characterized in that, The system includes the following modules: The acquisition module is used to acquire datasets; the datasets consist of original images containing the fruits of the target fruit tree variety. The fusion module is used to obtain the preserved occluded image and its corresponding adjusted image based on the dataset; The occluded image and its corresponding adjusted image are retained as an image control group, and the autoencoder network is trained using the image control group to obtain the trained autoencoder network. The network weights of the encoder in the trained autoencoder network are frozen, and the feature map output by the encoder with frozen weights is fused with the feature map of the corresponding layer in the YOLO backbone network to obtain the fused network. The fused network was trained using the dataset to obtain a trained fruit recognition model; The recognition module is used to recognize real-time fruit images collected by the robotic arm based on a trained fruit recognition model, and to control the robotic arm to perform grasping operations.

Images