Anti-shielding deep learning fruit identification and picking method and system

Abstract

The invention provides an anti-occlusion deep learning fruit identification and picking method and system, and the method comprises the steps: obtaining an image containing an occlusion fruit, extracting a visible region feature map and a contact edge feature map through a first neural network, and fusing the features into a comprehensive feature; calculating the area of the visible part of the fruit based on the visible region features as an initial size clue, constructing a candidate prototype subset based on the initial size clue, inputting the fusion features and the initial clue into a second neural network, and performing weighted summation on the candidate prototype subset by the network through predicting a weight coefficient to obtain a second neural network; and generating a complete shape mask of the sheltered fruit, calculating caliper diameters in multiple directions based on the complete mask contour, and taking the diameter statistical average value as the estimated size of the fruit.

Description

An occlusion-resistant deep learning method and system for fruit recognition and harvesting Technical Field

[0001] This application belongs to the field of image processing, and in particular relates to a deep learning-based method and system for fruit recognition and harvesting that is occlusion resistant. Background Technology

[0002] Under ideal conditions, when the fruit is completely unobstructed in the image, image segmentation algorithms can extract the fruit's full outline, allowing for relatively accurate size information to be obtained by calculating pixel area, fitting geometry, or measuring the maximum diameter. However, in natural growing environments such as orchards and greenhouses, fruits are often severely obstructed by branches, leaves, or other nearby fruits, resulting in only a portion of the fruit being visible in the image. When using drones or robots for harvesting, for obstructed fruits, the lack of overall fruit information often makes it impossible to determine whether to harvest, even if the robotic arm's current position is suitable for harvesting.

[0003] Some methods attempt to fit a complete geometric shape, such as a circle or ellipse, based on the visible contour of the fruit to estimate its size. However, these methods rely on overly strong assumptions, leading to significant estimation errors when the fruit shape is irregular or the visible portion is not representative. Other methods turn to deep learning-based non-modal instance segmentation or image inpainting techniques, attempting to recover the complete shape mask of an object from an occluded image. However, the models often fail to accurately complete the contours; the shape completion process lacks strong prior knowledge guidance, resulting in shapes that may not conform to the characteristics of specific fruit species; and many methods do not use the size of the visible portion as a clear constraint or initial clue during shape prediction, leading to poor adaptability to different target sizes and potentially significant discrepancies between the completed and actual sizes. Therefore, current technologies still fall short of meeting the practical needs of machine harvesting applications when dealing with occluded fruit in terms of size estimation accuracy. Summary of the Invention

[0004] This invention proposes an occlusion-resistant deep learning method for fruit recognition and harvesting, addressing the problem that existing technologies struggle to achieve accurate size estimation for occluded fruits, failing to meet harvesting requirements. The method includes the following steps: acquiring an image containing the occluded fruit to be estimated; processing the image using a first neural network to generate a first feature map representing the visible area of ​​the occluded fruit and a second feature map representing the contact edge between the fruit and the occluder; fusing the first and second feature maps to generate a fused feature map containing information about the visible area and the contact edge; calculating the area of ​​the visible portion of the fruit based on the first feature map as an initial size cue; and based on the initial size cue,... Multiple shape prototypes matching the size range are selected from a preset shape prototype encoding book to form a candidate prototype subset. The fusion feature and the initial size clue are input into a second neural network. The second neural network predicts the weight coefficients corresponding to each shape prototype in the candidate prototype subset based on the fusion feature, and performs a weighted summation of the shape prototypes in the candidate prototype subset based on the weight coefficients to generate a complete fruit shape mask. Based on the contour of the complete fruit shape mask, the caliper diameter of the contour in multiple preset directions is calculated. The statistical average of the multiple caliper diameters is determined as the estimated fruit size, and the fruit size is used to determine whether to harvest.

[0005] Optionally, the step of processing the image using a first neural network to generate a first feature map representing the visible area of ​​the occluded fruit and a second feature map representing the contact edge between the fruit and the occluder includes: using a neural network with an encoder-decoder structure as the first neural network; the encoder downsamples the image to extract depth features, the decoder upsamples the depth features, and generates the first feature map and the second feature map respectively through two independent output branches.

[0006] Optionally, the step of fusing the first feature map and the second feature map to generate a fused feature containing visible area and contact edge information includes: obtaining the fused feature by performing an element-wise addition operation between the first feature map and the second feature map.

[0007] Optionally, calculating the area of ​​the visible portion of the fruit based on the first feature map includes: thresholding the first feature map to generate a binary mask; counting the total number of pixels representing the fruit in the binary mask, and using the total number as the area of ​​the visible portion of the fruit.

[0008] Optionally, the step of selecting multiple shape prototypes with matching size ranges from a preset shape prototype codebook to form a candidate prototype subset includes: denoting the initial size clue as... Set a size range , where r is a preset scaling factor; K shape prototypes whose areas fall within the size range are selected from the shape prototype encoding to form a candidate prototype subset, where K is a preset positive integer.

[0009] Optionally, the step of calculating the caliper diameter of the outline of the complete fruit shape mask in multiple preset directions based on the outline of the complete fruit shape mask includes: selecting multiple directions as preset directions within a range of 0° to 180°, with preset angles as intervals; and for each preset direction, calculating the projection length of the outline of the complete fruit shape mask in that direction as the caliper diameter of that direction.

[0010] Optionally, determining the statistical average of the diameters of the plurality of calipers as the estimated fruit size includes: the calculation formula is: ;in, Estimate the size of the fruit. Let N be the caliper diameter calculated in the i-th preset direction, where N is the total number of preset directions.

[0011] The method of determining whether to pick based on fruit size specifically involves: comparing the estimated size of the fruit with a preset minimum picking size threshold; when the estimated size of the fruit is greater than or equal to the minimum picking size threshold, the fruit is determined to be ready for picking, and a picking instruction is generated.

[0012] Furthermore, this invention also relates to an anti-occlusion deep learning fruit recognition and harvesting system, comprising the following modules: an acquisition module, configured to acquire an image containing an occluded fruit to be estimated; process the image using a first neural network to generate a first feature map representing the visible area of ​​the occluded fruit, and a second feature map representing the contact edge between the fruit and the occluder; a first generation module, configured to fuse the first feature map and the second feature map to generate a fused feature containing information on the visible area and the contact edge; calculate the area of ​​the visible portion of the fruit based on the first feature map as an initial size cue; and a second generation module, configured to, based on the initial size cue, generate a fused feature from a preset shape prototype. The encoding process selects multiple shape prototypes that match a certain size range, forming a subset of candidate prototypes. The fusion features and the initial size clues are input into a second neural network. The second neural network predicts the weight coefficients corresponding to each shape prototype in the candidate prototype subset based on the fusion features, and performs a weighted summation of the shape prototypes in the candidate prototype subset based on the weight coefficients to generate a complete fruit shape mask. The calculation module is used to calculate the caliper diameter of the contour in multiple preset directions based on the contour of the complete fruit shape mask. The statistical average of the multiple caliper diameters is determined as the estimated fruit size, and the fruit size is used to determine whether to harvest.

[0013] Preferably, the step of processing the image using a first neural network to generate a first feature map representing the visible area of ​​the occluded fruit and a second feature map representing the contact edge between the fruit and the occluder includes: using a neural network with an encoder-decoder structure as the first neural network; the encoder downsamples the image to extract depth features, the decoder upsamples the depth features, and generates the first feature map and the second feature map respectively through two independent output branches.

[0014] Preferably, the step of fusing the first feature map and the second feature map to generate a fused feature containing visible area and contact edge information includes: obtaining the fused feature by performing an element-wise addition operation between the first feature map and the second feature map.

[0015] Preferably, calculating the area of ​​the visible portion of the fruit based on the first feature map includes: performing thresholding processing on the first feature map to generate a binary mask; counting the total number of pixels representing the fruit in the binary mask, and using the total number as the area of ​​the visible portion of the fruit.

[0016] Preferably, the step of selecting multiple shape prototypes with matching size ranges from a preset shape prototype codebook to form a candidate prototype subset includes: denoting the initial size clue as... Set a size range , where r is a preset scaling factor; K shape prototypes whose areas fall within the size range are selected from the shape prototype encoding to form a candidate prototype subset, where K is a preset positive integer.

[0017] Preferably, the step of calculating the caliper diameter of the outline of the complete fruit shape mask in multiple preset directions based on the outline of the complete fruit shape mask includes: selecting multiple directions as preset directions within a range of 0° to 180°, with preset angles as intervals; for each preset direction, calculating the projection length of the outline of the complete fruit shape mask in that direction as the caliper diameter of that direction.

[0018] Preferably, determining the estimated fruit size by the statistical average of the diameters of the plurality of calipers includes: the calculation formula is: ;in, Estimate the size of the fruit. Let N be the caliper diameter calculated in the i-th preset direction, where N is the total number of preset directions.

[0019] The method of determining whether to pick based on fruit size specifically involves: comparing the estimated size of the fruit with a preset minimum picking size threshold; when the estimated size of the fruit is greater than or equal to the minimum picking size threshold, the fruit is determined to be ready for picking, and a picking instruction is generated.

[0020] This invention provides a method for estimating the size of occluded fruits. A first neural network extracts and fuses the visible area features and contact edge features of the fruit in parallel. This allows the subsequent shape restoration process to fully utilize the key geometric clues contained in the occlusion boundary, rather than relying solely on the incomplete visible outline, thus improving the accuracy of the complete shape prediction. An initial size clue based on the visible area is used, combined with a shape prototype encoding template. The initial size clue constrains a second neural network to select and combine shape prototypes within a reasonable size range, thereby combining data-driven prediction with prior knowledge of the fruit's morphology. This ensures that the completed fruit mask is closer to reality in both shape and size. The size is determined by calculating the average diameter of calipers in multiple directions, reducing measurement errors caused by local irregularities in fruit posture or outline, enhancing the reliability of the estimation results, and improving harvesting efficiency. Attached Figure Description

[0021] Figure 1 is a flowchart of the first embodiment; Figure 2 is a schematic diagram of the first neural network processing; Figure 3 is a schematic diagram of feature map fusion. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] In the first embodiment, the present invention proposes an anti-occlusion deep learning fruit recognition and picking method, as shown in Figure 1, including the following steps: S1, acquiring an image containing the occluded fruit to be estimated; processing the image using a first neural network to generate a first feature map representing the visible area of ​​the occluded fruit, and a second feature map representing the contact edge between the fruit and the occluder; specifically, acquiring fruit images in an orchard or greenhouse environment using a camera, with an image resolution of, for example, 1920×1080 pixels, and preprocessing the acquired original image, including image scaling, normalization, and other operations, to adapt to the input requirements of the neural network.

[0024] The first neural network employs an encoder-decoder structure, such as the U-Net network. The encoder uses ResNet-50 as the backbone to extract deep features from the image, while the decoder contains two parallel branches. The first branch outputs a single-channel probability map through upsampling and convolution operations, where each pixel value represents the probability that the location belongs to the visible area of ​​the fruit. After thresholding, this yields the first feature map. The second branch outputs another single-channel probability map in a similar manner, representing the probability that each pixel belongs to the contact edge between the fruit and the occluder. After thresholding, this yields the second feature map.

[0025] In an optional embodiment, the step of processing the image using a first neural network to generate a first feature map representing the visible area of ​​the occluded fruit and a second feature map representing the contact edge between the fruit and the occluder includes: using a neural network with an encoder-decoder structure as the first neural network; the encoder downsampling the image to extract depth features, the decoder upsampling the depth features, and generating the first feature map and the second feature map respectively through two independent output branches.

[0026] An end-to-end deep learning model is employed, with an encoder-decoder structure at its core, similar to U-Net or SegNet. The encoder consists of a series of convolutional and pooling layers, with ResNet-50 preferably used as the backbone network, as mentioned above. Given a 256×256×3 fruit image as input, the encoder progressively reduces the spatial resolution of the feature map while increasing the number of channels through successive convolution and max-pooling operations, extracting deep features containing rich semantic information, such as an 8×8×2048 feature tensor.

[0027] The decoder's role is to restore the depth features extracted by the encoder to the resolution of the original image. It consists of a series of upsampling layers, such as transposed convolutions or bilinear interpolations, and convolutional layers. To preserve fine details of the image, the decoder fuses feature maps from corresponding layers generated by the encoder during downsampling, using skip connections. In the final layer of the decoder, the feature maps are fed into two independent output branches. Each branch consists of one or more 1×1 or 3×3 convolutional layers and an activation function such as the sigmoid function. The first branch is trained to predict the visible portion of the fruit, outputting a first feature map where the pixel value represents the probability that the pixel belongs to the visible fruit. The second branch is trained to predict the completion cues for occluded parts, outputting a second feature map.

[0028] The training process of the first neural network is based on a labeled dataset. This training set contains fruit images under different lighting and occlusion conditions, each image accompanied by two precise pixel-level ground truth masks. The first ground truth mask labels the visible area of ​​the fruit, and the second ground truth mask is a pixel-level label of the contact edge between the fruit and the occluder, with pixels along the contact edge labeled as 1 and the rest as 0. Before training, data augmentation operations such as rotation, scaling, and cropping are performed on the images to improve the model's generalization ability. The model's input is a fruit image of size H×W×3. During training, the loss function is a weighted sum of the differences between the prediction results of the two branches and their corresponding ground truth masks, for example, using a binary cross-entropy loss function. The total loss function can be expressed as... ,in and These are the predicted losses for the visible and occluded portions, respectively. The hyperparameters are used to balance the two tasks and are fine-tuned on the validation set through cross-validation to achieve optimal performance in visible part segmentation and occlusion region prediction. The model undergoes multiple epochs of iterative training, continuously adjusting network weights through backpropagation until the loss function converges. After training, the probability maps generated by the model's two output branches are a first feature map representing the visible region of the fruit and a second feature map representing the contact edge between the fruit and the occluder. The feature maps have the same size as the input image, as shown in Figure 2.

[0029] S2, fuse the first feature map and the second feature map to generate a fused feature containing information about the visible area and contact edges; calculate the area of ​​the visible part of the fruit based on the first feature map as an initial size cue; concatenate the first feature map and the second feature map along the channel dimension to form a multi-channel feature map, for example, concatenate two single-channel feature maps into a dual-channel fused feature, which simultaneously contains pixel position information of the visible part of the fruit and position information of the occlusion boundary. Count the total number of pixels with a pixel value of 1 in the first feature map; this total number represents the pixel area of ​​the visible part of the fruit, and is used as a scalar value as the initial size cue for subsequent networks.

[0030] In an optional embodiment, fusing the first feature map and the second feature map to generate a fused feature containing visible area and contact edge information includes: obtaining the fused feature by performing an element-wise addition operation between the first feature map and the second feature map.

[0031] The fusion operation is a linear combination process. The first neural network has generated two feature maps with the same spatial dimensions, namely the first feature map and the second feature map. Assume that the size of both feature maps is H×W, for example, 256×256. The pixel value of the first feature map F1... The position (i,j) represents the probability of the visible fruit portion, while each pixel value in the second feature map F2... This indicates the strength of the inferred occlusion at that location.

[0032] The fusion process involves computing a new feature map. The pixel value at any position (i,j) It equals the sum of the pixel values ​​of the two input feature maps at that location, i.e. For example, at a certain location (100, 150) in the image, if the pixel value of the first feature map is 0.9, it indicates that this location is likely to be visible fruit pulp, while the pixel value of the second feature map is 0.4, indicating that there is some occlusion completion signal at this location. Then, the pixel value at this location in the fused feature map will be 1.3. Through normalization processing, the fused feature value is reduced to the 0-1 range. By adding element-wise, the fused feature map not only strengthens the signal of the visible area but also integrates the inferred information of the occluded area, as shown in Figure 3.

[0033] In order to represent the size of the visible area of ​​the fruit from the first feature map, in an optional embodiment, the step of calculating the area of ​​the visible part of the fruit based on the first feature map includes: thresholding the first feature map to generate a binary mask; counting the total number of pixels representing the fruit in the binary mask, and using the total number as the area of ​​the visible part of the fruit.

[0034] The first feature map is a probability map where each pixel value ranges from 0 to 1, representing the probability that the pixel belongs to the visible fruit. A fixed threshold T is set, chosen to be 0.5. Each pixel in the first feature map is iterated through, and its value is compared with the threshold T. If the pixel value is greater than T, a value of 1 is assigned to the corresponding position in the new binary mask, indicating that the pixel is classified as fruit; if the pixel value is less than or equal to T, a value of 0 is assigned, representing the background.

[0035] After thresholding, a binary mask image containing only 0s and 1s is obtained. This mask clearly outlines the visible portion of the identified fruit. The area of ​​this binary mask is then calculated. The area is calculated by counting the total number of pixels with a value of 1 in the mask. For example, if the processed binary mask is a 512×512 matrix, and by traversing it or using a built-in function in an image processing library, it is found that there are 25,000 pixels with a value of 1, then the area of ​​the visible portion of the fruit is determined to be 25,000 pixels. This value will serve as an initial size clue. .

[0036] S3, based on the initial size clue, multiple shape prototypes matching the size range are selected from the preset shape prototype encoding book to form a candidate prototype subset. The fusion feature and the initial size clue are input into the second neural network. The second neural network predicts the weight coefficients corresponding to each shape prototype in the candidate prototype subset based on the fusion feature, and performs a weighted summation of the shape prototypes in the candidate prototype subset based on the weight coefficients to generate a complete fruit shape mask. Specifically, based on the initial size clue, a subset of similar sizes is selected from the encoding book, for example, K prototypes with an area within 1 to 2 times the visible area, where K is a preset positive integer, preferably 16-32. If there are not enough, the closest prototype is added; if there are too many, they are truncated according to area similarity. The shape prototype encoding book is pre-generated by training a large number of complete fruit mask datasets using K-means clustering or an autoencoder, and contains 128 representative fruit shape mask prototypes.

[0037] The second neural network employs a hybrid structure consisting of a convolutional encoder and a fully connected regressor, perfectly suited to the characteristics of the candidate prototype subset. Specifically, the network's convolutional module comprises a four-layer small convolutional neural network (CNN), whose input is the fused feature map of the dual channels. Each layer sequentially performs convolution, batch normalization, and ReLU activation operations, and undergoes two downsampling operations using a convolution with a stride of 2, ultimately flattening into a 512-dimensional feature vector to capture key spatial context information related to matching the candidate prototype; initial size cues. The size embedding vector is mapped to a 32-dimensional embedding vector through a separate encoder containing two fully connected layers, further enhancing the size constraint and the fit of the candidate subset. The 512-dimensional CNN feature vector and the 32-dimensional size embedding vector are concatenated along the feature dimension and input into a regression head consisting of two fully connected layers. The last layer of this regression head has its dimension strictly set to K and uses the Softmax activation function to output a set of non-negative weight coefficients corresponding to the candidate prototypes, with a weight sum of 1. Each coefficient represents the contribution of the corresponding prototype to the complete fruit shape. Optionally, binarization is performed after generating the mask.

[0038] The network training process also constructs supervised associations using a subset of candidate prototypes, which is a supervised end-to-end training: after processing the occluded image samples with fully labeled fruit masks through the S1 and S2 preprocessing steps, a subset of candidate prototypes is first obtained, and then the fused features are combined with... As network input; the supervision objective, on the one hand, calculates the K prototype weights that best fit the true mask through linear least squares optimization, which serve as the KL divergence loss. On the one hand, the supervisory signal is used; on the other hand, the prediction weights are weighted and summed with the candidate subset prototypes to generate a complete shape probability map, which is then compared with the true mask to calculate the binary cross-entropy loss. The total loss is the weighted sum of the two, i.e. , The balancing coefficient is preferably 0.2, and the network parameters are optimized end-to-end through backpropagation.

[0039] The predicted weight coefficients are used to perform a weighted summation of the shape prototypes in the subset to generate a probability map. Then, by setting a threshold such as 0.5, a binarized complete fruit shape mask is obtained.

[0040] In an optional embodiment, the step of selecting multiple shape prototypes with matching size ranges from a preset shape prototype codebook to form a candidate prototype subset includes: recording the initial size clue as... Set a size range , where r is a preset scaling factor; K shape prototypes whose areas fall within the size range are selected from the shape prototype encoding to form a candidate prototype subset, where K is a preset positive integer.

[0041] Using a prior knowledge base, namely a shape prototype codebook, we can assist in completing the shape of the fruit. Initial size clues are obtained from the above. ,For example Pixels. Set a scaling factor r, for example, r=0.5, to define a reasonable size range. According to the formula, the size range is calculated to be [25000, 37500]. The resulting range represents the possible size range of a complete fruit compatible with the observed visible portion size.

[0042] A pre-built shape prototype encoding book is retrieved, which stores a large number of standard, complete fruit shape masks and their corresponding area information. Each shape prototype in the encoding book is traversed, and its area is checked to see if it falls within the interval [25000, 37500]. K, for example 32, shape prototypes that meet the condition are selected to obtain a candidate prototype subset. When the number of prototypes matching the size range is less than 32, the closest matching area is selected. The prototypes are processed until the quantity reaches 32. When the number of prototypes conforming to the size range > 32, they are processed according to area and... Similarity is like 1 / |area- Sort the prototypes and take the first 32.

[0043] In another embodiment, the second neural network employs a lightweight encoder-predictor architecture to process a single-channel fused feature map obtained by element-wise addition of the first and second probability maps, and outputs weight coefficients for combining candidate shape prototypes. Specifically, the network first processes the fused feature map through an encoding module consisting of three convolutional blocks: each convolutional block contains a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function, where the first two blocks maintain spatial resolution, and the third block uses a convolution with a stride of 2 for downsampling, ultimately downsampling the input to half its original size and outputting a 256-dimensional feature tensor. Subsequently, global average pooling is performed on this feature tensor, transforming it into a 256-dimensional global feature vector. Simultaneously, initial size cues... The input is fed into a separate fully connected encoder, which consists of two fully connected layers, generating a 32-dimensional size embedding vector. The aforementioned 256-dimensional global feature vector is concatenated with the 32-dimensional size embedding vector to obtain a 288-dimensional joint feature vector. This joint feature vector is then fed into a predictor module, which consists of two fully connected layers, for example, 288-dimensional -> 144-dimensional -> K-dimensional, where K is the number of prototypes in the currently selected subset of candidate shape prototypes. The last layer uses the Softmax activation function, outputting K non-negative weight coefficients that sum to 1. The network is trained on a dataset containing occluded images and their complete fruit ground truth masks: for each training sample, a fused feature map is first generated through steps S1 and S2. Then, according to K candidate prototypes are selected from a pre-constructed shape prototype encoding template; then, the optimal weights for combining these K candidate prototypes into a truth mask are calculated as a supervision signal by solving a linear least squares problem; finally, the fused feature maps and As the network input, the optimal weights are used as the supervision target to minimize the KL divergence loss between the network predicted weights and the optimal weights, and the binary cross-entropy loss between the shape formed by the combination of network predicted weights and the ground truth mask is used as regularization to perform end-to-end training.

[0044] The K shape prototypes in the candidate prototype subset are weighted and summed according to the combination weights, so that the combined shape best matches the fused feature map obtained in the previous step in space, thereby generating a complete fruit shape mask that meets both size constraints and fits the image evidence.

[0045] S4. Based on the outline of the complete fruit shape mask, calculate the caliper diameter of the outline in multiple preset directions; determine the statistical average of the multiple caliper diameters as the estimated size of the fruit, and determine whether to pick the fruit based on the size of the fruit.

[0046] Specifically, a contour discovery algorithm, such as the Suzuki algorithm, is used to extract the outermost contour point set from the complete fruit shape mask. A set of rotation angles is set, for example, from 0° to 179°, with a step size of 10°, for a total of 18 directions. For each direction, the contour point set is rotated around the centroid of the mask by the corresponding angle, and the maximum span of the point set in the horizontal direction after rotation is calculated, i.e., the difference between the maximum and minimum horizontal coordinates. This span is the caliper diameter in that direction.

[0047] The caliper diameter values ​​calculated in the 18 directions are summed and then divided by the total number of directions, 18, to obtain the arithmetic mean. This mean is used as the estimated size of the fruit, expressed in pixels. This value can be converted to millimeters or other units later based on camera calibration parameters. If the estimated size falls within this ripeness standard range, the fruit is deemed suitable for harvesting, and a harvesting instruction is generated. Conversely, if the size is too small, the fruit is deemed unharvestable, and the robotic arm ignores it.

[0048] In some embodiments, the estimated size of the fruit still needs to meet basic maturity requirements. The reliability of this size is evaluated, which can be based on the degree of occlusion of the fruit: the smaller the occlusion area, the more visible the part, the more accurate the complete shape recovered by the neural network, and the higher the reliability of its size estimate. Furthermore, considering the current position and posture of the robotic arm, the feasibility and cost of harvesting the fruit are analyzed, such as whether there are obstacles in the path, whether the grasping angle is appropriate, and whether the moving distance is too long. Taking into account the three factors of size, reliability, and harvesting cost, for example, through a normalized weighted calculation, the harvesting command is prioritized for fruits that meet the size requirements, have high estimation reliability, and offer the highest harvesting benefits for the robotic arm, such as the shortest path and the easiest to grasp.

[0049] In an optional embodiment, the step of calculating the caliper diameter of the outline of the complete fruit shape mask in multiple preset directions based on the outline of the complete fruit shape mask includes: selecting multiple directions as preset directions within a range of 0° to 180°, with preset angles as intervals; and for each preset direction, calculating the projection length of the outline of the complete fruit shape mask in that direction as the caliper diameter of that direction.

[0050] This process is used to obtain multiple measurements of the fruit size, resisting the bias that may be introduced by measuring in a single direction. A set of directions for measurement is determined. A common practice is to perform equal-angle sampling within a semicircle from 0° to 180°. For example, 12 directions are selected as preset directions at 15° intervals: 0, 15, 30, 45, 60, 75, 90, 105, 120, 135, 150, and 165°. Before calculation, the outer contour needs to be extracted from the generated complete fruit shape mask to obtain a set of pixel coordinates representing the fruit boundary.

[0051] For each selected direction, such as 30°, calculate the caliper diameter in that direction. This is achieved by finding a pair of parallel lines perpendicular to the 30° direction line. These parallel lines should exactly enclose the entire outline of the fruit; that is, they are tangents to the outline. The perpendicular distance between the two parallel lines is the caliper diameter of the fruit in the 30° direction. This process can be implemented by rotating the outline point coordinate system or using a rotating bounding box algorithm. Repeat this process in all 12 preset directions to obtain 12 different caliper diameter measurements, for example... .

[0052] To derive a stable and representative estimate of the fruit size based on measurements from all directions, in an optional embodiment, determining the estimated fruit size by the statistical average of the multiple caliper diameters includes: The calculation formula is as follows: ;in, Estimate the size of the fruit. Let N be the caliper diameter calculated in the i-th preset direction, where N is the total number of preset directions.

[0053] In the steps above, the caliper diameters in N different directions have been calculated. For example, if 12 measurement directions are selected, then N equals 12, and 12 diameter data are obtained, such as 145.2 pixels, 148.1 pixels, ..., 150.5 pixels.

[0054] To obtain the estimated size of the fruit This method uses an arithmetic mean. According to the formula, the diameter values ​​of the 12 calipers are added together to obtain a sum. For example, if the sum of the 12 values ​​is 1788, the sum is divided by the total number of directions, N, which is 12. The result is 149. Therefore, the estimated size of the fruit is determined to be 149 pixels. This method of averaging the diameters in multiple directions can reduce measurement errors caused by irregular fruit shapes or tilted placement angles.

[0055] The method of determining whether to pick based on fruit size specifically involves: comparing the estimated size of the fruit with a preset minimum picking size threshold; when the estimated size of the fruit is greater than or equal to the minimum picking size threshold, the fruit is determined to be ready for picking, and a picking instruction is generated.

[0056] In a second embodiment, the present invention also provides an anti-occlusion deep learning fruit recognition and harvesting system, comprising the following modules: an acquisition module, configured to acquire an image containing an occluded fruit to be estimated; process the image using a first neural network to generate a first feature map representing the visible area of ​​the occluded fruit, and a second feature map representing the contact edge between the fruit and the occluder; a first generation module, configured to fuse the first feature map and the second feature map to generate a fused feature containing information on the visible area and the contact edge; calculate the area of ​​the visible portion of the fruit based on the first feature map as an initial size cue; and a second generation module, configured to, based on the initial size cue, generate a fused feature from a preset shape... Multiple shape prototypes matching the size range are selected from the prototype encoding sample to form a candidate prototype subset. The fusion features and the initial size clues are input into a second neural network. The second neural network predicts the weight coefficients corresponding to each shape prototype in the candidate prototype subset based on the fusion features, and performs a weighted summation of the shape prototypes in the candidate prototype subset based on the weight coefficients to generate a complete fruit shape mask. The calculation module is used to calculate the caliper diameter of the contour in multiple preset directions based on the contour of the complete fruit shape mask. The statistical average of the multiple caliper diameters is determined as the estimated fruit size, and the fruit size is used to determine whether to pick the fruit.

[0057] In this specification, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise limited, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this document, "a," "an," "the," "the," and "its" may also include plural forms unless the context clearly indicates otherwise. "Multiple" refers to at least two, such as 2, 3, 5, or 8, etc. "And / or" includes any and all combinations of the associated listed items.

[0058] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0059] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A deep learning-based fruit recognition and harvesting method with occlusion resistance, characterized in that, Includes the following steps: Acquire an image containing the occluded fruit to be estimated; process the image using a first neural network to generate a first feature map representing the visible area of ​​the occluded fruit and a second feature map representing the contact edge between the fruit and the occluder; fuse the first feature map and the second feature map to generate a fused feature containing information about the visible area and the contact edge. The area of ​​the visible part of the fruit is calculated based on the first feature map as an initial size clue; Based on the initial size clue, multiple shape prototypes with matching size ranges are selected from the preset shape prototype codebook to form a candidate prototype subset. The fusion feature and the initial size clue are input into the second neural network. The second neural network predicts the weight coefficients corresponding to each shape prototype in the candidate prototype subset based on the fusion feature, and performs a weighted summation on the shape prototypes in the candidate prototype subset based on the weight coefficients to generate a complete fruit shape mask. Based on the outline of the complete fruit shape mask, the caliper diameter of the outline in multiple preset directions is calculated; the statistical average of the multiple caliper diameters is determined as the estimated size of the fruit, and the fruit size is used to determine whether to pick it.

2. The method according to claim 1, characterized in that, The step of processing the image using a first neural network to generate a first feature map representing the visible area of ​​the occluded fruit and a second feature map representing the contact edge between the fruit and the occluder includes: using a neural network with an encoder-decoder structure as the first neural network; the encoder downsampling the image to extract depth features, the decoder upsampling the depth features, and generating the first feature map and the second feature map respectively through two independent output branches.

3. The method according to claim 1, characterized in that, The step of fusing the first feature map and the second feature map to generate a fused feature containing visible area and contact edge information includes: obtaining the fused feature by performing an element-wise addition operation between the first feature map and the second feature map.

4. The method according to claim 1, characterized in that, The step of calculating the area of ​​the visible portion of the fruit based on the first feature map includes: performing thresholding processing on the first feature map to generate a binary mask; counting the total number of pixels representing the fruit in the binary mask, and using the total number as the area of ​​the visible portion of the fruit.

5. The method according to claim 1, characterized in that, The step of selecting multiple shape prototypes with matching size ranges from a preset shape prototype codebook to form a candidate prototype subset includes: denoting the initial size clue as... Set a size range , where r is a preset scaling factor; K shape prototypes whose areas fall within the size range are selected from the shape prototype encoding to form a candidate prototype subset, where K is a preset positive integer.

6. The method according to any one of claims 1-5, characterized in that, The step of calculating the caliper diameter of the outline of the complete fruit shape mask in multiple preset directions based on the outline of the complete fruit shape mask includes: selecting multiple directions as preset directions within a range of 0° to 180° with preset angle intervals; for each preset direction, calculating the projection length of the outline of the complete fruit shape mask in the direction as the caliper diameter of the direction.

7. The method according to claim 1, characterized in that, The method of determining whether to pick based on fruit size specifically involves: comparing the estimated size of the fruit with a preset minimum picking size threshold; when the estimated size of the fruit is greater than or equal to the minimum picking size threshold, the fruit is determined to be ready for picking, and a picking instruction is generated.

8. An occlusion-resistant deep learning-based fruit recognition and harvesting system, characterized in that, The system includes the following modules: an acquisition module, used to acquire an image containing the occluded fruit to be estimated; to process the image using a first neural network to generate a first feature map representing the visible area of ​​the occluded fruit and a second feature map representing the contact edge between the fruit and the occluder; and a first generation module, used to fuse the first feature map and the second feature map to generate a fused feature containing information about the visible area and the contact edge. The area of ​​the visible part of the fruit is calculated based on the first feature map as an initial size clue; The second generation module is used to select multiple shape prototypes with matching size ranges from a preset shape prototype codebook based on the initial size clues, forming a candidate prototype subset. The fusion feature and the initial size clues are input into the second neural network. The second neural network predicts the weight coefficients corresponding to each shape prototype in the candidate prototype subset based on the fusion feature, and performs a weighted summation on the shape prototypes in the candidate prototype subset based on the weight coefficients to generate a complete fruit shape mask. The calculation module is used to calculate the caliper diameter of the outline in multiple preset directions based on the outline of the complete fruit shape mask; determine the statistical average of the multiple caliper diameters as the estimated size of the fruit; and determine whether to pick the fruit based on the size of the fruit.

9. The system according to claim 8, characterized in that, The step of processing the image using a first neural network to generate a first feature map representing the visible area of ​​the occluded fruit and a second feature map representing the contact edge between the fruit and the occluder includes: using a neural network with an encoder-decoder structure as the first neural network; the encoder downsampling the image to extract depth features, the decoder upsampling the depth features, and generating the first feature map and the second feature map respectively through two independent output branches.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program, when executed by a processor, implements the method as described in any one of claims 1-7.

Images