Fruit maturity detection and grading picking method and system

Abstract

The invention provides a fruit maturity detection and grading picking method and system, and the method comprises the steps: employing a network of a parallel color texture and space contour attention module to extract a fruit image fusion feature map, and enhancing the color texture and boundary information; generating an initial mask based on the feature map, establishing a joint prediction model for a mask overlapping region, calculating the membership probability of overlapping pixels to adjacent instances by using the boundary response of the feature map and the contour curvature of a non-overlapping region, and distributing the pixels to obtain a complete shape mask of each fruit; constructing a fruit maturity feature vector based on the complete mask and evaluating the shielding degree; the shielding degree is determined by the area ratio of the non-overlapping region to the overlapping region belonging to the fruit; inputting the feature vectors into a classifier to judge the maturity level; and establishing a cost function for the fruits reaching the standard to generate an optimal picking sequence.

Description

Technical Field

[0001] This application belongs to the field of vision, and in particular relates to a method and system for fruit ripeness detection, grading and harvesting. Background Technology

[0002] Machine vision-based fruit maturity detection and grading harvesting technologies typically employ image processing and deep learning methods to identify and analyze fruits in orchard environments. However, when fruits occlude each other, image segmentation algorithms or general instance segmentation networks struggle to separate each fruit, leading to overlapping segmentation masks, incomplete edges, or target loss. Because complete visual information of the fruit cannot be obtained, misjudgments of maturity are easily made when key areas for determining maturity are occluded. Furthermore, most harvesting systems generate harvesting sequences based on the determined maturity level after identifying the harvesting target, without considering the positional information of the robotic arm. Simultaneously, the maneuverability of the fruit itself, such as its shape and posture, is rarely analyzed to ensure it is suitable for robotic grasping. This single-dimensional decision-making approach results in harvesting robots planning suboptimal harvesting paths when facing dense fruit clusters, leading to low operational efficiency and even collisions with surrounding branches, leaves, or other fruits during grasping, reducing harvesting success rates and potentially damaging both the fruit and the plant. Therefore, how to make intelligent harvesting decisions by integrating multi-dimensional information under severe obstruction is a technical challenge that urgently needs to be solved in the field of automated harvesting. Summary of the Invention

[0003] This invention proposes a method for fruit ripeness detection and grading during harvesting, addressing the problem that existing technologies cannot make intelligent harvesting decisions by integrating multi-dimensional information when fruits are severely obscured. The method includes: A feature extraction network with parallel color texture attention module and spatial contour attention module is used to extract a fusion attention feature map of the fruit image, which enhances the fruit color texture and spatial boundary information in the feature map; Based on the fused attention feature map, an initial mask for each fruit is generated, and a joint prediction model is established for the overlapping area between the masks. The membership probability of each pixel in the overlapping area to the neighboring instance is calculated, and the pixel is assigned to the instance with the highest membership probability to generate the complete shape mask for each fruit. For the non-overlapping regions in the complete shape mask of the fruit, a first feature vector is generated on the color texture detail feature map through pooling operation; for the overlapping regions in the complete shape mask of the fruit, the color texture features are aggregated according to the membership probability of each pixel to generate a second feature vector; the first feature vector and the second feature vector are concatenated to obtain the maturity feature vector; the value obtained by dividing the number of overlapping region pixels belonging to the fruit by the total number of pixels in the complete shape mask is used as the degree of occlusion. The maturity feature vector is input into the maturity classifier to determine the maturity level of each fruit; for fruits that meet the picking criteria, a cost function is established based on the degree of shading and the maturity level to generate the optimal picking sequence.

[0004] Furthermore, this invention also relates to a fruit maturity detection and grading harvesting system, comprising the following modules: The enhancement module is used to extract a fusion attention feature map of the fruit image by using a feature extraction network with a parallel setting of a color texture attention module and a spatial contour attention module, and to enhance the fruit color texture and spatial boundary information in the feature map; The allocation module is used to generate an initial mask for each fruit based on the fused attention feature map, establish a joint prediction model for the overlapping area between the masks, calculate the membership probability of each pixel in the overlapping area to belong to the neighboring instance, and assign the pixel to the instance with the highest membership probability to generate the complete shape mask for each fruit. The construction module is used to generate a first feature vector on the color texture detail feature map for the non-overlapping regions in the fruit's complete shape mask through pooling operations; for the overlapping regions in the fruit's complete shape mask, the color texture features are aggregated according to the membership probability of each pixel to generate a second feature vector; the first feature vector and the second feature vector are concatenated to obtain a maturity feature vector; the value obtained by dividing the number of overlapping region pixels belonging to the fruit by the total number of pixels in the complete shape mask is used as the degree of occlusion. The picking module is used to input the maturity feature vector into the maturity classifier to determine the maturity level of each fruit; for fruits that meet the picking criteria, at least based on the degree of shading and maturity level, a cost function is established to generate the optimal picking sequence.

[0005] This invention obtains a highly recognizable fused attention feature map by parallel enhancing the color texture and spatial boundary information of fruit images. Furthermore, it can recover the complete shape of each fruit within an overlapping area based on the fruit's boundary response and the outline of its visible portion. By aggregating visible area features and probabilistically weighting occluded area features, a maturity feature vector reflecting the overall condition of the fruit is constructed, improving the reliability of maturity grading. The established harvesting decision cost function considers fruit value, operational success rate, and the movement cost due to occlusion, achieving reasonable harvesting sequence planning and improving the performance of automated harvesting operations. Attached Figure Description

[0006] Figure 1 A flowchart of the first embodiment; Figure 2 This is a schematic diagram of the feature extraction network structure; Figure 3 A schematic diagram is constructed for maturity characteristics; Figure 4 This is a schematic diagram for assessing the degree of occlusion. Detailed Implementation

[0007] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0008] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0009] In the first embodiment, the present invention proposes a method for fruit ripeness detection and grading during harvesting, such as... Figure 1 ,include: S1, a feature extraction network with parallel color texture attention module and spatial contour attention module is used to extract a fusion attention feature map of the fruit image, thereby enhancing the fruit color texture and spatial boundary information in the feature map; The ResNet-50 backbone network is used to extract basic feature maps. The color and texture attention module performs global average pooling on the basic feature maps to obtain channel representations. Then, the dependencies between channels are learned through two fully connected layers to generate channel attention weights, which are multiplied back into the basic feature maps. The spatial contour attention module performs max pooling and average pooling on the basic feature maps along the channel dimensions. The two results are concatenated and passed through a convolutional layer to generate a spatial attention map, which is then multiplied with the basic feature maps. The feature maps output by the two modules are added element-wise to generate a fused attention feature map.

[0010] In an optional embodiment, the feature extraction network employing a parallel configuration of a color texture attention module and a spatial contour attention module to extract a fused attention feature map of the fruit image includes: The color and texture attention module consists of multiple convolutional layers and is used to extract the color and texture detail feature maps of the fruit. The spatial contour attention module enhances the boundary contour feature map of the fruit through edge detection operators and pooling operations. The color texture detail feature map and the boundary contour feature map are concatenated and fused to output the fused attention feature map.

[0011] Specifically, features are extracted through a color texture attention module, which consists of three consecutive convolutional layers. The first convolutional block contains a 3×3 kernel with a stride of 2, followed by a batch normalization layer and a ReLU activation function for downsampling and extracting primary features. The subsequent two convolutional blocks both use 3×3 kernels with a stride of 1, followed by batch normalization layers and a ReLU activation function to deepen feature representation. Taking a 256×256×3 RGB image of a fruit as input, this module generates a 64×64×128 color texture detail feature map. This feature map can detect subtle color changes on the fruit surface, such as gradients from green to yellow, as well as high-frequency details of spots or patterns on the skin.

[0012] The input 256×256×3 image is converted to grayscale. This grayscale image is then processed using an edge detection operator such as Sobel to generate a 256×256×1 edge response map. Higher pixel values ​​in this map indicate a greater likelihood that the point is a boundary. A 4×4 max pooling operation with a stride of 4 is then performed on this edge response map to reduce its dimension to 64×64×1, resulting in the boundary contour feature map.

[0013] The 64×64×128 color texture detail feature map obtained in the first two steps is concatenated with the 64×64×1 boundary contour feature map along the channel dimension. The concatenated result is a fused attention feature map of size 64×64×129. This fused feature map combines detailed internal appearance information and external contour information of the fruit, enabling the network model to simultaneously focus on what the fruit is and where it is, such as... Figure 2 In this invention, unless otherwise specified, for cases with different dimensions, they can be converted to be consistent through scale transformation, which is obvious to those skilled in the art and will not be elaborated further.

[0014] The feature extraction network, trained as part of a larger instance segmentation framework, has a two-branch structure. The first branch is a color and texture attention module, consisting of three concatenated convolutional blocks. Each convolutional block contains a 3×3 kernel layer with a stride of 1, a batch normalization layer, and a ReLU activation function. The second branch is a spatial contour attention module, utilizing the Sobel operator and max pooling, but not participating in gradient updates. The output feature maps of the two branches are concatenated along the channel dimension to form a fused feature map. The training set contains thousands of fruit images from different scenes, each image annotated to provide a pixel-level segmentation mask and a corresponding maturity level label for each fruit instance. The network input is a normalized RGB image of fruit, 256×256×3 in size. The network output is a fused attention feature map, 64×64×129 in size, integrating color, texture, and spatial contour information.

[0015] S2, Generate an initial mask for each fruit based on the fused attention feature map, establish a joint prediction model for the overlapping area between the masks, calculate the membership probability of each pixel in the overlapping area to belong to the neighboring instance, and assign the pixel to the instance with the highest membership probability to generate the complete shape mask for each fruit. The fused attention feature map is input into the mask prediction head network to generate the initial mask for each fruit instance. The Sobel edge detection operator is applied to the fused attention feature map to obtain the boundary response map. The contours of the non-overlapping regions are extracted from each initial mask and the average curvature is calculated. For any pixel in the overlapping region, the joint prediction model is input with the value of the pixel in the boundary response map and its distance to the contours of adjacent instances. Combined with the average curvature of the non-overlapping region of the instance, the membership probability of the pixel belonging to this instance is calculated by a multilayer perceptron. After traversing all adjacent instances, the pixel is assigned to the instance with the highest membership probability, thus completing the assignment of pixels in all overlapping regions.

[0016] The mask prediction head network employs a lightweight fully convolutional structure, with its input being a 64×64×129 fused attention feature map output from the feature extraction network. This head network consists of two consecutive convolutional modules: the first module contains a 3×3 convolutional layer with 256 output channels, a batch normalization layer, and a ReLU activation function; the second module contains a 3×3 convolutional layer with 128 output channels and the same batch normalization and ReLU activation. Finally, a 1×1 convolutional layer maps the number of channels to K×M×M, where K is the model's preset maximum number of instances, and M is the resolution of the predicted mask, such as 28×28. Each M×M slice corresponds to the initial mask of a candidate instance.

[0017] In yet another embodiment, generating the initial mask for each fruit based on the fused attention feature map includes: Multi-scale feature mapping is performed on the fused attention feature map, and local and global response features of the fruit are extracted by setting convolution kernels with different receptive fields; Based on the multi-scale feature mapping, a pixel-level instance segmentation prediction network is used to perform foreground and background discrimination on each pixel in the fused attention feature map to obtain the probability response map of the fruit candidate region. Threshold segmentation and connected component analysis are performed on the probability response map to generate candidate regions corresponding to each fruit instance; Morphological operations are performed on the candidate regions to eliminate noise and fill gaps, resulting in the initial mask for each fruit.

[0018] Specifically, the fused attention feature map is input into a mask prediction head network to obtain initial masks for each fruit instance. The mask prediction head network employs a lightweight fully convolutional structure to perform end-to-end pixel-level prediction on the fused attention feature map, outputting a set of initial masks corresponding to multiple candidate fruit instances. A Sobel edge detection operator is applied to the fused attention feature map to obtain a boundary response map reflecting the strength of the fruit's spatial boundary. The outer contours of non-overlapping regions in each initial mask are extracted, and the average curvature of the corresponding contours is calculated to characterize the overall shape features of the fruit instance in an unoccluded state. For overlapping areas between the initial masks, a joint prediction model is constructed to complete pixel-level fine allocation. For any pixel within the overlapping area, the response value of the pixel on the boundary response map, its distance to the non-overlapping contour of the adjacent fruit instance, and the average curvature of the corresponding instance are used as joint features and input into the multilayer perceptron model to calculate the membership probability of the pixel to each adjacent fruit instance. After traversing all adjacent instances, the pixel is assigned to the fruit instance with the highest membership probability, thereby realizing point-by-point determination and assignment of pixels in the overlapping area and generating instance masks with clear boundaries and complete shapes for each fruit.

[0019] In an optional embodiment, establishing a joint prediction model for the overlapping regions between masks and calculating the membership probability of each pixel within the overlapping region belonging to a neighboring instance includes: Calculate the boundary response value of each pixel within the overlapping region in the fused attention feature map; Extract the symbolic distance from each pixel to the initial mask contour of each adjacent instance; Extract the contours of adjacent instances in the non-overlapping region and calculate the contour curvature index; The joint prediction model is a trainable multilayer perceptron. It takes the aforementioned boundary response value, symbolic distance, and contour curvature index as input features, and directly outputs the initial confidence that the pixel belongs to the current instance through network forward propagation.

[0020] The initial confidence scores of all competing instances are Softmax normalized to obtain the final membership probabilities.

[0021] Assume pixel P is located within the overlapping region of fruit A and fruit B. Through the above process, the feature vector of instance A is extracted. This includes the boundary response value of 0.8 and the distance to profile A. =5, the mean curvature of the profile of A is 0.2, and the eigenvectors of instance B. This includes the boundary response value of 0.8 and the distance to profile B. =12, the average curvature of the profile of B is 0.7. (The remaining text appears to be incomplete and requires further context.) and Input the trained joint prediction module to obtain the initial confidence level. and ,For example =0.85, =0.15. After Softmax normalization, the final membership probability of point P belonging to fruit A is P(P∈A)≈0.67, and the probability of belonging to fruit B is P(P∈B)≈0.33. This process is repeated for each pixel in the overlapping region to determine the final membership of each pixel. During training, the joint prediction module is optimized end-to-end along with the feature extraction network and the mask prediction head network. Its training signal comes from the ground truth labels of pixels in the overlapping region, and the loss function is cross-entropy loss. The weights of the MLP and the parameters of the feature extraction network are updated through backpropagation.

[0022] In an optional embodiment, establishing a joint prediction model for the overlapping regions between masks and calculating the membership probability of each pixel within the overlapping region belonging to a neighboring instance includes: Calculate the boundary response value of each pixel within the overlapping region in the fused attention feature map; Extract the contours of adjacent instances in the non-overlapping region and calculate the contour curvature index; The membership probability is determined based on the boundary response value, the contour curvature index of the instance, and the distance from the pixel to the contour of the non-overlapping region of the instance, and then normalized.

[0023] Assuming pixel P is located within the overlapping region of fruit A and fruit B, the value corresponding to point P is found in the boundary contour channel of the fused attention feature map; this value is the boundary response value, for example, 0.8. Simultaneously, contour segments of fruit A and fruit B in the non-overlapping regions are extracted, and the curvature of each point on these segments is calculated using the difference method. The average value is then used to obtain the respective contour curvature index. For example, fruit A has a smooth contour with a curvature index of 0.2, while fruit B has a more curved contour with a curvature index of 0.7.

[0024] For pixel P, calculate the shortest Euclidean distance from the pixel to the non-overlapping contours of fruit A and fruit B, assuming they are respectively... Pixels and Pixels are obtained by performing min-max normalization on the distance features. , Plan a membership scoring function for each instance; for example, for fruit A, the score is... Linear models are ,in , , represents the weighting coefficient. Based on the example data, scores will tend to favor instances that are close together and have smooth contours.

[0025] Calculate the membership scores of pixel P for fruits A and B. and Then, the probability of belonging to each instance is obtained by normalization using the Softmax function. For example, the probability of point P belonging to fruit A is... If the calculation yields , ,but , Repeat the process for each pixel within the overlapping region to determine the possible affiliation of each pixel.

[0026] S3, Based on the complete shape mask, construct the maturity feature vector of each fruit and evaluate the degree of occlusion; For each fruit's complete shape mask, extract all pixels within the non-overlapping region, calculate the H-channel color histogram of the pixel in the HSV color space, and the energy and contrast texture features based on the gray-level co-occurrence matrix; extract pixels within the overlapping region, weight each pixel according to the membership probability calculated in step S2, and calculate the weighted H-channel color histogram and weighted texture features; concatenate the features of the non-overlapping region with the weighted features of the overlapping region to form a maturity feature vector; simultaneously, calculate the total number of pixels A in the non-overlapping region of the fruit and the total number of pixels B in the overlapping region belonging to it, with the occlusion degree being B / (A+B).

[0027] More specifically, for the non-overlapping regions in the fruit's complete shape mask, a first feature vector is generated on the color texture detail feature map through pooling operations; for the overlapping regions in the fruit's complete shape mask, the color texture features are aggregated according to the membership probability of each pixel to generate a second feature vector; the first feature vector and the second feature vector are concatenated to obtain a maturity feature vector; the value obtained by dividing the number of overlapping region pixels belonging to the fruit by the total number of pixels in the complete shape mask is used as the degree of occlusion.

[0028] For a specific fruit, a full shape mask is used to identify regions that do not overlap with other fruits. Assume this region contains 2000 pixels. The positions of these 2000 pixels are mapped onto a 64×64×128-dimensional color and texture detail feature map generated by the color and texture attention module, and the corresponding 2000 128-dimensional feature vectors are extracted. Global average pooling is performed on these 2000 feature vectors, i.e., they are summed element-wise and then averaged to generate a 128-dimensional first feature vector. This vector represents the core color and texture attributes of the clearly visible part of the fruit.

[0029] The region in the fruit mask that overlaps with other fruits is processed, assuming this region contains 500 pixels. For any pixel i among these 500 pixels, the membership probability of belonging to the current fruit has been calculated. It can also extract the corresponding 128-dimensional feature vector from the color texture detail feature map. Information is aggregated by performing a probability-weighted summation on the feature vectors to generate a second feature sub-vector: This allows pixels with higher membership probabilities to contribute more to the features, thus extracting feature information related to the current fruit from regions where membership has not yet been determined.

[0030] Optionally, the second feature vector is generated by aggregating the color and texture features of the overlapping regions in the complete shape mask of the fruit based on the membership probability of each pixel, including: For each pixel within the overlapping region, obtain its corresponding pixel-level features on the color texture detail feature map; The pixel-level feature is multiplied by the membership probability of the pixel belonging to the current target fruit instance to obtain the weighted feature of the pixel; The weighted features of all overlapping region pixels belonging to the current target fruit instance are summed or pooled to obtain the second feature vector.

[0031] Specifically, for any target fruit, pixels are divided into non-overlapping and overlapping regions based on its complete shape mask. For the non-overlapping regions, the corresponding features of pixels within these regions in the color texture detail feature map are uniformly indexed, and global average pooling or max pooling is used to converge the features, resulting in a first feature vector representing the color distribution and texture structure of the clearly visible part of the fruit. This first feature vector primarily reflects the intrinsic maturity characteristics of the fruit under unobstructed conditions, reducing the impact of occlusion and background interference on maturity judgment. For each pixel within the overlapping region, its corresponding pixel-level feature vector in the color texture detail feature map is extracted, and this feature vector is multiplied by the membership probability of the pixel belonging to the current fruit instance to obtain a weighted feature. Summation or normalized pooling is performed on all weighted features to obtain a second feature vector. The first and second feature vectors are concatenated to obtain a maturity feature vector used for maturity judgment. The ratio of the number of overlapping region pixels allocated to the current fruit to the total number of pixels in the fruit's complete shape mask is defined as the occlusion degree, used to indicate the severity of occlusion of the fruit. Optionally, the first feature vector and the second feature vector can be normalized before splicing.

[0032] The 128-dimensional first feature vector and the 128-dimensional second feature vector obtained above are concatenated along the dimensional direction. This produces a 256-dimensional vector that contains both the deterministic features of the unobstructed portion of the fruit and the probabilistic features of the obstructed portion, forming a comprehensive representation of the fruit's complete state, i.e., the maturity feature vector, as shown below. Figure 3 .

[0033] For each fruit instance identified in the image, area is calculated based on the segmentation mask. Total area of ​​the complete shape mask. This refers to the total number of pixels belonging to the fruit, including both non-overlapping areas and the portion of the overlapping area allocated to the fruit. The non-overlapping area refers to the total number of pixels within the fruit mask that do not intersect with any other fruit mask. For example, for fruit A, the non-overlapping area is 3000 pixels. The area allocated to fruit A in the overlapping area is B = 400 pixels. The complete area... =A+B=3400 pixels.

[0034] The formula for calculating the degree of occlusion D is: Substituting the obtained example data into the formula, the degree of occlusion of fruit A is: The value ranges from [0,1], with larger values ​​indicating a higher proportion of the fruit being obscured, resulting in lower visibility. For example... Figure 4 .

[0035] S4, input the maturity feature vector into the maturity classifier to determine the maturity level of each fruit; for fruits that meet the picking standards, establish a cost function based on at least the degree of shading and the maturity level to generate the optimal picking sequence.

[0036] A maturity feature vector, formed by concatenating the first feature vector of the non-overlapping region and the second feature vector of the overlapping region, is input into a pre-trained maturity classifier to determine the maturity status of each fruit. In one embodiment, the maturity classifier is a support vector machine classifier, and its output includes three maturity levels: unripe, ripe, and overripe. Only fruits determined to be ripe or overripe are included in the subsequent picking decision set to avoid ineffective planning for fruits that do not meet the picking criteria.

[0037] For each fruit that meets the picking conditions, a cost function is constructed using maturity level, occlusion degree, graspability, and picking motion cost. The occlusion degree is derived from the ratio of the number of pixels in the overlapping area of ​​the fruit calculated in the preceding steps to the total number of pixels in the complete shape mask. This ratio measures the degree to which the fruit is occluded by branches, leaves, or other fruits from the current viewpoint. A higher occlusion degree indicates a more limited space for direct access to the fruit, resulting in a higher risk of collision or failed grasping during picking. Therefore, it is used as a penalty factor in the cost function calculation.

[0038] In the specific implementation, the outline of the fruit is fitted with a minimum bounding ellipse based on the complete shape mask of the fruit. The graspability of the fruit is characterized by calculating the ratio of the length of the major axis to the minor axis of the ellipse. The closer the ratio is to 1, the more regular the fruit shape and the more favorable the posture, resulting in a higher graspability score. Secondly, the picking motion cost is based on the Euclidean distance from the current position of the robot's end effector to the centroid of the fruit, and the degree of occlusion is introduced as a weighted penalty term, so that fruits with a higher degree of occlusion correspond to a greater motion cost. The cost function is the sum of the picking motion cost and the occlusion penalty, minus the weighted sum of the maturity level score and the graspability score. All fruits to be picked are regarded as a set of nodes, and a picking path optimization problem is constructed. A greedy algorithm or path planning algorithm is used to generate the optimal picking sequence that meets the requirements of maturity priority, occlusion avoidance, and motion efficiency by minimizing the total cost.

[0039] In an optional embodiment, inputting the maturity feature vector into a maturity classifier to determine the maturity level of each fruit includes: The maturity classifier is a support vector machine classifier; Multiple maturity levels are predefined, and the classifier maps the input maturity feature vector to one of the levels.

[0040] Before deployment, a labeled training dataset needs to be collected. This dataset contains images of hundreds of fruits. For each fruit, a 256-dimensional maturity feature vector is extracted, and human experts label the maturity level. The predefined maturity levels are divided into three categories: Level 1 represents immature, Level 2 represents semi-ripe, and Level 3 represents fully ripe. A multi-class support vector machine model is trained using these feature vectors and their corresponding level labels. During training, a radial basis function (RBF) kernel is typically used to find the optimal classification hyperplane by maximizing the inter-class margin.

[0041] In practical applications, once a new fruit is detected and a 256-dimensional maturity feature vector is extracted, this vector is fed as input to a pre-trained support vector machine (SVM) classifier. The classifier calculates the distance between this feature vector and the decision boundary corresponding to each level, and classifies it into the maturity level with the closest distance or the highest score based on the discriminant function. For example, if the classifier outputs a value of 3 after calculating a given feature vector, it indicates that the fruit is fully ripe.

[0042] In an optional embodiment, the step of generating an optimal harvesting sequence for fruits that meet the harvesting criteria, at least based on the degree of shading and maturity level, includes establishing a cost function and including: The graspability G of the fruit is determined based on the ratio of the major and minor axes of the ellipse fitted by the complete shape mask. Obtain the Euclidean distance L from the robot's current position to the centroid of each fruit; Define the cost function Where M represents the value corresponding to the maturity level, and D represents the degree of shading. , , , For preset weighting coefficients, The normalized Euclidean distance; A greedy algorithm is used to generate the picking sequence.

[0043] Specifically, for each fruit to be evaluated, the graspability G is calculated. By fitting an ellipse to the complete shape mask of the fruit, the major axis a and minor axis b of the ellipse are obtained, then G = a / b. A nearly spherical fruit has a G value close to 1, indicating it is easy to grasp. A value M is assigned according to the maturity level, for example, fully ripe corresponds to M=10, semi-ripe to M=5, and unripe to M=1. The Euclidean distance L from the robot's current position to the centroid of the fruit is calculated, and the... For example, let's set the distance to the farthest fruit. =3 meters, for a fruit with L=1.2 meters, then =0.4. Optionally, before calculation, the values ​​of G, D, and 1 / M are normalized.

[0044] Setting weighting coefficients to balance the importance of different factors needs to be adjusted according to the actual scenario, for example... , , , Assume a fruit is judged to be fully ripe (M=10), nearly round in shape (G=1.05), and has good visibility (D=0.15). The distance... =0.4. Therefore, the cost value J is calculated as: J=0.745, yielding a comprehensive cost value for each fruit.

[0045] A greedy algorithm is used to select the fruit with the smallest J value from the fruits to be picked as the next target. After picking, the robot position is updated to the position of that fruit, and the L and J values ​​of the remaining fruits to the new position are recalculated. This process is repeated until all fruits have been picked.

[0046] In a second embodiment, the present invention also proposes a fruit maturity detection and grading harvesting system, comprising the following modules: The enhancement module is used to extract a fusion attention feature map of the fruit image by using a feature extraction network with a parallel setting of a color texture attention module and a spatial contour attention module, and to enhance the fruit color texture and spatial boundary information in the feature map; The allocation module is used to generate an initial mask for each fruit based on the fused attention feature map, establish a joint prediction model for the overlapping area between the masks, calculate the membership probability of each pixel in the overlapping area to belong to the neighboring instance, and assign the pixel to the instance with the highest membership probability to generate the complete shape mask for each fruit. The construction module is used to generate a first feature vector on the color texture detail feature map for the non-overlapping regions in the fruit's complete shape mask through pooling operations; for the overlapping regions in the fruit's complete shape mask, the color texture features are aggregated according to the membership probability of each pixel to generate a second feature vector; the first feature vector and the second feature vector are concatenated to obtain a maturity feature vector; the value obtained by dividing the number of overlapping region pixels belonging to the fruit by the total number of pixels in the complete shape mask is used as the degree of occlusion. The picking module is used to input the maturity feature vector into the maturity classifier to determine the maturity level of each fruit; for fruits that meet the picking criteria, at least based on the degree of shading and maturity level, a cost function is established to generate the optimal picking sequence.

[0047] In some embodiments, the feature extraction network employing a parallel configuration of a color texture attention module and a spatial contour attention module to extract a fused attention feature map of the fruit image includes: The color and texture attention module consists of multiple convolutional layers and is used to extract the color and texture detail feature maps of the fruit. The spatial contour attention module enhances the boundary contour feature map of the fruit through edge detection operators and pooling operations. The color texture detail feature map and the boundary contour feature map are concatenated and fused to output the fused attention feature map.

[0048] In some embodiments, generating an initial mask for each fruit based on the fused attention feature map includes: Multi-scale feature mapping is performed on the fused attention feature map, and local and global response features of the fruit are extracted by setting convolution kernels with different receptive fields; Based on the multi-scale feature mapping, a pixel-level instance segmentation prediction network is used to perform foreground and background discrimination on each pixel in the fused attention feature map to obtain the probability response map of the fruit candidate region. Threshold segmentation and connected component analysis are performed on the probability response map to generate candidate regions corresponding to each fruit instance; Morphological operations are performed on the candidate regions to eliminate noise and fill gaps, resulting in the initial mask for each fruit.

[0049] In some embodiments, establishing a joint prediction model for the overlapping regions between masks and calculating the membership probability of each pixel within the overlapping region belonging to a neighboring instance includes: Calculate the boundary response value of each pixel within the overlapping region in the fused attention feature map; Extract the contours of adjacent instances in the non-overlapping region and calculate the contour curvature index; The membership probability is determined based on the boundary response value, the contour curvature index of the instance, and the distance from the pixel to the contour of the non-overlapping region of the instance, and then normalized.

[0050] In some embodiments, the aggregation of color and texture features in the overlapping region of the fruit's complete shape mask according to the membership probability of each pixel to generate a second feature vector includes: For each pixel within the overlapping region, obtain its corresponding pixel-level features on the color texture detail feature map; The pixel-level feature is multiplied by the membership probability of the pixel belonging to the current target fruit instance to obtain the weighted feature of the pixel; The weighted features of all overlapping region pixels belonging to the current target fruit instance are summed or pooled to obtain the second feature vector.

[0051] In some embodiments, inputting the maturity feature vector into a maturity classifier to determine the maturity level of each fruit includes: The maturity classifier is a support vector machine classifier; Multiple maturity levels are predefined, and the classifier maps the input maturity feature vector to one of the levels.

[0052] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0053] It should be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0054] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0055] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for fruit maturity detection and grading during harvesting, characterized in that, Includes the following steps: A feature extraction network with parallel color texture attention module and spatial contour attention module is used to extract a fusion attention feature map of the fruit image, which enhances the fruit color texture and spatial boundary information in the feature map; Based on the fused attention feature map, an initial mask for each fruit is generated, and a joint prediction model is established for the overlapping area between the masks. The membership probability of each pixel in the overlapping area to the neighboring instance is calculated, and the pixel is assigned to the instance with the highest membership probability to generate the complete shape mask for each fruit. For the non-overlapping regions in the complete shape mask of the fruit, a first feature vector is generated on the color texture detail feature map by pooling operation; For the overlapping areas in the complete shape mask of the fruit, the color and texture features are aggregated according to the membership probability of each pixel to generate a second feature vector; the first feature vector and the second feature vector are concatenated to obtain the maturity feature vector; the value obtained by dividing the number of overlapping area pixels belonging to the fruit by the total number of pixels in the complete shape mask is used as the degree of occlusion. The maturity feature vector is input into the maturity classifier to determine the maturity level of each fruit; for fruits that meet the picking criteria, a cost function is established based on the degree of shading and the maturity level to generate the optimal picking sequence.

2. The method according to claim 1, characterized in that, The feature extraction network, which employs a parallel configuration of a color texture attention module and a spatial contour attention module, extracts a fused attention feature map of the fruit image, including: The color and texture attention module consists of multiple convolutional layers and is used to extract the color and texture detail feature maps of the fruit. The spatial contour attention module enhances the boundary contour feature map of the fruit through edge detection operators and pooling operations. The color texture detail feature map and the boundary contour feature map are concatenated and fused to output the fused attention feature map.

3. The method according to claim 1, characterized in that, The process of generating an initial mask for each fruit based on the fused attention feature map includes: Multi-scale feature mapping is performed on the fused attention feature map, and local and global response features of the fruit are extracted by setting convolution kernels with different receptive fields; Based on the multi-scale feature mapping, a pixel-level instance segmentation prediction network is used to perform foreground and background discrimination on each pixel in the fused attention feature map to obtain the probability response map of the fruit candidate region. Threshold segmentation and connected component analysis are performed on the probability response map to generate candidate regions corresponding to each fruit instance; Morphological operations are performed on the candidate regions to eliminate noise and fill gaps, resulting in the initial mask for each fruit.

4. The method according to claim 1 or 2, characterized in that, The establishment of a joint prediction model for the overlapping regions between masks, and the calculation of the membership probability of each pixel within the overlapping region belonging to its neighboring instance, includes: Calculate the boundary response value of each pixel within the overlapping region in the fused attention feature map; Extract the contours of adjacent instances in the non-overlapping region and calculate the contour curvature index; The membership probability is determined based on the boundary response value, the contour curvature index of the instance, and the distance from the pixel to the contour of the non-overlapping region of the instance, and then normalized.

5. The method according to claim 1, characterized in that, The overlapping regions in the complete fruit shape mask are aggregated based on the membership probability of each pixel to generate a second feature vector, including: For each pixel within the overlapping region, obtain its corresponding pixel-level features on the color texture detail feature map; The pixel-level feature is multiplied by the membership probability of the pixel belonging to the current target fruit instance to obtain the weighted feature of the pixel; The weighted features of all overlapping region pixels belonging to the current target fruit instance are summed or pooled to obtain the second feature vector.

6. The method according to claim 1, characterized in that, The step of inputting the maturity feature vector into the maturity classifier to determine the maturity level of each fruit includes: The maturity classifier is a support vector machine classifier; Multiple maturity levels are predefined, and the classifier maps the input maturity feature vector to one of the levels.

7. A fruit maturity detection and grading harvesting system, characterized in that, Includes the following modules: The enhancement module is used to extract a fusion attention feature map of the fruit image by using a feature extraction network with a parallel setting of a color texture attention module and a spatial contour attention module, and to enhance the fruit color texture and spatial boundary information in the feature map; The allocation module is used to generate an initial mask for each fruit based on the fused attention feature map, establish a joint prediction model for the overlapping area between the masks, calculate the membership probability of each pixel in the overlapping area to belong to the neighboring instance, and assign the pixel to the instance with the highest membership probability to generate the complete shape mask for each fruit. The construction module is used to generate a first feature vector on the color texture detail feature map by pooling the non-overlapping regions in the complete shape mask of the fruit; For the overlapping areas in the complete shape mask of the fruit, the color and texture features are aggregated according to the membership probability of each pixel to generate a second feature vector; the first feature vector and the second feature vector are concatenated to obtain the maturity feature vector; the value obtained by dividing the number of overlapping area pixels belonging to the fruit by the total number of pixels in the complete shape mask is used as the degree of occlusion. The picking module is used to input the maturity feature vector into the maturity classifier to determine the maturity level of each fruit; for fruits that meet the picking criteria, at least based on the degree of shading and maturity level, a cost function is established to generate the optimal picking sequence.

8. The system according to claim 7, characterized in that, The feature extraction network, which employs a parallel configuration of a color texture attention module and a spatial contour attention module, extracts a fused attention feature map of the fruit image, including: The color and texture attention module consists of multiple convolutional layers and is used to extract the color and texture detail feature maps of the fruit. The spatial contour attention module enhances the boundary contour feature map of the fruit through edge detection operators and pooling operations. The color texture detail feature map and the boundary contour feature map are concatenated and fused to output the fused attention feature map.

9. The system according to claim 7, characterized in that, The process of generating an initial mask for each fruit based on the fused attention feature map includes: Multi-scale feature mapping is performed on the fused attention feature map, and local and global response features of the fruit are extracted by setting convolution kernels with different receptive fields; Based on the multi-scale feature mapping, a pixel-level instance segmentation prediction network is used to perform foreground and background discrimination on each pixel in the fused attention feature map to obtain the probability response map of the fruit candidate region. Threshold segmentation and connected component analysis are performed on the probability response map to generate candidate regions corresponding to each fruit instance; Morphological operations are performed on the candidate regions to eliminate noise and fill gaps, resulting in the initial mask for each fruit.

10. The system according to claim 7 or 8, characterized in that, The establishment of a joint prediction model for the overlapping regions between masks, and the calculation of the membership probability of each pixel within the overlapping region belonging to its neighboring instance, includes: Calculate the boundary response value of each pixel within the overlapping region in the fused attention feature map; Extract the contours of adjacent instances in the non-overlapping region and calculate the contour curvature index; The membership probability is determined based on the boundary response value, the contour curvature index of the instance, and the distance from the pixel to the contour of the non-overlapping region of the instance, and then normalized.

Images