Pineapple identification and positioning method and system

Abstract

The invention provides a pineapple identification and positioning method and system, and the method comprises the steps: obtaining an initial RGB image and an initial depth image of a to-be-picked pineapple region, carrying out the image preprocessing of the initial RGB image and the initial depth image, and generating a target RGB image and a target depth image; performing instance segmentation on the target RGB image by adopting a target deep learning model to generate a target pineapple mask and pineapple label data; mapping the target pineapple mask to a target depth map based on the pineapple label data to perform point cloud data extraction, and performing clustering segmentation and dynamic filtering optimization on the point cloud data to generate target pineapple leaf point cloud data and target pineapple main body point cloud data; and according to the target pineapple leaf point cloud data and the target pineapple main body point cloud data, calculating grabbing points and grabbing posture data of pineapples in the to-be-picked pineapple area, and generating pineapple identification positioning data. Through multi-modal data alignment, point cloud de-noising dynamic optimization and geometric analysis, high-precision segmentation, grabbing point positioning and attitude estimation of the pineapple are realized, so that the accuracy of the obtained positioning result is high.

Description

Technical Field

[0001] The present invention relates to the field of agricultural automation technology and computer vision recognition and positioning technology, and in particular to a pineapple recognition and positioning method and system. Background Art

[0002] As an important economic crop in tropical regions, pineapple picking operations have long relied on manual experience, resulting in low efficiency, high costs, and labor shortages. With the development of agricultural automation technology, robotic picking has become an important research direction. The core difficulty lies in how to accurately identify pineapple targets and locate the grasping points in complex field environments. Although deep learning-based detection methods have been applied, they are often difficult to accurately match and segment target point cloud data due to insufficient feature extraction, insufficient cross-scale information fusion, and loss of details of distant targets, resulting in limited accuracy in identifying and positioning target differences. Therefore, existing pineapple recognition and positioning still face the following technical bottlenecks:

[0003] 1. Low segmentation accuracy: Existing pineapple recognition methods rely on two-dimensional image features. In complex field environments, pineapple plants are planted in rows of 330mm and columns of 500mm. During the harvest season, mature pineapples are densely packed together, making it difficult to accurately segment the leaves from the main body due to occlusion, leading to mis-segmentation and misidentification.

[0004] 2. Poor anti-interference ability: In orchard environments, occlusion by branches and leaves, changes in illumination, and point cloud data processing are often affected by environmental noise, making it difficult to stably extract target point cloud data, affecting positioning stability.

[0005] 3. Large spatial positioning errors: The system lacks the ability to accurately calculate the three-dimensional coordinates of the grasping point using depth information. The posture estimation method does not fully utilize the three-dimensional geometric features, resulting in a high grasping failure rate. At the same time, multiple pineapples can be easily identified through the camera's field of view, requiring an additional screening mechanism to handle multiple label results. Summary of the Invention

[0006] In view of this, the present invention proposes a pineapple identification and positioning method and system, which solves the technical problem that the existing pineapple identification and positioning method lacks three-dimensional information processing capabilities, resulting in low segmentation accuracy, poor anti-interference ability and large spatial positioning errors in the pineapple identification and positioning process, resulting in low accuracy of positioning results.

[0007] The present invention provides a pineapple identification and positioning method, comprising the following steps:

[0008] Obtaining an initial RGB image and an initial depth map of the area to be picked pineapples, performing image preprocessing on the initial RGB image and the initial depth map to generate a target RGB image and a target depth map;

[0009] Perform instance segmentation on the target RGB image using a target deep learning model to generate target pineapple mask and pineapple label data;

[0010] Based on the pineapple label data, the target pineapple mask is mapped to the target depth map to extract point cloud data, and the point cloud data is clustered and segmented and dynamically filtered to generate target pineapple leaf point cloud data and target pineapple body point cloud data;

[0011] The grasping point and grasping posture data of the pineapples in the pineapple area to be picked are calculated based on the target pineapple leaf point cloud data and the target pineapple body point cloud data to generate pineapple recognition and positioning data.

[0012] Optionally, the step of performing image preprocessing on the initial RGB image and the initial depth map to generate a target RGB image and a target depth map includes:

[0013] Performing temporal alignment and spatial registration using the initial RGB image and the initial depth image to generate an intermediate RGB image and an intermediate depth image;

[0014] Performing image enhancement on the intermediate RGB image to generate an RGB enhanced image;

[0015] Using bilateral filtering to remove noise in the RGB enhanced image to generate a target RGB image;

[0016] Normalizing the depth values ​​of the intermediate depth map to generate a normalized depth map;

[0017] Fill holes in the normalized depth map to generate a target depth map.

[0018] Optionally, before the step of performing instance segmentation on the target RGB image using the target deep learning model to generate target pineapple mask and pineapple label data, the step further includes:

[0019] Obtain a historical pineapple image set in a pineapple planting scenario, perform regional category labeling on the historical pineapple images in the historical pineapple image set, and generate a plurality of target historical pineapple images;

[0020] Preprocess the target historical pineapple images to generate model training data;

[0021] Inputting the model training data into the initial deep learning model for feature extraction and feature fusion to generate a historical pineapple segmentation map;

[0022] Performing classification prediction and segmentation prediction on the historical pineapple segmentation graph, and performing loss value calculation to generate a total loss value;

[0023] When the total loss value is less than the preset loss value threshold, the initial deep learning model at the current moment is used as the target initial deep learning model.

[0024] Optionally, the step of performing classification prediction and segmentation prediction on the historical pineapple segmentation map, and calculating the loss value to generate a total loss value includes:

[0025] Performing classification prediction on the historical pineapple segmentation graph to generate classification prediction data;

[0026] Perform segmentation prediction on the historical pineapple segmentation graph to generate segmentation prediction data;

[0027] A cross entropy loss function is used to calculate the loss value between the classification prediction data and the true classification label corresponding to the model training data to generate a classification loss value;

[0028] Using the Diesel loss function to calculate the loss value between the segmentation prediction data and the true pixel-level mask corresponding to the model training data, to generate a segmentation loss value;

[0029] The classification loss value and the segmentation loss value are weighted and summed to generate a total loss value.

[0030] Optionally, the target deep learning model includes a backbone network, a focused encoder, a cross-scale fusion module, and a focused decoder; and the step of performing instance segmentation on the target RGB image using the target deep learning model to generate target pineapple masks and pineapple label data includes:

[0031] Extracting a hierarchical feature map from the target RGB image through the backbone network to generate a first initial hierarchical feature map, a second initial hierarchical feature map, a third initial hierarchical feature map, and a fourth primary hierarchical feature map;

[0032] Performing feature encoding on the fourth initial level feature map by the focused encoder to generate a spatial feature map;

[0033] Performing weighted fusion on the first initial level feature map, the second initial level feature map, the third initial level feature map, and the spatial feature map through the cross-scale fusion module to generate a first target level feature map, a second target level feature map, a third target level feature map, and a fourth target level feature map;

[0034] Performing pineapple region recognition on the second target-level feature map, the third target-level feature map, and the fourth target-level feature map through the focus decoder to generate pineapple label data and a pineapple segmentation map;

[0035] Perform a matrix multiplication operation on the pineapple segmentation map and the first target level feature map to generate a target pineapple mask.

[0036] Optionally, the step of performing feature encoding on the fourth initial level feature map by the focused encoder to generate a spatial feature map includes:

[0037] Flattening the fourth initial level feature map in spatial dimension to generate a flattened feature sequence;

[0038] Adding the position code corresponding to the flattened feature sequence to the corresponding feature vector to generate an initial input sequence;

[0039] Performing a self-attention operation on the initial input sequence to generate an intermediate input sequence;

[0040] Performing a nonlinear transformation on the intermediate input sequence through a feedforward neural network to generate a target input sequence;

[0041] The target input sequence is converted into a spatial feature map to generate a spatial feature map.

[0042] Optionally, the step of performing pineapple region recognition on the second target-level feature map, the third target-level feature map, and the fourth target-level feature map by the focused decoder to generate pineapple label data and a pineapple segmentation map includes:

[0043] Upsampling the second target-level feature map, the third target-level feature map, and the fourth target-level feature map respectively using a progressive upsampling structure to generate a second sampling feature map, a third sampling feature map, and a fourth sampling feature map;

[0044] Performing cross-scale feature fusion on the second sampling feature map, the third sampling feature map, and the fourth sampling feature map to generate a multi-scale feature map;

[0045] filtering background noise in the multi-scale feature map through a spatial attention gating mechanism to determine a pineapple region feature map;

[0046] The pineapple region feature map is subjected to image segmentation to generate pineapple label data and a pineapple segmentation map.

[0047] Optionally, the step of mapping the target pineapple mask to the target depth map based on the pineapple label data to extract point cloud data, and performing clustering segmentation and dynamic filtering optimization on the point cloud data to generate target pineapple leaf point cloud data and target pineapple body point cloud data includes:

[0048] Mapping the target pineapple mask to the target depth map, extracting depth information of the pineapple leaves and the pineapple body according to the pineapple label data, and generating initial pineapple leaf point cloud data and initial pineapple body point cloud data;

[0049] Clustering and segmenting the initial pineapple leaf point cloud data and the initial pineapple body point cloud data to generate pineapple leaf segmentation data and pineapple body segmentation data;

[0050] Point cloud data greater than or equal to a preset clustering threshold value in the pineapple leaf segmentation data and the pineapple body segmentation data are respectively screened out to generate target pineapple leaf point cloud data and target pineapple body point cloud data.

[0051] Optionally, the step of calculating the grasping point and grasping posture data of the pineapples in the area to be picked according to the target pineapple leaf point cloud data and the target pineapple body point cloud data to generate pineapple identification and positioning data includes:

[0052] Calculating the point cloud data centroid of the target pineapple main body point cloud data to generate three-dimensional centroid coordinates;

[0053] Merging the target pineapple leaf point cloud data and the target pineapple body point cloud data to generate pineapple entity point cloud data;

[0054] Centralizing the pineapple entity point cloud data to generate centralized point cloud data;

[0055] The centralized point cloud data is used to construct a matrix to obtain a covariance matrix;

[0056] Performing eigendecomposition on the covariance matrix to generate a plurality of matrix eigenvalues ​​and a plurality of matrix eigenvectors;

[0057] The matrix eigenvector corresponding to the maximum value of the matrix eigenvalues ​​is used as the main axis direction of the point cloud to generate the target grasping posture axis;

[0058] The target grasping posture axis, all the matrix eigenvalues ​​and all the matrix eigenvectors are used to construct pineapple recognition and positioning data.

[0059] The present invention also provides a pineapple identification and positioning system comprising:

[0060] An image generation module is used to obtain an initial RGB image and an initial depth image of the area to be picked pineapples, perform image preprocessing on the initial RGB image and the initial depth image, and generate a target RGB image and a target depth map;

[0061] A mask and label data generation module is used to perform instance segmentation on the target RGB image using a target deep learning model to generate target pineapple mask and pineapple label data;

[0062] a point cloud data generation module, configured to map the target pineapple mask to the target depth map based on the pineapple label data to extract point cloud data, and perform clustering segmentation and dynamic filtering optimization on the point cloud data to generate target pineapple leaf point cloud data and target pineapple body point cloud data;

[0063] The pineapple identification and positioning data generation module is used to calculate the grasping point and grasping posture data of the pineapple in the pineapple area to be picked according to the target pineapple leaf point cloud data and the target pineapple body point cloud data, and generate pineapple identification and positioning data.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] The present invention simultaneously acquires an initial RGB image and an initial depth map corresponding to the area of ​​pineapples to be picked. The initial RGB image and initial depth map are subjected to image preprocessing to generate a target RGB image and a target depth map, ensuring pixel-level spatial alignment and providing a high-precision data foundation for subsequent segmentation and point cloud generation. The target RGB image is input into a target deep learning model for instance segmentation, generating a target pineapple mask and pineapple label data. The deep learning model accurately distinguishes the pineapple from the background and individual pineapples in the RGB image, determines the outline of each pineapple, and assigns it a category label. The position and range of the pineapple in the image are precisely located, clarifying the target area for subsequent point cloud data extraction, improving pineapple recognition accuracy, and facilitating subsequent operations to focus on the pineapple itself. Based on the pineapple label data, the target pineapple mask is mapped to the target depth map for point cloud data extraction. Cluster segmentation and dynamic filtering optimization are then performed on the point cloud data to generate target pineapple leaf point cloud data and target pineapple body point cloud data. This accurately describes the pineapple's morphology from a three-dimensional perspective, providing critical spatial data support for the subsequent determination of grasping points and grasping postures, enabling the robot to better perceive the pineapple's actual morphology. Based on the point cloud data of the target pineapple leaves and the target pineapple body, the robot calculates the grasping point and grasping posture data for the pineapples in the area to be picked, generating pineapple identification and positioning data. This determines the robot's precise operating position and angle when picking pineapples, enabling precise identification and positioning of the pineapples, improving the success rate and efficiency of picking, reducing damage to the pineapples, and enhancing the quality of automated picking. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only preferred embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0067] Figure 1A flowchart of a pineapple identification and positioning method provided in Example 1 of the present invention;

[0068] Figure 2 Schematic diagram of the structure of the target deep learning model in Example 1 of the present invention;

[0069] Figure 3 This is a structural block diagram of a pineapple identification and positioning system provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0070] The embodiments of the present invention provide a pineapple identification and positioning method and system for solving the technical problem that the existing pineapple identification and positioning methods lack three-dimensional information processing capabilities, resulting in low segmentation accuracy, poor anti-interference ability and large spatial positioning errors in the pineapple identification and positioning process, leading to low accuracy of positioning results.

[0071] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0072] Example 1

[0073] See also Figure 1 , Figure 1 This is a flowchart of the steps of a pineapple identification and positioning method provided in Example 1 of the present invention.

[0074] The present invention provides a pineapple identification and positioning method, comprising:

[0075] Step 101: Obtain an initial RGB image and an initial depth image of the area to be picked pineapples, perform image preprocessing on the initial RGB image and the initial depth image, and generate a target RGB image and a target depth map.

[0076] In this embodiment of the present invention, a 3D camera is used to simultaneously acquire an initial RGB image and an initial depth map corresponding to the area of ​​the pineapple to be picked. This ensures pixel-level spatial alignment of the pineapple RGB image and depth map, providing a high-precision data foundation for subsequent segmentation and point cloud generation.

[0077] Furthermore, the initial RGB image and initial depth map can also be collected using a lidar or a ToF camera, and the RGB image and point cloud data can be combined to achieve high-precision collection of depth information.

[0078] By temporally aligning and spatially registering the initial RGB image and depth map, and denoising the depth map using a filtering algorithm to eliminate sensor noise and environmental interference, we obtain the target RGB image and depth map. This makes the target RGB image and depth map clearer and more standardized, improving image quality and providing a higher-quality data foundation for subsequent instance segmentation and point cloud data extraction, reducing misidentification and data errors.

[0079] Furthermore, step 101 may include the following sub-steps S11-S15:

[0080] S11, using the initial RGB image and the initial depth image to perform temporal alignment and spatial registration to generate an intermediate RGB image and an intermediate depth image;

[0081] S12, performing image enhancement on the intermediate RGB image to generate an RGB enhanced image;

[0082] S13, using bilateral filtering to remove noise in the RGB enhanced image to generate a target RGB image;

[0083] S14, normalizing the depth values ​​of the intermediate depth map to generate a normalized depth map;

[0084] S15: Fill the holes in the normalized depth map to generate a target depth map.

[0085] In an embodiment of the present invention, the initial RGB image and the initial depth map are temporally aligned and spatially registered to ensure that the initial RGB image and the initial depth image correspond to each other in time and that their pixels correspond accurately, that is, the positions of the same objects in the two images are consistent in space, thereby obtaining an intermediate RGB image and an intermediate depth map.

[0086] Due to the large fluctuations in light intensity in the pineapple growing environment, images captured at different times exhibit significant brightness differences. Using histogram equalization, the histogram of the intermediate RGB image is adjusted to achieve a more uniform distribution of pixel grayscale values, enhancing overall image contrast and reducing the impact of illumination variations on image features. This results in an enhanced RGB image. Alternatively, adaptive histogram equalization (CLAHE) can be used to obtain the enhanced RGB image. This method performs histogram equalization on different local regions, enhancing contrast while avoiding image distortion caused by global adjustments and better preserving pineapple image details. When capturing images in outdoor environments, they are susceptible to noise. Bilateral filtering is used to smooth the enhanced RGB image and remove noise, resulting in the target RGB image. While removing noise, bilateral filtering effectively preserves image edge information, making the pineapple's outline clearer.

[0087] Due to different acquisition distances and device parameters, the depth value range of the depth map may be inconsistent. Therefore, the intermediate depth map is normalized to obtain a normalized depth map, and the depth value is normalized to a specific range (such as [0, 1] or [0, 255]) to facilitate subsequent processing and model training. Specifically, a linear transformation method can be used, such as:

[0088]

[0089] Among them, d norm is the normalized depth value; d is the original depth value; d min is the minimum depth value in the intermediate depth map; d max is the maximum depth value in the intermediate depth map.

[0090] Due to occlusion or acquisition errors, depth maps may contain holes. Holes in the normalized depth map obtained above are filled using a neighborhood interpolation method, such as bilinear interpolation or bicubic interpolation. Taking bilinear interpolation as an example, the depth value at the hole is calculated through linear interpolation based on the depth values ​​of the four neighboring points around the hole, making the target depth map more complete.

[0091] Step 102: Use the target deep learning model to perform instance segmentation on the target RGB image to generate target pineapple mask and pineapple label data.

[0092] In the embodiment of the present invention, Figure 2 As shown, the target RGB image is input into a pre-trained end-to-end deep learning model, the target deep learning model, for instance segmentation. The model then outputs target pineapple masks for the leaves and body of the pineapple, as well as pineapple label data for each leaf (label 0) and body (label 1) in the image. Using the deep learning model, the pineapple is accurately distinguished from the background and individual pineapples in the RGB image. The outline of each pineapple (pineapple mask) is determined and its category label (pineapple label data) is assigned. The pineapple's position and range in the image are precisely located, clarifying the target area for subsequent point cloud data extraction, improving pineapple recognition accuracy, and facilitating subsequent operations to focus on the pineapple itself.

[0093] Furthermore, before step 102, steps 1 to 5 are also included:

[0094] Step 1: Obtain a historical pineapple image set under a pineapple planting scenario, perform regional category labeling on the historical pineapple images in the historical pineapple image set, and generate multiple target historical pineapple images;

[0095] Step 2: Preprocess the target historical pineapple images to generate model training data;

[0096] Step 3: Input the model training data into the initial deep learning model for feature extraction and feature fusion to generate a historical pineapple segmentation map;

[0097] Step 4: Perform classification prediction and segmentation prediction on the historical pineapple segmentation map, calculate the loss value, and generate the total loss value;

[0098] Step 5: When the total loss value is less than the preset loss value threshold, the initial deep learning model at the current moment is used as the target initial deep learning model.

[0099] In an embodiment of the present invention, steps 1 to 5 are the process of training a deep learning model. This process can be performed between steps 101 and 102, or before step 101. Historical images (RGB images) of pineapple planting scenes are collected, covering different lighting conditions, occlusion levels, growth stages (such as mature / unripe pineapples), shooting angles, etc., to form a historical pineapple image set. The historical pineapple images in the historical pineapple image set are respectively annotated with regional categories to generate multiple target historical pineapple images. The annotation content includes two types of information: one is to assign a pineapple classification label to each image, such as classification information such as pineapple variety and maturity; the other is to generate a pixel-level mask that clearly indicates whether each pixel in the image belongs to the pineapple region. Among them, the classification label of the target historical pineapple image can be set as: 0 for the pineapple crown, 1 for the mature pineapple body, and 2 for the immature pineapple body. Data preprocessing operations are performed on the collected target historical pineapple images, such as adjusting the image size and normalizing the pixel values, to ensure data consistency and model training effect. At the same time, the processed data is divided into training set, validation set and test set to obtain model training data for model training, evaluation and testing.

[0100] The model inputs training data from the training dataset. Using EfficientNetV2-S as the backbone network, the model extracts hierarchical feature maps (S1-S4) of the image. These feature maps are downsampled by 4x, 8x, 16x, and 32x, respectively. S1 represents shallow-level detail features such as texture; S2 represents mid-level structural features such as the local shape and segmented regions of the pineapple; S3 represents high-level semantic features such as the overall geometric structure of the pineapple and inference of occluded regions; and S4 represents deep-level features such as the overall contour. The focused encoder further encodes these features. The cross-scale fusion module performs a weighted fusion of the S1-S4 feature maps using a bidirectional feature pyramid and incorporates an adaptive attention mechanism to enhance the representation of key surface features of the pineapple. The focused decoder employs a progressive upsampling architecture, fusing cross-scale features with high-resolution mask information. Background noise is filtered through a spatial attention gating mechanism, resulting in a historical pineapple segmentation map. This historical pineapple segmentation map is then used for classification and segmentation predictions, followed by loss calculation to produce a total loss. When the total loss value exceeds the preset loss threshold, the model parameters are updated using a backpropagation algorithm (such as the Adam optimizer) to reduce the total loss value. Steps 3-4 are repeated until the total loss value steadily decreases. When the total loss value is less than the preset loss threshold, the current initial deep learning model is used as the target initial deep learning model. The preset loss threshold is a critical value set based on the difficulty of the task, such as 0.3.

[0101] Furthermore, step 4 may include the following sub-steps 41-45:

[0102] Step 41: classify and predict the historical pineapple segmentation graph to generate classification prediction data;

[0103] Step 42: Perform segmentation prediction on the historical pineapple segmentation graph to generate segmentation prediction data;

[0104] Step 43: Use the cross entropy loss function to calculate the loss value between the classification prediction data and the true classification label corresponding to the model training data to generate a classification loss value;

[0105] Step 44: Use the Diesel loss function to calculate the loss value between the segmentation prediction data and the true pixel-level mask corresponding to the model training data to generate a segmentation loss value;

[0106] Step 45: Perform a weighted summation of the classification loss value and the segmentation loss value to generate a total loss value.

[0107] In an embodiment of the present invention, the historical pineapple segmentation map generated in step 3 is input into a model-specific classification module (such as a fully connected layer, etc.) in training. Based on the features extracted from the historical pineapple segmentation map, the module predicts the category of the pineapple in the image (such as maturity, variety, etc., which are not listed in detail in the document, but can be inferred by analogy), outputs the probability value corresponding to each category, and obtains classification prediction data.

[0108] The focused decoder of the model under training is used to perform pixel-level segmentation prediction on the historical pineapple segmentation map, judging the probability of each pixel belonging to the pineapple leaf (label 0) or the pineapple body (label 1), and obtaining a prediction result map of the same size as the original image. The value of each pixel in the map represents the possibility of it belonging to the corresponding category, and the segmentation prediction data is obtained.

[0109] Cross-Entropy Loss is used to measure the difference between the classification prediction result and the true classification label. That is, the cross-entropy loss function is used to calculate the loss value between the classification prediction data and the true classification label corresponding to the model training data to obtain the classification loss value. The cross-entropy loss function can effectively measure the distance between two probability distributions. Its formula is:

[0110]

[0111] Among them, L ds is the classification loss value; C is the number of classification categories; y i is the i-th component of the true classification label (usually 0 or 1); p i The probability of the i-th category predicted by the model is the classification prediction data.

[0112] The Dice loss function is used to calculate the loss between the segmentation prediction data and the true pixel-level mask corresponding to the model training data to obtain the segmentation loss value. The Dice coefficient is a commonly used indicator to measure the similarity between two sets. The Dice loss is 1 minus the Dice coefficient. Its formula is:

[0113]

[0114] Among them, L seg is the segmentation loss value; X is the segmentation prediction data; Y is the true pixel-level mask.

[0115] The classification loss value and the segmentation loss value are weighted and summed to get the total loss value:

[0116] L=αL ds +βL seg ;

[0117] Among them, L is the total loss value; Lseg is the segmentation loss value; L ds is the classification loss value; β is the segmentation weight coefficient; α is the classification weight coefficient.

[0118] Furthermore, the target deep learning model includes a backbone network, a focused encoder, a cross-scale fusion module, and a focused decoder. Step 102 may include the following sub-steps S21-S25:

[0119] S21, extracting the hierarchical feature map in the target RGB image through the backbone network to generate a first initial hierarchical feature map, a second initial hierarchical feature map, a third initial hierarchical feature map and a fourth primary hierarchical feature map;

[0120] S22, feature encoding the fourth initial level feature map through a focused encoder to generate a spatial feature map;

[0121] S23, performing weighted fusion on the first initial level feature map, the second initial level feature map, the third initial level feature map, and the spatial feature map through a cross-scale fusion module to generate a first target level feature map, a second target level feature map, a third target level feature map, and a fourth target level feature map;

[0122] S24, performing pineapple region recognition on the second target level feature map, the third target level feature map, and the fourth target level feature map through a focused decoder to generate pineapple label data and a pineapple segmentation map;

[0123] S25. Perform matrix multiplication on the pineapple segmentation map and the first target level feature map to generate a target pineapple mask.

[0124] Furthermore, step S22 may include the following sub-steps S221-S225:

[0125] S221, flattening the fourth initial level feature map in spatial dimension to generate a flattened feature sequence;

[0126] S222, adding the position code corresponding to the flattened feature sequence to the corresponding feature vector to generate an initial input sequence;

[0127] S223, perform a self-attention operation on the initial input sequence to generate an intermediate input sequence;

[0128] S224, performing nonlinear transformation on the intermediate input sequence through a feedforward neural network to generate a target input sequence;

[0129] S225. Convert the target input sequence into a spatial feature map to generate a spatial feature map.

[0130] Furthermore, step S24 may include the following sub-steps S241-S244:

[0131] S241, using a progressive upsampling structure to upsample the second target level feature map, the third target level feature map, and the fourth target level feature map respectively to generate a second sampling feature map, a third sampling feature map, and a fourth sampling feature map;

[0132] S242, performing cross-scale feature fusion on the second sampling feature map, the third sampling feature map, and the fourth sampling feature map to generate a multi-scale feature map;

[0133] S243, filtering the background noise in the multi-scale feature map through the spatial attention gating mechanism to determine the pineapple region feature map;

[0134] S244. Perform image segmentation on the pineapple region feature map to generate pineapple label data and a pineapple segmentation map.

[0135] In the embodiment of the present invention, Figure 2 As shown, the backbone network extracts hierarchical feature maps from the target RGB image, obtaining a first initial hierarchical feature map S1, a second initial hierarchical feature map S2, a third initial hierarchical feature map S3, and a fourth initial hierarchical feature map S4. EfficientNetV2-S is preferably used as the backbone network.

[0136] By focusing the encoder to encode the feature map of the fourth initial level, the model can effectively model global context information. Combining position encoding and residual connection, it ensures that the features encoded by the Transformer not only retain the spatial structure but also obtain richer semantic features. The specific encoding process is as follows:

[0137] (1) Input data: Each spatial position in the S4 feature map (the fourth initial level feature map) is treated as a word vector, and then these vectors are flattened into a sequence. The S4 feature map has dimensions [B, C, H, W]. Flattening the spatial dimensions into a sequence yields a flattened feature sequence [B, H × W, C].

[0138] (2) Positional encoding: The transformer requires positional encoding to preserve the positional information of the elements in the sequence. For each spatial position in the S4 feature map, a corresponding positional encoding can be generated and added to the feature vector. That is, the shape of the positional encoding is [H*W, C]. After adding it to the shape of the input sequence [B, H*W, C], the input sequence with positional information is obtained, that is, the initial input sequence.

[0139] (3) Self-attention mechanism: The self-attention mechanism allows the model to pay attention to all other positions in the sequence while processing each position. For the input sequence [B, H*W, C], a query vector (Q), key vector (K), and value vector (V) matrix are generated through linear transformation, and then the attention score is calculated:

[0140]

[0141] Among them, Attention(Q, K, V) is the attention score; Q is the query vector; K is the key vector; V is the value vector; softmax is an activation function that converts the input value into a probability distribution. Here, the attention score is converted into the probability of each position being paid attention to, so that the model can perform a weighted sum of V according to these probabilities to obtain the output of the self-attention mechanism; d k is the dimension of the Q and K vectors. It is introduced into the denominator to normalize the attention scores, preventing them from being too large, which would cause the softmax function to enter the gradient saturation region and make training difficult. The multi-head attention mechanism allows multiple self-attention operations to be performed in parallel, enhancing the model's expressiveness and generating intermediate input sequences.

[0142] (4) Feedforward Neural Network (FFN): Each Transformer layer also contains a feedforward neural network, which usually consists of two linear transformations and an activation function:

[0143] FFN(x)=ReLU(xW1+b1)W2+b2;

[0144] Where FFN(x) is the output value of the feedforward neural network; x is the input feature vector of a pineapple image; W1 is the first weight matrix; W2 is the second weight matrix; b1 is the first bias term; b2 is the second bias term; and ReLU is the activation function, which introduces nonlinearity to prevent the model from degenerating into a linear transformation. Feedforward neural networks enhance the representational power of each position through nonlinear transformations while maintaining the consistency of input and output dimensions. This results in a target input sequence that can be used in conjunction with residual connections and layer normalization to facilitate the training of deep networks.

[0145] (5) Output processing: After multiple Transformer layers, the output target input sequence is still in the shape of [B, H*W, C]. In order to be combined with the subsequent Focused Decoder, the sequence needs to be restored to the form of a spatial feature map. That is, the target input sequence is converted into a spatial feature map to obtain a spatial feature map [B, C, H, W].

[0146] The cross-scale fusion module performs a weighted fusion of the first, second, and third initial-level feature maps and the spatial feature map through a bidirectional feature pyramid. Combined with an adaptive attention mechanism, it enhances the representation of key features such as the pineapple's surface color and patches. Through multi-level feature interaction at different levels (top-down and bottom-up), and the complementarity of semantics and details, the fused multi-scale feature map is ultimately output, resulting in the first, second, third, and fourth target-level feature maps.

[0147] By focusing the decoder and adopting a progressive upsampling structure, the cross-scale features and high-resolution mask information are fused, and the background noise is filtered through the spatial attention gating mechanism, the pineapple area is accurately located, and the pineapple segmentation map and pineapple label data are output. The specific decoding process is: the second target level feature map, the third target level feature map and the fourth target level feature map are processed separately using a progressive upsampling structure. Upsampling is an operation to increase the size of the feature map. For example, through interpolation (such as bilinear interpolation) and other methods, a feature map with a small size, rich semantic information but low spatial resolution is restored to a larger size to generate a second sampling feature map, a third sampling feature map and a fourth sampling feature map. The second sampling feature map, the third sampling feature map and the fourth sampling feature map obtained by upsampling are fused. The fusion method may be a simple addition of feature maps, or splicing based on the channel dimension. By fusing features at different scales to complement each other, a multi-scale feature map is generated. Feature maps of different scales contain information at different levels. Cross-scale fusion allows the model to comprehensively utilize this information, enhance the overall grasp of pineapple features, and help to more accurately identify and segment pineapple areas. Specifically, Figure 2 In this example, X represents the matrix algorithm; C represents the concatenation of the S4 encoded data obtained through the Focused Encoder and the feature data obtained by weighted fusion of S3 and S2 using the bidirectional feature pyramid. The output data is a multi-scale fused feature map. The classification loss (Cross-Entropy Loss) and the segmentation loss (Dice Loss) are jointly optimized to achieve dual supervision of classification and localization.

[0148] The multi-scale feature map is processed using a spatial attention gating mechanism. This mechanism analyzes information at different spatial locations in the feature map and assigns different weights to different locations based on the characteristic differences between the pineapple region and the background. The spatial attention gating mechanism allows the model to focus on the pineapple region, suppressing the impact of background noise on subsequent segmentation tasks and improving segmentation accuracy. The feature map of the pineapple region, after filtering out background noise, is input into an image segmentation module (such as a deep learning-based segmentation network structure). Based on the feature information at different locations in the feature map, the segmentation module determines which part of the pineapple each pixel belongs to (such as the pineapple leaf or the pineapple body). It then generates pineapple label data (a set of labels that records the category to which each pixel belongs) and a pineapple segmentation map containing the pineapple leaf and pineapple body categories. Finally, the pineapple segmentation map is matrix multiplied with the first target level feature map to generate the target pineapple mask.

[0149] Step 103: Map the target pineapple mask to the target depth map based on the pineapple label data to extract point cloud data, and perform clustering segmentation and dynamic filtering optimization on the point cloud data to generate target pineapple leaf point cloud data and target pineapple body point cloud data.

[0150] Furthermore, step 103 may include the following sub-steps S31-S33:

[0151] S31, the target pineapple mask is mapped to the target depth map, the depth information of the pineapple leaf and the pineapple body is extracted according to the pineapple label data, and the initial pineapple leaf point cloud data and the initial pineapple body point cloud data are generated;

[0152] S32, initial pineapple leaf point cloud data and initial pineapple main body point cloud data are clustered and segmented to generate pineapple leaf segmentation data and pineapple main body segmentation data;

[0153] S33. Filter out the point cloud data greater than or equal to the preset clustering threshold value in the pineapple leaf segmentation data and the pineapple body segmentation data respectively, and generate target pineapple leaf point cloud data and target pineapple body point cloud data.

[0154] In an embodiment of the present invention, a mask is mapped to a depth map to generate a point cloud, and the depth information of the pineapple leaves and the main body is obtained according to the pineapple label data to generate initial pineapple leaf point cloud data and initial pineapple main body point cloud data. After cluster segmentation and dynamic filtering optimization, the valid pineapple target point cloud is retained to obtain the target pineapple leaf point cloud data and the target pineapple main body point cloud data. The specific extraction process is: cluster segmentation based on Euclidean distance to generate pineapple leaf segmentation data and pineapple main body segmentation data; according to the point cloud density formula ρ = N / V (N is the total number of point clouds, V is the point cloud volume), the minimum valid cluster point number threshold, that is, the preset cluster threshold N, is dynamically adjusted. min=k*ρ (k is a proportional coefficient. It plays a regulatory role and is used to determine the minimum number of valid points N based on the point cloud density ρ. min ρ is the point cloud density). A shortest distance-based screening mechanism was designed to select a set of pineapple body and crown point clouds that matched the physical characteristics of the pineapple from multiple sets of targets, combining them into a pineapple point cloud. This process involved performing Euclidean clustering on the initial point cloud, setting a cluster threshold, and checking whether the number of points in the current point cloud cluster met the minimum valid cluster size threshold. Clusters with fewer than the preset number of points were then removed, and the cluster with the largest number of points was retained. This resulted in the denoised pineapple leaf and pineapple body point clouds, respectively.

[0155] Furthermore, statistical outlier filtering or radius filtering can be used instead of Euclidean clustering to dynamically adjust parameters for different noise patterns.

[0156] Step 104: Calculate the grasping point and grasping posture data of the pineapple in the area to be picked according to the target pineapple leaf point cloud data and the target pineapple body point cloud data, and generate pineapple recognition and positioning data.

[0157] In an embodiment of the present invention, the center of mass of the point cloud is calculated from the pineapple body and used as the three-dimensional coordinates of the robot's grasping point. The center of mass is the weighted average of the point cloud coordinates, and the weight is determined by the point cloud density distribution function. A principal component analysis (PCA) is performed on the pineapple entity point cloud, and a set of spatial bases is obtained based on the spatial principal axis to represent the posture characteristics of the pineapple point cloud, that is, the target grasping posture, and obtain pineapple identification and positioning data. The principal component analysis (PCA) fuses the leaf and main body point clouds, extracts the main direction axis to determine the optimal grasping posture. Based on the grasping point coordinates and the posture axis, the motion trajectory of the robot end effector is generated, and the grasping posture is adjusted according to the posture characteristics of the end. The posture direction is consistent with the end direction, the grasping posture yaw angle is ±10°, and the grasping position pitch angle is dynamically adjusted to drive the robotic arm to perform the grasping action.

[0158] Furthermore, template matching can be performed based on the geometric features of the pineapple body (such as cylindrical approximation) to replace the center of mass calculation, which is suitable for positioning the grasping points that can be fitted into axisymmetric targets.

[0159] Furthermore, step 104 may include the following sub-steps S41-S47:

[0160] S41, calculating the centroid of the point cloud data of the target pineapple main body point cloud data to generate three-dimensional centroid coordinates;

[0161] S42, merging the target pineapple leaf point cloud data and the target pineapple body point cloud data to generate pineapple entity point cloud data;

[0162] S43, centralizing the pineapple entity point cloud data to generate centralized point cloud data;

[0163] S44. Use the centralized point cloud data to construct a matrix to obtain a covariance matrix;

[0164] S45, performing eigendecomposition on the covariance matrix to generate multiple matrix eigenvalues ​​and multiple matrix eigenvectors;

[0165] S46, taking the matrix eigenvector corresponding to the maximum value of the matrix eigenvalues ​​as the main axis direction of the point cloud to generate the target grasping posture axis;

[0166] S47. Use the target grasping posture axis, all matrix eigenvalues ​​and all matrix eigenvectors to construct pineapple recognition and positioning data.

[0167] In an embodiment of the present invention, the point cloud data centroid of the target pineapple main body point cloud data is calculated to generate three-dimensional centroid coordinates. The calculation formula corresponding to the three-dimensional centroid coordinates is:

[0168]

[0169] in, is the three-dimensional centroid coordinate; N is the number of point clouds; (x i ,y i , z i ) is the point cloud coordinate; i is an index variable used to traverse each point in the pineapple point cloud dataset.

[0170] Merge the target pineapple leaf point cloud data and the target pineapple body point cloud data to obtain the pineapple entity point cloud data. Perform principal component analysis (PCA) on the entire point cloud; select the first three principal component directions as the pineapple grasping posture axes to determine the optimal grasping direction. The principal component analysis calculation process is as follows:

[0171] First, the data is centralized and the pineapple point cloud coordinates are translated to the center of mass. That is, the target pineapple leaf point cloud data and the target pineapple body point cloud data are merged to generate the pineapple entity point cloud data.

[0172] Secondly, the centralized point cloud data is used to construct the matrix and obtain the covariance matrix:

[0173]

[0174] Among them, Cov is the covariance matrix; N is the number of point clouds; P i is the point cloud coordinate, representing the coordinate vector of the i-th point in three-dimensional space; are the three-dimensional centroid coordinates.

[0175] Next, perform eigendecomposition on the covariance matrix to generate multiple matrix eigenvalues ​​and multiple matrix eigenvectors. Calculate the matrix eigenvalues ​​(λ1≥λ2≥λ3) and the corresponding matrix eigenvectors (v1, v2, v3) of the covariance matrix.

[0176] Finally, the principal axis direction is determined. The matrix eigenvector v1 corresponding to the maximum eigenvalue is the principal axis direction of the point cloud, which is the target grasping posture axis of the pineapple. The target grasping posture axis, all matrix eigenvalues, and all matrix eigenvectors are used to construct the pineapple recognition and positioning data.

[0177] The robot's end-effector trajectory is generated based on the grasping point coordinates and attitude axis. The grasping point coordinates and attitude vector are then transmitted to the robot control system. The grasping posture is adjusted based on the end-effector's posture characteristics, ensuring that the posture orientation is consistent with the end-effector's orientation. The grasping posture yaw angle is ±10°, and the grasping position pitch angle is dynamically adjusted. This drives the robotic arm to execute the grasping action, achieving precise grasping.

[0178] In this embodiment of the present invention, by fusing RGB images and depth maps from a 3D camera, combined with a deep learning model, cross-scale dynamic weighted fusion adaptively fuses features from different levels using learnable weight parameters, addressing the problem of traditional FPN's insufficient response to small target features. Mask-guided focused decoding introduces a masked supervision signal during the decoding process, utilizing a spatial attention mechanism to enhance pineapple feature reconstruction and reduce interference from lighting and occlusion. This achieves precise instance segmentation of pineapple leaves and the main body, maintaining a high recognition accuracy of over 95% even in complex lighting and occlusion scenarios.

[0179] The Euclidean clustering segmentation technology is used to dynamically eliminate environmental noise point clouds and adaptively retain target point clouds, improving the point cloud denoising efficiency by more than 30%. At the same time, a fruit body and crown matching strategy is introduced for scenes with multiple targets. The fruit body and crown of the same target object are matched through the distance segmentation of different fruit bodies and crowns to ensure accurate point cloud matching. This method is particularly suitable for complex environments in orchards with intertwined branches and leaves and interference from foreign objects.

[0180] Based on centroid positioning and principal component analysis (PCA), the three-dimensional coordinates of the pineapple grasping point and the optimal grasping posture are automatically calculated. At the same time, the grasping posture is adjusted according to the posture characteristics of the end of the mechanical part. The posture orientation is consistent with the end orientation, the grasping posture yaw angle is ±10°, and the pitch angle of the grasping position is dynamically adjusted, which solves the positioning deviation problem caused by traditional methods relying on manual experience.

[0181] The method framework used in this paper is adaptable to a variety of fruits and vegetables (e.g., picking oval or cylindrical fruits and vegetables like mangoes and apples). Rapid migration is achieved by simply adjusting the model training data, reducing the development cost of agricultural robots for multi-scenario applications. By dynamically filtering invalid detection results (e.g., background noise), the amount of data required for subsequent point cloud processing is reduced, and system memory usage is reduced by 40%, meeting the real-time requirements of embedded devices.

[0182] Furthermore, by simply modifying the target category label and geometric analysis logic, the present invention can be applied to scenarios such as industrial sorting parts grabbing and logistics warehousing package positioning.

[0183] Example 2

[0184] See also Figure 3 , Figure 3 This is a structural block diagram of a pineapple identification and positioning system provided in Example 2 of the present invention.

[0185] An embodiment of the present invention provides a pineapple identification and positioning system, comprising:

[0186] An image generation module 301 is configured to obtain an initial RGB image and an initial depth image of the area to be picked, perform image preprocessing on the initial RGB image and the initial depth image, and generate a target RGB image and a target depth image;

[0187] A mask and label data generation module 302 is used to perform instance segmentation on the target RGB image using a target deep learning model to generate target pineapple mask and pineapple label data;

[0188] The point cloud data generation module 303 is used to map the target pineapple mask to the target depth map based on the pineapple label data to extract the point cloud data, and perform clustering segmentation and dynamic filtering optimization on the point cloud data to generate the target pineapple leaf point cloud data and the target pineapple body point cloud data;

[0189] The pineapple identification and positioning data generation module 304 is used to calculate the grasping point and grasping posture data of the pineapple in the pineapple area to be picked based on the target pineapple leaf point cloud data and the target pineapple body point cloud data, and generate pineapple identification and positioning data.

[0190] Optionally, the image generation module 301 may perform the following steps:

[0191] The initial RGB image and the initial depth image are used for temporal alignment and spatial registration to generate an intermediate RGB image and an intermediate depth image;

[0192] Perform image enhancement on the intermediate RGB image to generate an RGB enhanced image;

[0193] Bilateral filtering is used to remove noise from the RGB enhanced image and generate the target RGB image;

[0194] Normalize the depth value of the intermediate depth map to generate a normalized depth map;

[0195] Fill the holes in the normalized depth map to generate the target depth map.

[0196] Optionally, the system further includes a model training module, which can perform the following steps:

[0197] Obtain a historical pineapple image set in a pineapple planting scenario, perform regional category labeling on the historical pineapple images in the historical pineapple image set, and generate multiple target historical pineapple images;

[0198] Preprocess the target historical pineapple images to generate model training data;

[0199] The model training data is input into the initial deep learning model for feature extraction and feature fusion to generate a historical pineapple segmentation map;

[0200] Perform classification prediction and segmentation prediction on the historical pineapple segmentation map, calculate the loss value, and generate the total loss value;

[0201] When the total loss value is less than the preset loss value threshold, the initial deep learning model at the current moment is used as the target initial deep learning model.

[0202] Optionally, the model training module can also perform the following steps:

[0203] Perform classification prediction on the historical pineapple segmentation map to generate classification prediction data;

[0204] Perform segmentation prediction on the historical pineapple segmentation graph to generate segmentation prediction data;

[0205] The cross entropy loss function is used to calculate the loss value between the classification prediction data and the true classification label corresponding to the model training data to generate the classification loss value;

[0206] The Dise loss function is used to calculate the loss between the segmentation prediction data and the true pixel-level mask corresponding to the model training data to generate the segmentation loss value;

[0207] The classification loss value and the segmentation loss value are weighted and summed to generate the total loss value.

[0208] Optionally, the target deep learning model includes a backbone network, a focused encoder, a cross-scale fusion module, and a focused decoder. The mask and label data generation module 302 can perform the following steps:

[0209] Extracting hierarchical feature maps from the target RGB image through the backbone network to generate a first initial hierarchical feature map, a second initial hierarchical feature map, a third initial hierarchical feature map, and a fourth primary hierarchical feature map;

[0210] Performing feature encoding on the fourth initial level feature map through a focused encoder to generate a spatial feature map;

[0211] The first initial level feature map, the second initial level feature map, the third initial level feature map and the spatial feature map are weightedly fused by the cross-scale fusion module to generate a first target level feature map, a second target level feature map, a third target level feature map and a fourth target level feature map;

[0212] The second target level feature map, the third target level feature map, and the fourth target level feature map are used by the focused decoder to identify the pineapple region and generate pineapple label data and pineapple segmentation map;

[0213] Perform matrix multiplication on the pineapple segmentation map and the first target level feature map to generate the target pineapple mask.

[0214] Optionally, the mask and label data generation module 302 may further perform the following steps:

[0215] Flatten the spatial dimension of the fourth initial level feature map to generate a flattened feature sequence;

[0216] Add the position code corresponding to the flattened feature sequence to the corresponding feature vector to generate the initial input sequence;

[0217] Perform self-attention operation on the initial input sequence to generate an intermediate input sequence;

[0218] Perform nonlinear transformation on the intermediate input sequence through a feedforward neural network to generate the target input sequence;

[0219] The target input sequence is converted into a spatial feature map to generate a spatial feature map.

[0220] Optionally, the mask and label data generation module 302 may further perform the following steps:

[0221] The second target level feature map, the third target level feature map, and the fourth target level feature map are respectively upsampled using a progressive upsampling structure to generate a second sampling feature map, a third sampling feature map, and a fourth sampling feature map;

[0222] Perform cross-scale feature fusion on the second sampling feature map, the third sampling feature map, and the fourth sampling feature map to generate a multi-scale feature map;

[0223] The spatial attention gating mechanism is used to filter the background noise in the multi-scale feature map and determine the pineapple region feature map;

[0224] Perform image segmentation on the pineapple region feature map to generate pineapple label data and pineapple segmentation map.

[0225] Optionally, the point cloud data generation module 303 may perform the following steps:

[0226] Map the target pineapple mask to the target depth map, extract the depth information of the pineapple leaves and pineapple body according to the pineapple label data, and generate the initial pineapple leaf point cloud data and the initial pineapple body point cloud data;

[0227] Clustering and segmenting the initial pineapple leaf point cloud data and the initial pineapple body point cloud data to generate pineapple leaf segmentation data and pineapple body segmentation data;

[0228] Point cloud data greater than or equal to the preset clustering threshold are respectively filtered out from the pineapple leaf segmentation data and the pineapple body segmentation data to generate target pineapple leaf point cloud data and target pineapple body point cloud data.

[0229] Optionally, the pineapple identification and positioning data generation module 304 may perform the following steps:

[0230] Calculate the centroid of the point cloud data of the target pineapple main body and generate the three-dimensional centroid coordinates;

[0231] Merge the target pineapple leaf point cloud data and the target pineapple body point cloud data to generate pineapple entity point cloud data;

[0232] Centralize the pineapple entity point cloud data to generate centralized point cloud data;

[0233] Use centralized point cloud data to construct the matrix and obtain the covariance matrix;

[0234] Perform eigendecomposition on the covariance matrix to generate multiple matrix eigenvalues ​​and multiple matrix eigenvectors;

[0235] The matrix eigenvector corresponding to the maximum value of the matrix eigenvalue is used as the main axis direction of the point cloud to generate the target grasping posture axis;

[0236] The target grasping posture axis, all matrix eigenvalues ​​and all matrix eigenvectors are used to construct the pineapple recognition and positioning data.

[0237] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A pineapple identification and positioning method, characterized in that: The following steps are involved: Obtaining an initial RGB image and an initial depth map of the area to be picked pineapples, performing image preprocessing on the initial RGB image and the initial depth map to generate a target RGB image and a target depth map; Perform instance segmentation on the target RGB image using a target deep learning model to generate target pineapple mask and pineapple label data; Based on the pineapple label data, the target pineapple mask is mapped to the target depth map to extract point cloud data, and the point cloud data is clustered and segmented and dynamically filtered to generate target pineapple leaf point cloud data and target pineapple body point cloud data; The grasping point and grasping posture data of the pineapples in the pineapple area to be picked are calculated based on the target pineapple leaf point cloud data and the target pineapple body point cloud data to generate pineapple recognition and positioning data.

2. A pineapple identification and positioning method according to claim 1, characterized in that, The step of performing image preprocessing on the initial RGB image and the initial depth map to generate a target RGB image and a target depth map includes: Performing temporal alignment and spatial registration using the initial RGB image and the initial depth image to generate an intermediate RGB image and an intermediate depth image; Performing image enhancement on the intermediate RGB image to generate an RGB enhanced image; Using bilateral filtering to remove noise in the RGB enhanced image to generate a target RGB image; Normalizing the depth values ​​of the intermediate depth map to generate a normalized depth map; Fill holes in the normalized depth map to generate a target depth map.

3. A pineapple identification and positioning method according to claim 1, characterized in that, Before the step of using the target deep learning model to perform instance segmentation on the target RGB image to generate target pineapple mask and pineapple label data, the method further includes: Obtain a historical pineapple image set in a pineapple planting scenario, perform regional category labeling on the historical pineapple images in the historical pineapple image set, and generate a plurality of target historical pineapple images; Preprocess the target historical pineapple images to generate model training data; Inputting the model training data into the initial deep learning model for feature extraction and feature fusion to generate a historical pineapple segmentation map; Performing classification prediction and segmentation prediction on the historical pineapple segmentation graph, and performing loss value calculation to generate a total loss value; When the total loss value is less than the preset loss value threshold, the initial deep learning model at the current moment is used as the target initial deep learning model.

4. A pineapple identification and positioning method according to claim 3, characterized in that, The step of performing classification prediction and segmentation prediction on the historical pineapple segmentation map, and calculating the loss value to generate a total loss value includes: Performing classification prediction on the historical pineapple segmentation graph to generate classification prediction data; Perform segmentation prediction on the historical pineapple segmentation graph to generate segmentation prediction data; A cross entropy loss function is used to calculate the loss value between the classification prediction data and the true classification label corresponding to the model training data to generate a classification loss value; Using the Diesel loss function to calculate the loss value between the segmentation prediction data and the true pixel-level mask corresponding to the model training data, to generate a segmentation loss value; The classification loss value and the segmentation loss value are weighted and summed to generate a total loss value.

5. A pineapple identification and positioning method according to claim 1 or 3, characterized in that, The target deep learning model includes a backbone network, a focused encoder, a cross-scale fusion module, and a focused decoder; the step of using the target deep learning model to perform instance segmentation on the target RGB image to generate target pineapple masks and pineapple label data includes: Extracting a hierarchical feature map from the target RGB image through the backbone network to generate a first initial hierarchical feature map, a second initial hierarchical feature map, a third initial hierarchical feature map, and a fourth primary hierarchical feature map; Performing feature encoding on the fourth initial level feature map by the focused encoder to generate a spatial feature map; Performing weighted fusion on the first initial level feature map, the second initial level feature map, the third initial level feature map, and the spatial feature map through the cross-scale fusion module to generate a first target level feature map, a second target level feature map, a third target level feature map, and a fourth target level feature map; Performing pineapple region recognition on the second target-level feature map, the third target-level feature map, and the fourth target-level feature map through the focus decoder to generate pineapple label data and a pineapple segmentation map; Perform a matrix multiplication operation on the pineapple segmentation map and the first target level feature map to generate a target pineapple mask.

6. A pineapple identification and positioning method according to claim 5, characterized in that, The step of performing feature encoding on the fourth initial level feature map by the focused encoder to generate a spatial feature map comprises: Flattening the fourth initial level feature map in spatial dimension to generate a flattened feature sequence; Adding the position code corresponding to the flattened feature sequence to the corresponding feature vector to generate an initial input sequence; Performing a self-attention operation on the initial input sequence to generate an intermediate input sequence; Performing a nonlinear transformation on the intermediate input sequence through a feedforward neural network to generate a target input sequence; The target input sequence is converted into a spatial feature map to generate a spatial feature map.

7. A pineapple identification and positioning method according to claim 5, characterized in that, The step of performing pineapple region recognition on the second target-level feature map, the third target-level feature map, and the fourth target-level feature map by the focused decoder to generate pineapple label data and a pineapple segmentation map includes: Upsampling the second target-level feature map, the third target-level feature map, and the fourth target-level feature map respectively using a progressive upsampling structure to generate a second sampling feature map, a third sampling feature map, and a fourth sampling feature map; Performing cross-scale feature fusion on the second sampling feature map, the third sampling feature map, and the fourth sampling feature map to generate a multi-scale feature map; filtering background noise in the multi-scale feature map through a spatial attention gating mechanism to determine a pineapple region feature map; The pineapple region feature map is subjected to image segmentation to generate pineapple label data and a pineapple segmentation map.

8. A pineapple identification and positioning method according to claim 1, characterized in that, The step of mapping the target pineapple mask to the target depth map based on the pineapple label data to extract point cloud data, and performing clustering segmentation and dynamic filtering optimization on the point cloud data to generate target pineapple leaf point cloud data and target pineapple body point cloud data includes: Mapping the target pineapple mask to the target depth map, extracting depth information of the pineapple leaves and the pineapple body according to the pineapple label data, and generating initial pineapple leaf point cloud data and initial pineapple body point cloud data; Clustering and segmenting the initial pineapple leaf point cloud data and the initial pineapple body point cloud data to generate pineapple leaf segmentation data and pineapple body segmentation data; Point cloud data greater than or equal to a preset clustering threshold value in the pineapple leaf segmentation data and the pineapple body segmentation data are respectively screened out to generate target pineapple leaf point cloud data and target pineapple body point cloud data.

9. A pineapple identification and positioning method according to claim 1, characterized in that, The step of calculating the grasping point and grasping posture data of the pineapples in the area to be picked according to the target pineapple leaf point cloud data and the target pineapple body point cloud data, and generating pineapple identification and positioning data, comprises: Calculating the point cloud data centroid of the target pineapple main body point cloud data to generate three-dimensional centroid coordinates; Merging the target pineapple leaf point cloud data and the target pineapple body point cloud data to generate pineapple entity point cloud data; Centralizing the pineapple entity point cloud data to generate centralized point cloud data; Using the centralized point cloud data to construct a matrix to obtain a covariance matrix; Performing eigendecomposition on the covariance matrix to generate a plurality of matrix eigenvalues ​​and a plurality of matrix eigenvectors; The matrix eigenvector corresponding to the maximum value of the matrix eigenvalues ​​is used as the main axis direction of the point cloud to generate the target grasping posture axis; The target grasping posture axis, all the matrix eigenvalues ​​and all the matrix eigenvectors are used to construct pineapple recognition and positioning data.

10. A pineapple identification and positioning system, characterized in that: include: An image generation module is used to obtain an initial RGB image and an initial depth image of the area to be picked pineapples, perform image preprocessing on the initial RGB image and the initial depth image, and generate a target RGB image and a target depth map; A mask and label data generation module is used to perform instance segmentation on the target RGB image using a target deep learning model to generate target pineapple mask and pineapple label data; a point cloud data generation module, configured to map the target pineapple mask to the target depth map based on the pineapple label data to extract point cloud data, and perform clustering segmentation and dynamic filtering optimization on the point cloud data to generate target pineapple leaf point cloud data and target pineapple body point cloud data; The pineapple identification and positioning data generation module is used to calculate the grasping point and grasping posture data of the pineapple in the pineapple area to be picked according to the target pineapple leaf point cloud data and the target pineapple body point cloud data, and generate pineapple identification and positioning data.

Images