A method for distinguishing the three-dimensional picking point of tea bud leaves under the shelter condition

Abstract

The present application provides a kind of tea bud leaf three-dimensional picking point prediction method distinguishing sheltered condition, if tea bud leaf stalk picking point is not sheltered, then the real three-dimensional point cloud of tea bud leaf stalk is extracted using multi-view information fusion mode.The geometric center of real three-dimensional point cloud is calculated, and the real three-dimensional coordinates of tea bud leaf stalk picking point are obtained.If tea bud leaf stalk picking point is sheltered, then tea bud leaf stalk is predicted using multi-resolution point cloud estimation model, and the three-dimensional point cloud after prediction is obtained.The geometric center of three-dimensional point cloud after prediction is calculated, and the predicted three-dimensional coordinates of tea bud leaf stalk picking point are obtained.The feature extraction and fusion of visible point cloud of sheltered tea bud leaf picking point are carried out through hybrid encoder, and the fusion feature containing local details and global information is obtained, then the fusion feature is decoded through point pyramid decoder, and the three-dimensional point cloud after prediction of tea bud leaf stalk is obtained, so as to enhance the perception ability of model to multi-scale morphology.

Description

Technical Field

[0001] This invention relates to the field of intelligent agricultural control technology, specifically a method for predicting the three-dimensional picking point of tea buds and leaves by distinguishing shading conditions. Background Technology

[0002] Tea is one of my country's major economic crops, and its harvesting process demands extremely high standards of quality and efficiency. Precise positioning of the stem picking point is crucial to ensure the integrity of the buds and leaves. Traditional harvesting methods rely heavily on manual labor, resulting in high labor intensity and costs, and are limited by seasonality and time constraints, making it difficult to meet the needs of large-scale production. With the development of agricultural automation technology, automated tea harvesting has become a core direction for industrial upgrading, and the precise positioning of the stem picking point is a key technological bottleneck in achieving automated harvesting.

[0003] Current automated harvesting methods struggle to match the precision of manual harvesting in selecting tenderness and size, leading to inconsistent tea quality and potential damage to leaves, thus affecting the appearance and quality of the tea. Furthermore, overlapping leaves and intertwined branches result in some tea bud and leaf picking points being obscured, making it difficult for existing technologies to infer the complete stem structure from visual information, resulting in a significant decrease in positioning accuracy. Current technologies for uncrowded targets often rely on single-view images or simple point cloud clustering, lacking the fusion and utilization of multi-source information. For obscured targets, the lack of an effective morphological reasoning mechanism prevents the complete prediction of stem structure.

[0004] Therefore, there is a need for a three-dimensional tea bud picking point prediction method that differentiates between shading conditions and picking point location based on whether the picking point is shaded or unshaded in the tea garden environment. This method would allow for precise picking point location and improve the stability and accuracy of picking point identification. Summary of the Invention

[0005] To overcome the problems existing in related technologies, the purpose of this invention is to provide a three-dimensional tea bud and leaf picking point prediction method that distinguishes between shading conditions. This method adopts different picking point positioning methods for shading and unshading conditions of tea bud and leaf picking points in the tea garden environment, thereby achieving accurate picking point positioning and improving the stability and accuracy of picking point identification.

[0006] A method for predicting three-dimensional picking points of tea buds and leaves based on shading conditions includes:

[0007] Obtain information on the shading conditions at the tea bud, leaf, and stem picking points;

[0008] If the picking point of tea buds, leaves and stems is not obscured, the real three-dimensional point cloud of tea buds, leaves and stems is extracted using a multi-view information fusion method.

[0009] Calculate the geometric center of the real three-dimensional point cloud to obtain the real three-dimensional coordinates of the tea bud, leaf, and stem picking point;

[0010] If the picking point of tea buds, leaves and stems is obscured, a multi-resolution point cloud estimation model is used to predict the tea buds, leaves and stems, and the predicted three-dimensional point cloud is obtained.

[0011] The geometric center of the predicted three-dimensional point cloud is calculated to obtain the predicted three-dimensional coordinates of the tea bud, leaf, and stem picking point.

[0012] In a preferred embodiment of the present invention, calculating the geometric center of the true three-dimensional point cloud to obtain the true three-dimensional coordinates of the tea bud, leaf, and stem picking point includes:

[0013] The geometric center of a real 3D point cloud is calculated using the following formula:

[0014] ;

[0015] ;

[0016] ;

[0017] Where M is the total number of points in the real 3D point cloud, x i Let y be the coordinate value of the i-th point in the X-axis direction. i Let z be the coordinate value of the i-th point in the Y-axis direction. i Let x be the coordinate of the i-th point in the Z-axis direction, x be the coordinate of the geometric center in the X-axis direction, y be the coordinate of the geometric center in the Y-axis direction, and z be the coordinate of the geometric center in the Z-axis direction.

[0018] In a preferred embodiment of the present invention, the step of using a multi-resolution point cloud estimation model to predict tea buds, leaves, and stems to obtain a predicted three-dimensional point cloud includes:

[0019] Obtain the visible point cloud of the tea bud picking point that is obscured;

[0020] A hybrid encoder is used to extract and fuse features from the visual point cloud to obtain fused features;

[0021] The fused features are decoded using a point pyramid decoder to obtain the predicted 3D point cloud.

[0022] In a preferred embodiment of the present invention, the step of using a hybrid encoder to extract and fuse features from the visible point cloud to obtain fused features includes:

[0023] Multi-resolution point cloud branches are used to extract features of different resolutions from the visible point cloud to obtain multi-scale features; wherein, the multi-resolution point cloud branches include high-resolution branches, medium-resolution branches and low-resolution branches;

[0024] The multi-scale features are fused using a multi-scale perception self-attention module to obtain fused features.

[0025] In a preferred embodiment of the present invention, the step of fusing the multi-scale features using a multi-scale perceptual self-attention module to obtain fused features includes:

[0026] The following formula is used to extract the fusion features:

[0027] ;

[0028] ;

[0029] ;

[0030] ;

[0031] Where Q is the query matrix, K is the key matrix, V is the value matrix, and F is the value matrix. in For the aforementioned multi-scale features, W q To query the weight matrix, W k W is the key weight matrix. v Here is the value weight matrix, where Att is the attention weight. For the normalized exponential function, K T Let be the transpose of the key matrix, and d be the number of channels; For point-by-point multiplication, F a For attention features, MLP stands for Multilayer Perceptron, F out This refers to the fusion feature.

[0032] In a preferred embodiment of the present invention, the step of using a point pyramid decoder to decode the fused features to obtain the predicted 3D point cloud includes:

[0033] The fused features are split using a fully connected layer of a point pyramid decoder to obtain a first resolution feature layer, a second resolution feature layer, and a third resolution feature layer; wherein the resolution of the first resolution feature layer is less than the resolution of the second resolution feature layer, and the resolution of the second resolution feature layer is less than the resolution of the third resolution feature layer.

[0034] Prediction is performed on the first resolution feature layer, the second resolution feature layer and the third resolution feature layer respectively to obtain the first predicted point cloud, the second predicted point cloud and the third predicted point cloud;

[0035] Hierarchical feature fusion and multi-scale refinement are performed on the first, second, and third predicted point clouds to obtain the predicted 3D point cloud.

[0036] In a preferred embodiment of the present invention, before obtaining the shading status of the tea bud and stem picking point, the method further includes:

[0037] A semantic segmentation model is used to extract a mask image from a tea bud and leaf image. The mask image includes the tea leaf region and the stem region.

[0038] The stem region of the masked image is analyzed to obtain the occlusion status of the tea bud and leaf stem picking point.

[0039] In a preferred embodiment of the present invention, the step of extracting the true three-dimensional point cloud of tea buds, leaves, and stems using a multi-view information fusion method includes:

[0040] The camera on the robotic arm is controlled to dynamically capture tea buds and leaves from different angles, resulting in multiple images of tea buds and leaves; the camera moves with the robotic arm.

[0041] Multi-view information fusion was performed on multiple tea bud and leaf images to obtain a true three-dimensional point cloud of tea buds, leaves, and stems.

[0042] In a preferred embodiment of the present invention, before predicting the first resolution feature layer, the second resolution feature layer, and the third resolution feature layer respectively, the method further includes:

[0043] The first resolution feature layer is transposed convolution or upsampled to align the first resolution feature layer with the second resolution feature layer.

[0044] Based on the attention mechanism, the second resolution feature layer is corrected using the first resolution feature layer;

[0045] The second resolution feature layer is transposed or upsampled to align the second resolution feature layer with the third resolution feature layer.

[0046] Based on the attention mechanism, the third resolution feature layer is modified using the second resolution feature layer.

[0047] In a preferred embodiment of the present invention, the hybrid encoder employs a combined multilayer perceptron as a feature extractor, wherein the combined multilayer perceptron comprises multiple stacked multilayer perceptrons, and the last N layers of the combined multilayer perceptron are max pooling layers; wherein N≥2.

[0048] The beneficial effects of this invention are as follows:

[0049] The method for predicting the three-dimensional picking points of tea buds and leaves based on occlusion conditions provided by this invention includes: obtaining the occlusion status of tea bud and leaf stem picking points; if the tea bud and leaf stem picking points are not occluded, then extracting the true three-dimensional point cloud of the tea bud and leaf stem using a multi-view information fusion method; calculating the geometric center of the true three-dimensional point cloud to obtain the true three-dimensional coordinates of the tea bud and leaf stem picking points; if the tea bud and leaf stem picking points are occluded, then predicting the tea bud and leaf stem using a multi-resolution point cloud estimation model to obtain the predicted three-dimensional point cloud; calculating the geometric center of the predicted three-dimensional point cloud to obtain the predicted three-dimensional coordinates of the tea bud and leaf stem picking points. By acquiring information on the occlusion status of tea bud and stem picking points, including both unoccluded and occluded cases, the following methods are used: For unoccluded picking points, a multi-view information fusion approach is employed to extract the true 3D point cloud of the tea bud and stem. By calculating the average coordinates of all points in the true 3D point cloud, the coordinates of the geometric center along the X, Y, and Z axes are obtained, thus acquiring the true 3D coordinates of the tea bud and stem picking point. For occluded picking points, a multi-resolution point cloud estimation model is used to predict the occlusion status of the tea bud and stem. A hybrid encoder extracts and fuses features from the visible point cloud of the occluded tea bud and stem picking point, obtaining fused features containing local details and global information. A point pyramid decoder then decodes these fused features to obtain the predicted 3D point cloud of the tea bud and stem. The joint structure of a hybrid encoder and a point pyramid decoder enhances the perception capability of the multi-resolution point cloud estimation model for multi-scale morphology. The effectiveness of feature fusion is improved by using a multi-scale perception self-attention module. Finally, by calculating the average coordinates of all points in the predicted 3D point cloud, the coordinates of the geometric center in the X-axis, Y-axis, and Z-axis directions are obtained, thus obtaining the predicted 3D coordinates of the tea bud and stem picking point. This achieves accurate positioning of the tea bud and stem picking point, provides reliable technical support for automated tea picking, and significantly improves the stability and accuracy of tea bud and stem picking point identification. Attached Figure Description

[0050] Figure 1 This is a flowchart of the method for predicting the three-dimensional picking point of tea buds and leaves based on shading conditions according to the present invention.

[0051] Figure 2 This is a structural diagram of the hybrid encoder of the present invention;

[0052] Figure 3 This is a structural diagram of the dot pyramid decoder of the present invention. Detailed Implementation

[0053] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0054] Example 1

[0055] like Figure 1 As shown, this embodiment provides a method for predicting the three-dimensional picking point of tea buds and leaves based on shading conditions, including:

[0056] S1: Obtain information on the shading status of tea bud, leaf, and stem picking points;

[0057] S2: If the picking point of tea buds, leaves and stems is not obscured, the real three-dimensional point cloud of tea buds, leaves and stems is extracted by multi-view information fusion.

[0058] S3: Calculate the geometric center of the real three-dimensional point cloud to obtain the real three-dimensional coordinates of the tea bud, leaf and stem picking point;

[0059] S4: If the tea bud and stem picking point is obstructed, a multi-resolution point cloud estimation model is used to predict the tea bud and stem, and the predicted three-dimensional point cloud is obtained.

[0060] S5: Calculate the geometric center of the predicted three-dimensional point cloud to obtain the predicted three-dimensional coordinates of the tea bud and stem picking point.

[0061] The method of extracting the true three-dimensional point cloud of tea buds, leaves, and stems using multi-view information fusion includes:

[0062] The camera on the robotic arm is controlled to dynamically capture tea buds and leaves from different angles, resulting in multiple images of tea buds and leaves; the camera moves with the robotic arm.

[0063] Multi-view information fusion was performed on multiple tea bud and leaf images to obtain a true three-dimensional point cloud of tea buds, leaves, and stems.

[0064] Due to the complex environment of tea gardens, tea buds and stems are slender and often obscured by leaves, making it almost impossible to see the complete stem from a single perspective. Therefore, obtaining the true 3D point cloud of tea buds and stems through multi-view information fusion is a key technology for achieving precise robotic grasping and separation during tea bud harvesting. When collecting tea bud and leaf images, a robotic arm is controlled to move or stop in front of the tea bush. The robotic arm is typically equipped with multiple cameras, which simultaneously capture multiple images of tea buds and leaves from different angles, showing multiple images of the same tea bush. Multi-view information fusion of multiple tea bud and leaf images is the most crucial step. The goal is to find the pixels corresponding to the same physical point in different images. There are generally two methods: sparse fusion based on feature points and depth map fusion based on dense matching. Sparse fusion based on feature points uses algorithms or deep learning networks to detect key points in each tea bud and leaf image, such as nodes on the stem, edge intersections, and texture feature points, and calculates their feature descriptors. Then, the descriptors of feature points in different tea bud and leaf images are compared to find matching pairs. For each successful matching point, the precise coordinates of the point in three-dimensional space are calculated using the triangulation principle based on the previously calibrated geometric relationship between cameras. Through matching a large number of feature points, a series of sparse three-dimensional points representing key positions on the stem can be obtained, i.e., the true three-dimensional point cloud of the tea bud and leaf stem. The depth map fusion based on dense matching calculates the depth map of the tea bud and leaf image for each camera view using a stereo matching algorithm or deep learning network. Each pixel value in the depth map represents the distance from the point to the camera. Since there are multiple viewpoints, multiple depth maps are generated. By transforming all these depth maps from different viewpoints into the same global coordinate system and fusing them, the true three-dimensional point cloud of the tea bud and leaf stem is finally obtained.

[0065] The calculation of the geometric center of the true 3D point cloud to obtain the true 3D coordinates of the tea bud, leaf, and stem picking point includes:

[0066] The geometric center of a real 3D point cloud is calculated using the following formula:

[0067] ;

[0068] ;

[0069] ;

[0070] Where M is the total number of points in the real 3D point cloud, x i Let y be the coordinate value of the i-th point in the X-axis direction. i Let z be the coordinate value of the i-th point in the Y-axis direction. iLet x be the coordinate of the i-th point in the Z-axis direction, x be the coordinate of the geometric center in the X-axis direction, y be the coordinate of the geometric center in the Y-axis direction, and z be the coordinate of the geometric center in the Z-axis direction.

[0071] After obtaining the true 3D point cloud of tea buds, leaves, and stems, the geometric center is obtained by calculating the average coordinates of all points in the true 3D point cloud. First, the average coordinates of all points in the true 3D point cloud along the X-axis are calculated to obtain the coordinates of the geometric center in the X-axis direction. Specifically, the coordinates of all points in the true 3D point cloud along the X-axis are added together to obtain the total coordinates of all points in the X-axis direction, and then divided by the total number of points M to obtain the average coordinates of all points in the X-axis direction, which is the coordinate of the geometric center in the X-axis direction. Then, the same method is used to calculate the average coordinates of all points in the true 3D point cloud along the Y-axis and the Z-axis, thus obtaining the coordinates of the geometric center in the Y-axis and Z-axis directions.

[0072] The method for predicting the three-dimensional picking points of tea buds and leaves based on occlusion conditions provided in this embodiment includes: obtaining the occlusion status of the picking points of tea buds, leaves, and stems; if the picking points of tea buds, leaves, and stems are not occluded, then a multi-view information fusion method is used to extract the true three-dimensional point cloud of the tea buds, leaves, and stems; the geometric center of the true three-dimensional point cloud is calculated to obtain the true three-dimensional coordinates of the picking points of the tea buds, leaves, and stems; if the picking points of tea buds, leaves, and stems are occluded, then a multi-resolution point cloud estimation model is used to predict the tea buds, leaves, and stems to obtain the predicted three-dimensional point cloud; the geometric center of the predicted three-dimensional point cloud is calculated to obtain the predicted three-dimensional coordinates of the picking points of the tea buds, leaves, and stems. By acquiring information on the occlusion status of tea bud and stem picking points, including both unoccluded and occluded cases, the following methods are used: For unoccluded picking points, a multi-view information fusion approach is employed to extract the true 3D point cloud of the tea bud and stem. By calculating the average coordinates of all points in the true 3D point cloud, the coordinates of the geometric center along the X, Y, and Z axes are obtained, thus acquiring the true 3D coordinates of the tea bud and stem picking point. For occluded picking points, a multi-resolution point cloud estimation model is used to predict the occlusion status of the tea bud and stem. A hybrid encoder extracts and fuses features from the visible point cloud of the occluded tea bud and stem picking point, obtaining fused features containing local details and global information. A point pyramid decoder then decodes these fused features to obtain the predicted 3D point cloud of the tea bud and stem. The combined structure of a hybrid encoder and a point pyramid decoder enhances the perception capability of the multi-resolution point cloud estimation model for multi-scale morphology. A multi-scale perception self-attention module improves the effectiveness of feature fusion. Finally, by calculating the average coordinates of all points in the predicted 3D point cloud, the coordinates of the geometric center in the X, Y, and Z axes are obtained, thus acquiring the predicted 3D coordinates of the tea bud and stem picking points. This achieves precise localization of the tea bud and stem picking points, providing reliable technical support for automated tea harvesting and significantly improving the stability and accuracy of tea bud and stem picking point identification. Combined with the geometric center solving algorithm, it effectively addresses issues such as mutual occlusion of tea buds and leaves and missing point cloud data in the tea garden environment, achieving precise localization of picking points and providing reliable technical support for automated tea harvesting, significantly improving the stability and accuracy of picking point identification.

[0073] Example 2

[0074] like Figure 1 As shown, this embodiment provides a method for predicting the three-dimensional picking point of tea buds and leaves based on shading conditions, including:

[0075] S1: Obtain information on the shading status of tea bud, leaf, and stem picking points;

[0076] S2: If the picking point of tea buds, leaves and stems is not obscured, the real three-dimensional point cloud of tea buds, leaves and stems is extracted by multi-view information fusion.

[0077] S3: Calculate the geometric center of the real three-dimensional point cloud to obtain the real three-dimensional coordinates of the tea bud, leaf and stem picking point;

[0078] S4: If the tea bud and stem picking point is obstructed, a multi-resolution point cloud estimation model is used to predict the tea bud and stem, and the predicted three-dimensional point cloud is obtained.

[0079] S5: Calculate the geometric center of the predicted three-dimensional point cloud to obtain the predicted three-dimensional coordinates of the tea bud and stem picking point.

[0080] The method employs a multi-resolution point cloud estimation model to predict tea buds, leaves, and stems, resulting in a predicted 3D point cloud, including:

[0081] S41: Obtain the visible point cloud of the occluded tea bud picking point;

[0082] S42: Use a hybrid encoder to extract and fuse features from the visible point cloud to obtain fused features;

[0083] S43: The fused features are decoded using a point pyramid decoder to obtain the predicted 3D point cloud.

[0084] like Figure 2 As shown, the hybrid encoder uses a combined multilayer perceptron as the feature extractor. This combined multilayer perceptron comprises multiple stacked multilayer perceptrons, with the last N layers being max-pooling layers; where N ≥ 2. High-resolution, medium-resolution, and low-resolution clouds each correspond to a multilayer perceptron. The multilayer perceptron performs multiple data processing steps on the input point cloud, each including linear transformation and nonlinear activation, to obtain corresponding feature vectors, such as low-resolution, medium-resolution, and high-resolution feature vectors. These three feature vectors are concatenated to obtain a concatenated vector. The concatenated vector undergoes latent feature mapping and perceptron processing to obtain a fused feature, which takes the form of a final feature vector.

[0085] For cases where the tea bud and stem picking points are not obscured, a multi-resolution point cloud estimation model is used to predict the 3D point cloud of the tea bud and stem. The multi-resolution point cloud estimation model includes a hybrid encoder and a point pyramid decoder. The hybrid encoder uses a combined multilayer perceptron (CMLP) as a feature extractor. By stacking multiple MLPs, higher-level feature representations are gradually extracted. In the last N layers of the CMLP, max pooling is used to extract key features. The feature vectors of different levels are integrated to form a comprehensive feature vector, providing a more representative feature representation for the prediction of the 3D point cloud of tea bud and stem.

[0086] The method employs a hybrid encoder to extract and fuse features from the visible point cloud to obtain fused features, including:

[0087] S42': Use multi-resolution point cloud branches to extract features of different resolutions from the visible point cloud to obtain multi-scale features; wherein, the multi-resolution point cloud branches include high-resolution branches, medium-resolution branches and low-resolution branches;

[0088] S42'': The multi-scale features are fused using a multi-scale perception self-attention module to obtain fused features.

[0089] First, a visible point cloud of the tea bud and leaf picking points under occlusion is obtained. Specifically, the visible point cloud refers to the set of all 3D points within the camera's field of view that are not obscured by other leaves, taken from the perspective of a camera on a robotic arm. Next, a hybrid encoder is used to extract features from the visible point cloud. This encoder consists of multi-resolution point cloud branches, specifically low-resolution, medium-resolution, and high-resolution branches. The low-resolution branch has a larger receptive field and is mainly used to extract scene-level semantic features, such as identifying the approximate area of ​​the tea canopy and separating the background from the target tea tree. The medium-resolution branch is mainly used to extract body component features, identifying targets through the shape and structural features of objects, such as locating and distinguishing specific tea buds, leaves, and stems within the canopy. The high-resolution branch has a smaller receptive field and is mainly used to extract millimeter-level detail features, such as precisely locating the optimal picking point on the identified stems or buds. After extracting features at different resolutions from the visible point cloud using multi-resolution point cloud branches, multi-scale features are finally obtained.

[0090] The multi-scale features are fused using a multi-scale perceptual self-attention module to obtain fused features, including:

[0091] The following formula is used to extract the fusion features:

[0092] ;

[0093] ;

[0094] ;

[0095] ;

[0096] Where Q is the query matrix, K is the key matrix, V is the value matrix, and F is the value matrix. in For the aforementioned multi-scale features, W q To query the weight matrix, W k W is the key weight matrix. vHere is the value weight matrix, where Att is the attention weight. For the normalized exponential function, K T Let be the transpose of the key matrix, and d be the number of channels; For point-by-point multiplication, F a For attention features, MLP stands for Multilayer Perceptron, F out This refers to the fusion feature.

[0097] The multi-scale perception self-attention module is based on the Transformer architecture, which offers advantages in global dependency modeling and dynamic weight allocation. The self-attention module allows any two elements in a sequence to interact directly, perfectly capturing long-distance dependencies. Attention weights are dynamically calculated, enabling the model to adaptively focus on the most important parts based on the input content. Multi-scale perception can simultaneously understand macro-context and micro-details, solving problems related to varying target sizes and distances. The multi-scale perception self-attention module fuses multi-scale features, capturing global features and contextual information to obtain fused features. First, the multi-scale features F are obtained. in The query matrix Q, key matrix K, and value matrix V are extracted through linear operations, specifically through multi-scale features F. in respectively with the query weight matrix W q Key weight matrix W k Sum weight matrix W v Perform a dot product to obtain the mapping of multi-scale features in three spaces, where the query weight matrix W is used. q Key weight matrix W k Sum weight matrix W v Both are learnable weight matrices. Next, the attention weights Att are calculated using the Softmax function, and then multiplied by the value matrix V to obtain the attention features F. a Finally, the multi-scale features F in With attention feature F a Element-wise subtraction, followed by multi-layer perceptron (MLP) mapping and multi-scale feature F in Adding them together yields the fused feature F. out .

[0098] This embodiment employs a hybrid encoder to extract and fuse features from the visible point cloud, obtaining fused features. The hybrid encoder consists of multi-resolution point cloud branches, specifically low-resolution, medium-resolution, and high-resolution branches. First, the visible point cloud of the occluded tea bud picking point is acquired. Then, the multi-resolution point cloud branches are used to extract features from the visible point clouds at different resolutions, obtaining multi-scale features containing local details and global information. Finally, a multi-scale perceptual self-attention module based on the Transformer architecture is used to fuse the multi-scale features, capturing global features and contextual information to obtain fused features.

[0099] Example 3

[0100] like Figure 1 As shown, this embodiment provides a method for predicting three-dimensional picking points of tea buds and leaves based on occlusion conditions, including:

[0101] S1: Obtain information on the shading status of tea bud, leaf, and stem picking points;

[0102] S2: If the picking point of tea buds, leaves and stems is not obscured, the real three-dimensional point cloud of tea buds, leaves and stems is extracted by multi-view information fusion.

[0103] S3: Calculate the geometric center of the real three-dimensional point cloud to obtain the real three-dimensional coordinates of the tea bud, leaf and stem picking point;

[0104] S4: If the picking point of tea buds, leaves and stems is blocked, a multi-resolution point cloud estimation model is used to predict the tea buds, leaves and stems, and the predicted three-dimensional point cloud is obtained.

[0105] S5: Calculate the geometric center of the predicted three-dimensional point cloud to obtain the predicted three-dimensional coordinates of the tea bud and stem picking point.

[0106] The method employs a multi-resolution point cloud estimation model to predict tea buds, leaves, and stems, resulting in a predicted 3D point cloud, including:

[0107] S41: Obtain the visible point cloud of the occluded tea bud picking point;

[0108] S42: Use a hybrid encoder to extract and fuse features from the visible point cloud to obtain fused features;

[0109] S43: The fused features are decoded using a point pyramid decoder to obtain the predicted 3D point cloud.

[0110] Use point cloud recognition software such as CloudCompare to open the txt point cloud file of the occluded tea bud picking point, and convert the txt point cloud file into an image or 3D model, which is a visual point cloud, thereby visualizing the point cloud.

[0111] By using a hybrid encoder to extract and fuse features from the visible point cloud, a feature vector containing local details and global information is obtained. The hybrid encoder consists of multi-resolution point cloud branches, which extract features from point clouds of different resolutions.

[0112] The step of decoding the fused features using a point pyramid decoder to obtain the predicted 3D point cloud includes:

[0113] S431: The fused features are split using a fully connected layer of a point pyramid decoder to obtain a first resolution feature layer, a second resolution feature layer, and a third resolution feature layer; wherein the resolution of the first resolution feature layer is less than the resolution of the second resolution feature layer, and the resolution of the second resolution feature layer is less than the resolution of the third resolution feature layer.

[0114] S432: Predict the first resolution feature layer, the second resolution feature layer and the third resolution feature layer respectively to obtain the first predicted point cloud, the second predicted point cloud and the third predicted point cloud;

[0115] S433: Perform hierarchical feature fusion and multi-scale refinement on the first, second, and third predicted point clouds to obtain the predicted 3D point cloud.

[0116] The point pyramid decoder is currently used for point cloud completion, point cloud generation, and upsampling tasks. It breaks down an end-to-end completion process into several sub-steps from coarse to fine. Each step is only responsible for generating the residual defects at the current resolution, and then uses the result of the previous step as the seed for the next level, refining it layer by layer.

[0117] like Figure 3 As shown, the point pyramid decoder has three fully connected layers: FC1, FC2, and FC3. FC1, the first fully connected layer, has 256 neurons, FC2, the second fully connected layer, has 512 neurons, and FC3, the third fully connected layer, has 1024 neurons. Each fully connected layer is responsible for predicting the point cloud at a different resolution. Starting with FC1, it predicts the center point once; FC2 predicts the relative coordinates of the secondary center points; and FC3 predicts the relative coordinates of the final center point.

[0118] After decomposing the fused features using a fully connected layer of a point pyramid decoder, three feature layers with different resolutions are obtained: a first-resolution feature layer, a second-resolution feature layer, and a third-resolution feature layer. The first-resolution feature layer has the lowest resolution, corresponding to high-level features. The features extracted by the first-resolution feature layer differ the most from the real data and can reflect the hidden features of the real data, such as the semantic information of objects. Prediction is performed on the first-resolution feature layer to obtain the first predicted point cloud. The second-resolution feature layer has a higher resolution than the first-resolution feature layer, corresponding to medium-resolution features. The features extracted by the second-resolution feature layer are between those extracted by the first and third-resolution feature layers. Prediction is performed on the second-resolution feature layer to obtain the second predicted point cloud. The third-resolution feature layer has the highest resolution, corresponding to low-level features. The features extracted by the third-resolution feature layer are closest to the real data, such as the edges and textures of objects. Prediction is performed on the third-resolution feature layer to obtain the third predicted point cloud. Hierarchical feature fusion and multi-scale refinement are then performed on the first, second, and third predicted point clouds to obtain the predicted 3D point cloud.

[0119] The upper-layer decoder only outputs sparse skeleton points, which can preserve low-frequency geometric features well, but it leads to the loss of high-frequency geometric features. The lower-layer decoder requires high-frequency details. If only the point coordinates of the upper-layer decoder are used as input, the local context is insufficient, which easily leads to misalignment and artifacts. Therefore, this embodiment dynamically injects the local features and global latency extracted by the hybrid encoder at each scale into the corresponding level decoder, thereby achieving cross-scale information fusion.

[0120] Before predicting the first resolution feature layer, the second resolution feature layer, and the third resolution feature layer respectively, the method further includes:

[0121] S4311': Perform transpose convolution or upsampling on the first resolution feature layer to align the first resolution feature layer with the second resolution feature layer;

[0122] S4312': Based on the attention mechanism, the second resolution feature layer is corrected using the first resolution feature layer;

[0123] S4313': Perform transposed convolution or upsampling on the second resolution feature layer to align the second resolution feature layer with the third resolution feature layer;

[0124] S4314': Based on the attention mechanism, the third resolution feature layer is modified using the second resolution feature layer.

[0125] Transposed convolution is a learnable upsampling technique that stretches each element of the input feature map by a stride to form a sparse network before performing regular convolution. The purpose of transposed convolution is to enlarge the size and change the channels. The coordinates of feature points at each level obtained by the encoder's downsampling are not in the same geometric space as the coordinates of the same level point cloud generated by the decoder. If these coordinates are directly concatenated during training, offset drift will occur, leading to misalignment of local details. Therefore, transposed convolution or upsampling is used to enlarge the size of the first-resolution feature layer, increasing its resolution so that the first and second-resolution feature layers have the same resolution, thus aligning them. Similarly, transposed convolution or upsampling is performed on the second-resolution feature layer to make it and the third-resolution feature layer have the same resolution, thus aligning them.

[0126] During the layer-by-layer upsampling process, errors can be amplified cascaded. Therefore, attention mechanisms or weighted summation are needed to correct low-level features using high-level features, which contain local details, while low-level features contain global structure. In this embodiment, the multi-scale local features extracted by the hybrid encoder are subjected to weighted cross-attention with the global latent, followed by transposed convolution or upsampling, before being input into the corresponding fully connected layer to predict the offset.

[0127] Before obtaining the shading status of the tea bud and stem picking point, the process also includes:

[0128] A semantic segmentation model is used to extract a mask image from a tea bud and leaf image. The mask image includes the tea leaf region and the stem region.

[0129] The stem region of the masked image is analyzed to obtain the occlusion status of the tea bud and leaf stem picking point.

[0130] The semantic segmentation model identifies tea leaves and stems, acquiring a mask image of the target and then combining it with a depth image to generate a point cloud. If the stem is visible, its geometric center is directly calculated and used as the 3D coordinates of the picking point. If the stem is not visible, a multi-resolution point cloud estimation model predicts the point cloud of the occluded stem. The geometric center is calculated based on the prediction result to obtain the 3D coordinates of the picking point; that is, the stem point cloud is predicted first, and then the picking point is located. The multi-resolution point cloud estimation model includes a hybrid encoder and a point pyramid decoder.

[0131] The hybrid encoder in this embodiment extracts and fuses features from the visual point cloud to obtain a feature vector containing both local details and global information. The point pyramid decoder decodes the encoded feature vector into three feature layers of different resolutions, each responsible for predicting the output. This embodiment uses the hybrid encoder and point pyramid decoder together to enhance the model's ability to perceive multi-scale morphology.

[0132] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0133] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for predicting three-dimensional picking points of tea buds and leaves based on occlusion conditions, characterized in that, include: Obtain information on the shading conditions at the tea bud, leaf, and stem picking points; If the picking point of tea buds, leaves and stems is not obscured, the real three-dimensional point cloud of tea buds, leaves and stems is extracted using a multi-view information fusion method. Calculate the geometric center of the real three-dimensional point cloud to obtain the real three-dimensional coordinates of the tea bud, leaf, and stem picking point; If the picking point of tea buds, leaves and stems is obscured, a multi-resolution point cloud estimation model is used to predict the tea buds, leaves and stems, and the predicted three-dimensional point cloud is obtained. Calculate the geometric center of the predicted three-dimensional point cloud to obtain the predicted three-dimensional coordinates of the tea bud, leaf, and stem picking point; The method employs a multi-resolution point cloud estimation model to predict tea buds, leaves, and stems, resulting in a predicted 3D point cloud, including: Obtain the visible point cloud of the tea bud picking point that is obscured; A hybrid encoder is used to extract and fuse features from the visual point cloud to obtain fused features; The fused features are decoded using a point pyramid decoder to obtain the predicted 3D point cloud; The method employs a hybrid encoder to extract and fuse features from the visible point cloud to obtain fused features, including: Multi-resolution point cloud branches are used to extract features of different resolutions from the visible point cloud to obtain multi-scale features; wherein, the multi-resolution point cloud branches include high-resolution branches, medium-resolution branches and low-resolution branches; The multi-scale features are fused using a multi-scale perception self-attention module to obtain fused features; The step of decoding the fused features using a point pyramid decoder to obtain the predicted 3D point cloud includes: The fused features are split using a fully connected layer of a point pyramid decoder to obtain a first resolution feature layer, a second resolution feature layer, and a third resolution feature layer; wherein the resolution of the first resolution feature layer is less than the resolution of the second resolution feature layer, and the resolution of the second resolution feature layer is less than the resolution of the third resolution feature layer. Prediction is performed on the first resolution feature layer, the second resolution feature layer and the third resolution feature layer respectively to obtain the first predicted point cloud, the second predicted point cloud and the third predicted point cloud; Hierarchical feature fusion and multi-scale refinement are performed on the first, second, and third predicted point clouds to obtain the predicted 3D point cloud.

2. The method for predicting three-dimensional picking points of tea buds and leaves based on shading conditions according to claim 1, characterized in that, The calculation of the geometric center of the true 3D point cloud to obtain the true 3D coordinates of the tea bud, leaf, and stem picking point includes: The geometric center of a real 3D point cloud is calculated using the following formula: ； ； ； Where M is the total number of points in the real 3D point cloud, x i Let y be the coordinate value of the i-th point in the X-axis direction. i Let z be the coordinate value of the i-th point in the Y-axis direction. i Let x be the coordinate of the i-th point in the Z-axis direction, x be the coordinate of the geometric center in the X-axis direction, y be the coordinate of the geometric center in the Y-axis direction, and z be the coordinate of the geometric center in the Z-axis direction.

3. The method for predicting three-dimensional picking points of tea buds and leaves based on shading conditions according to claim 1, characterized in that, The multi-scale features are fused using a multi-scale perceptual self-attention module to obtain fused features, including: The following formula is used to extract the fusion features: ； ； ； ； Where Q is the query matrix, K is the key matrix, V is the value matrix, and F is the value matrix. in For the aforementioned multi-scale features, W q To query the weight matrix, W k W is the key weight matrix. v Here is the value weight matrix, where Att is the attention weight. For the normalized exponential function, K T Let be the transpose of the key matrix, and d be the number of channels; For point-by-point multiplication, F a For attention features, MLP stands for Multilayer Perceptron, F out This refers to the fusion feature.

4. The method for predicting three-dimensional picking points of tea buds and leaves based on shading conditions according to claim 1, characterized in that, Before obtaining the shading status of the tea bud and stem picking point, the process also includes: A semantic segmentation model is used to extract a mask image from a tea bud and leaf image. The mask image includes the tea leaf region and the stem region. The stem region of the masked image is analyzed to obtain the occlusion status of the tea bud and leaf stem picking point.

5. The method for predicting three-dimensional picking points of tea buds and leaves based on shading conditions according to claim 1, characterized in that, The method of extracting the true three-dimensional point cloud of tea buds, leaves, and stems using multi-view information fusion includes: The camera on the robotic arm is controlled to dynamically capture tea buds and leaves from different angles, resulting in multiple images of tea buds and leaves; the camera moves with the robotic arm. Multi-view information fusion was performed on multiple tea bud and leaf images to obtain a true three-dimensional point cloud of tea buds, leaves, and stems.

6. The method for predicting three-dimensional picking points of tea buds and leaves based on shading conditions according to claim 1, characterized in that, Before predicting the first resolution feature layer, the second resolution feature layer, and the third resolution feature layer respectively, the method further includes: The first resolution feature layer is transposed convolution or upsampled to align the first resolution feature layer with the second resolution feature layer. Based on the attention mechanism, the second resolution feature layer is corrected using the first resolution feature layer; The second resolution feature layer is transposed or upsampled to align the second resolution feature layer with the third resolution feature layer. Based on the attention mechanism, the third resolution feature layer is modified using the second resolution feature layer.

7. The method for predicting three-dimensional picking points of tea buds and leaves based on shading conditions according to claim 1, characterized in that, The hybrid encoder employs a combined multilayer perceptron as a feature extractor, wherein multiple multilayer perceptrons are stacked within the combined multilayer perceptron, and the last N layers of the combined multilayer perceptron are max pooling layers; wherein N≥2.

Images