Fruit pose estimation method and system suitable for deep information missing scene

Abstract

The application discloses a kind of forest fruit pose estimation method and system suitable for depth information missing scene, belong to picking robot environment perception field, wherein the method includes: using trained model to carry out forest fruit pose estimation to orchard fruit image, model includes: target detection network, feature enhancement module and pose prediction head, use fruit image sample to train model, target detection network extracts multi-scale feature map from fruit image sample, and generates target detection frame;Feature enhancement module extracts global semantic Token from multi-scale feature map, combines target feature in target detection frame, constructs feature sequence, extracts global feature from feature sequence;Pose prediction head predicts target pose using global feature, trains model to convergence, obtains trained model.The application improves the accuracy of pose estimation under the condition of lacking depth information, has strong generalization ability, so as to enhance the work robustness and reliability of picking robot in complex outdoor orchard environment.

Description

Technical Field

[0001] This invention belongs to the field of environmental perception for harvesting robots, and more specifically, relates to a method and system for estimating the pose of forest fruits in scenarios where depth information is lacking. Background Technology

[0002] Fruit pose estimation is a key technology in agricultural automation, primarily used in orchard harvesting robots and yield monitoring. It is a crucial step in realizing the core functions of harvesting robots. In outdoor orchard environments, due to external interference and the limitations of prolonged robot operation, depth cameras often experience missing or even complete loss of depth information when acquiring depth images, preventing the robot from accurately locating the target fruit. Therefore, fruit pose estimation under conditions lacking depth information is particularly important.

[0003] Currently, pose estimation for such scenarios mainly employs image feature-based methods. These methods extract key points from RGB images and match them with points in a 3D model to construct 2D-3D point pairs, then use the Perspective-n-Point (PnP) algorithm to solve for the spatial pose of the target. However, due to the sparse and small differences in surface texture features of forest fruits, coupled with significant differences in size, shape, and color among different varieties, these methods have limited generalization ability in complex natural environments, making it difficult to guarantee pose estimation accuracy.

[0004] It is evident that existing forest and fruit pose estimation techniques without depth information suffer from limited generalization ability and low accuracy. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method and system for estimating the pose of forest fruits in scenarios lacking depth information, thereby solving the technical problems of limited generalization ability and low accuracy in existing forest fruit pose estimation technologies under conditions without depth information.

[0006] To achieve the above objectives, according to one aspect of the present invention, a training method for a forest fruit pose estimation model suitable for scenarios lacking deep information is provided. The forest fruit pose estimation model includes: an object detection network, a feature enhancement module, and a pose prediction head. The training method includes:

[0007] Acquire orchard fruit image samples and the true pose of targets in the fruit image samples, wherein the targets are fruit targets;

[0008] The forest fruit image samples are used to train the forest fruit pose estimation model, and the target detection network extracts multi-scale feature maps from the forest fruit image samples and generates target detection boxes.

[0009] The feature enhancement module extracts global semantic tokens from multi-scale feature maps, combines the global semantic tokens with the target features within the target detection box to construct a feature sequence, and uses the normalized center coordinates, width, and height of the target detection box for position encoding to generate a target query. The feature enhancement module includes an encoder and a decoder. The encoder encodes the feature sequence, and the decoder uses the target query to decode the encoded feature sequence to obtain the global features of the target.

[0010] The pose prediction head uses the global features of the target to predict the target pose, uses the error between the predicted target pose and the target's true pose as the loss function, backpropagates to update the model parameters, and trains the forest and fruit pose estimation model until it converges, thus obtaining the trained forest and fruit pose estimation model.

[0011] Furthermore, the pose prediction head includes a rotation regression head and a translation regression head. The rotation regression head uses the global features of the target to predict the six-dimensional vector of the target, and the translation regression head uses the global features of the target to predict the depth value of the target and the two-dimensional offset of the target relative to a reference point, where the reference point is the center point of the normalized target detection box.

[0012] The true pose of the target in the forest fruit image sample includes the true coordinates, true depth value and true six-dimensional vector of the target in the forest fruit image sample;

[0013] The sum of the predicted target's two-dimensional offset relative to the reference point and the reference point's coordinates is used as the predicted target coordinates. Two independent three-dimensional rotation vectors are extracted from the predicted target's six-dimensional vector to construct the predicted rotation matrix. Two independent three-dimensional rotation vectors are extracted from the true six-dimensional vector to construct the true rotation matrix. The errors between the predicted rotation matrix and the true rotation matrix, between the predicted target coordinates and the true target coordinates, and between the predicted target depth value and the true target depth value are fused to form the loss function.

[0014] Furthermore, the depth prediction branches of the rotation regression head and the translation regression head have the same structure as the offset prediction branch of the translation regression head. The rotation regression head includes a pose initialization module and a pose iteration optimization module, with a depth-separable convolutional layer between the pose initialization module and the pose iteration optimization module.

[0015] The pose initialization module obtains the initial rotation features of the target from the global features of the target. Then, it uses a depthwise separable convolutional layer to compress the channel dimension of the initial rotation features to obtain the initial rotation estimation vector. The initial rotation features and the initial rotation estimation vector are concatenated in the channel dimension and input into the pose iteration optimization module to output the residual vector of the rotation features. The residual vector of the rotation features is added to the initial rotation estimation vector to obtain the predicted six-dimensional vector of the target.

[0016] Furthermore, the pose initialization module normalizes the spatial coordinates in the global features of the target and maps them to the interval [-1, 1], generating two coordinate feature channels containing spatial position information. The spatial position information and the global features are concatenated in the two coordinate feature channel dimensions. The concatenated features are then processed sequentially through a standard convolutional layer and a depthwise separable convolutional layer to obtain the initial rotation features of the target.

[0017] Furthermore, both the encoder and decoder of the feature enhancement module have deformable attention modules.

[0018] After the target query enters the bias network in the deformable attention module, it undergoes depth-variable convolution, non-linear activation, and convolution operations in sequence to generate multiple bias values. The deformable attention module combines the reference point with multiple bias values ​​to obtain multiple deformable points. These deformable points are used as sampling points in the multi-scale feature map. The feature vectors of the sampling points are linearly transformed through the query weight matrix, key weight matrix, and value weight matrix in the multi-head attention to generate the query matrix, key matrix, and value matrix. The positions of the keys in the key matrix are adjusted by the offset of the bias values ​​and then combined with the query matrix to calculate the attention weights at the sampling points. The attention weights at all sampling points are weighted and summed with the corresponding value matrices to obtain the output features of the deformable attention module.

[0019] According to another aspect of the present invention, a method for estimating the pose of forest fruits in scenarios lacking depth information is provided, comprising:

[0020] Images of orchard fruits are collected, and a fruit pose estimation model trained using a training method suitable for scenarios with missing depth information is used to estimate the fruit poses in the orchard images, thus obtaining the fruit poses.

[0021] Furthermore, the fruit pose includes a six-dimensional vector of the target, a depth value, and a two-dimensional offset of the target relative to a reference point. The fruit pose is converted into the translation vector and rotation matrix required for the harvesting robot to pick the fruit in the following manner:

[0022] The target coordinates are obtained by adding the two-dimensional offset of the target relative to the reference point to the coordinates of the reference point. The target coordinates and the target depth value are combined with the camera intrinsic parameters to obtain the translation vector of the target in three-dimensional space through back projection. Two independent three-dimensional rotation vectors are extracted from the six-dimensional vector to construct the rotation matrix.

[0023] According to another aspect of the present invention, a forest fruit pose estimation system suitable for scenarios lacking depth information is provided, comprising:

[0024] The preprocessing module is used to acquire orchard fruit image samples and the true pose of targets in the fruit image samples, wherein the targets are fruit targets;

[0025] The training module trains a forest fruit pose estimation model using forest fruit image samples. The model includes an object detection network, a feature enhancement module, and a pose prediction head. The object detection network extracts multi-scale feature maps from the forest fruit image samples and generates object detection boxes. The feature enhancement module extracts global semantic tokens from the multi-scale feature maps, combines these tokens with target features within the object detection boxes to construct a feature sequence, and uses the normalized center coordinates, width, and height of the object detection boxes for position encoding to generate a target query. The feature enhancement module includes an encoder and a decoder. The encoder encodes the feature sequence, and the decoder decodes the encoded feature sequence using the target query to obtain the global features of the target. The pose prediction head uses the global features of the target to predict the target pose, uses the error between the predicted and true target poses as a loss function, backpropagates to update the model parameters, and trains the forest fruit pose estimation model until convergence, resulting in a well-trained model.

[0026] The prediction module uses a trained orchard pose estimation model to estimate the pose of orchard fruits in orchard images, thus obtaining the fruit poses.

[0027] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed by a processor, it controls the device where the storage medium is located to execute a training method for a forest fruit pose estimation model suitable for scenarios with missing depth information and / or a forest fruit pose estimation method suitable for scenarios with missing depth information.

[0028] According to another aspect of the present invention, a harvesting robot is provided. The harvesting robot uses a training method for a forest fruit pose estimation model suitable for scenarios with missing depth information to train a forest fruit pose estimation model to estimate the pose of forest fruits in an orchard image, obtain the forest fruit pose, and convert the forest fruit pose into a translation vector and a rotation matrix for harvesting forest fruits.

[0029] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0030] (1) In this invention, the target detection network extracts multi-scale features and generates detection box information. Multi-scale features help improve the detection capability of small targets such as orchards, while the detection box information serves as a positional prior for subsequent feature enhancement modules to further improve the accuracy of target representation. The feature enhancement module combines global semantic tokens with target features within the target detection box to establish a semantic association between the target and the global scene, introduces target queries to process the feature sequence, and generates global features with context awareness. Compared to methods that directly perform pose regression, this invention uses global features with context awareness, which can effectively improve the accuracy of pose estimation. The model trained by this invention can improve the pose estimation accuracy of orchards in the absence of depth information, has strong generalization ability, and thus enhances the robustness and reliability of the harvesting robot in complex outdoor orchard environments.

[0031] (2) This invention includes a pose prediction head comprising a rotation regression head and a translation regression head. Traditional pose estimation methods typically use quaternions or Euler angles to parameterize the target rotation. However, these methods suffer from problems such as discontinuous representation, singularities, and complex optimization spaces in practical applications, which can easily lead to difficulties in convergence or a decrease in prediction accuracy during neural network training, especially when the rotation angle is close to the boundary region. To overcome these problems, this invention uses a rotation regression head for 6D pose estimation. Compared with traditional techniques, six-dimensional vectors have stronger continuity and a simpler expression, exhibiting good convergence performance and prediction accuracy in multiple pose estimation tasks. In terms of translation estimation, a translation regression head is used to predict depth values ​​and offsets, further improving prediction accuracy. The loss function considers depth value loss, displacement loss, and 6D pose loss, which can improve the prediction accuracy of the model.

[0032] (3) This invention integrates the initial rotation estimation vector with shallower local detail features (residual vectors of rotation features) in the network, enabling the model to capture fine-grained changes in the target pose while maintaining global structure awareness. This mechanism not only effectively improves the accuracy and robustness of rotation estimation but also provides a solid feature foundation for subsequent high-precision pose regression. This invention employs a pose prediction head based on an iterative refinement strategy to progressively optimize and predict the target's six-dimensional vector (six-degree-of-freedom pose). Based on the initial estimation, the pose prediction head iteratively refines and corrects the pose, thereby achieving a more accurate final regression result. Compared to traditional one-time regression methods, this invention effectively mitigates the impact of initial prediction errors and significantly improves the accuracy and robustness of pose estimation.

[0033] (4) The translation regression head introduces two independent estimation branches, each with a structure consistent with the rotation regression head. The pose initialization module concatenates positional information and global features along the channel dimension to fuse spatial structure and semantic information. Both translation and rotation regression employ a combination of standard convolution and depthwise separable convolution for feature extraction. Through this design, the model can more accurately capture changes in the spatial position of the target and maintain good translation and rotation estimation performance even in orchard scenes with weak textures and complex backgrounds.

[0034] (5) The feature enhancement module of this invention introduces a deformable attention mechanism, which effectively combines the spatial sparse location awareness capability of deformable convolution with the global modeling capability of the feature enhancement module, taking into account both the flexibility of feature selection and the context awareness capability. Specifically, for each query position, this mechanism no longer performs global attention calculation on the entire feature map, but dynamically selects several key positions (deformation points) around the reference point as sampling points to perform local attention operations, thereby significantly reducing the amount of computation between each query and all keys, and greatly reducing the overall computational complexity. In addition, deformable attention also has good training characteristics: by focusing on semantically related regions in space, it can effectively alleviate the problem of the attention weights of the feature enhancement module being too scattered in the early stage of training, and accelerate the convergence speed of the model. More importantly, the computational complexity of this mechanism is linearly related to the number of sampling points, rather than directly linked to the spatial dimension of the feature map, thereby breaking through the efficiency bottleneck of the standard attention mechanism when processing high-resolution images, and improving the scalability and practicality of the model in visual tasks.

[0035] (6) The model trained in this invention decouples the translation and rotation components when performing pose estimation. For translation estimation, convolutional neural networks, due to their translation invariance, are better at extracting semantic features of local regions, but weaker at modeling the absolute spatial position of the target in the input image. This characteristic leads to limited accuracy when directly regressing the 3D translation vector. The network is less effective at predicting absolute coordinates than relative coordinates. Therefore, this invention does not directly model the target's translation vector T(t) x , t y , t z ) T Instead of performing end-to-end regression, this invention employs a decoupled modeling strategy to construct the translation prediction process in a more reasonable way. The translation part is decoupled using camera intrinsic parameters, while the rotation part is modeled using two 3D vectors. Compared to the traditional unified modeling approach, this invention's decoupling strategy better balances the feature representation of translation and rotation, avoiding mutual interference between the two during the modeling process. Attached Figure Description

[0036] Figure 1This is a flowchart of a forest fruit pose estimation method applicable to scenarios with missing depth information, provided by an embodiment of the present invention;

[0037] Figure 2 This is a structural diagram of the pose estimation model provided in an embodiment of the present invention;

[0038] Figure 3 This is a structural diagram of the feature enhancement module based on deformable Transformer provided in an embodiment of the present invention;

[0039] Figure 4 This is a structural diagram of the deformable attention module based on position prior provided in an embodiment of the present invention;

[0040] Figure 5 This is a structural diagram of the pose prediction head provided in an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0042] like Figure 1 As shown, a training method for a forest fruit pose estimation model suitable for scenarios lacking deep information is presented. The forest fruit pose estimation model includes: an object detection network, a feature enhancement module, and a pose prediction head. The training method includes:

[0043] Acquire orchard fruit image samples and the true pose of targets in the fruit image samples, wherein the targets are fruit targets;

[0044] The forest fruit image samples are used to train the forest fruit pose estimation model, and the target detection network extracts multi-scale feature maps from the forest fruit image samples and generates target detection boxes.

[0045] The target detection network can adopt any target detection network model with multi-scale feature fusion capability.

[0046] The feature enhancement module extracts global semantic tokens from multi-scale feature maps, combines the global semantic tokens with the target features within the target detection box to construct a feature sequence, and uses the normalized center coordinates, width, and height of the target detection box for position encoding to generate a target query. The feature enhancement module includes an encoder and a decoder. The encoder encodes the feature sequence, and the decoder uses the target query to decode the encoded feature sequence to obtain the global features of the target.

[0047] The pose prediction head uses the global features of the target to predict the target pose, uses the error between the predicted target pose and the target's true pose as the loss function, backpropagates to update the model parameters, and trains the forest and fruit pose estimation model until it converges, thus obtaining the trained forest and fruit pose estimation model.

[0048] Example 1

[0049] Example 1 describes in detail the structure of the pose estimation model and its training method.

[0050] like Figure 2 As shown, the pose estimation model includes an object detection network, a feature enhancement module, and a pose prediction head, which are trained in the following way:

[0051] Collect orchard fruit image samples and depth values ​​of the fruit within the image samples.

[0052] Forest fruit image samples are input into the target detection network to extract multi-scale feature maps and generate target detection boxes;

[0053] like Figure 3 As shown, the feature enhancement module is a deformable Transformer-based feature enhancement module, including five encoders and five decoders. Each encoder is configured with a deformable attention module, residual connection and layer normalization (Add&Norm), feedforward neural network (FFN), and Add&Norm in sequence from input to output. Each decoder is configured with multi-head attention, Add&Norm, Deformable Attention, Add&Norm, FFN, and Add&Norm in sequence from input to output.

[0054] Residual connections are used to add the input value to the output value, thereby enhancing the network's performance. Because stacking multiple deep networks can easily lead to vanishing or exploding gradients, residual connections can ensure that information flows within the network, avoiding these problems.

[0055] Layer normalization refers to standardizing each input sample so that the mean of each feature is 0 and the variance is 1. This makes the importance of each feature more balanced, avoiding overfitting caused by overemphasizing certain features in the network. Therefore, layer normalization can improve the generalization ability of the network.

[0056] In the Transformer network with a five-layer encoder-decoder structure, a deformable attention mechanism and positional priors of the detection box information are introduced. First, global semantic tokens are extracted from multi-scale feature maps. The global semantic tokens are combined with the target features in the target detection box to construct a feature sequence. Then, the sequence is input into the Transformer network to perform semantic modeling between the target and the global scene, and to establish contextual dependencies between multiple targets.

[0057] During the encoding and decoding stages, a deformable attention mechanism is used to sparsely sample the input features. At the same time, the center coordinates, width, and height of the normalized target detection box are used for position encoding to generate a target query, which serves as the input to the Transformer decoder.

[0058] The implementation method for generating the target query embedding is as follows:

[0059] For each detection box, extract its four parameters (x, x) c y c (w, h), where x c y c The coordinates represent the center coordinates of the detection box, and w and h represent the width and height of the detection box; these coordinates and dimensions are then normalized to the image dimensions. Where W and H are the width and height of the original image; the four normalized values ​​are concatenated into a position vector. Finally, the position vector is encoded using an MLP (Multilayer Perceptron). The encoded vector q pos Used as the target query input in the Transformer decoder, replacing the traditional learnable query embedding.

[0060] Figure 4 The bias network in the deformable attention module includes DWConv (depth-variable convolution), GRLU (non-linear activation function), and Conv (convolution). After the target query enters the bias network, it undergoes depth-variable convolution, non-linear activation, and convolution operations in sequence to obtain the bias value Θ. offset .

[0061] The deformable attention module uses the center point of the normalized target detection box as a reference point. This reference point is combined with the bias value to obtain deformable points. After the target query enters the bias network, multiple bias values ​​are dynamically generated. These multiple bias values ​​are combined with the reference point to obtain multiple deformable points, forming a deformable point set R. These deformable points are used as sampling points in the multi-scale feature map X. The feature vectors of the sampling points are passed through the query weight matrix W in the multi-head attention module. Q Key weight matrix W K Sum weight matrix W VPerform a linear transformation to generate the query matrix Q, key matrix K, and value matrix V. Calculate the attention weights at the sampling points using the following formula:

[0062]

[0063] Where K+Δp represents the position of the key adjusted by the offset value Δp. The Softmax function is used to calculate the attention weights, such that the sum of all attention weights is 1, d k Q represents the number of columns in the key matrix K, and Q has the same number of columns as K.

[0064] The output features of the deformable attention module are obtained by weighting and summing the attention weights at all sampling points with the corresponding value matrices.

[0065] The deformable attention mechanism effectively combines the spatial sparse location awareness of deformable convolution with the global modeling capability of Transformer, balancing the flexibility of feature selection with context awareness. Specifically, for each query location, this mechanism no longer performs global attention calculation on the entire feature map. Instead, it dynamically selects several key locations around the reference point as sampling points and performs local attention operations, thereby significantly reducing the computational cost between each query and all keys, and greatly reducing the overall computational complexity.

[0066] Furthermore, deformable attention possesses excellent training properties: by spatially focusing on semantically relevant regions, it effectively alleviates the problem of overly dispersed attention weights in the early stages of Transformer training, accelerating model convergence. More importantly, the computational complexity of this mechanism is linearly related to the number of sampling points, rather than directly linked to the spatial dimension of the feature map. This overcomes the efficiency bottleneck of standard attention mechanisms when processing high-resolution images, improving the scalability and practicality of the model in visual tasks.

[0067] This invention designs an iterative refinement strategy to progressively approximate the final 6D pose of a target, thereby improving the accuracy and stability of pose estimation. This strategy achieves more accurate pose estimation by introducing feature residuals for multiple refinement regressions based on the initial prediction. The pose prediction head includes a rotation regression head and a translation regression head. The structure of the rotation regression head is consistent with that of the translation regression head network, as shown below. Figure 5 As shown.

[0068] Taking the rotary regressor head as an example, the relevant explanation is as follows:

[0069] First, obtain the global feature representation output by the deformable Transformer-based feature enhancement module. Then, the model first performs preliminary rotational pose estimation to obtain the initial rotational features r of the target. pre Specifically, firstly, regarding F... i The spatial coordinates (x, y) are normalized and then mapped to the interval [-1, 1] to generate two coordinate feature channels containing spatial location information. Subsequently, this location information is compared with the global feature F output by the deformable Transformer-based feature enhancement module. i Channel-level concatenation is performed to fuse spatial structure and semantic information. The fused features are then processed sequentially through a standard convolutional layer (Conv Module) and three depthwise separable convolutional layers (DWConv) to extract more discriminative pose-related features layer by layer. Group normalization and the Swish activation function are introduced after each convolutional layer to enhance the model's non-linear expressiveness and training stability. The final output is the initial rotation feature r. pre This will serve as the foundational input for subsequent refinement and prediction processes. The calculation method is as follows:

[0070] r pre =DSConv 3 (Conv(C(F i coord(F) i ))))

[0071] Where DSConv(g) represents depthwise separable convolution, Conv(g) represents standard convolution, C(g) is the concatenation operation, and coord(g) is the spatial coordinate feature extraction operation.

[0072] Then, an additional depthwise separable convolutional layer is used to compress the number of feature channels to 6, resulting in the initial rotation estimation vector r. init The above process can be represented as:

[0073] r init =DSConv(r pre )

[0074] Finally, the initial prediction vector r pre With the initial rotation estimation vector r init The data is concatenated along the channel dimension and input into the Pose Iteration Module (PItM). This module consists of two depthwise separable convolutional layers, each followed by group normalization and a Swish activation function to enhance nonlinear expressiveness and extract higher-order rotation features. Subsequently, the features undergo channel compression through an additional depthwise separable convolutional layer, outputting a residual vector Δr of the rotation features, which is used to further refine the rotation estimation results.

[0075] Δx=DSConv 3 (C(r pre r init ))

[0076] The residual vector Δr of the rotation feature is compared with the initial rotation estimation vector r. init Add them together to obtain the final optimized 6D rotation prediction result r. 6d :

[0077] r 6d =r init +Δr

[0078] The above process constitutes a complete pose iteration optimization process. By fusing the initial rotation prediction results with shallower local detail features in the network, the model can further capture fine-grained changes in the target pose while maintaining global structure perception capabilities. This mechanism not only effectively improves the accuracy and robustness of rotation estimation, but also provides a solid feature foundation for subsequent high-precision pose regression.

[0079] In terms of translation estimation, to further improve prediction accuracy, this invention does not simply use the reference point coordinates as the target centroid coordinates, but instead adopts a more flexible modeling strategy. Specifically, the translation regression head introduces two independent estimation branches, which predict the target's two-dimensional pixel offset (Δx, Δy) and depth value t in the image, respectively. z This approach achieves decoupled modeling of 3D translation vectors. Each branch structure is consistent with the rotation regression head, employing a combination of standard convolution and depthwise separable convolution, along with group normalization and activation functions for feature extraction. Through this design, the model can more accurately capture changes in the spatial position of the target, maintaining good translation estimation performance even in orchard scenes with weak textures and complex backgrounds.

[0080] Traditional pose estimation methods typically use quaternions or Euler angles to parameterize the target rotation. However, these methods suffer from problems such as discontinuous representations, singularities, and complex optimization spaces in practical applications, which can easily lead to convergence difficulties or decreased prediction accuracy during neural network training, especially when the rotation angle is close to the boundary region.

[0081] To overcome the above problems, this invention introduces a six-dimensional rotation representation (6DRotationRepresentation) based on high-dimensional vectors. Compared with traditional techniques, the six-dimensional representation has stronger continuity, a simpler expression form, and has been proven to have good convergence performance and prediction accuracy in multiple pose estimation tasks.

[0082] In the rotation representation used in this invention, the rotation consists of two three-dimensional vectors, each corresponding to one of the two orthogonal directions (i.e., column vectors) in the first three columns of the rotation matrix, thus implicitly constructing the orthogonal structure of the rotation matrix. Specifically, the model directly obtains a six-dimensional vector r through the rotation regression head. 6d From the given information (r1, r2, r3, r4, r5, r6), extract two independent three-dimensional vectors v1(r1, r2, r3) and v2(r4, r5, r6). Based on v1 and v2, construct standard bases e1, e2, e3, using the following formula:

[0083] e1 = Φ(v1)

[0084] e2 = Φ(v2)

[0085] e3 = e1 × e2

[0086] Where r1, r2, and r3 represent the rotation angles of the XYZ axes, r4, r5, and r6 represent the redundant rotation angles of the XYZ axes, Φ(·) represents vector normalization, and × represents the vector cross product.

[0087] Using this set of orthogonal bases, we can further construct the rotation matrix R:

[0088] R = [e1] T e2 T e3 T ]

[0089] To more accurately quantify the angular difference between the predicted rotation and the actual rotation, geodesic loss was used as the optimization objective in the rotation estimation task. Its mathematical expression is as follows:

[0090]

[0091] Where tr(·) represents the trace of the matrix, Let R represent the actual rotation matrix. pred This represents the predicted rotation matrix.

[0092] For translation estimation, convolutional neural networks, due to their translation invariance, are better at extracting semantic features of local regions, but weaker at modeling the absolute spatial position of targets in the input image. This characteristic leads to limited accuracy when directly regressing the 3D translation vector. Therefore, this invention does not directly model the target's translation vector T(t) x , t y , t z ) TInstead of performing end-to-end regression, a decoupled modeling strategy is adopted to construct the translation prediction process in a more reasonable way. The network is less effective at predicting absolute coordinates than relative coordinates. Specifically, considering the clear perspective projection relationship between the target's two-dimensional projection position in the image (image center point) and its actual three-dimensional spatial position, a pinhole camera model can be used for modeling. This relationship can be expressed as:

[0093] t z (p x p y ,1) T =K(t) x , t y ,t z ) T

[0094] Where K is the camera intrinsic parameter matrix, (p x p y ) represents the projected coordinates of the target's 3D center point on the image plane.

[0095] Based on the above relationship, the three-dimensional translation T is first decoupled into the two-dimensional projected coordinates (p) of the target on the image. x p y ) and its corresponding depth t z To improve the network's ability to model target locations, the two-dimensional pixel coordinates (p... x p y ) is further decomposed into reference point coordinates (c x c y The target's two-dimensional offset (Δx, Δy) relative to the reference point and the target's depth value t are also considered. Therefore, the network's translation branch only needs to predict the target's two-dimensional offset (Δx, Δy) relative to the reference point and the target's depth value t. z Finally, by predicting the two-dimensional offset (Δx, Δy) and the target depth t... z This allows us to obtain the target's 3D translation vector. Combined with the camera's intrinsic parameters, we can then recover the target's 3D translation vector T through backprojection.

[0096]

[0097] t z =t z

[0098] Among them, f x f y p x p y This can be obtained through the camera's internal parameters.

[0099] To effectively supervise the translation prediction, the L2 norm is used as the loss function in the 2D projection coordinate prediction stage to measure the error between the predicted coordinates and the true projected coordinates. Its mathematical expression is as follows:

[0100]

[0101] Among them, (t) x , t y ) pred and (t) x , t y ) gt These represent the predicted and actual coordinates, respectively. Here, the actual coordinates refer to the coordinates of the target in the image.

[0102] In depth prediction, considering the potential for occlusion, lighting variations, or noise interference in the scene, directly using L2 loss might be overly sensitive to outliers, thus affecting overall training stability. Therefore, L1 absolute error is used as the loss function for depth to enhance the model's robustness in depth estimation tasks. Its mathematical expression is as follows:

[0103]

[0104] Among them, t zpred and t zgt These represent the predicted and actual target depth values, respectively.

[0105] Compared to traditional unified modeling methods, the decoupling strategy of this invention can better balance the feature representation of translation and rotation, avoiding mutual interference between the two during the modeling process. Specifically, rotation estimation relies more on the target's appearance structure and local morphological changes, such as edge contours and surface textures, and has strong local discriminative power; while translation estimation focuses on the overall spatial position information of the target in the image, including its relative layout and scale features with respect to the environment.

[0106] Example 2

[0107] Example 2 describes in detail how the harvesting robot uses the present invention.

[0108] When the harvesting robot uses the model trained by this invention to harvest orchard fruit, it inputs the real-time collected images of orchard fruit into the trained pose estimation model to predict the two-dimensional pixel offset and depth value of the fruit target in the image, as well as the six-dimensional vector.

[0109] The two-dimensional pixel offset and depth value of the fruit target in the image are combined with camera intrinsic parameters to reconstruct the translation vector of the fruit target in three-dimensional space through back projection. Two independent three-dimensional vectors are extracted from the six-dimensional vector to construct a rotation matrix. The harvesting robot harvests the fruit based on the translation vector and the rotation matrix.

[0110] The orchard contains fruit trees including: citrus, oranges, jujubes, apples, cherries, pomegranates, and persimmons.

[0111] The model trained by this invention can improve the accuracy of fruit pose estimation in orchards even in the absence of depth information, and has strong generalization ability, thereby enhancing the robustness and reliability of the harvesting robot in complex outdoor orchard environments.

[0112] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A training method for a forest fruit pose estimation model suitable for scenarios lacking depth information, characterized in that, The forest fruit pose estimation model includes: a target detection network, a feature enhancement module, and a pose prediction head; the training method includes: Acquire orchard fruit image samples and the true pose of targets in the fruit image samples, wherein the targets are fruit targets; The forest fruit image samples are used to train the forest fruit pose estimation model, and the target detection network extracts multi-scale feature maps from the forest fruit image samples and generates target detection boxes. The feature enhancement module extracts global semantic tokens from multi-scale feature maps, combines the global semantic tokens with the target features within the target detection box to construct a feature sequence, and uses the normalized center coordinates, width, and height of the target detection box for position encoding to generate a target query. The feature enhancement module includes an encoder and a decoder. The encoder encodes the feature sequence, and the decoder uses the target query to decode the encoded feature sequence to obtain the global features of the target. The pose prediction head uses the global features of the target to predict the target pose, uses the error between the predicted target pose and the true target pose as the loss function, backpropagates to update the model parameters, trains the forest and fruit pose estimation model until convergence, and obtains the trained forest and fruit pose estimation model. The pose prediction head includes a rotation regression head and a translation regression head. The rotation regression head uses the global features of the target to predict the six-dimensional vector of the target, and the translation regression head uses the global features of the target to predict the depth value of the target and the two-dimensional offset of the target relative to a reference point, which is the center point of the target detection box after normalization. The true pose of the target in the forest fruit image sample includes the true coordinates, true depth value and true six-dimensional vector of the target in the forest fruit image sample; The sum of the predicted target's two-dimensional offset relative to the reference point and the reference point's coordinates is used as the predicted target coordinates. Two independent three-dimensional rotation vectors are extracted from the predicted target's six-dimensional vector to construct the predicted rotation matrix. Two independent three-dimensional rotation vectors are extracted from the true six-dimensional vector to construct the true rotation matrix. The errors between the predicted rotation matrix and the true rotation matrix, between the predicted target coordinates and the true target coordinates, and between the predicted target depth value and the true target depth value are fused to form the loss function. The depth prediction branch of the rotation regression head and the offset prediction branch of the translation regression head have the same structure. The rotation regression head includes a pose initialization module and a pose iteration optimization module, and there is a depth-separable convolutional layer between the pose initialization module and the pose iteration optimization module. The pose initialization module obtains the initial rotation features of the target from the global features of the target. Then, it uses a depthwise separable convolutional layer to compress the channel dimension of the initial rotation features to obtain the initial rotation estimation vector. The initial rotation features and the initial rotation estimation vector are concatenated in the channel dimension and input into the pose iteration optimization module to output the residual vector of the rotation features. The residual vector of the rotation features is added to the initial rotation estimation vector to obtain the predicted six-dimensional vector of the target.

2. The training method for a forest fruit pose estimation model applicable to scenarios lacking depth information, as described in claim 1, is characterized in that... The pose initialization module normalizes the spatial coordinates in the global features of the target and then maps them to an interval. Within the system, two coordinate feature channels containing spatial location information are generated. The spatial location information and global features are concatenated along the dimensions of the two coordinate feature channels. The concatenated features are then processed sequentially through a standard convolutional layer and a depthwise separable convolutional layer to obtain the initial rotation features of the target.

3. A training method for a forest fruit pose estimation model applicable to scenarios lacking depth information, as described in claim 1 or 2, characterized in that... Both the encoder and decoder of the feature enhancement module have deformable attention modules. After the target query enters the bias network in the deformable attention module, it undergoes depth-variable convolution, non-linear activation, and convolution operations in sequence to generate multiple bias values. The deformable attention module combines the reference point with multiple bias values ​​to obtain multiple deformable points. These deformable points are used as sampling points in the multi-scale feature map. The feature vectors of the sampling points are linearly transformed through the query weight matrix, key weight matrix, and value weight matrix in the multi-head attention to generate the query matrix, key matrix, and value matrix. The positions of the keys in the key matrix are adjusted by the offset of the bias values ​​and then combined with the query matrix to calculate the attention weights at the sampling points. The attention weights at all sampling points are weighted and summed with the corresponding value matrices to obtain the output features of the deformable attention module.

4. A method for estimating the pose of forest fruits in scenarios lacking depth information, characterized in that, include: Images of orchard fruits are collected, and the fruit pose estimation model trained using the training method of any one of claims 1-3, which is suitable for scenarios with missing depth information, is used to estimate the fruit pose in the orchard images to obtain the fruit pose.

5. The forest fruit pose estimation method applicable to scenarios lacking depth information as described in claim 4, characterized in that, The fruit pose includes a six-dimensional vector of the target, a depth value, and a two-dimensional offset of the target relative to a reference point. The fruit pose is converted into the translation vector and rotation matrix required for the harvesting robot to pick the fruit in the following manner: The target coordinates are obtained by adding the two-dimensional offset of the target relative to the reference point to the coordinates of the reference point. The target coordinates and the target depth value are combined with the camera intrinsic parameters to obtain the translation vector of the target in three-dimensional space through back projection. Two independent three-dimensional rotation vectors are extracted from the six-dimensional vector to construct the rotation matrix.

6. A forest fruit pose estimation system suitable for scenarios lacking depth information, used to execute the forest fruit pose estimation method for scenarios lacking depth information as described in claim 4 or 5, characterized in that, include: The preprocessing module is used to acquire orchard fruit image samples and the true pose of targets in the fruit image samples, wherein the targets are fruit targets; The training module trains a forest fruit pose estimation model using forest fruit image samples. The model includes an object detection network, a feature enhancement module, and a pose prediction head. The object detection network extracts multi-scale feature maps from the forest fruit image samples and generates object detection boxes. The feature enhancement module extracts global semantic tokens from the multi-scale feature maps, combines these tokens with target features within the object detection boxes to construct a feature sequence, and uses the normalized center coordinates, width, and height of the object detection boxes for position encoding to generate a target query. The feature enhancement module includes an encoder and a decoder. The encoder encodes the feature sequence, and the decoder decodes the encoded feature sequence using the target query to obtain the global features of the target. The pose prediction head uses the global features of the target to predict the target pose, uses the error between the predicted and true target poses as a loss function, backpropagates to update the model parameters, and trains the forest fruit pose estimation model until convergence, resulting in a well-trained model. The prediction module uses a trained orchard pose estimation model to estimate the pose of orchard fruits in orchard images, thus obtaining the fruit poses.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed by a processor, it controls the device where the storage medium is located to execute a training method for a forest fruit pose estimation model applicable to scenarios with missing depth information as described in any one of claims 1-3 and / or a forest fruit pose estimation method applicable to scenarios with missing depth information as described in claim 4 or 5.

8. A harvesting robot, characterized in that, The harvesting robot uses the training method of the fruit pose estimation model for scenarios with missing depth information as described in any one of claims 1-3 to estimate the pose of the fruit in the orchard image, obtain the fruit pose, and convert the fruit pose into translation vectors and rotation matrices for harvesting fruit.

Images