Forest and fruit pose estimation method and system suitable for depth information missing scene

Abstract

The invention discloses a forest fruit pose estimation method and system suitable for a depth information loss scene, and belongs to the field of picking robot environment awareness, and the method comprises the steps: carrying out the forest fruit pose estimation of a forest fruit image of an orchard through employing a trained model, and enabling the model to comprise a target detection network, a feature enhancement module and a pose prediction head, using the forest fruit image sample training model and the target detection network to extract a multi-scale feature map from the forest fruit image sample, and generating a target detection frame; a feature enhancement module extracts global semantic Token from the multi-scale feature map, constructs a feature sequence in combination with target features in a target detection frame, and extracts global features from the feature sequence; the pose prediction head predicts the target pose by using the global features, and trains the model to convergence to obtain a trained model. According to the method, the pose estimation precision is improved in the absence of depth information, and the generalization ability is high, so that the operation robustness and reliability of the picking robot in a complex outdoor orchard environment are enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of environmental perception of picking robots, and more specifically, relates to a method and system for estimating the posture of fruits and trees in scenarios where depth information is missing. Background Art

[0002] Orchard pose estimation is a key technology in agricultural automation, primarily used in scenarios such as orchard harvesting robots and yield monitoring. It is crucial for realizing the core functions of harvesting robots. In outdoor orchards, due to external interference and the long operating time constraints of harvesting robots, depth cameras often experience missing or even complete loss of depth information when capturing depth images, making it difficult for robots to accurately locate the target fruit. Therefore, pose estimation for orchard fruit without depth information is particularly important.

[0003] Currently, pose estimation for such scenes primarily relies on image feature-based methods. These methods extract key points from RGB images and match them with points in a 3D model to construct 2D-3D point pairs. The spatial pose of the target is then determined using the PnP (Perspective-n-Point) algorithm. However, due to the sparse and minimally differentiated surface texture features of fruit, and the significant variations in size, shape, and color between varieties, these methods have limited generalization capabilities in complex natural environments, making pose estimation accuracy difficult to guarantee.

[0004] It can be seen that the existing orchard pose estimation technology without depth information has technical problems such as limited generalization ability and low accuracy. Summary of the Invention

[0005] In response to the above defects or improvement needs of the existing technology, the present invention provides a method and system for estimating the pose of fruits and trees in scenarios where depth information is missing, thereby solving the technical problems of limited generalization ability and low accuracy in the existing technology for estimating the pose of fruits and trees in the absence of depth information.

[0006] To achieve the above objectives, according to one aspect of the present invention, a training method for a Linguo pose estimation model suitable for use in depth information missing scenarios is provided. The Linguo pose estimation model includes: a target detection network, a feature enhancement module, and a pose prediction head. The training method includes:

[0007] Obtaining a fruit orchard image sample and a true position of a target in the fruit orchard image sample, wherein the target is a fruit orchard target;

[0008] The fruit and orchard image samples are used to train the fruit and orchard pose estimation model. The object detection network extracts multi-scale feature maps from the fruit and orchard image samples and generates object detection frames.

[0009] The feature enhancement module extracts global semantic tokens from multi-scale feature maps, combines the global semantic tokens with the target features within the target detection box, constructs a feature sequence, and uses the normalized center coordinates, width, and height of the target detection box for position encoding to generate a target query. The feature enhancement module includes an encoder and a decoder. The encoder encodes the feature sequence, and the decoder decodes the encoded feature sequence using the target query to obtain the global features of the target.

[0010] The pose prediction head uses the global features of the target to predict the target pose, takes the error between the predicted target pose and the true pose of the target as the loss function, backpropagates to update the model parameters, trains the Lin Guo pose estimation model until convergence, and obtains the trained Lin Guo pose estimation model.

[0011] Furthermore, the pose prediction head includes a rotation regression head and a translation regression head. The rotation regression head uses the global features of the target to predict the six-dimensional vector of the target. The translation regression head uses the global features of the target to predict the target depth value and the two-dimensional offset of the target relative to a reference point. The reference point is the center point of the normalized target detection frame.

[0012] The real position of the target in the forest fruit image sample includes the real coordinates, real depth value and real six-dimensional vector of the target in the forest fruit image sample;

[0013] The sum of the predicted two-dimensional offset of the target relative to the reference point and the coordinates of the reference point is used as the predicted target coordinates. Two independent three-dimensional rotation vectors are extracted from the predicted six-dimensional vector of the target to construct a predicted rotation matrix. Two independent three-dimensional rotation vectors are extracted from the real six-dimensional vector to construct a real rotation matrix. The error between the predicted rotation matrix and the real rotation matrix, the error between the predicted target coordinates and the real coordinates of the target, and the error between the predicted target depth value and the real depth value of the target are fused to form a loss function.

[0014] Furthermore, the depth value prediction branch of the rotation regression head and the translation regression head has the same structure as the offset prediction branch of the translation regression head. The rotation regression head includes: a pose initialization module and a pose iterative optimization module. There is a depth-separable convolution layer between the pose initialization module and the pose iterative optimization module.

[0015] The pose initialization module obtains the initial rotation feature of the target from the global feature of the target. Then, a depth-wise separable convolution layer is used to compress the channel dimension of the initial rotation feature to obtain the initial rotation estimation vector. The initial rotation feature and the initial rotation estimation vector are concatenated in the channel dimension and input into the pose iterative optimization module. The residual vector of the rotation feature is output and added to the initial rotation estimation vector to obtain the predicted six-dimensional vector of the target.

[0016] Furthermore, the pose initialization module normalizes the spatial coordinates in the global features of the target and maps them to the interval [-1, 1], generating two coordinate feature channels containing spatial position information, and splicing the spatial position information and the global features in the two coordinate feature channel dimensions. The spliced features are processed in turn by the standard convolution layer and the depth-wise separable convolution layer to obtain the initial rotation features of the target.

[0017] Furthermore, both the encoder and decoder of the feature enhancement module have a deformable attention module.

[0018] After the target query enters the bias network in the deformable attention module, it performs deep variable convolution, nonlinear activation and convolution operations in sequence to generate multiple bias values. The deformable attention module combines the reference point with multiple bias values to obtain multiple deformation points, and uses the deformation points as sampling points in the multi-scale feature map. The feature vectors of the sampling points are linearly transformed through the query weight matrix, key weight matrix and value weight matrix in the multi-head attention to generate the query matrix, key matrix and value matrix. The position of the key in the key matrix is adjusted by the offset of the bias value and then combined with the query matrix to calculate the attention weight at the sampling point. The attention weights at all sampling points are weighted and summed with the corresponding value matrix to obtain the output features of the deformable attention module.

[0019] According to another aspect of the present invention, a method for estimating the pose of a tree fruit in a scene where depth information is missing is provided, comprising:

[0020] Fruit and vegetable images in an orchard are collected, and the fruit and vegetable pose estimation model trained by a training method suitable for a fruit and vegetable pose estimation model in a scene with missing depth information is used to estimate the fruit and vegetable pose in the orchard images to obtain the fruit and vegetable pose.

[0021] Furthermore, the fruit pose includes the six-dimensional vector, depth value, and two-dimensional offset of the target relative to the reference point. The fruit pose is converted into the translation vector and rotation matrix required for the picking robot to pick the fruit in the following way:

[0022] The target coordinates are obtained by adding the two-dimensional offset of the target relative to the reference point to the coordinates of the reference point. The target coordinates and the target depth value are combined with the camera intrinsic parameters to obtain the translation vector of the target in three-dimensional space by back projection. Two independent three-dimensional rotation vectors are extracted from the six-dimensional vector to construct the rotation matrix.

[0023] According to another aspect of the present invention, a system for estimating the pose of a tree fruit tree in a scene where depth information is missing is provided, comprising:

[0024] A preprocessing module is used to obtain a sample of an orchard's fruit image and a true position of a target in the sample of the orchard's fruit image, wherein the target is a fruit target;

[0025] A training module uses forest fruit image samples to train a forest fruit pose estimation model. The forest fruit pose estimation model includes: a target detection network, a feature enhancement module and a pose prediction head. The target detection network extracts a multi-scale feature map from the forest fruit image sample and generates a target detection frame; the feature enhancement module extracts a global semantic token from the multi-scale feature map, combines the global semantic token with the target feature in the target detection frame, constructs a feature sequence, uses the normalized center coordinates, width and height of the target detection frame for position encoding, and generates a target query. The feature enhancement module includes an encoder and a decoder. The encoder encodes the feature sequence, and the decoder decodes the encoded feature sequence using the target query to obtain the global features of the target; the pose prediction head uses the global features of the target to predict the target pose, and uses the error between the predicted target pose and the true pose of the target as the loss function. The model parameters are updated through back propagation, and the forest fruit pose estimation model is trained until convergence to obtain a trained forest fruit pose estimation model.

[0026] The prediction module uses the trained fruit pose estimation model to estimate the fruit pose of the orchard fruit images to obtain the fruit pose.

[0027] According to another aspect of the present invention, a computer-readable storage medium is provided, which includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute a training method for a Linguo pose estimation model suitable for a scenario where depth information is missing and / or a Linguo pose estimation method suitable for a scenario where depth information is missing.

[0028] According to another aspect of the present invention, a picking robot is provided. The picking robot uses a fruit pose estimation model trained by a training method for a fruit pose estimation model in a scenario where depth information is missing to estimate the pose of fruits in an orchard, obtain the pose of the fruits, and convert the pose of the fruits into a translation vector and a rotation matrix for picking fruits.

[0029] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0030] (1) The target detection network in the present invention extracts multi-scale features and generates detection frame information. Multi-scale features help improve the detection capability of small targets such as fruits and orchards, while the detection frame information is used as a position prior in the subsequent feature enhancement module to further improve the accuracy of target representation. The feature enhancement module combines the global semantic token with the target features in the target detection frame, establishes a semantic association relationship between the target and the global scene, introduces the target query to process the feature sequence, and generates global features with context-aware capabilities. Compared with the method of directly performing pose regression, the present invention uses global features with context-aware capabilities, which can effectively improve the accuracy of pose estimation. The model trained by the present invention can improve the pose estimation accuracy of fruits and orchards in the absence of depth information, and has strong generalization ability, thereby enhancing the robustness and reliability of the picking robot in a complex outdoor orchard environment.

[0031] (2) The present invention includes a rotation regression head and a translation regression head in the pose prediction head. Traditional pose estimation methods usually use quaternions or Euler angles to parameterize the target rotation. However, these methods have problems such as discontinuous representation, singular points, and complex optimization space in practical applications, which can easily lead to convergence difficulties or reduced prediction accuracy during neural network training, especially when the rotation angle is close to the boundary area. To overcome the above problems, the present invention uses a rotation regression head for 6D pose estimation. Compared with traditional technologies, the six-dimensional vector has stronger continuity and a simpler expression form, and has good convergence performance and prediction accuracy in multiple pose estimation tasks. In terms of translation estimation, a translation regression head is used to predict the depth value and offset to further improve the prediction accuracy. The loss function takes into account the depth value loss, displacement loss and 6D pose loss, which can improve the prediction accuracy of the model.

[0032] (3) The present invention fuses the initial rotation estimation vector with the shallower local detail features in the network (the residual vector of the rotation feature), so that the model can further capture the fine-grained changes in the target posture while maintaining the global structural perception capability. This mechanism not only effectively improves the accuracy and robustness of the rotation estimation, but also provides a solid feature foundation for subsequent high-precision posture regression. The present invention adopts a posture prediction head based on an iterative refinement strategy to gradually optimize the prediction of the six-dimensional vector (six-degree-of-freedom posture) of the target. Based on the initial estimate, the posture prediction head iteratively refines and corrects the posture to achieve a more accurate final regression result. Compared with the traditional one-time regression method, the present invention can effectively alleviate the impact of the initial prediction error and significantly improve the accuracy and robustness of the posture estimation.

[0033] (4) The translation regression head introduces two independent estimation branches, each with the same structure as the rotation regression head. The pose initialization module concatenates position information and global features in the channel dimension to fuse spatial structure and semantic information. Both translation and rotation regression use a structure combining standard convolution and depthwise separable convolution for feature extraction. This design allows the model to more accurately capture the spatial position changes of the target and maintain good translation and rotation estimation performance in orchard scenes with weak textures and complex backgrounds.

[0034] (5) The feature enhancement module of the present invention introduces a deformable attention mechanism, which effectively combines the spatial sparse position perception capability of the deformable convolution with the global modeling capability of the feature enhancement module, taking into account the flexibility of feature selection and context perception capability. Specifically, for each query position, the mechanism no longer performs global attention calculations on the entire feature map, but dynamically selects several key positions (deformation points) around the reference point as sampling points to perform local attention operations, thereby significantly reducing the amount of calculations between each query and all keys, and greatly reducing the overall computational complexity. In addition, deformable attention also has good training characteristics: by focusing on semantically related areas in space, it can effectively alleviate the problem of excessive dispersion of attention weights in the feature enhancement module in the early stages of training, and accelerate the convergence of the model. More importantly, the computational complexity of this mechanism is linearly related to the number of sampling points, rather than being directly linked to the spatial dimension of the feature map, thereby breaking through the efficiency bottleneck of the standard attention mechanism in processing high-resolution images and improving the scalability and practicality of the model in visual tasks.

[0035] (6) The model trained by the present invention decouples the translation and rotation parts for modeling when performing pose estimation. For translation estimation, since the convolutional neural network has the property of translation invariance, it is better at extracting the semantic features of local areas, but has a weaker ability to model the absolute spatial position of the target in the input image. This characteristic results in the network having limited accuracy when directly regressing the three-dimensional translation vector. The network's prediction of absolute coordinates is not as effective as that of relative coordinates. Therefore, the present invention does not directly regress the translation vector T(t x , t y , t z ) T Instead of performing end-to-end regression, we employ a decoupled modeling strategy to construct a more rational translation prediction process. The translation component is decoupled using camera intrinsics, while the rotation component is modeled using two 3D vectors. Compared to traditional unified modeling, this decoupling strategy better balances the feature representation of both translation and rotation, preventing mutual interference between the two during the modeling process. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1This is a flow chart of a method for estimating the pose of a tree or fruit tree in a scene where depth information is missing, provided by an embodiment of the present invention;

[0037] Figure 2 is a structural diagram of a pose estimation model provided by an embodiment of the present invention;

[0038] Figure 3 is a structural diagram of a feature enhancement module based on a deformable Transformer provided by an embodiment of the present invention;

[0039] Figure 4 is a structural diagram of a position prior-based deformable attention module provided by an embodiment of the present invention;

[0040] Figure 5 2 is a structural diagram of a posture prediction head provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0042] like Figure 1 As shown, a training method for a Linguo pose estimation model suitable for scenarios where depth information is missing, the Linguo pose estimation model includes: a target detection network, a feature enhancement module and a pose prediction head, and the training method includes:

[0043] Obtaining a fruit orchard image sample and a true position of a target in the fruit orchard image sample, wherein the target is a fruit orchard target;

[0044] The fruit and orchard image samples are used to train the fruit and orchard pose estimation model. The object detection network extracts multi-scale feature maps from the fruit and orchard image samples and generates object detection frames.

[0045] The target detection network can adopt any target detection network model with multi-scale feature fusion capability.

[0046] The feature enhancement module extracts global semantic tokens from multi-scale feature maps, combines the global semantic tokens with the target features within the target detection box, constructs a feature sequence, and uses the normalized center coordinates, width, and height of the target detection box for position encoding to generate a target query. The feature enhancement module includes an encoder and a decoder. The encoder encodes the feature sequence, and the decoder decodes the encoded feature sequence using the target query to obtain the global features of the target.

[0047] The pose prediction head uses the global features of the target to predict the target pose, takes the error between the predicted target pose and the true pose of the target as the loss function, backpropagates to update the model parameters, trains the Lin Guo pose estimation model until convergence, and obtains the trained Lin Guo pose estimation model.

[0048] Example 1

[0049] The structure of the pose estimation model and its training method are described in detail through Example 1.

[0050] like Figure 2 As shown in Figure 1, the pose estimation model includes an object detection network, a feature enhancement module, and a pose prediction head, which are trained as follows:

[0051] Collect the orchard's fruit image samples and the depth values of the fruit in the fruit image samples.

[0052] Input the fruit image samples into the target detection network, extract the multi-scale feature map, and generate the target detection frame;

[0053] like Figure 3 As shown, the feature enhancement module is a feature enhancement module based on the deformable Transformer, including: five encoders and five decoders. Each encoder (Encoder) is sequentially set with a deformable attention module (Deformable Attention), residual connection and layer normalization (Add&Norm), feedforward neural network (FFN), Add&Norm from the input end to the output end. Each decoder (Decoder) is sequentially set with multi-head attention, Add&Norm, Deformable Attention, Add&Norm, FFN, Add&Norm from the input end to the output end.

[0054] Residual connections add the input value to the output value, thereby enhancing the network's performance. Because stacking multiple deep networks can easily lead to vanishing or exploding gradients, residual connections ensure that information flows through the network, preventing these gradients from vanishing or exploding.

[0055] Layer normalization normalizes each input sample so that the mean of each feature is 0 and the variance is 1. This balances the importance of each feature and avoids overfitting caused by overemphasizing certain features in the network. Therefore, layer normalization can improve the generalization ability of the network.

[0056] In the five-layer encoder-decoder structure of the Transformer network, a deformable attention mechanism and the position prior of the detection box information are introduced. First, the global semantic token is extracted from the multi-scale feature map. The global semantic token is combined with the target features in the target detection box to construct a feature sequence. This sequence is then input into the Transformer network to perform semantic modeling between the target and the global scene, and at the same time establish the contextual dependency relationship between multiple targets.

[0057] In the encoding and decoding stages, a deformable attention mechanism is used to perform sparse position sampling of input features. At the same time, the normalized center coordinates, width, and height of the target detection box are used for position encoding to generate the target query as the input of the Transformer decoder.

[0058] The implementation method for generating target query embedding is:

[0059] For each detection box, extract its four parameters (x c ,y c , w, h), where x c ,y c Represents the center coordinates of the detection frame, w and h represent the width and height of the detection frame; the above coordinates and sizes are normalized according to the image size Where W and H are the width and height of the original image; the four normalized values are concatenated into a position vector Finally, the position vector is encoded by MLP (multi-layer perceptron) The encoded vector q pos Used as the target query input in the Transformer decoder, replacing the traditional learnable query embedding.

[0060] Figure 4 The bias network in the deformable attention module includes DWConv (deep variable convolution), GRLU (non-linear activation function) and Conv (convolution). After the target query enters the bias network, deep variable convolution, non-linear activation and convolution operations are performed in sequence to obtain the bias value Θ offset .

[0061] The deformable attention module uses the normalized center point of the target detection box as the reference point. The reference point is combined with the deviation value to obtain the deformation point. After the target query enters the deviation network, multiple deviation values will be dynamically generated. Multiple deviation values are combined with the reference point to obtain multiple deformation points, forming a deformation point set R. The deformation point is used as the sampling point in the multi-scale feature map X. The feature vector of the sampling point is obtained through the query weight matrix W in the multi-head attention. Q , key weight matrix W K Sum value weight matrix W VPerform linear transformation to generate query matrix Q, key matrix K and value matrix V. Calculate the attention weight at the sampling point, the formula is as follows:

[0062]

[0063] Among them, K+Δp represents the position of the key after the offset adjustment of the deviation value Δp. The Softmax function is used to calculate the attention weight so that the sum of all attention weights is 1, d k Indicates the number of columns of the key matrix K. Q and K have the same number of columns.

[0064] The attention weights at all sampling points are weighted and summed with the corresponding value matrix to obtain the output features of the deformable attention module.

[0065] The deformable attention mechanism effectively combines the spatially sparse position perception of deformable convolutions with the global modeling capabilities of the Transformer, balancing flexibility in feature selection with contextual awareness. Specifically, for each query position, the mechanism no longer performs global attention calculations on the entire feature map. Instead, it dynamically selects several key locations around the reference point as sampling points for local attention operations, significantly reducing the amount of computation between each query and all keys, greatly reducing overall computational complexity.

[0066] Deformable attention also has excellent training properties: by spatially focusing on semantically related regions, it effectively alleviates the problem of overly dispersed attention weights in the Transformer during early training, accelerating model convergence. More importantly, the computational complexity of this mechanism is linearly related to the number of sampling points, rather than directly linked to the spatial dimensions of the feature map. This overcomes the efficiency bottleneck of the standard attention mechanism when processing high-resolution images, improving the model's scalability and practicality for visual tasks.

[0067] The present invention designs an iterative refinement strategy to gradually approach the final 6D pose of the target to improve the accuracy and stability of pose estimation. This strategy introduces feature residuals to perform multiple refinement regressions based on the initial prediction, thereby achieving more accurate pose estimation. The pose prediction head includes a rotation regression head and a translation regression head. The structure of the rotation regression head is consistent with the network structure of the translation regression head. Its structure is as follows Figure 5 shown.

[0068] Taking the rotary return head as an example, the following description is provided:

[0069] First, we obtain the global feature representation output by the feature enhancement module based on the deformable Transformer. After that, the model first performs a preliminary rotation pose estimation to obtain the initial rotation feature r of the target pre Specifically, first of all, F i The spatial coordinates (x, y) are normalized and then mapped to the interval [-1, 1] to generate two coordinate feature channels containing spatial position information. Subsequently, the position information is combined with the global feature F output by the feature enhancement module based on the deformable Transformer. i The channel dimension is spliced to fuse the spatial structure and semantic information. The fused features are processed sequentially through a standard convolutional layer (Conv Module) and three depth-wise separable convolutional layers (DWConv) to extract more discriminative posture-related features layer by layer. Group normalization and Swish activation function are introduced after each convolutional layer to enhance the nonlinear expression ability and training stability of the model. The final output is the preliminary rotation feature r pre It will serve as the basic input for the subsequent refinement and prediction process. The calculation method is:

[0070] r pre =DSConv 3 (Conv(C(F i ,coord(F i ))))

[0071] Among them, DSConv(g) represents depth-wise separable convolution, Conv(g) represents standard convolution, C(g) is the concatenation operation, and coord(g) is the spatial coordinate feature extraction operation.

[0072] Then, an additional depth-wise separable convolutional layer is used to compress the number of feature channels to 6, and the initial rotation estimation vector r is obtained. init The above process can be expressed as:

[0073] r init =DSConv(r pre )

[0074] Finally, the initial prediction vector r pre With the initial rotation estimate vector r init The features are concatenated along the channel dimension and input into the Pose Iteration Module (PItM), which consists of two depthwise separable convolutional layers, each followed by group normalization and Swish activation to enhance nonlinear expression and extract higher-order rotation features. Subsequently, the features are compressed through an additional depthwise separable convolutional layer, which outputs a residual vector Δr of the rotation features, which is used to further refine the rotation estimate:

[0075] Δx=DSConv 3 (C(r pre , r init ))

[0076] The residual vector Δr of the rotation feature is combined with the initial rotation estimation vector r init Add together to obtain the final optimized 6D rotation prediction result r 6d :

[0077] r 6d =r init +Δr

[0078] The above process constitutes a complete iterative pose optimization process. By fusing the initial rotation prediction results with the shallower local detail features in the network, the model can further capture the fine-grained changes in the target pose while maintaining the global structural perception ability. This mechanism not only effectively improves the accuracy and robustness of rotation estimation, but also provides a solid feature foundation for subsequent high-precision pose regression.

[0079] In terms of translation estimation, in order to further improve the prediction accuracy, the present invention does not simply use the reference point coordinates as the target centroid coordinates, but adopts a more flexible modeling strategy. Specifically, the translation regression head introduces two independent estimation branches to predict the two-dimensional pixel offset (Δx, Δy) and depth value t of the target in the image respectively. z , to achieve decoupled modeling of 3D translation vectors. Each branch structure is consistent with the rotation regression head, employing a combination of standard convolution and depthwise separable convolution, along with group normalization and activation functions for feature extraction. This design enables the model to more accurately capture the spatial position changes of the target, maintaining excellent translation estimation performance in orchard scenes with weak textures and complex backgrounds.

[0080] Traditional pose estimation methods typically use quaternions or Euler angles to parameterize the target rotation. However, these methods suffer from problems in practical applications, such as representation discontinuities, singularities, and complex optimization spaces. These problems can lead to convergence difficulties or reduced prediction accuracy during neural network training, especially when the rotation angle approaches boundary regions.

[0081] To overcome these issues, this paper introduces a high-dimensional vector-based six-dimensional rotation representation (6DRotationRepresentation). Compared to traditional techniques, the six-dimensional representation offers greater continuity and a simpler expression, and has been demonstrated to have good convergence performance and prediction accuracy in multiple pose estimation tasks.

[0082] In the rotation representation adopted by the present invention, the rotation consists of two three-dimensional vectors, which correspond to the two orthogonal directions (i.e., column vectors) of the first three columns in the rotation matrix, thereby implicitly constructing the orthogonal structure of the rotation matrix. Specifically, the model directly obtains a six-dimensional vector r by rotating the regression head. 6d (r1, r2, r3, r4, r5, r6), extract two independent three-dimensional vectors v1 (r1, r2, r3) and v2 (r4, r5, r6) from it, and construct the standard basis e1, e2, e3 based on v1 and v2. The specific formula is:

[0083] e1=Φ(v1)

[0084] e2=Φ(v2)

[0085] e3=e1×e2

[0086] Where r1, r2, and r3 represent the rotation angles of the XYZ axis, r4, r5, and r6 represent the redundant rotation angles of the XYZ axis, Φ(·) represents vector normalization, and × represents vector cross product.

[0087] This set of orthogonal bases is used to further construct the rotation matrix R:

[0088] R=[e1 T , e2 T , e3 T ]

[0089] In order to more accurately quantify the angular difference between the predicted rotation and the true rotation, the geodesic loss is used as the optimization objective in the rotation estimation task. Its mathematical expression is:

[0090]

[0091] where tr(·) represents the trace of the matrix, represents the true rotation matrix, R pred Represents the predicted rotation matrix.

[0092] For translation estimation, since convolutional neural networks are translation invariant, they are better at extracting semantic features of local areas, but are weak in modeling the absolute spatial position of the target in the input image. This characteristic leads to limited accuracy when the network directly regresses the three-dimensional translation vector. Therefore, the present invention does not directly regress the translation vector T(t x , t y , t z ) TInstead of performing end-to-end regression, we adopt a decoupled modeling strategy to construct a more reasonable translation prediction process. The network's prediction of absolute coordinates is not as effective as relative coordinates. Specifically, considering that there is a clear perspective projection relationship between the target's 2D projection position in the image (the image center point) and its actual 3D spatial position, we can model it based on the pinhole camera model. This relationship can be expressed as:

[0093] t z (p x , p y , 1) T =K(t x , t y ,t z ) T

[0094] Among them, K is the camera internal parameter matrix, (p x , p y ) is the projection coordinate of the target 3D center point on the image plane.

[0095] Based on the above relationship, the three-dimensional translation T is first decoupled into the two-dimensional projection coordinates of the target on the image (p x , p y ) and its corresponding depth t z In order to improve the network's ability to model the target position, the two-dimensional pixel coordinates (p x , p y ) is decomposed into reference point coordinates (c x , c y ) and the offset (Δx, Δy) relative to the reference point. Therefore, the translation branch of the network only needs to predict the two-dimensional offset (Δx, Δy) of the target relative to the reference point and the depth value t of the target z Finally, by predicting the two-dimensional offset (Δx, Δy) and the target depth t z , the three-dimensional translation vector of the target can be obtained, and the translation vector T of the target in three-dimensional space can be restored by back-projection in combination with the camera intrinsic parameters, that is:

[0096]

[0097] t z =t z

[0098] Among them, f x , f y , p x , p y It can be obtained through the camera's internal parameters.

[0099] In order to effectively supervise the translation prediction, the L2 norm is used as the loss function in the two-dimensional projection coordinate prediction stage to measure the error between the predicted coordinates and the true projection coordinates. Its mathematical expression is as follows:

[0100]

[0101] Among them, (t x , t y ) pred and (t x , t y ) gt Represent the predicted and real coordinates respectively, where the real coordinates refer to the coordinates of the target in the image.

[0102] In depth prediction, considering that there may be a certain degree of occlusion, illumination changes, or noise interference in the scene, directly using L2 loss may be too sensitive to outliers, thus affecting the overall training stability. Therefore, the depth loss function uses L1 absolute error to enhance the robustness of the model in the depth estimation task. Its mathematical expression is as follows:

[0103]

[0104] Among them, t zpred and t zgt Represent the predicted and true target depth values respectively.

[0105] Compared to traditional unified modeling approaches, the proposed decoupling strategy better accounts for both translation and rotational feature representation, preventing mutual interference during the modeling process. Specifically, rotation estimation relies more on the target's appearance structure and local morphological variations, such as edge contours and surface texture, and possesses strong local discriminability; whereas translation estimation focuses on the target's overall spatial position within the image, including its relative layout to the environment and scale characteristics.

[0106] Example 2

[0107] The manner in which the picking robot uses the present invention is described in detail through Example 2.

[0108] When the picking robot uses the trained model of the present invention to pick fruits, the real-time collected orchard and fruit images are input into the trained pose estimation model to predict the two-dimensional pixel offset and depth value, as well as the six-dimensional vector of the fruit target in the image.

[0109] The 2D pixel offset and depth of the fruit object in the image are combined with the camera's intrinsic parameters to recover the object's translation vector in 3D space through back-projection. Two independent 3D vectors are extracted from the 6D vector to construct a rotation matrix. The harvesting robot then uses this translation vector and rotation matrix to pick the fruit.

[0110] The fruits in the orchard include: tangerines, oranges, winter dates, apples, cherries, pomegranates and persimmons.

[0111] The model trained by the present invention can improve the accuracy of pose estimation of fruits in orchards in the absence of depth information, and has strong generalization ability, thereby enhancing the robustness and reliability of the picking robot in complex outdoor orchard environments.

[0112] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A training method for a pose estimation model for forest fruit trees in a scene with missing depth information, characterized in that: The forest fruit pose estimation model includes: a target detection network, a feature enhancement module and a pose prediction head, and the training method includes: Obtaining a fruit orchard image sample and a true position of a target in the fruit orchard image sample, wherein the target is a fruit orchard target; The fruit and orchard image samples are used to train the fruit and orchard pose estimation model. The object detection network extracts multi-scale feature maps from the fruit and orchard image samples and generates object detection frames. The feature enhancement module extracts global semantic tokens from multi-scale feature maps, combines the global semantic tokens with the target features within the target detection box, constructs a feature sequence, and uses the normalized center coordinates, width, and height of the target detection box for position encoding to generate a target query. The feature enhancement module includes an encoder and a decoder. The encoder encodes the feature sequence, and the decoder decodes the encoded feature sequence using the target query to obtain the global features of the target. The pose prediction head uses the global features of the target to predict the target pose, takes the error between the predicted target pose and the true pose of the target as the loss function, backpropagates to update the model parameters, trains the Lin Guo pose estimation model until convergence, and obtains the trained Lin Guo pose estimation model.

2. The training method of a pose estimation model for a forest fruit tree in a scene with missing depth information according to claim 1, characterized in that: The pose prediction head includes a rotation regression head and a translation regression head. The rotation regression head uses the global features of the target to predict the six-dimensional vector of the target. The translation regression head uses the global features of the target to predict the target depth value and the two-dimensional offset of the target relative to the reference point. The reference point is the center point of the target detection frame after normalization. The real position of the target in the forest fruit image sample includes the real coordinates, real depth value and real six-dimensional vector of the target in the forest fruit image sample; The sum of the predicted two-dimensional offset of the target relative to the reference point and the coordinates of the reference point is used as the predicted target coordinates. Two independent three-dimensional rotation vectors are extracted from the predicted six-dimensional vector of the target to construct a predicted rotation matrix. Two independent three-dimensional rotation vectors are extracted from the real six-dimensional vector to construct a real rotation matrix. The error between the predicted rotation matrix and the real rotation matrix, the error between the predicted target coordinates and the real coordinates of the target, and the error between the predicted target depth value and the real depth value of the target are fused to form a loss function.

3. The training method of a pose estimation model for a forest fruit in a scene with missing depth information according to claim 2, characterized in that: The depth value prediction branch of the rotation regression head and the translation regression head has the same structure as the offset prediction branch of the translation regression head. The rotation regression head includes: a pose initialization module and a pose iterative optimization module. There is a depth-separable convolution layer between the pose initialization module and the pose iterative optimization module. The pose initialization module obtains the initial rotation feature of the target from the global feature of the target. Then, a depth-wise separable convolution layer is used to compress the channel dimension of the initial rotation feature to obtain the initial rotation estimation vector. The initial rotation feature and the initial rotation estimation vector are concatenated in the channel dimension and input into the pose iterative optimization module. The residual vector of the rotation feature is output and added to the initial rotation estimation vector to obtain the predicted six-dimensional vector of the target.

4. The training method of a pose estimation model for a forest fruit in a scene with missing depth information according to claim 3, characterized in that: The pose initialization module normalizes the spatial coordinates in the global features of the target and maps them to the interval [-1, 1], generating two coordinate feature channels containing spatial position information, and splicing the spatial position information and the global features in the two coordinate feature channel dimensions. The spliced features are processed in turn by a standard convolutional layer and a depth-wise separable convolutional layer to obtain the initial rotation features of the target.

5. A training method for a pose estimation model for a fruit tree suitable for a scene with missing depth information according to any one of claims 1 to 4, characterized in that: The encoder and decoder of the feature enhancement module both have deformable attention modules. After the target query enters the bias network in the deformable attention module, it performs deep variable convolution, nonlinear activation and convolution operations in sequence to generate multiple bias values. The deformable attention module combines the reference point with multiple bias values to obtain multiple deformation points, and uses the deformation points as sampling points in the multi-scale feature map. The feature vectors of the sampling points are linearly transformed through the query weight matrix, key weight matrix and value weight matrix in the multi-head attention to generate the query matrix, key matrix and value matrix. The position of the key in the key matrix is adjusted by the offset of the bias value and then combined with the query matrix to calculate the attention weight at the sampling point. The attention weights at all sampling points are weighted and summed with the corresponding value matrix to obtain the output features of the deformable attention module.

6. A method for estimating the pose of fruit trees in a scene with missing depth information, characterized in that: include: Collect images of fruits and trees in an orchard, and use a fruit and tree pose estimation model trained by a training method for a fruit and tree pose estimation model suitable for a scene with missing depth information as described in any one of claims 1-5 to estimate the fruit and tree pose of the images of the orchard to obtain the fruit and tree pose.

7. The method for estimating the pose of a fruit tree in a scene with missing depth information according to claim 6, wherein: The fruit pose includes the six-dimensional vector, depth value, and two-dimensional offset of the target relative to the reference point. The fruit pose is converted into the translation vector and rotation matrix required for the picking robot to pick the fruit in the following way: The target coordinates are obtained by adding the two-dimensional offset of the target relative to the reference point to the coordinates of the reference point. The target coordinates and the target depth value are combined with the camera intrinsic parameters to obtain the translation vector of the target in three-dimensional space by back projection. Two independent three-dimensional rotation vectors are extracted from the six-dimensional vector to construct the rotation matrix.

8. A system for estimating the pose of fruit trees in scenes with missing depth information, characterized in that: include: A preprocessing module is used to obtain a sample of an orchard's fruit image and a true position of a target in the sample of the orchard's fruit image, wherein the target is a fruit target; A training module uses forest fruit image samples to train a forest fruit pose estimation model. The forest fruit pose estimation model includes: a target detection network, a feature enhancement module and a pose prediction head. The target detection network extracts a multi-scale feature map from the forest fruit image sample and generates a target detection frame; the feature enhancement module extracts a global semantic token from the multi-scale feature map, combines the global semantic token with the target feature in the target detection frame, constructs a feature sequence, uses the normalized center coordinates, width and height of the target detection frame for position encoding, and generates a target query. The feature enhancement module includes an encoder and a decoder. The encoder encodes the feature sequence, and the decoder decodes the encoded feature sequence using the target query to obtain the global features of the target; the pose prediction head uses the global features of the target to predict the target pose, and uses the error between the predicted target pose and the true pose of the target as the loss function. The model parameters are updated through back propagation, and the forest fruit pose estimation model is trained until convergence to obtain a trained forest fruit pose estimation model. The prediction module uses the trained fruit pose estimation model to estimate the fruit pose of the orchard fruit images to obtain the fruit pose.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed by the processor, the device where the storage medium is located is controlled to execute a training method for a Linguo pose estimation model suitable for a scenario where depth information is missing as described in any one of claims 1-5 and / or a Linguo pose estimation method suitable for a scenario where depth information is missing as described in claim 6 or 7.

10. A picking robot, characterized in that: The picking robot uses a fruit pose estimation model trained using a training method for a fruit pose estimation model suitable for a scene with missing depth information as described in any one of claims 1-5 to estimate the pose of fruits in an orchard, obtain the pose of fruits, and convert the pose of fruits into a translation vector and a rotation matrix for picking fruits.

Images