An object pose determination method and apparatus, an electronic device, and a medium

Abstract

The application discloses a pose determination method and device of an object, an electronic device and a medium, and is applied to a pose determination model. A to-be-processed image of an object is acquired, and target semantic features of the to-be-processed image are determined. Initial structured regression parameters of a three-dimensional bounding box of the object on the to-be-processed image are determined according to the target semantic features, including projection scale information, a center point projection coordinate and a normalized offset parameter of an angle point projection coordinate relative to the center point projection coordinate. Center point residual errors, angle point residual errors and scale residual errors of the initial structured regression parameters are determined according to the initial structured regression parameters and the target semantic features, and the initial structured regression parameters are adjusted by using the center point residual errors, the angle point residual errors and the scale residual errors to obtain target structured regression parameters. Two-dimensional Gaussian heat maps of the center points and the angle points are generated according to the to-be-processed image, and pose information of the object is determined based on the target structured regression parameters and the two-dimensional Gaussian heat maps, so that the object pose determination accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of pose determination, and more particularly to a method, apparatus, electronic device, and medium for determining the pose of an object. Background Technology

[0002] In related technologies, the core task of six-degree-of-freedom pose estimation of a target object is to determine the position and orientation of the target object in three-dimensional space based on the images acquired by the sensor.

[0003] In related technologies, target object pose estimation methods typically involve acquiring an image of the target object, extracting image features, and then directly performing absolute two-dimensional coordinate regression on multiple key points of the target object to obtain the projected two-dimensional coordinates of the key points on the image. Then, based on these projected two-dimensional coordinates, the target object's pose is estimated. However, because the regression of absolute two-dimensional coordinates in this method has a high degree of freedom, the obtained projected two-dimensional coordinates of the key points on the image are inaccurate, leading to inaccurate target object pose estimation. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and medium for determining the pose of an object, in order to solve the problem that in the pose estimation methods of target objects in the related art, the regression degree of the absolute two-dimensional coordinates is high, which leads to inaccurate projection two-dimensional coordinates of the key points of the target object on the image, and thus inaccurate pose estimation of the target object.

[0005] To address the aforementioned technical problems, embodiments of this application provide a method for determining the pose of an object, applied to a pose determination model. The method for determining the pose of an object includes: Acquire the image of the object to be processed, and determine the target semantic features of the image to be processed based on the image to be processed; Based on the target semantic features, the initial structured regression parameters of the preset three-dimensional bounding box outside the object on the image to be processed are determined; the initial structured regression parameters include the projection scale information of the three-dimensional bounding box on the image to be processed, the center point projection coordinates, and the normalized offset parameters of the corner point projection coordinates relative to the center point projection coordinates. Based on the initial structured regression parameters and the target semantic features, the center point residual, corner point residual, and scale residual of the initial structured regression parameters are determined, and the center point projection coordinates, the normalized offset parameter, and the projection scale information are adjusted using the center point residual, the corner point residual, and the scale residual to obtain the target structured regression parameters of the object. Based on the image to be processed, a two-dimensional Gaussian heatmap of the center point and corner points of the three-dimensional bounding box is generated, and the pose information of the object is determined based on the target structured regression parameters and the two-dimensional Gaussian heatmap.

[0006] In this embodiment, by acquiring the image to be processed and determining its target semantic features, a feature foundation with global context awareness is provided for subsequent regression of structured parameters. Based on this, initial structured regression parameters (including projection scale information, center point projection coordinates, and normalized offset parameters) are first predicted according to the target semantic features. Then, the initial structured regression parameters are adjusted using center point residuals, corner point residuals, and scale residuals, realizing a two-stage regression mechanism from coarse to fine adjustment of the structured regression parameters. The coarse adjustment stage uses a structured expression of "center projection coordinates—projection scale parameters—normalized offset parameters." The fine adjustment stage refines the coarse adjustment results through residual correction, obtaining target structured regression parameters including projection scale information, center point projection coordinates, and normalized offset parameters. This reduces the difficulty required for the pose determination model to directly regress the absolute coordinates of all key points and improves the accuracy of the coordinates in the target structured regression parameters. Finally, through the synergistic effect of the target structured regression parameters and heatmap localization, the accuracy of object pose determination can be improved.

[0007] Optionally, determining the pose information of the object based on the target structured regression parameters and the two-dimensional Gaussian heatmap includes: The center point and corner points of the 3D bounding box are used as key points for determining the pose of the object, and the key point projection coordinates on the image to be processed are determined based on the target structured regression parameters. The pose information of the object is determined based on the projected coordinates of the key points and the two-dimensional Gaussian heatmap.

[0008] In this embodiment, the center point and corner points of the 3D bounding box are used as key points. The key point projection coordinates on the image to be processed are determined based on the target structured regression parameters. Then, pose information is determined based on the key point projection coordinates and the 2D Gaussian heatmap, achieving the synergistic utilization of structured regression results and heatmap localization results. Since the key point projection coordinates originate from structured regression that integrates scale information and geometric constraints, they naturally possess overall geometric consistency; while the 2D Gaussian heatmap provides pixel-level probability distribution information within the key point's neighborhood. Combining these two methods can improve the accuracy of local localization while maintaining the overall rationality of the key point layout, thus providing a more reliable 2D key point information foundation for determining the pose of the target object.

[0009] Optionally, the two-dimensional Gaussian heatmap of any of the key points includes probability information that at least one pixel around the key point's projected coordinates is the target projected coordinate of the key point; the step of determining the pose information of the object based on the key point's projected coordinates and the two-dimensional Gaussian heatmap includes: For any of the key points, based on the probability information of at least one pixel around the projected coordinates of the key point, the normalized probability information of any pixel around the projected coordinates of the key point is determined. Based on the normalized probability information, the continuous heatmap coordinates of the key points are determined; According to a preset coordinate fusion method, the projected coordinates of the key point and the continuous heat map coordinates of the key point are fused to obtain the target projected coordinates of the key point. Based on the target projection coordinates of the key points, the pose information of the object is determined.

[0010] In this embodiment, for any key point, normalized probability information is determined based on the probability information of at least one pixel around the key point's projected coordinates. This normalized probability information is then used to determine the continuous coordinates of the heatmap. The normalized probability information converts the original heatmap response into a standardized probability distribution, eliminating interference from differences in response amplitude across different regions and making the extraction of continuous heatmap coordinates more accurate and reliable. The key point's projected coordinates and the continuous heatmap coordinates are fused according to a preset coordinate fusion method, achieving a complementarity between the macroscopic geometric constraints of the regressed key points and the local precise positioning of the key points in the heatmap: the key point's projected coordinates provide an overall layout conforming to the target's geometric structure, while the continuous heatmap coordinates provide pixel-level or even sub-pixel-level fine positional information. Through this fusion mechanism, target projected coordinates with both geometric consistency and local precision are obtained, thereby improving the accuracy and reliability of pose determination.

[0011] Optionally, the pose determination model includes a pose solving module; determining the pose information of the object based on the target projection coordinates of the key points includes: Obtain the three-dimensional coordinates of the key point in a preset three-dimensional coordinate system, and determine the correspondence information between the three-dimensional coordinates of the key point and the target projection coordinates based on the three-dimensional coordinates and the target projection coordinates; The corresponding relationship information is input into the pose solving module to obtain the pose information of the object.

[0012] In this embodiment, by obtaining the three-dimensional coordinates of key points in a preset three-dimensional coordinate system, establishing a correspondence between the three-dimensional coordinates and the target projection coordinates, and then inputting this correspondence information into the pose solving module, a geometric mapping from two-dimensional key point prediction to three-dimensional pose recovery is achieved. The pose solving module uses this correspondence to perform geometric solving, and the resulting pose information has high numerical stability and interpretability, thereby ensuring the reliability and accuracy of the pose determination method in real-world application scenarios.

[0013] Optionally, determining the target semantic features of the image to be processed based on the image to be processed includes: Extract the initial spatial detail features and initial semantic discriminant features of the image to be processed, and perform convolution processing on the initial spatial detail features and the initial semantic discriminant features according to a preset convolution strategy, so that the channel dimensions of the initial spatial detail features and the initial semantic discriminant features are aligned. Based on the initial spatial detail features and the initial semantic discrimination features aligned to the channel dimension, the attention weight of at least one region on the image to be processed is determined; The initial spatial detail features and the initial semantic discrimination features are processed using the attention weights respectively to obtain the spatial detail features to be processed and the semantic discrimination features to be processed. The spatial detail features and semantic discrimination features to be processed are processed using a preset spatial alignment strategy to make the spatial resolution of the spatial detail features and semantic discrimination features to be processed the same, thereby obtaining target spatial detail features and target semantic discrimination features; The target spatial detail features and the target semantic discrimination features are fused to obtain the fused features of the image to be processed, and the target semantic features of the image to be processed are determined based on the fused features and the preset target query vector.

[0014] In this embodiment, by extracting initial spatial detail features and initial semantic discriminative features from the image to be processed, and aligning the channel dimensions according to a preset convolution strategy, the problem of inconsistent feature channel numbers at different levels and the inability to directly fuse them is solved. Based on this, attention weights are determined based on the channel-aligned features, and these attention weights are used to enhance the two feature paths respectively, enabling the model to adaptively suppress redundant background region responses and highlight the feature expressions of target-related regions. Subsequently, a preset spatial alignment strategy is used to make the spatial resolution of the spatial detail features and the semantic discriminative features to be processed the same, and the two are fused to obtain a fused feature that simultaneously possesses mid-level detail information and high-level semantic discriminative information. This fused feature, while preserving target edges, contours, and local texture information, also has the semantic discriminative ability to distinguish target objects from background regions, providing a more complete input foundation for subsequent semantic feature extraction based on target query vectors.

[0015] Optionally, the pose determination model includes an encoder and a decoder; determining the target semantic features of the image to be processed based on the fused features and a preset target query vector includes: The encoder is used to model the feature relationship between the fusion features of at least two regions of the image to be processed, so as to obtain the global memory features of the image to be processed. Using the decoder, the global memory features are queried according to the target query vector to obtain the target semantic features of the image to be processed.

[0016] In this embodiment, the pose determination model includes an encoder and a decoder. After obtaining the fused features, the encoder first uses a global model to model the feature relationships between the fused features of at least two regions in the image to be processed, obtaining global memory features. This enables the encoder to use a multi-head self-attention mechanism to ensure that features at each spatial location can perceive information from all other locations in the entire image, thereby establishing long-distance dependencies and capturing complex contextual information. Based on this, the decoder queries the global memory features according to a preset target query vector, and extracts the high-level semantic information most relevant to the geometric structure of the target object from the global memory through a cross-attention mechanism, obtaining the target semantic features. This process uses the feature interaction results of different regions as the basis for the query, allowing the decoder to focus specifically on the target object itself, reducing the interference of background information on subsequent structured regression, and providing a more semantically focused and geometrically rich feature representation for the pose determination of the target object.

[0017] Optionally, generating a two-dimensional Gaussian heatmap of the center point and corner points of the three-dimensional bounding box based on the image to be processed includes: Based on the global memory features or the fusion features, a two-dimensional Gaussian heatmap of the center point and the corner points is generated.

[0018] In this embodiment, the generation of the 2D Gaussian heatmap is based on global memory features or fused features, rather than relying on single-scale convolutional features. Global memory features contain long-range dependencies and contextual information modeled globally by the encoder, while fused features contain mid-level details and high-level semantic information fused across multiple scales; both possess rich target perception capabilities. The Gaussian heatmap generated based on these features can form a more accurate and smooth probability distribution response in the neighborhood of key points, providing high-quality input for subsequent normalized probability information calculation and continuous coordinate extraction from the heatmap. Furthermore, since heatmap generation can share intermediate computational results with the encoder or fused features, eliminating the need for repeated forward propagation, it helps reduce overall computational overhead and improves the inference efficiency and engineering practicality of the pose determination method.

[0019] This application also discloses a pose determination device for an object, applied to a pose determination model, the pose determination device for the object comprising: The target semantic feature determination module is used to acquire the image of the object to be processed, and determine the target semantic features of the image to be processed based on the image to be processed; The initial structured regression parameter determination module is used to determine the initial structured regression parameters of the preset three-dimensional bounding box outside the object on the image to be processed based on the target semantic features; the initial structured regression parameters include the projection scale information of the three-dimensional bounding box on the image to be processed, the center point projection coordinates, and the normalized offset parameters of the corner point projection coordinates relative to the center point projection coordinates. The target structured regression parameter acquisition module is used to determine the center point residual, corner point residual, and scale residual of the initial structured regression parameters based on the initial structured regression parameters and the target semantic features, and to adjust the center point projection coordinates, the normalized offset parameter, and the projection scale information using the center point residual, the corner point residual, and the scale residual to obtain the target structured regression parameters of the object. The pose information determination module is used to generate a two-dimensional Gaussian heatmap of the center point and corner points of the three-dimensional bounding box based on the image to be processed, and to determine the pose information of the object based on the target structured regression parameters and the two-dimensional Gaussian heatmap.

[0020] Optionally, the pose information determination module includes: The key point projection coordinate determination submodule is used to use the center point and corner points of the 3D bounding box as key points for determining the pose of the object, and to determine the key point projection coordinates of the key points on the image to be processed based on the target structured regression parameters. The pose information determination submodule is used to determine the pose information of the object based on the key point projection coordinates and the two-dimensional Gaussian heatmap.

[0021] Optionally, the two-dimensional Gaussian heatmap of any of the key points includes probability information that at least one pixel around the key point's projected coordinates is the target projected coordinate of the key point; the pose information determination submodule includes; The normalized probability information determination unit is used to determine the normalized probability information of any pixel around the projected coordinates of the key point, based on the probability information of at least one pixel around the projected coordinates of the key point. The heatmap continuous coordinate determination unit is used to determine the heatmap continuous coordinates of the key points based on the normalized probability information. The target projection coordinate obtaining unit is used to fuse the key point projection coordinates and the key point heatmap continuous coordinates according to a preset coordinate fusion method to obtain the target projection coordinates of the key point. The pose information determination unit is used to determine the pose information of the object based on the target projection coordinates of the key points.

[0022] Optionally, the pose determination model includes a pose solving module; the pose information determination unit includes: The correspondence information determination subunit is used to obtain the three-dimensional coordinates of the key point in a preset three-dimensional coordinate system, and determine the correspondence information between the three-dimensional coordinates of the key point and the target projection coordinates based on the three-dimensional coordinates and the target projection coordinates; The pose information acquisition sub-unit is used to input the correspondence information into the pose solving module to obtain the pose information of the object.

[0023] Optionally, the target semantic feature determination module includes: The feature extraction submodule is used to extract the initial spatial detail features and initial semantic discriminant features of the image to be processed, and to perform convolution processing on the initial spatial detail features and the initial semantic discriminant features according to a preset convolution strategy, so that the channel dimensions of the initial spatial detail features and the initial semantic discriminant features are aligned. The attention weight determination submodule is used to determine the attention weight of at least one region on the image to be processed based on the initial spatial detail features and the initial semantic discrimination features aligned to the channel dimension. The submodule for obtaining semantic discriminative features to be processed is used to process the initial spatial detail features and the initial semantic discriminative features using the attention weights respectively, to obtain spatial detail features to be processed and semantic discriminative features to be processed; The target semantic discrimination feature acquisition submodule is used to process the spatial detail features to be processed and the semantic discrimination features to be processed using a preset spatial alignment strategy, so that the spatial resolution of the spatial detail features to be processed and the semantic discrimination features to be processed are the same, thereby obtaining the target spatial detail features and the target semantic discrimination features; The target semantic feature determination submodule is used to fuse the target spatial detail features and the target semantic discrimination features to obtain the fused features of the image to be processed, and to determine the target semantic features of the image to be processed based on the fused features and the preset target query vector.

[0024] Optionally, the pose determination model includes an encoder and a decoder; the target semantic feature determination submodule includes: A global memory feature acquisition unit is used to model the feature relationship between the fusion features of at least two regions of the image to be processed using the encoder, so as to obtain the global memory features of the image to be processed. The target semantic feature obtaining unit is used to use the decoder to query the global memory features according to the target query vector to obtain the target semantic features of the image to be processed.

[0025] Optionally, the pose information determination module includes: The two-dimensional Gaussian heatmap generation submodule is used to generate two-dimensional Gaussian heatmaps of the center point and the corner points based on the global memory features or the fusion features.

[0026] This application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method described in this application.

[0027] This application also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in this application. Attached Figure Description

[0028] Figure 1 This is a flowchart of the steps of a method for determining the pose of an object provided in an embodiment of this application; Figure 2 This is a schematic diagram of a pose determination model provided in an embodiment of this application; Figure 3This is a schematic diagram of the application process of a detection converter provided in the embodiments of this application; Figure 4 This is a schematic diagram illustrating the process of determining the target projection coordinates of a key point according to an embodiment of this application; Figure 5 This is a schematic diagram of the 6D pose recovery process of a target object based on two-dimensional keypoint prediction and PnP solution provided in the embodiments of this application. Figure 6 This is a structural block diagram of an object pose determination device provided in the embodiments of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0030] It should be understood that the step numbers used in the text are for ease of description only and are not intended to limit the order in which the steps are performed.

[0031] To facilitate understanding of the technical solutions and effects of the embodiments of this application, the relevant technologies of this application will be briefly described below.

[0032] In related technologies, six-DOF pose estimation of target objects is a key foundational technology in fields such as robot grasping, industrial assembly, intelligent inspection, augmented reality, and vision-guided control. Its core task is to determine the position and orientation of a target object in three-dimensional space based on images or point cloud information acquired by sensors. Unlike ordinary target detection tasks, pose estimation not only needs to identify the target category and location but also needs to recover the rotation and translation parameters of the object relative to the camera coordinate system. Therefore, it places higher demands on spatial perception accuracy, geometric modeling capabilities, and robustness in complex scenes.

[0033] In related technologies, methods for estimating the pose of target objects typically include point cloud-based methods, RGB-D (Red-Green-Blue – Depth)-based methods, monocular RGB (Red-Green-Blue) image-based methods, and DETR (Detection Transformer) network-based methods. However, these methods suffer from at least the following problems: (1) Point cloud or RGB-D based methods can directly establish strong geometric constraints with the help of the depth information of the target object. Under the condition that the target structure is complete and the depth information is of high quality, they have good positioning accuracy. However, such methods often rely on depth sensors, point cloud registration or multimodal fusion, which have problems such as high hardware cost, system complexity and insufficient deployment flexibility.

[0034] (2) The method based on monocular RGB images can complete target perception with the help of ordinary industrial cameras, which has the advantages of low cost, convenient deployment and wide applicability. However, since monocular imaging naturally lacks direct depth information, when the target object is in a complex background, partially occluded, with changes in lighting or surface reflection, the image features are prone to degradation, which causes the two-dimensional key point positioning of the target object's key points projected on the image to drift, ultimately affecting the stability and accuracy of the PnP (Perspective-n-Point) pose solution results.

[0035] (3) The DETR-based method introduces DETR-based networks into the target object pose estimation task. It can extract image features through the encoding and decoding structure of the convolutional backbone network and the Transformer, and then directly regress the two-dimensional coordinates or pose parameters of key points such as the corner points of the three-dimensional bounding box of the target object in the image. Combined with the PnP algorithm, it recovers its 6D (six-dimensional) pose. This method can improve the perception of the target object to a certain extent by utilizing the global context of the image through the self-attention mechanism of the Transformer. However, in scenarios where the image has complex backgrounds, occlusions, weak textures and reflections, it is prone to problems such as insufficient utilization of multi-scale features, unstable local localization of key points and decreased pose solution accuracy.

[0036] Pose estimation methods based on DETR-like networks can extract high-level semantic features, mid-level texture edge information, and local spatial details from target object images. However, these methods typically focus more on high-level semantic feature modeling, while underutilizing mid-level texture edge information and local spatial details. This leads to problems such as incomplete target region response and increased keypoint prediction errors when the target object image has blurred contours, sparse texture, or large scale variations.

[0037] In addition, the aforementioned keypoint-based pose estimation method directly regresses the absolute two-dimensional coordinates of multiple keypoints of the target object to determine its six-degree-of-freedom pose, which offers a high degree of freedom. However, during network model training, pose estimation performance is easily affected by sample distribution, target scale variations, and local occlusion, resulting in slow convergence, unstable optimization, and insufficient geometric consistency. This leads to a lack of effective structural constraints between keypoints. Insufficient geometric consistency refers to the fact that the relative positional relationships between multiple keypoints predicted by the model do not conform to the inherent rigid geometric structure of the target object itself.

[0038] The methods described above rely solely on the regression branch to output keypoint coordinates, which typically fails to fully utilize pixel-level spatial distribution information within the keypoint's neighborhood in the image, resulting in limited representation capabilities for local peak locations and edge responses. Specifically, the ability to represent local peak locations refers to the network's ability to accurately pinpoint the pixel location of the keypoint using the probability response map; the ability to represent edge responses refers to the network's ability to capture the linear response band formed by edge gradients around the keypoint and utilize this structural information to aid in localization. When abnormal regions such as noise interference, blurred edges, or partial occlusion exist in the image, the regression results for keypoint coordinates are easily influenced by these abnormal regions, leading to inaccurate predictions.

[0039] In addition, while such methods can improve pose detection accuracy by introducing depth information, multi-stage iterative optimization, or large-scale models, they also bring problems such as strong sensor dependence, high computational complexity, and increased inference latency, making it difficult to meet the comprehensive requirements of cost, speed, stability, and deployability in industrial scenarios.

[0040] Reference Figure 1 The diagram illustrates a flowchart of a method for determining the pose of an object according to an embodiment of this application. Applied to a pose determination model, the method may specifically include the following steps: Step 101: Obtain the image of the object to be processed, and determine the target semantic features of the image to be processed based on the image to be processed; In this embodiment, the object can be a robot, etc. A single image of the object to be processed can be captured by a camera, and the image can be input into a pose determination model to obtain the rotation matrix R and translation vector T of the object relative to the camera coordinate system, thereby recovering the six-degree-of-freedom pose of the object. The pose determination model can be an improved DETR main network.

[0041] Reference Figure 2This diagram illustrates a pose determination model provided in an embodiment of this application. The image to be processed is an RGB image of an object. The pose determination model includes a Detection Transformer and a pose prediction module. The Detection Transformer includes a ResNet-34 (34-layer residual network), a multi-scale fusion module, a position encoding module, a Transformer Encoder, an object query (i.e., a preset target query vector), and a Transformer Decoder. The pose prediction module includes a heatmap branch, a coarse-tuning branch, and a fine-tuning branch. The pose prediction module outputs the pose estimation result of the object.

[0042] In the embodiments of this application, the Detection Transformer can be used to determine the target semantic features of the image to be processed based on the image to be processed.

[0043] In some embodiments of this application, determining the target semantic features of the image to be processed based on the image to be processed includes: Extract the initial spatial detail features and initial semantic discriminant features of the image to be processed, and perform convolution processing on the initial spatial detail features and the initial semantic discriminant features according to a preset convolution strategy, so that the channel dimensions of the initial spatial detail features and the initial semantic discriminant features are aligned. Based on the initial spatial detail features and the initial semantic discrimination features aligned to the channel dimension, the attention weight of at least one region on the image to be processed is determined; The initial spatial detail features and the initial semantic discrimination features are processed using the attention weights respectively to obtain the spatial detail features to be processed and the semantic discrimination features to be processed. The spatial detail features and semantic discrimination features to be processed are processed using a preset spatial alignment strategy to make the spatial resolution of the spatial detail features and semantic discrimination features to be processed the same, thereby obtaining target spatial detail features and target semantic discrimination features; The target spatial detail features and the target semantic discrimination features are fused to obtain the fused features of the image to be processed, and the target semantic features of the image to be processed are determined based on the fused features and the preset target query vector.

[0044] In this embodiment, a single image to be processed is input into a convolutional backbone network for feature extraction. It should be noted that ResNet-34 (a deep convolutional neural network) is used as the feature extraction backbone.

[0045] In this embodiment, ResNet-34 includes multiple residual stages, enabling hierarchical feature extraction from low-level texture information to high-level semantic information. For example, the extracted hierarchical features may include λ1, λ2, λ3, and λ4.

[0046] In this embodiment, the initial spatial detail features L3 extracted in the third residual stage of ResNet-34 and the initial semantic discriminant features L4 extracted in the fourth residual stage can be selected as the basic inputs for subsequent pose estimation. Among them, the initial spatial detail features have relatively high spatial resolution, preserving more edge, contour and local texture information of objects; the initial semantic discriminant features have stronger semantic expressive power and can more effectively distinguish objects and background regions in the image.

[0047] In this embodiment of the application, since the initial spatial detail features and the initial semantic discriminative features differ in channel dimension and spatial size, MSFF (Multi-Scale Feature Fusion) is set after ResNet-34.

[0048] Using this multi-scale feature fusion module, initial spatial detail features and initial semantic discriminative features can be convolved according to a preset convolution strategy, aligning their channel dimensions. The preset convolution strategy can be: applying a 1×1 convolution to both the initial spatial detail features and the initial semantic discriminative features respectively to achieve channel alignment.

[0049] Subsequently, using this multi-scale feature fusion module, attention weights for at least one region in the image to be processed can be determined based on the initial spatial detail features and initial semantic discriminant features aligned to the channel dimensions. Specifically, for the initial spatial detail features and initial semantic discriminant features aligned to the channel dimensions, normalization processing is first performed on these two features respectively, adjusting the initial spatial detail feature response values ​​and initial semantic discriminant feature response values ​​of at least one region in the image to be processed to a uniform numerical range, so as to eliminate the bias caused by the difference in numerical scale between different features; then, the normalized feature response values ​​are mapped to the interval between 0 and 1 using an activation function, obtaining the attention weights corresponding to the initial spatial detail features and initial semantic discriminant features with the same feature space size, where the weight coefficient of each region is used to quantify the importance of its corresponding original image region.

[0050] Based on this, attention weights are applied to process the corresponding initial spatial detail features and initial semantic discriminant features to obtain the spatial detail features and semantic discriminant features to be processed. Specifically, the attention weight of at least one region is multiplied element-wise with the initial spatial detail feature response and the initial semantic discriminant feature response of the corresponding region to complete gating enhancement. This ensures that the feature responses of the original image regions with larger weight coefficients are preserved and enhanced, while the feature responses of the original image regions with smaller weight coefficients are suppressed. Through this process, the feature response intensity of redundant background regions in the original image can be reduced, while the feature response intensity of the region where the object is located can be highlighted, thus obtaining the spatial detail features and semantic discriminant features to be processed of the image to be processed.

[0051] Subsequently, a preset spatial alignment strategy is used to process the spatial detail features and semantic discriminant features to be processed, ensuring that their spatial resolutions are the same, thus obtaining the target spatial detail features and target semantic discriminant features. The spatial alignment strategy is as follows: the semantic discriminant features to be processed are upsampled to align their spatial resolution with that of the spatial detail features to be processed.

[0052] Finally, element-wise fusion is performed on the spatially aligned target spatial detail features and target semantic discriminative features to obtain a unified feature representation that combines mid-level texture details and high-level semantic information, i.e., the fused feature. Specifically, the target semantic discriminative features and target spatial detail features with channel alignment and spatial size alignment have the same height, width, and number of channels. Element-wise operations are performed on the two features according to the same spatial position and the same channel number to obtain the fused feature F. For any spatial position (i, j) and any channel c, the feature value F3(i, j, c) of the target spatial detail feature and the feature value F4(i, j, c) of the target semantic discriminative feature at that position are taken respectively, and element-wise fusion is performed. The fusion operation can be performed by element-wise addition, i.e.: F(i,j,c)=F3(i,j,c)+F4(i,j,c) The above method can effectively superimpose the edge, texture and local structural information contained in the target spatial detail features with the high-level semantic and global context information contained in the target semantic discrimination features at the same spatial location, thereby forming a fusion feature that takes into account both detail expression ability and semantic discrimination ability. This can effectively solve the problem of insufficient expression of single-scale features and improve the target representation ability in complex backgrounds and weak texture scenes.

[0053] In this embodiment, by extracting initial spatial detail features and initial semantic discriminative features from the image to be processed, and aligning the channel dimensions according to a preset convolution strategy, the problem of inconsistent feature channel numbers at different levels and the inability to directly fuse them is solved. Furthermore, by simultaneously utilizing the initial spatial detail features and initial semantic discriminative features, a feature foundation that balances detail and semantics can be provided for the subsequent keypoint localization of the 3D bounding box outside the object. Based on this, attention weights are determined based on the channel-aligned features, and these attention weights are used to enhance the two feature paths respectively, enabling the model to adaptively suppress redundant background region responses and highlight the feature expressions of target-related regions. Subsequently, a preset spatial alignment strategy is used to make the spatial resolution of the spatial detail features and the semantic discriminative features to be processed the same, and the two are fused to obtain a fused feature that simultaneously possesses mid-level detail information and high-level semantic discriminative information. This fused feature, while preserving target edges, contours, and local texture information, also has the semantic discriminative ability to distinguish target objects from background regions, providing a more complete input foundation for subsequent semantic feature extraction based on the target query vector.

[0054] In some embodiments of this application, the pose determination model includes an encoder and a decoder; determining the target semantic features of the image to be processed based on the fused features and a preset target query vector includes: The encoder is used to model the feature relationship between the fusion features of at least two regions of the image to be processed, so as to obtain the global memory features of the image to be processed. Using the decoder, the global memory features are queried according to the target query vector to obtain the target semantic features of the image to be processed.

[0055] In this embodiment, after obtaining the fused features, they are flattened into a sequence to meet the encoder's format requirements for the input structure. Then, the flattened fused features are added to the two-dimensional positional encoding and input into the encoder. The positional encoding can be a fixed two-dimensional sine and cosine positional encoding, used to explicitly inject spatial positional information in the image plane, enabling the encoder to perceive the relative relationships between different spatial positions during self-attention operations.

[0056] The encoder can use a multi-head self-attention mechanism to model the feature relationships in the global scope, that is, to model the feature relationships between the fused features of at least two regions of the image to be processed, and obtain the global memory features of the image to be processed, thereby enhancing the network's ability to understand long-distance dependencies, occlusion relationships and complex contexts.

[0057] The global memory features output by the encoder are then input into the decoder. The decoder can receive a learnable, pre-defined target query vector. Based on the target query vector, it queries the global memory features and extracts the high-level semantic information most relevant to the object's geometric structure from the encoded memory through a cross-attention mechanism, forming the target semantic features required for subsequent keypoint regression and heatmap localization. Two-dimensional sine and cosine position encoding can be represented as: Where x is the horizontal coordinate of the spatial location, y is the vertical coordinate of the spatial location, i is the dimension index of the location code, and d is the total dimension of the location code. The second i-th dimension component is the horizontal position encoding. The (2i+1)th dimension component for horizontal position encoding. The second i-th dimension component is encoded for the vertical position. The (2i+1)th dimension component is used to encode the vertical position.

[0058] The input encoder after feature fusion and flattening takes the following form: Where Z is the input sequence of the encoder, As a feature of fusion, This refers to the operation of converting the features of a two-dimensional arrangement into a one-dimensional sequence. It refers to two-dimensional position coding, which is obtained by combining horizontal and vertical position coding.

[0059] Reference Figure 3 The diagram illustrates an application flow diagram of a detection converter provided in an embodiment of this application.

[0060] The ResNet-34 network outputs initial spatial detail features in its third stage and initial semantic discriminative features in its fourth stage. The initial spatial detail features are 256×40×40 in size, representing a feature map with 256 channels, a height of 40, and a width of 40. The initial semantic discriminative features are 512×20×20 in size, representing a feature map with 512 channels, a height of 20, and a width of 20.

[0061] Both the initial spatial detail features and the initial semantic discriminant features are processed through a convolutional layer (Conv1d), group normalization, and activation functions to obtain the attention weights corresponding to the initial spatial detail features and the initial semantic discriminant features, respectively. Then, the attention weights are used to process the corresponding initial spatial detail features and initial semantic discriminant features to obtain the spatial detail features and semantic discriminant features to be processed.

[0062] The semantic discriminative features to be processed are upsampled to align their spatial resolution with the spatial detail features to be processed, resulting in target spatial detail features and target semantic discriminative features. Element-wise fusion is then performed on the spatially aligned target spatial detail features and target semantic discriminative features to obtain fused features. After obtaining the fused features, they are flattened into a sequence form. The flattened fused features are then added to the two-dimensional positional code and input into the encoder.

[0063] In this embodiment, the pose determination model includes an encoder and a decoder. After obtaining the fused features, the encoder first uses a global model to model the feature relationships between the fused features of at least two regions in the image to be processed, obtaining global memory features. This enables the encoder to use a multi-head self-attention mechanism to ensure that features at each spatial location can perceive information from all other locations in the entire image, thereby establishing long-distance dependencies and capturing complex contextual information. Based on this, the decoder queries the global memory features according to a preset target query vector, and extracts the high-level semantic information most relevant to the geometric structure of the target object from the global memory through a cross-attention mechanism, obtaining the target semantic features. This process uses the feature interaction results of different regions as the basis for the query, allowing the decoder to focus specifically on the target object itself, reducing the interference of background information on subsequent structured regression, and providing a more semantically focused and geometrically rich feature representation for the pose determination of the target object. Compared to traditional methods that rely solely on the local receptive field of convolution, this application further enhances the modeling ability for the overall structure of the object and complex scene relationships through the Transformer structure.

[0064] Step 102: Determine the initial structured regression parameters of the preset three-dimensional bounding box outside the object on the image to be processed based on the target semantic features; the initial structured regression parameters include the projection scale information of the three-dimensional bounding box on the image to be processed, the center point projection coordinates, and the normalized offset parameters of the corner point projection coordinates relative to the center point projection coordinates. In this embodiment, the object may have a preset three-dimensional bounding box. The three-dimensional bounding box has a center point and eight corner points, which can serve as nine key points for determining the object's pose. This nine-point representation can reflect the overall spatial range and center position of the object, and also provide sufficient geometric constraints for subsequent PnP pose solving.

[0065] In this embodiment of the application, after obtaining the target semantic features output by the decoder, the absolute two-dimensional coordinates of the nine key points (i.e. the target projection coordinates of the key points in the image to be processed) are not directly regressed in one go. Instead, a two-stage key point regression branch of coarse adjustment and fine adjustment is constructed, and a heat map key point localization branch is introduced for auxiliary correction to obtain the target projection coordinates of the key points in the image to be processed.

[0066] In this embodiment, the coarse adjustment stage branch can determine the initial structured regression parameters of the preset 3D bounding box outside the object on the image to be processed based on the target semantic features. The initial structured regression parameters include the projection scale information of the 3D bounding box outside the object on the image to be processed, the normalized offset parameters of the center point projection coordinates and the corner point projection coordinates relative to the center point projection coordinates.

[0067] In this embodiment, compared to directly regressing the absolute two-dimensional coordinates of keypoints, this expression method of "center point projection coordinates - projection scale information - normalized offset parameters" significantly reduces the degrees of freedom in the regression space. This allows the pose determination model to first learn the approximate position and scale range of the object, and then learn the geometric distribution relationship between the corner points and the center point among the keypoints. This approach improves the stability of the coarse prediction stage and enhances the geometric consistency of the overall keypoint layout. Based on the projection scale information, the center point projection coordinates, and the normalized offset parameters of the corner point projection coordinates relative to the center point projection coordinates, the coarse prediction results for nine keypoints can be recovered.

[0068] The key points for coarse adjustment are represented as follows: in, This represents the coarse adjustment result of the center point projection coordinates. This represents the coarse-tuned result of the corner projection coordinates of the i-th corner point. This expression constrains the point set within a unified geometric framework, significantly reducing the degrees of freedom in the regression space. These are the projected coordinates of the center point of the 3D bounding box. The projection scale parameter of the 3D bounding box. The normalized offset parameter is the corner projection coordinates of the i-th corner point relative to the center point projection coordinates.

[0069] Step 103: Based on the initial structured regression parameters and the target semantic features, determine the center point residual, corner point residual, and scale residual of the initial structured regression parameters, and use the center point residual, the corner point residual, and the scale residual to adjust the center point projection coordinates, the normalized offset parameter, and the projection scale information to obtain the target structured regression parameters of the object. In this embodiment, during the fine-tuning stage, the target semantic features and the initial structured regression parameters from the coarse-tuning are further input into the fine-tuning keypoint refinement module, which outputs keypoint residuals and scale residuals. The keypoint residuals include center point residuals and corner point residuals. The function of the fine-tuning keypoint refinement module is not to regenerate all keypoints, but to perform constrained local corrections based on the initial structured regression parameters from the coarse-tuning to obtain the target structured regression parameters. Based on the target structured regression parameters, the keypoint projection coordinates of the nine keypoints can be determined.

[0070] Specifically, center point residuals and corner point residuals are used to update the coordinates of keypoints in the coarse adjustment stage, and scale residuals are used to compensate for the estimated projection scale information in the coarse adjustment stage, so that the correction range is adapted to the imaging scale of the object. In this way, when the target is large, the keypoints can be adjusted at a wider range of pixels; when the target is small, the keypoint correction range remains more refined. Through a two-stage regression process of coarse-to-fine, the optimization difficulty of direct end-to-end accurate regression can be reduced, and the stability and consistency of the final keypoint results can be improved.

[0071] The fine-tuning and updating process can be represented as: in, This indicates the fine-tuned projection scale information. This represents the keypoint projection coordinates of the i-th keypoint after fine-tuning. Represents the interval cutoff function. This refers to the projection scale information predicted during the coarse adjustment stage. For scale residuals, This is the lower threshold for projection scale information. This represents the upper threshold for projection scale information. Let i be the coordinate residual of the i-th key point. This is the threshold for truncating the coordinate residuals. By scaling the keypoint residuals with scale involvement, the correction magnitude can be ensured to match the target size.

[0072] Step 104: Based on the image to be processed, generate a two-dimensional Gaussian heatmap of the center point and corner points of the three-dimensional bounding box, and determine the pose information of the object based on the target structured regression parameters and the two-dimensional Gaussian heatmap.

[0073] In this embodiment of the application, to further enhance the local precision positioning capability, a heatmap key point positioning branch is set. This branch can generate a two-dimensional Gaussian heatmap of the center point and corner points of the three-dimensional bounding box.

[0074] In the embodiments of this application, the pose information of an object can be determined based on the two-dimensional Gaussian heatmap of the key points output by the heatmap key point localization branch and the target structured regression parameters in the fine-tuning stage.

[0075] In some embodiments of this application, generating a two-dimensional Gaussian heatmap of the center point and corner points of the three-dimensional bounding box based on the image to be processed includes: Based on the global memory features or the fusion features, a two-dimensional Gaussian heatmap of the center point and the corner points is generated.

[0076] In this embodiment, the heatmap keypoint localization branch can generate two-dimensional Gaussian heatmaps for each of the nine keypoints based on the global memory features output by the encoder or the fused shared features. The two-dimensional Gaussian heatmap of each keypoint has the highest response value near the keypoint projection coordinates of the corresponding keypoint and decays smoothly towards the surrounding area, thereby transforming the keypoint prediction problem into a continuous spatial probability distribution modeling problem.

[0077] In this embodiment of the application, during the training of the heatmap key point localization branch, a target Gaussian heatmap is constructed with the real key point projection coordinates of the key point as the center, and the predicted heatmap is trained using the target Gaussian heatmap as a supervision signal until the error between the predicted heatmap and the target Gaussian heatmap is less than a preset threshold, thus obtaining the trained heatmap key point localization branch.

[0078] Let the coordinates of the center of the Gaussian heatmap of the i-th key point be... Then the target Gaussian heatmap of the i-th key point can be represented as: Where (u, v) are the coordinates in the heatmap plane. , The vertical coordinate in the heatmap plane, Let be the response value of the target Gaussian heatmap of the i-th key point at coordinates (u, v) on the heatmap plane. The standard deviation is Gaussian, used to control the diffusion range of the heatmap.

[0079] In the embodiments of this application, compared with simple coordinate regression, the heat map key point localization branch can provide optimized signals for multiple pixel positions in the neighborhood of the key point. Therefore, it can still maintain good local peak localization capability in scenarios with blurred edges, partial occlusion, weak texture and reflective interference.

[0080] In this embodiment, the generation of the 2D Gaussian heatmap is based on global memory features or fused features, rather than relying on single-scale convolutional features. Global memory features contain long-range dependencies and contextual information modeled globally by the encoder, while fused features contain mid-level details and high-level semantic information fused across multiple scales; both possess rich target perception capabilities. The Gaussian heatmap generated based on these features can form a more accurate and smooth probability distribution response in the neighborhood of key points, providing high-quality input for subsequent normalized probability information calculation and continuous coordinate extraction from the heatmap. Furthermore, since heatmap generation can share intermediate computational results with the encoder or fused features, eliminating the need for repeated forward propagation, it helps reduce overall computational overhead and improves the inference efficiency and engineering practicality of the pose determination method.

[0081] In this embodiment, by acquiring the image to be processed and determining its target semantic features, a feature foundation with global context awareness is provided for subsequent regression of structured parameters. Based on this, initial structured regression parameters (including projection scale information, center point projection coordinates, and normalized offset parameters) are first predicted according to the target semantic features. Then, the initial structured regression parameters are adjusted using center point residuals, corner point residuals, and scale residuals, realizing a two-stage regression mechanism from coarse to fine adjustment of the structured regression parameters. The coarse adjustment stage uses a structured expression of "center projection coordinates—projection scale parameters—normalized offset parameters." The fine adjustment stage refines the coarse adjustment results through residual correction, obtaining target structured regression parameters including projection scale information, center point projection coordinates, and normalized offset parameters. This reduces the difficulty required for the pose determination model to directly regress the absolute coordinates of all key points and improves the accuracy of the coordinates in the target structured regression parameters. Finally, through the synergistic effect of the target structured regression parameters and heatmap localization, the accuracy of object pose determination can be improved.

[0082] In some embodiments of this application, determining the pose information of the object based on the target structured regression parameters and the two-dimensional Gaussian heatmap includes: The center point and corner points of the 3D bounding box are used as key points for determining the pose of the object, and the key point projection coordinates on the image to be processed are determined based on the target structured regression parameters. The pose information of the object is determined based on the projected coordinates of the key points and the two-dimensional Gaussian heatmap.

[0083] In this embodiment, the center point and corner points of the three-dimensional bounding box can be used as key points for determining the pose of the object, and the key point projection coordinates on the image to be processed can be determined based on the target structured regression parameters.

[0084] In the embodiments of this application, the pose information of an object can be determined based on the projected coordinates of key points and the two-dimensional Gaussian heatmap of the key points.

[0085] In this embodiment, the center point and corner points of the 3D bounding box are used as key points. The key point projection coordinates on the image to be processed are determined based on the target structured regression parameters. Then, pose information is determined based on the key point projection coordinates and the 2D Gaussian heatmap, achieving the synergistic utilization of structured regression results and heatmap localization results. Since the key point projection coordinates originate from structured regression that integrates scale information and geometric constraints, they naturally possess overall geometric consistency; while the 2D Gaussian heatmap provides pixel-level probability distribution information within the key point's neighborhood. Combining these two methods can improve the accuracy of local localization while maintaining the overall rationality of the key point layout, thus providing a more reliable 2D key point information foundation for determining the pose of the target object.

[0086] In some embodiments of this application, the two-dimensional Gaussian heatmap of any of the key points includes probability information that at least one pixel around the key point's projected coordinates is the target projected coordinate of the key point; the step of determining the pose information of the object based on the key point's projected coordinates and the two-dimensional Gaussian heatmap includes: For any of the key points, based on the probability information of at least one pixel around the projected coordinates of the key point, the normalized probability information of any pixel around the projected coordinates of the key point is determined. Based on the normalized probability information, the continuous heatmap coordinates of the key points are determined; According to a preset coordinate fusion method, the projected coordinates of the key point and the continuous heat map coordinates of the key point are fused to obtain the target projected coordinates of the key point. Based on the target projection coordinates of the key points, the pose information of the object is determined.

[0087] In the embodiments of this application, the two-dimensional Gaussian heatmap of any key point includes probability information that at least one pixel around the key point's key point projection coordinates is the target projection coordinate of the key point.

[0088] In this embodiment, for any key point, the soft-argmax operation can convert the heat map of that key point into continuous coordinates of the heat map. Specifically, let the two-dimensional Gaussian heat map of any key point be... Then its normalized probability distribution can be expressed as: Where, ∑ p,q P represents the summation over all locations in the heatmap plane. i(u,v) represents the normalized probability information of the i-th key point at coordinates (u,v) on the heatmap plane. Let be the raw response value of the heatmap of the i-th keypoint predicted by the network at coordinates (p, q), and its meaning be the same as... Similarly, (p,q) are only temporary coordinate variables when summing.

[0089] The continuous coordinates of the heatmap for the i-th keypoint can be obtained through the soft-argmax operation, i.e., based on the normalized probability distribution P. i (u,v) is a weighted summation of all locations in the heatmap plane: in, Let be the continuous coordinates of the heatmap for the i-th key point.

[0090] In this embodiment, the projected coordinates and continuous heatmap coordinates of key points can be weighted and fused using preset weights, or adaptive fusion can be performed based on confidence information to obtain the target projected coordinates of the key points. When the local edges and textures of the image are clear, the heatmap branch can usually provide more accurate peak positions; when the local area is occluded or disturbed by noise, the global geometric prior carried by the fine-tuned regression branch can suppress heatmap response drift. By complementing the two information paths, the unity of "global structural stability" and "local pixel accuracy" is achieved, and the accurate positions of the nine key points in the two-dimensional image are finally output, laying the foundation for subsequent pose solving.

[0091] The formula for fusing the projected coordinates of key points and the continuous coordinates of key points in the heatmap can be expressed as: in, To preset weights, ∈[0,1]; Let be the target projection coordinates of the i-th key point; Let be the keypoint projection coordinates of the i-th keypoint.

[0092] In this embodiment, for any key point, normalized probability information is determined based on the probability information of at least one pixel around the key point's projected coordinates. This normalized probability information is then used to determine the continuous coordinates of the heatmap. The normalized probability information converts the original heatmap response into a standardized probability distribution, eliminating interference from differences in response amplitude across different regions and making the extraction of continuous heatmap coordinates more accurate and reliable. The key point's projected coordinates and the continuous heatmap coordinates are fused according to a preset coordinate fusion method, achieving a complementarity between the macroscopic geometric constraints of the regressed key points and the local precise positioning of the key points in the heatmap: the key point's projected coordinates provide an overall layout conforming to the target's geometric structure, while the continuous heatmap coordinates provide pixel-level or even sub-pixel-level fine positional information. Through this fusion mechanism, target projected coordinates with both geometric consistency and local precision are obtained, thereby improving the accuracy and reliability of pose determination.

[0093] In some embodiments of this application, the pose determination model includes a pose solving module; determining the pose information of the object based on the target projection coordinates of the key points includes: Obtain the three-dimensional coordinates of the key point in a preset three-dimensional coordinate system, and determine the correspondence information between the three-dimensional coordinates of the key point and the target projection coordinates based on the three-dimensional coordinates and the target projection coordinates; The corresponding relationship information is input into the pose solving module to obtain the pose information of the object.

[0094] In this embodiment, the 3D coordinates of key points in a preset 3D coordinate system are pre-calculated from the 3D model of the object. After obtaining the target projection coordinate set of 9 key points, since the target projection coordinates of the 3D key points and the 3D key points on the 3D model of the object maintain the same semantic order, a one-to-one correspondence between the 3D coordinates of the key points and the target projection coordinates can be established, thus obtaining the correspondence information. Combined with the camera intrinsic parameters, the correspondence information is input into the PnP pose solving module to obtain the rotation matrix R and translation vector T of the object relative to the camera coordinate system, thereby recovering the six-degree-of-freedom pose of the object.

[0095] Compared with direct regression of pose parameters, this method makes full use of the geometric prior of the target object, which is beneficial to improving the interpretability and numerical stability of pose estimation.

[0096] Let X be the i-th 3D key point of the target object. i The target projection coordinates of the key point are hat{k} i If the camera intrinsic parameter matrix is ​​K, then the PnP pose solution process can be expressed as the following reprojection error minimization problem: in, Represents the perspective projection function. and Let these represent the final optimal rotation matrix and translation vector, respectively. Let be the observed pixel coordinates of the i-th keypoint.

[0097] In the training phase of the pose determination model in this application, a joint loss function is used to optimize the overall network. Preferably, the total loss is a weighted sum of the predicted point coordinate loss, the coarse-tuning stage supervision loss, the heatmap supervision loss, and the heatmap coordinate loss. Specifically, the predicted point coordinate loss constrains the error between the final fused keypoints and the real 2D keypoints; the coarse-tuning stage supervision loss constrains the geometric rationality of the coarse-tuning results; the heatmap supervision loss constrains the distribution difference between the predicted heatmap and the target Gaussian heatmap; and the heatmap coordinate loss constrains the deviation between the heatmap coordinates obtained by soft-argmax decoding and the real coordinates. Through the joint constraints of multiple losses, this application can simultaneously improve coarse localization capability, fine regression capability, and local heatmap response capability, thereby achieving end-to-end stable training.

[0098] The joint loss function can be expressed as: in, This indicates the final key point prediction loss. This indicates the monitoring losses during the rough adjustment phase. Indicates heatmap monitoring loss, This indicates the loss of heatmap coordinates. , , These are the weighting coefficients for each loss term.

[0099] During training, data augmentation operations such as random rotation, random cropping, and scaling can be performed on the input image to enhance the model's generalization ability to different viewpoints, distances, and local truncation conditions. After training, the model only needs a single RGB image as input during the inference phase to sequentially complete feature extraction, multi-scale fusion, global semantic modeling, coarse-tuned regression, fine-tuning, heatmap localization, keypoint fusion, and PnP pose solving, ultimately outputting the 6D pose result of the target object. This process requires no additional depth sensor or multi-view input, and has advantages such as clear structure, low hardware dependency, and convenient engineering deployment.

[0100] In this embodiment, by obtaining the three-dimensional coordinates of key points in a preset three-dimensional coordinate system, establishing a correspondence between the three-dimensional coordinates and the target projection coordinates, and then inputting this correspondence information into the pose solving module, a geometric mapping from two-dimensional key point prediction to three-dimensional pose recovery is achieved. The pose solving module uses this correspondence to perform geometric solving, and the resulting pose information has high numerical stability and interpretability, thereby ensuring the reliability and accuracy of the pose determination method in real-world application scenarios.

[0101] Reference Figure 4 This diagram illustrates a process for determining the target projection coordinates of a key point according to an embodiment of this application. The target semantic features output by the Transformer Decoder are input into a coarse-tuning stage. The coarse-tuning stage predicts the center point projection coordinates, projection scale information, and normalized offset parameters for eight corner points. The center point projection coordinates are 2D vectors, the projection scale information is a 3D vector, and the normalized offset parameters are 8×2D vectors. Combining these parameters recovers the coarse-tuned key point coordinates for nine key points, including the eight corner points and the center point of the 3D bounding box.

[0102] The coarse-tuned keypoint coordinates and the target semantic features output by the Transformer Decoder are input into the fine-tuning stage. The fine-tuning stage refines the coarse-tuned keypoint coordinates through residual prediction and scale adaptive correction, and outputs the fine-tuned keypoint projected coordinates.

[0103] Global memory features or fused features are input into a Conv-BN-ReLU (convolution-batch normalization-linear rectified unit) structure for further transformation. The output is then input into the softmax (normalized exponential function) branch and the view (dimensionality reconstruction) branch, respectively. The output of the softmax branch is then processed through the view (dimensionality reconstruction) and soft-argmax (softening maximum value) operations to obtain the continuous coordinates of the heatmap branch. The fine-tuning keypoint projection coordinates of the fine-tuning branch are then fused with the continuous coordinates of the heatmap after the view (dimensionality reconstruction) operation to obtain the target projection coordinates of 9 keypoints.

[0104] Finally, the target projection coordinates of the nine key points are input into the PnP pose solving module. Combined with the coordinates of the three-dimensional reference key points of the target object, the rotation matrix R of the target object relative to the camera coordinate system is obtained. 6D Translation vector t SITE This allows us to obtain the object's pose. (Refer to...) Figure 5 This illustration shows a schematic diagram of the target object 6D pose recovery process based on two-dimensional key point prediction and PnP solution provided in the embodiments of this application. Figure 5 A total of 16 stages of posture recovery were demonstrated.

[0105] In this embodiment, a single RGB image is used as input, and the eight corner points and center point of the target object's 3D bounding box are used as 3D reference keypoints. Through the collaborative work of a convolutional backbone network, a multi-scale feature fusion module, a Transformer encoder-decoder module, a coarse-fine adjustment keypoint regression branch, a heatmap keypoint localization branch, and a PnP pose solving module, the six-DOF pose of the target object relative to the camera coordinate system is output. Overall, this application does not directly regress the rotation matrix and translation vector. Instead, it first predicts the projection positions of the nine keypoints of the target object in the 2D image, and then recovers the spatial pose of the target object based on the correspondence between the 2D keypoints and the 3D reference keypoints, thus balancing network prediction stability and geometric solution interpretability.

[0106] To address the technical challenges of existing monocular RGB pose estimation methods in complex backgrounds, weak textures, occlusion, and reflective scenes, such as insufficient target feature representation, inadequate utilization of multi-scale information, poor stability and lack of geometric constraints in direct keypoint regression, and low local localization accuracy, this application constructs a monocular RGB pose estimation system collaboratively composed of a multi-scale feature fusion module, a Transformer global semantic modeling module, a coarse-fine two-stage keypoint regression branch, a Gaussian heatmap localization branch, and a PnP geometry solution module. The system achieves the following technical effects: First, multi-scale feature fusion enhances target perception capabilities in complex scenes. Addressing the technical challenges of insufficient target feature representation and keypoint drift in complex backgrounds, occlusions, and reflective scenes in existing monocular RGB pose estimation methods, this application introduces a multi-scale feature fusion module between the convolutional backbone network and the Transformer encoder. This module performs channel alignment, gating enhancement, spatial alignment, and element-wise fusion on the third-stage feature L3 and the fourth-stage feature L4, enabling the network to simultaneously utilize mid-level detailed texture information and high-level semantic information. This improves the target region response quality, background suppression capability, and keypoint prediction stability in complex scenes, overcoming the limitation of single-scale features in simultaneously addressing localization and discrimination.

[0107] Second, structured coarse-tuning and residual fine-tuning enhance geometric consistency and training stability. Addressing the technical problems of traditional keypoint regression methods that directly output all absolute coordinates, have high degrees of freedom in the regression space, and lack sufficient geometric constraints during training, this application proposes a two-stage coarse-fine-tuning keypoint regression mechanism. In the coarse-tuning stage, a structured expression of "center coordinates—scale parameters—corner point relative offsets" is used to initially estimate nine keypoints, transforming keypoint prediction into an estimate of the overall position, size, and geometric distribution of the object, significantly reducing the degrees of freedom in the regression space. In the fine-tuning stage, keypoint residuals and scale residuals are used to constrain and refine the coarse-tuning results, and the scale parameter is used to participate in residual scaling, ensuring that the correction magnitude matches the target size. This reduces the difficulty of network optimization and improves the geometric consistency between keypoints, scale adaptation capability, and prediction stability under different target imaging conditions.

[0108] Third, the fusion of heatmap localization and regression branch improves local localization accuracy and anti-interference capability. Addressing the technical problem that relying solely on the regression branch is insufficient to fully utilize local peak distribution information, and that localization is inaccurate under conditions of edge blurring and local occlusion, this application further incorporates a Gaussian heatmap keypoint localization branch. This branch generates two-dimensional Gaussian heatmaps for each of the nine keypoints, transforming keypoint prediction into a continuous spatial probability distribution modeling problem, and converting the heatmaps into continuous coordinates through a soft-argmax operation. After obtaining the prediction results from the fine-tuning branch and the heatmap branch, the two results are weighted and fused. This enables the system to simultaneously possess the global geometric prior of the regression branch and the local fine-localization capability of the heatmap branch, improving the accuracy of keypoint coordinate prediction, enhancing the perception of weakly textured regions, locally missing regions, and edge detail regions, while reducing the negative impact of local anomaly responses on the final pose estimation result.

[0109] Fourth, single RGB image input reduces hardware dependence and deployment costs. Addressing the technical challenge that some existing high-precision pose estimation methods rely on depth sensors, complex multi-stage optimization, or large-scale inference computation, making it difficult to meet the requirements of low cost, loose coupling, and real-time deployment in industrial settings, this application achieves high-precision 6D pose recovery of target objects under single RGB image input conditions by combining an improved DETR network, key point fusion localization, and PnP pose solving. The entire inference process requires only a single RGB image, eliminating the need for additional depth sensors, multi-view input, and complex post-processing iterative optimization. Therefore, while ensuring a clear computational process and low implementation cost, it improves the hardware friendliness, engineering feasibility, and scenario scalability of the pose estimation system, making it more suitable for widespread application in industrial settings, vision guidance, robot grasping, and low-cost intelligent equipment.

[0110] Fifth, a joint loss function ensures stable end-to-end training. Addressing the challenges of collaborative optimization and slow convergence among branches during network training, this application employs a joint loss function to optimize the overall network. The total loss is a weighted average of the final keypoint prediction loss, the coarse-tuning stage supervision loss, the heatmap supervision loss, and the heatmap coordinate loss. The final keypoint prediction loss constrains the error between the fused keypoint coordinates and the true coordinates; the coarse-tuning stage supervision loss constrains the geometric rationality of the structured regression parameters; the heatmap supervision loss constrains the distribution difference between the predicted heatmap and the target Gaussian heatmap; and the heatmap coordinate loss constrains the deviation between the heatmap coordinates obtained from soft-argmax decoding and the true coordinates. Through the joint constraints of these four losses, each module of the network obtains effective gradient supervision signals, achieving a synergistic improvement in coarse-tuning localization capabilities, fine-tuning regression capabilities, and heatmap response capabilities, thus ensuring the stability and convergence efficiency of end-to-end training.

[0111] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0112] Reference Figure 6 This application also discloses a structural block diagram of an object pose determination device, applied to a pose determination model, including: The target semantic feature determination module 601 is used to acquire the image of the object to be processed, and determine the target semantic features of the image to be processed based on the image to be processed; The initial structured regression parameter determination module 602 is used to determine the initial structured regression parameters of the preset three-dimensional bounding box outside the object on the image to be processed based on the target semantic features; the initial structured regression parameters include the projection scale information of the three-dimensional bounding box on the image to be processed, the center point projection coordinates, and the normalized offset parameters of the corner point projection coordinates relative to the center point projection coordinates. The target structured regression parameter acquisition module 603 is used to determine the center point residual, corner point residual, and scale residual of the initial structured regression parameters based on the initial structured regression parameters and the target semantic features, and to adjust the center point projection coordinates, the normalized offset parameter, and the projection scale information using the center point residual, the corner point residual, and the scale residual to obtain the target structured regression parameters of the object. The pose information determination module 604 is used to generate a two-dimensional Gaussian heatmap of the center point and corner points of the three-dimensional bounding box based on the image to be processed, and to determine the pose information of the object based on the target structured regression parameters and the two-dimensional Gaussian heatmap.

[0113] In one optional embodiment of this application, the pose information determination module includes: The key point projection coordinate determination submodule is used to use the center point and corner points of the 3D bounding box as key points for determining the pose of the object, and to determine the key point projection coordinates of the key points on the image to be processed based on the target structured regression parameters. The pose information determination submodule is used to determine the pose information of the object based on the key point projection coordinates and the two-dimensional Gaussian heatmap.

[0114] In one optional embodiment of this application, the two-dimensional Gaussian heatmap of any of the key points includes probability information that at least one pixel around the key point's key point projection coordinates is the target projection coordinate of the key point; the pose information determination submodule includes; The normalized probability information determination unit is used to determine the normalized probability information of any pixel around the projected coordinates of the key point, based on the probability information of at least one pixel around the projected coordinates of the key point. The heatmap continuous coordinate determination unit is used to determine the heatmap continuous coordinates of the key points based on the normalized probability information. The target projection coordinate obtaining unit is used to fuse the key point projection coordinates and the key point heatmap continuous coordinates according to a preset coordinate fusion method to obtain the target projection coordinates of the key point. The pose information determination unit is used to determine the pose information of the object based on the target projection coordinates of the key points.

[0115] In one optional embodiment of this application, the pose determination model includes a pose solving module; the pose information determination unit includes: The correspondence information determination subunit is used to obtain the three-dimensional coordinates of the key point in a preset three-dimensional coordinate system, and determine the correspondence information between the three-dimensional coordinates of the key point and the target projection coordinates based on the three-dimensional coordinates and the target projection coordinates; The pose information acquisition sub-unit is used to input the correspondence information into the pose solving module to obtain the pose information of the object.

[0116] In one optional embodiment of this application, the target semantic feature determination module includes: The feature extraction submodule is used to extract the initial spatial detail features and initial semantic discriminant features of the image to be processed, and to perform convolution processing on the initial spatial detail features and the initial semantic discriminant features according to a preset convolution strategy, so that the channel dimensions of the initial spatial detail features and the initial semantic discriminant features are aligned. The attention weight determination submodule is used to determine the attention weight of at least one region on the image to be processed based on the initial spatial detail features and the initial semantic discrimination features aligned to the channel dimension. The submodule for obtaining semantic discriminative features to be processed is used to process the initial spatial detail features and the initial semantic discriminative features using the attention weights respectively, to obtain spatial detail features to be processed and semantic discriminative features to be processed; The target semantic discrimination feature acquisition submodule is used to process the spatial detail features to be processed and the semantic discrimination features to be processed using a preset spatial alignment strategy, so that the spatial resolution of the spatial detail features to be processed and the semantic discrimination features to be processed are the same, thereby obtaining the target spatial detail features and the target semantic discrimination features; The target semantic feature determination submodule is used to fuse the target spatial detail features and the target semantic discrimination features to obtain the fused features of the image to be processed, and to determine the target semantic features of the image to be processed based on the fused features and the preset target query vector.

[0117] In one optional embodiment of this application, the pose determination model includes an encoder and a decoder; the target semantic feature determination submodule includes: A global memory feature acquisition unit is used to model the feature relationship between the fusion features of at least two regions of the image to be processed using the encoder, so as to obtain the global memory features of the image to be processed. The target semantic feature obtaining unit is used to use the decoder to query the global memory features according to the target query vector to obtain the target semantic features of the image to be processed.

[0118] In one optional embodiment of this application, the pose information determination module includes: The two-dimensional Gaussian heatmap generation submodule is used to generate two-dimensional Gaussian heatmaps of the center point and the corner points based on the global memory features or the fusion features.

[0119] It is understood that the above-described device embodiments correspond to the method embodiments of this application, and can implement the method provided by any of the above-described method embodiments of this application.

[0120] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0121] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided in this application, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0122] Based on the embodiments of the above methods, another embodiment of this application provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the method described in any embodiment of this application.

[0123] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete this application. The one or more module units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0124] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0125] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0126] Based on the above-described method embodiments, another embodiment of this application provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the method described in any of the above-described method embodiments of this application.

[0127] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0128] Based on the above-described method embodiments, another embodiment of this application provides a computer program product, including a computer program or instructions, which, when executed by a communication device, implements the method described in any of the above-described method embodiments of this application.

[0129] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. In particular, it should be noted that any modifications, equivalent substitutions, or improvements made by those skilled in the art within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for determining the pose of an object, characterized in that, Applied to a pose determination model, the pose determination method for the object includes: Acquire the image of the object to be processed, and determine the target semantic features of the image to be processed based on the image to be processed; Based on the target semantic features, the initial structured regression parameters of the preset three-dimensional bounding box outside the object on the image to be processed are determined; the initial structured regression parameters include the projection scale information of the three-dimensional bounding box on the image to be processed, the center point projection coordinates, and the normalized offset parameters of the corner point projection coordinates relative to the center point projection coordinates. Based on the initial structured regression parameters and the target semantic features, the center point residual, corner point residual, and scale residual of the initial structured regression parameters are determined, and the center point projection coordinates, the normalized offset parameter, and the projection scale information are adjusted using the center point residual, the corner point residual, and the scale residual to obtain the target structured regression parameters of the object. Based on the image to be processed, a two-dimensional Gaussian heatmap of the center point and corner points of the three-dimensional bounding box is generated, and the pose information of the object is determined based on the target structured regression parameters and the two-dimensional Gaussian heatmap.

2. The method for determining the pose of an object according to claim 1, characterized in that, The step of determining the pose information of the object based on the target structured regression parameters and the two-dimensional Gaussian heatmap includes: The center point and corner points of the 3D bounding box are used as key points for determining the pose of the object, and the key point projection coordinates on the image to be processed are determined based on the target structured regression parameters. The pose information of the object is determined based on the projected coordinates of the key points and the two-dimensional Gaussian heatmap.

3. The method for determining the pose of an object according to claim 2, characterized in that, The two-dimensional Gaussian heatmap of any of the key points includes probability information that at least one pixel around the key point's projected coordinates is the target projected coordinate of the key point; the step of determining the pose information of the object based on the key point's projected coordinates and the two-dimensional Gaussian heatmap includes: For any of the key points, based on the probability information of at least one pixel around the projected coordinates of the key point, the normalized probability information of any pixel around the projected coordinates of the key point is determined. Based on the normalized probability information, the heatmap continuous coordinates of the key points are determined; According to a preset coordinate fusion method, the projected coordinates of the key point and the continuous heat map coordinates of the key point are fused to obtain the target projected coordinates of the key point. Based on the target projection coordinates of the key points, the pose information of the object is determined.

4. The method for determining the pose of an object according to claim 3, characterized in that, The pose determination model includes a pose solving module; determining the pose information of the object based on the target projection coordinates of the key points includes: Obtain the three-dimensional coordinates of the key point in a preset three-dimensional coordinate system, and determine the correspondence information between the three-dimensional coordinates of the key point and the target projection coordinates based on the three-dimensional coordinates and the target projection coordinates; The corresponding relationship information is input into the pose solving module to obtain the pose information of the object.

5. The method for determining the pose of an object according to claim 1, characterized in that, The step of determining the target semantic features of the image to be processed based on the image to be processed includes: Extract the initial spatial detail features and initial semantic discriminant features of the image to be processed, and perform convolution processing on the initial spatial detail features and the initial semantic discriminant features according to a preset convolution strategy, so that the channel dimensions of the initial spatial detail features and the initial semantic discriminant features are aligned. Based on the initial spatial detail features and the initial semantic discrimination features aligned to the channel dimension, the attention weight of at least one region on the image to be processed is determined; The initial spatial detail features and the initial semantic discrimination features are processed using the attention weights respectively to obtain the spatial detail features to be processed and the semantic discrimination features to be processed. The spatial detail features and semantic discrimination features to be processed are processed using a preset spatial alignment strategy to make the spatial resolution of the spatial detail features and semantic discrimination features to be processed the same, thereby obtaining target spatial detail features and target semantic discrimination features; The target spatial detail features and the target semantic discrimination features are fused to obtain the fused features of the image to be processed, and the target semantic features of the image to be processed are determined based on the fused features and the preset target query vector.

6. The method for determining the pose of an object according to claim 5, characterized in that, The pose determination model includes an encoder and a decoder; the determination of the target semantic features of the image to be processed based on the fused features and a preset target query vector includes: The encoder is used to model the feature relationship between the fusion features of at least two regions of the image to be processed, so as to obtain the global memory features of the image to be processed. Using the decoder, the global memory features are queried according to the target query vector to obtain the target semantic features of the image to be processed.

7. The method for determining the pose of an object according to claim 6, characterized in that, The step of generating a two-dimensional Gaussian heatmap of the center point and corner points of the three-dimensional bounding box based on the image to be processed includes: Based on the global memory features or the fusion features, a two-dimensional Gaussian heatmap of the center point and the corner points is generated.

8. A device for determining the pose of an object, characterized in that, The pose determination device for an object, applied to a pose determination model, includes: The target semantic feature determination module is used to acquire the image of the object to be processed, and determine the target semantic features of the image to be processed based on the image to be processed; The initial structured regression parameter determination module is used to determine the initial structured regression parameters of the preset three-dimensional bounding box outside the object on the image to be processed based on the target semantic features; the initial structured regression parameters include the projection scale information of the three-dimensional bounding box on the image to be processed, the center point projection coordinates, and the normalized offset parameters of the corner point projection coordinates relative to the center point projection coordinates. The target structured regression parameter acquisition module is used to determine the center point residual, corner point residual, and scale residual of the initial structured regression parameters based on the initial structured regression parameters and the target semantic features, and to adjust the center point projection coordinates, the normalized offset parameter, and the projection scale information using the center point residual, the corner point residual, and the scale residual to obtain the target structured regression parameters of the object. The pose information determination module is used to generate a two-dimensional Gaussian heatmap of the center point and corner points of the three-dimensional bounding box based on the image to be processed, and to determine the pose information of the object based on the target structured regression parameters and the two-dimensional Gaussian heatmap.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images