Fusion prior factor and multi-layer interaction of vehicle 3D shape and posture joint analysis method

Abstract

This invention discloses a method for joint analysis of vehicle 3D shape and pose by integrating prior factors and multi-layered interactions, belonging to the field of electronic digital data processing technology. It enables efficient and robust joint reconstruction and analysis of vehicle 3D shape and pose. The invention includes: converting the input image to camera space, extracting 2D primitive features, and further constructing target fusion features; using a prior factor-driven attention module for vehicle 3D shape reconstruction, effectively aggregating prior information; using a hierarchical interactive attention module for vehicle 6DoF pose estimation, accurately modeling the interaction relationships between targets and between targets and the scene; and supervising the training of the joint analysis model using multi-level, multi-angle loss functions to output accurate shape and pose joint estimation results. This invention is applicable to vehicle perception in open and complex traffic environments.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction technology for computer graphics in general image data processing or generation techniques, and particularly to a method for joint analysis of vehicle 3D shape and attitude that integrates prior factors and multi-layer interactions. Background Technology

[0002] Vehicle 3D shape reconstruction refers to reconstructing the 3D shape of a vehicle simultaneously by regressing its 6DoF pose given an image. This technology has wide applications in fields with strong human-computer interaction, such as autonomous driving. However, the complex scenarios in real-world environments present significant challenges to this task. First, monocular 3D vehicle pose estimation is a typical ill-posed problem, primarily due to the lack of depth information and viewpoint limitations. Second, complex interactions within the scene significantly impact the final result. Finally, there remains a clear trade-off between the accuracy of 3D shape reconstruction and computational efficiency. Therefore, a joint analysis method for vehicle 3D shape and pose that integrates prior factors and multi-layered interactions is urgently needed to address these issues.

[0003] Early methods, largely based on geometric analysis, had significant limitations and were subsequently replaced by data-driven deep learning approaches. Existing methods have addressed some challenges. In the 6DoF (6 Degrees of Freedom Pose Estimation) pose regression task for vehicles, convolutional network-based methods utilize 3D bounding box annotations to obtain dense planar representations, mitigating the ill-conditioned nature of 2D-to-3D mapping. However, they neglect the impact of global context and inter-object interactions on the final regression results. In 3D shape reconstruction of vehicles, inspired by human shape reconstruction techniques, some studies have attempted to design principal component analysis (PCA) bases for vehicle shape reconstruction. However, limitations in detail capture are due to the linear assumption and sensitivity to noise.

[0004] Therefore, how to balance the lack of depth information, the complex interaction between targets, and the need for efficient modeling in complex open scenarios to achieve efficient and robust joint reconstruction analysis of vehicle 3D shape and pose has become a problem that needs to be studied and solved. Summary of the Invention

[0005] The embodiments of the present invention provide a method for joint analysis of vehicle 3D shape and attitude that integrates prior factors and multi-layer interactions, which can achieve efficient and robust joint reconstruction analysis of vehicle 3D shape and attitude.

[0006] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0007] A method for joint analysis of vehicle 3D shape and attitude that integrates prior factors and multi-layer interactions includes:

[0008] Step 1: Extract the feature map of the target image and convert it to camera space;

[0009] Step 2: Extract 2D primitive features from the feature map and generate target fusion features;

[0010] Step 3: Utilize prior factors and target fusion features to construct the first attention module for vehicle 3D shape reconstruction in the joint analysis model;

[0011] Step 4: Utilize target fusion features and scene features to construct a second attention module for vehicle 6DoF pose estimation in the joint analysis model;

[0012] Step 5: Establish a multi-level, multi-angle loss function to supervise the training of the joint analysis model, and then use the trained joint analysis model to perform 3D shape and attitude joint analysis on the vehicle.

[0013] In computer vision-related analysis and modeling techniques, four main coordinate systems are typically established: world coordinate system, camera coordinate system, image physical coordinate system, and pixel coordinate system. This embodiment focuses on the transformation from the pixel coordinate system to the camera coordinate system. Specifically, step one includes: obtaining feature maps of the target image using Mask R-CNN extended from Res2Net; and transforming the information of the obtained feature maps in pixel space to camera space, including: From pixel space conversion to The camera spatial approach is as follows: , , , Where: the bounding box of the feature map is represented in pixel space as the coordinates of the object's center. , , These are the x and y values ​​of the object's center coordinates; the corresponding bounding box width is... The height is The camera's internal parameters are , , The values ​​on the x and y axes corresponding to the focal length per unit pixel, , ) represents the principal point of the image center; Z represents the fixed scaling factor used to match the image shape.

[0014] Specifically, step two includes: extracting 2D primitive features from the feature map; constructing a keypoint heatmap using the extracted 2D primitive features; and then generating target fusion features based on the keypoint heatmap. This involves processing the 2D primitive features through an embedding network, whereby the keypoint heatmap is overlaid onto the corresponding keypoints. The 2D primitive features include bounding boxes. Bounding box features 2D key points and key point heatmap The processed 2D primitive features are stacked with the bounding box and its corresponding bounding box features to obtain the target fusion feature. , , , It is an embedding network that enhances the features of the bounding box. It is an embedding network that enhances the features of the bounding box. It is an embedding network that enhances the features of 2D key points. It is an embedding network that enhances the features of keypoint heatmaps, where n represents the number of detected vehicles and c represents the fused feature dimension. Represents the real number field. This indicates a fully connected layer.

[0015] Specifically, step three includes: constructing a prior set and obtaining prior factors. The prior set includes: prior features. Prior average and prior offset Among them, a total of =79 prior shapes, each consisting of v=1352 vertices, and prior factors are obtained by processing the prior features through a feature aggregation module. n represents the number of detected vehicles; c represents the fused feature dimension; v represents the number of vertices in each prior shape; 3D shape reconstruction of vehicles is performed using an attention module driven by the acquired prior factors. Specifically, target features are fused through linear transformation. Mapped to query vector , prior features Mapped to key vectors respectively Sum value vector Calculate attention score ,in, and These are all intermediate representations of the attention score calculation process. and The reconstructed 3D shape of the vehicle is represented as follows: .

[0016] Specifically, step four includes: constructing scene features. Preprocessing is then performed, where d is the dimension of the scene features; the result of the preprocessing is expressed as... and , , tanh() represents the dynamic tanh function. It is a learnable scaling factor used to dynamically adjust the scaling ratio based on the range of the input; and These are learnable scaling and translation parameters used to fit the real input-output relationship. The interaction between the target and the scene is decoupled into two categories: object-level interaction-aware attention. The object-level interaction-aware attention score is represented as ; and, context-aware attention at the scene level The scene-level context-aware attention score is represented as In the second attention module, a translation regressor is used. Predicting vehicle displacement PT using a rotational regressor Predict the vehicle's rotation PR; where, , , and For the corresponding and The learnable factor.

[0017] Specifically, the multi-level, multi-angle loss function in step five includes: a regression loss sub-function, a 3D reconstruction loss sub-function, an object detection loss sub-function, and a world-frame-guided shape-pose joint loss sub-function; among which, the translation loss in the regression loss sub-function is: , Let T represent the ground truth translation vector; the rotation loss is: , R represents the ground truth value of the rotation vector, and R represents the predicted rotation vector; the 3D reconstruction loss function is: , Represents the ground truth of the mesh, i represents the vertex number of the object mesh, v represents the number of vertices for each prior shape, and m represents the network reconstruction result of the target. This represents the 3D coordinate index of the i-th vertex of the object mesh; the object detection loss function is: , Represents the network loss of the region proposal in Maskr-CNN. Indicates the 2D bounding box loss. This indicates the loss of 2D key points.

[0018] The world-frame guided shape-pose joint loss function is: Among them, the 3D mesh loss in the world coordinate system Rotational space loss Translational space loss , The rotation matrix corresponding to the estimated object rotation. Represents the translation vector. This represents the actual rotation matrix corresponding to the object. The true translation vector corresponding to the object is represented by the multi-level, multi-angle loss function, which is: ,in, These are the weighting coefficients for each loss function.

[0019] The present invention provides a method for joint analysis of vehicle 3D shape and pose that integrates prior factors and multi-layered interactions. By integrating prior factors to drive an attention module and comprehensively considering the interaction between the target and the scene, it achieves efficient and robust joint reconstruction of vehicle 3D shape and pose. The invention includes: extracting and locating target bounding boxes from the input image and converting them to camera space; extracting 2D primitive features and further constructing target fusion features; designing a prior factor-driven attention module for vehicle 3D shape reconstruction, which significantly improves shape reconstruction accuracy and reduces computational cost through effective aggregation of prior information; designing a hierarchical interactive attention module for vehicle 6DoF pose estimation, accurately modeling the interaction relationships between targets and between the target and the scene, and improving pose estimation accuracy; and designing a multi-level, multi-angle loss function to supervise the training of the joint analysis model, ultimately outputting accurate shape and pose joint estimation results, thereby achieving efficient and robust joint reconstruction analysis of vehicle 3D shape and pose. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the overall framework process provided for an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a priori factor-driven attention module provided in an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of a hierarchical interaction attention module provided in an embodiment of the present invention;

[0024] Figure 4A schematic diagram illustrating the comparison with existing algorithms on the ApolloCar3D dataset provided in this embodiment of the invention;

[0025] Figure 5 This is a schematic diagram illustrating the effect of the present invention on the ApolloCar3D dataset provided in this embodiment of the invention;

[0026] Figure 6 This is a schematic diagram of the method flow provided in an embodiment of the present invention; Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Embodiments of the present invention will be described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in the specification of the present invention means the presence of features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say that an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0028] The formula model involved in the specific implementation is shown below.

[0029]

[0030] Embodiments of the present invention provide a method for joint analysis of vehicle 3D shape and attitude that integrates prior factors and multi-layer interactions, such as... Figure 6 As shown, it includes:

[0031] S1. For the input image, extract and locate the target bounding box, and convert it to camera space;

[0032] S2. Extract 2D primitive features and further construct target fusion features;

[0033] S3. Design a first attention module (i.e., an attention module driven by prior factors) for vehicle 3D shape reconstruction. By effectively aggregating prior information, the shape reconstruction accuracy is significantly improved and the computational cost is reduced.

[0034] S4. Design a second attention module (i.e., a hierarchical interactive attention module) for vehicle 6DoF pose estimation, accurately model the interaction relationships between targets and between targets and the scene, and improve the pose estimation accuracy.

[0035] S5. Design multi-level, multi-angle loss functions to supervise the training of the joint analysis model, and finally output accurate joint estimation results of shape and pose.

[0036] In this embodiment, S1 includes: extending Mask R-CNN through ReS2Net network to obtain feature maps of the target image; and converting information in pixel space to camera space.

[0037] Specifically, such as Figure 1 Mask R-CNN is extended using a ReS2Net backbone network to obtain feature maps of the target image. The bounding box is represented in pixel space as the coordinates of the object's center. and the width of the corresponding bounding box and height Camera intrinsics are defined as follows: ,in , Focal length per unit pixel, ( , () is the principal point of the image center. A fixed scaling factor is introduced. To match the shape of the image, From pixel space conversion to In camera space, as shown in formulas (1)-(4).

[0038] In this embodiment, S2 includes: processing the feature map through RoIAlign to obtain bounding box features; obtaining 2D key points by extending the detection branch of Mask R-CNN; designing an embedded network to regress a key point heatmap; and fusing the bounding box, bounding box features, 2D key points, and key point heatmap to construct target fusion features.

[0039] Specifically, such as Figure 1As shown, to obtain the appearance features of vehicle instances, RoIAlign is used to process the bounding boxes, thereby obtaining bounding box features. Keypoints not only provide relative position and size information within the instance but also reflect vehicle shape information. This is crucial for shape estimation because it involves the local structure of the vehicle. Considering global information, this invention introduces an embedded network to regress a 2D keypoint heatmap to more accurately represent the visibility of keypoints, helping the network determine whether a local area of ​​the instance is occluded.

[0040] Finally, the corresponding 2D primitive features are processed through an embedding network, and keypoint heatmaps are overlaid onto the corresponding keypoints to enhance instance information. These heatmaps are then stacked with bounding boxes and their corresponding bounding box features through two fully connected layers. Complete target fusion features The construction of the model further enhances its ability to recognize objects and estimate poses, as shown in formula (5). Wherein, It is a heatmap of key points. For 2D key point information, For bounding box information, This refers to the bounding box feature.

[0041] In this embodiment, in S3, a first attention module (i.e., an attention module driven by prior factors) is designed for vehicle 3D shape reconstruction. The shape reconstruction accuracy is significantly improved and the computational cost is reduced by effectively aggregating prior information. This includes: constructing a prior set to obtain prior factors; and completing the vehicle 3D shape reconstruction through the attention module mechanism driven by prior factors.

[0042] Specifically, such as Figure 3 As shown, the prior set consists of three parts: prior features, prior average, and prior bias. Among them, the prior features... Prior factors will be obtained through feature aggregation module. .use =79 prior shapes, each consisting of v=1352 vertices. Therefore, the prior average is expressed as The prior offset is expressed as .

[0043] Target features are fused through linear transformation. Mapped to query vector , prior features Mapped to key vectors respectively value vector Simultaneously, prior features are processed through a feature aggregation module to obtain prior factors. To avoid full computation between the query and the key. As in formulas (6) and (7), where, , These are all intermediate representations of the attention score calculation process. Further combined... Calculate attention score For example (8). To prevent the loss of target feature information during forward propagation, the present invention sets up a skip connection to complete the vehicle shape. The reconstruction. For example (9).

[0044] In this embodiment, in S4, a second attention module (i.e., a hierarchical interaction attention module) is designed for vehicle 6DoF pose estimation to accurately model the interaction relationships between targets and between targets and the scene, thereby improving the pose estimation accuracy. This includes: constructing scene features to enrich contextual information; decoupling the interaction relationship between targets and the scene into two categories: object-object interaction and object-scene interaction; and completing vehicle 6DoF pose estimation by designing the second attention module (i.e., a hierarchical interaction attention module).

[0045] Specifically, such as Figure 4 As shown, the input to this module is the target fusion feature. and scene features Unlike the construction method of target fusion features, scene features contain rich contextual information, originating from backbone feature maps and embedded through an embedding network. Scene features are obtained after processing ,in This is the dimension of scene features. We first use sub-networks... and To each and To enhance the expressive power of features, processing is performed. Traditional methods typically employ layer normalization to standardize feature distribution in order to improve training stability and optimization efficiency. However, this operation requires calculating the mean and variance of each neuron dimension, resulting in high computational cost. Therefore, we introduce a dynamic tanh function to eliminate the need for statistical calculations. Accordingly, the following definitions apply (Equations 10 and 11). Where, It is a learnable scaling factor used to dynamically adjust the scaling ratio based on the range of the input. and These are learnable scaling and translation parameters, used to fit the true input-output relationship. The final result is the preprocessed output. , .

[0046] For object-to-object interactions, this invention designs an object-level interaction-aware attention mechanism (OLIAA). Similarly, for object-to-scene interactions, we design a scene-level context-aware attention mechanism (SLCAA), calculated as follows (Equations 12 and 13). In Equation (12), It is mapped to queries, keys, values, and interaction information between modeling objects. In equation (13) It is mapped to a query, extracting rich contextual information from the scene. Mapped to key, value. For object-level interaction-aware attention scores, Context-aware attention scores are given at the scene level.

[0047] and The output is determined by learnable factors. and The method guides and fuses features at the feature layer, thereby improving the expressive power of pose estimation. Finally, this invention designs a translation regressor. With Rotary Regressor , are used to predict the vehicle's displacement and rotation information respectively (Equations 14 and 15). Where, Represents the three-dimensional translation of the target , Indicates the rotation angle of the target .

[0048] In this embodiment, in S5, a multi-level, multi-angle loss function is designed to supervise the training of the joint analysis model, and finally outputs an accurate shape and pose joint estimation result, including: constructing regression loss and 3D reconstruction loss; constructing target detection loss; constructing world frame-guided shape-pose joint loss to form the final loss function.

[0049] Specifically, regression loss and 3D reconstruction loss are constructed as shown in equations (16, 17). This invention defines the translation loss as follows: Rotational loss is .in, This represents the ground truth value of the translation vector. This represents the ground truth value of the rotation vector. Given that rotation pose regression is essentially a unimodal optimization problem, this invention limits the range of values ​​for each rotation axis to within... Within the range.

[0050] This invention uses L2 loss To monitor the accuracy of 3D shape reconstruction. Among them, Represents the true value of the grid ground.

[0051] Constructing the object detection loss: This invention extends Mask R-CNN. Therefore, it introduces RPN (Region Proposal Network) loss. 2D bounding box loss and 2D keypoint loss The detection loss is defined as the sum of their values, as in equation (19).

[0052] Construct a world-frame-guided shape-pose joint loss to form the final loss function: Let Rotation of the estimated object The corresponding rotation matrix. Using the rotation matrix and translation vector... We transform the predicted 3D mesh vertices in camera space to world space to calculate the 3D mesh loss in world coordinates, as shown in Equation (20).

[0053] Furthermore, to more clearly decouple the contributions of rotation and translation within the world framework to the final 3D prediction, we additionally consider the losses in rotation space and translation space. Specifically, we define the corresponding 3D mesh vertex representations in rotation space and translation space, respectively. and As shown in equations (21, 22). In summary, the world-frame guided shape-pose joint loss is defined as... As shown in (23). The loss function ultimately used to train the model. Defined as equation (24): where, These are the weighting coefficients for each loss function, used to balance the impact of different losses on model training.

[0054] The method proposed in this invention was tested on the public dataset ApolloCar3D, such as... Figure 4 As shown, comparative experiments of this invention with GSNet and BAAM are presented under various challenging road conditions. Red boxes represent shape failure cases, yellow boxes represent translation failure cases, white boxes represent rotation failure cases, and blue boxes represent successful cases. This invention efficiently and robustly completes the joint reconstruction of the vehicle's 3D shape and attitude. Figure 5 As shown, this invention uses four representative scenarios (reverse case, same-direction case, dense occlusion case, and turning case) to visually demonstrate the performance of the proposed method in shape reconstruction and pose estimation tasks, further verifying the robustness and efficiency of the model in complex real-world environments. For each scenario, the figure presents the input image, reconstruction result, and 3D output from a bird's-eye view (BEV), intuitively reflecting the model's performance under different perspectives and complex road conditions.

[0055] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The above descriptions are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for joint analysis of vehicle 3D shape and attitude integrating prior factors and multi-layer interactions, characterized in that, include: Step 1: Extract the feature map of the target image and convert it to camera space; Step 2: Extract 2D primitive features from the feature map and generate target fusion features; Step 3: Using prior factors and the target fusion features, construct the first attention module for vehicle 3D shape reconstruction in the joint analysis model; Step 4: Utilize target fusion features and scene features to construct a second attention module for vehicle 6DoF pose estimation in the joint analysis model; Step 5: Establish a multi-level, multi-angle loss function to supervise the training of the joint analysis model, and then use the trained joint analysis model to perform 3D shape and attitude joint analysis on the vehicle.

2. The method according to claim 1, characterized in that, Step one includes: Feature maps of the target image are obtained by using Mask R-CNN, which is an extension of the Res2Net network. The information of the acquired feature map in pixel space is transformed into camera space.

3. The method according to claim 2, characterized in that, The methods by which the information of the acquired feature maps in pixel space is transformed into camera space include: Will From pixel space conversion to The camera spatial approach is as follows: , , , ,in: The bounding box of the feature map is represented in pixel space as the coordinates of the object's center. , , These are the x and y coordinates of the object's center, corresponding to the width of the bounding box. The height is The camera's internal parameters are , Indicates camera spatial intrinsic parameters. , The values ​​on the x and y axes corresponding to the focal length per unit pixel, , ) represents the principal point of the image center, and Z represents the fixed scaling factor used to match the image shape.

4. The method according to claim 1, characterized in that, Step two includes: Extracting 2D primitive features from the feature map; A key point heatmap is constructed using the extracted 2D primitive features, and then target fusion features are generated based on the key point heatmap.

5. The method according to claim 4, characterized in that, The process of generating target fusion features includes: 2D primitive features are processed through an embedded network, where keypoint heatmaps are overlaid on the corresponding keypoints. The 2D primitive features include bounding boxes. Bounding box features 2D key points and key point heatmap , Represents the real number field; The processed 2D primitive features are stacked with the bounding box and its corresponding bounding box features to obtain the target fusion feature. ,in, , , This represents an embedding network that enhances the features of the bounding box. This represents an embedding network that enhances the features of the bounding box. This represents an embedding network that enhances the features of 2D keypoints. This represents an embedding network that enhances the features of a keypoint heatmap, where n represents the number of detected vehicles and c represents the fused feature dimension. This indicates a fully connected layer.

6. The method according to claim 4, characterized in that, Step three includes: Construct a prior set and obtain prior factors, wherein the prior set includes: prior features Prior average and prior offset Among them, a total of =79 prior shapes, each shape is composed of Composed of 1352 vertices, prior features are processed by a feature aggregation module to obtain prior factors. Where n represents the number of detected vehicles, and c represents the fused feature dimension. Represents the prior shape number. This represents the number of vertices for each prior shape; The 3D shape of the vehicle is reconstructed using an attention module driven by the acquired prior factors.

7. The method according to claim 6, characterized in that, Methods for reconstructing the 3D shape of a vehicle using an attention module driven by acquired prior factors include: Target features are fused through linear transformation. Mapped to query vector , prior features Mapped to key vectors respectively Sum value vector ; Calculate attention score ,in, and These are all intermediate representations of the attention score calculation process. , ; The reconstructed 3D shape of the vehicle is represented by M: .

8. The method according to claim 6, characterized in that, Step four includes: Constructing scene features Preprocessing is then performed, where n represents the number of detected vehicles and d is the dimension of the scene features; the result of the preprocessing is expressed as... and , , tanh() represents the dynamic tanh function; This represents two learnable scaling factors used to dynamically adjust the scaling ratio based on the range of the input. and These represent two learnable scaling parameters and two translation parameters, respectively. The interaction between the target and the scene is decoupled into two categories, including: object-level interaction-aware attention. and context-aware attention at the scene level The object-level interaction-aware attention score is represented as The scene-level context-aware attention score is represented as ; In the second attention module, through the translation regressor Predicting vehicle displacement PT using a rotational regressor Predict the vehicle's rotation PR; where, , , and For the corresponding and The learnable factor.

9. The method according to claim 1, characterized in that, The multi-level, multi-angle loss functions in step five include: regression loss function, 3D reconstruction loss function, object detection loss function, and world frame-guided shape-pose joint loss function; The translation loss in the regression loss subfunction is: , The ground truth value of the translation vector is represented by , and T represents the predicted translation vector. The rotational loss is: , R represents the ground truth value of the rotation vector, and R represents the predicted rotation vector. The 3D reconstruction loss function is: , Represents the ground truth of the mesh, where i represents the vertex number of the object's mesh. Let m represent the number of vertices for each prior shape, and m represent the network reconstruction result of the target. Represents the three-dimensional coordinate index of the i-th vertex; The target detection loss function is: , Represents the regional proposal network loss in mask r-cnn. Indicates the 2D bounding box loss. This indicates the loss of 2D key points.

10. The method according to claim 9, characterized in that, The world-frame-guided shape-pose joint loss function is: Among them, the 3D mesh loss in the world coordinate system , Rotational space loss , Translation space loss , The rotation matrix corresponding to the estimated object rotation. Represents the translation vector. This represents the actual rotation matrix corresponding to the object. Represents the actual translation vector corresponding to the object; The multi-level, multi-angle loss function is expressed as follows: ,in, These are the weighting coefficients for each loss function.

Images