Fusion prior factor and multi-layer interaction of vehicle 3D shape and posture joint analysis method

Abstract

The embodiment of the application discloses a vehicle 3D shape and pose joint analysis method fusing prior factors and multi-layer interaction, relates to the technical field of electric digital data processing, and can realize efficient and high-robust vehicle 3D shape and pose joint reconstruction analysis.The application comprises: converting input pictures to camera space, extracting 2D primitive features and further constructing target fusion features;an attention module driven by prior factors is used for vehicle 3D shape reconstruction, and effective aggregation of prior information is realized;an attention module of layered interaction is used for vehicle 6DoF pose estimation, and the interaction relationship between targets and the interaction relationship between targets and scenes are accurately modeled; a multi-level and multi-angle loss function is used for supervising training of the joint analysis model, and accurate shape and pose joint estimation results are output.The application is suitable for vehicle perception in an open and complex traffic environment.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction technology for computer graphics in general image data processing or generation techniques, and particularly to a method for joint analysis of vehicle 3D shape and attitude that integrates prior factors and multi-layer interactions. Background Technology

[0002] Vehicle 3D shape reconstruction refers to reconstructing the 3D shape of a vehicle simultaneously by regressing its 6DoF pose given an image. This technology has wide applications in fields with strong human-computer interaction, such as autonomous driving. However, the complex scenarios in real-world environments present significant challenges to this task. First, monocular 3D vehicle pose estimation is a typical ill-posed problem, primarily due to the lack of depth information and viewpoint limitations. Second, complex interactions within the scene significantly impact the final result. Finally, there remains a clear trade-off between the accuracy of 3D shape reconstruction and computational efficiency. Therefore, a joint analysis method for vehicle 3D shape and pose that integrates prior factors and multi-layered interactions is urgently needed to address these issues.

[0003] Early methods, largely based on geometric analysis, had significant limitations and were subsequently replaced by data-driven deep learning approaches. Existing methods have addressed some challenges. In the 6DoF (6 Degrees of Freedom Pose Estimation) pose regression task for vehicles, convolutional network-based methods utilize 3D bounding box annotations to obtain dense planar representations, mitigating the ill-conditioned nature of 2D-to-3D mapping. However, they neglect the impact of global context and inter-object interactions on the final regression results. In 3D shape reconstruction of vehicles, inspired by human shape reconstruction techniques, some studies have attempted to design principal component analysis (PCA) bases for vehicle shape reconstruction. However, limitations in detail capture are due to the linear assumption and sensitivity to noise. For example, existing solutions like GSNet and BAAM are problematic. GSNet mainly relies on PCA shape basis and geometric supervision to complete vehicle shape and pose estimation, but it lacks the ability to represent local vehicle details and does not adequately model the occlusion relationship between targets and the interaction relationship between targets and the road scene. Although BAAM introduces dual contextual attention and attention-guided modeling, its shape reconstruction process requires complex correlation calculations between target features and multiple shape priors, resulting in high computational overhead, and the modeling of rotation and translation information is relatively separate.

[0004] Therefore, how to balance the lack of depth information, the complex interaction between targets, and the need for efficient modeling in complex open scenarios to achieve efficient and robust joint reconstruction analysis of vehicle 3D shape and pose has become a problem that needs to be studied and solved. Summary of the Invention

[0005] The embodiments of the present invention provide a method for joint analysis of vehicle 3D shape and attitude that integrates prior factors and multi-layer interactions, which can achieve efficient and robust joint reconstruction analysis of vehicle 3D shape and attitude.

[0006] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0007] A method for joint analysis of vehicle 3D shape and attitude that integrates prior factors and multi-layer interactions includes:

[0008] Step 1: Extract the feature map of the target image and convert it to camera space;

[0009] Step 2: Extract 2D primitive features from the feature map and generate target fusion features;

[0010] Step 3: Utilize prior factors and target fusion features to construct the first attention module for vehicle 3D shape reconstruction in the joint analysis model;

[0011] Step 4: Utilize target fusion features and scene features to construct a second attention module for vehicle 6DoF pose estimation in the joint analysis model;

[0012] Step 5: Establish a multi-level, multi-angle loss function to supervise the training of the joint analysis model, and then use the trained joint analysis model to perform 3D shape and attitude joint analysis on the vehicle.

[0013] In computer vision-related analysis and modeling techniques, four main coordinate systems are typically established: world coordinate system, camera coordinate system, image physical coordinate system, and pixel coordinate system. This embodiment focuses on the transformation from the pixel coordinate system to the camera coordinate system. Specifically, step one includes: obtaining feature maps of the target image using Mask R-CNN extended from Res2Net; and transforming the information of the obtained feature maps in pixel space to camera space, including: From pixel space to The camera spatial approach is as follows: Where: the bounding box of the feature map is represented in pixel space as the coordinates of the object's center. , , These are the x and y values ​​of the object's center coordinates; the corresponding bounding box width is... The height is The camera's internal parameters are: , , The values ​​on the x and y axes corresponding to the focal length per unit pixel, , ) represents the principal point of the image center; z represents the fixed scaling factor used to match the shape of the image.

[0014] Specifically, step two includes: extracting 2D primitive features from the feature map; constructing a keypoint heatmap using the extracted 2D primitive features; and then generating target fusion features based on the keypoint heatmap. This involves processing the 2D primitive features through an embedding network, whereby the keypoint heatmap is overlaid onto the corresponding keypoints. The 2D primitive features include bounding boxes. Bounding box features 2D key points and key point heatmap The processed 2D primitive features are stacked with the bounding box and its corresponding bounding box features to obtain the target fusion feature. , , , It is an embedding network that enhances the features of the bounding box. It is an embedding network that enhances the features of the bounding box. It is an embedding network that enhances the features of 2D keypoints. It is an embedding network that enhances the features of keypoint heatmaps, where n represents the number of detected vehicles and c represents the fused feature dimension. Represents the real number field. Indicates a fully connected layer. Operators that concatenate variables along their channel dimensions.

[0015] Specifically, step three includes: constructing a prior set and obtaining prior factors. The prior set includes: prior features. Prior average and prior offset Among them, a total of =79 prior shapes, each consisting of v=1352 vertices, and prior factors are obtained by processing the prior features through a feature aggregation module. n represents the number of detected vehicles; c represents the fused feature dimension; v represents the number of vertices in each prior shape; 3D shape reconstruction of vehicles is performed using an attention module driven by the acquired prior factors. Specifically, target features are fused through linear transformation. Mapped to query vector , prior features Mapped to key vectors respectively Sum value vector ;

[0016] Calculate attention score ,in, , The reconstructed 3D shape of the vehicle is represented by M: .

[0017] Specifically, step four includes: constructing scene features. Preprocessing is then performed, where d is the dimension of the scene features; the result of the preprocessing is expressed as... and , , tanh() represents the dynamic tanh function. It is a learnable scaling factor used to dynamically adjust the scaling ratio based on the range of the input; and These are learnable scaling and translation parameters, used to fit the true input-output relationship.

[0018] The interaction between the target and the scene is decoupled into two categories, including: object-level interaction-aware attention. and context-aware attention at the scene level The object-level interaction-aware attention score is represented as Scene-level context-aware attention scores are represented as In the second attention module, a translation regressor is used. Predicting vehicle displacement via rotational regressor Predict vehicle rotation ;in, , , and For the corresponding and The learnable factor.

[0019] Specifically, the multi-level, multi-angle loss function in step five includes: regression loss function, 3D reconstruction loss function, object detection loss function, and world frame-guided shape-pose joint loss function;

[0020] The translation loss in the regression loss subfunction is: , This represents the ground truth value of the translation vector. This represents the predicted translation vector. This indicates the calculation of the L1 norm;

[0021] The rotational loss is: , Represents the ground truth value of the rotation vector. Represents the predicted rotation vector;

[0022] The 3D reconstruction loss function is: , Represents the ground truth of the mesh, where i represents the vertex number of the object mesh, and v represents the number of vertices for each prior shape. This represents the network reconstruction result of the target;

[0023] The target detection loss function is: , Represents the regional proposal network loss in mask r-cnn. Indicates the 2D bounding box loss. This indicates the loss of 2D key points.

[0024] The world-frame-guided shape-pose joint loss function is: Among them, the 3D mesh loss in the world coordinate system ,

[0025] Rotational space loss ,

[0026] Translation space loss , The rotation matrix corresponding to the estimated object rotation. This represents the actual rotation matrix corresponding to the object. Represents the actual translation vector corresponding to the object;

[0027] The multi-level, multi-angle loss function is expressed as follows: ,in, They represent the corresponding The weighting coefficients.

[0028] The present invention provides a method for joint analysis of vehicle 3D shape and pose that integrates prior factors and multi-layered interactions. By integrating prior factors to drive an attention module and comprehensively considering the interaction between the target and the scene, it achieves efficient and robust joint reconstruction of vehicle 3D shape and pose. The invention includes: extracting and locating target bounding boxes from the input image and converting them to camera space; extracting 2D primitive features and further constructing target fusion features; designing a prior factor-driven attention module for vehicle 3D shape reconstruction, which significantly improves shape reconstruction accuracy and reduces computational cost through effective aggregation of prior information; designing a hierarchical interactive attention module for vehicle 6DoF pose estimation, accurately modeling the interaction relationships between targets and between the target and the scene, and improving pose estimation accuracy; and designing a multi-level, multi-angle loss function to supervise the training of the joint analysis model, ultimately outputting accurate shape and pose joint estimation results, thereby achieving efficient and robust joint reconstruction analysis of vehicle 3D shape and pose. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the overall framework process provided for an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of a priori factor-driven attention module provided in an embodiment of the present invention;

[0032] Figure 3 A schematic diagram of a hierarchical interaction attention module provided in an embodiment of the present invention;

[0033] Figure 4 A schematic diagram illustrating the comparison with existing algorithms on the ApolloCar3D dataset provided in this embodiment of the invention;

[0034] Figure 5 This is a schematic diagram illustrating the effect of the present invention on the ApolloCar3D dataset provided in this embodiment of the invention;

[0035] Figure 6 This is a schematic diagram of the method flow provided in an embodiment of the present invention; Detailed Implementation

[0036] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Embodiments of the present invention will be described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in the specification of the present invention means the presence of features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when an element is referred to as “connected” or “coupled” to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein may include wireless connections or couplings. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0037] The formula model involved in the specific implementation is shown below.

[0038]

[0039]

[0040] Embodiments of the present invention provide a method for joint analysis of vehicle 3D shape and attitude that integrates prior factors and multi-layer interactions, such as... Figure 6 As shown, it includes:

[0041] S1. For the input image, extract and locate the target bounding box, and convert it to camera space;

[0042] S2. Extract 2D primitive features and further construct target fusion features;

[0043] S3. Design a first attention module (i.e., an attention module driven by prior factors) for vehicle 3D shape reconstruction. By effectively aggregating prior information, the shape reconstruction accuracy is significantly improved and the computational cost is reduced.

[0044] S4. Design a second attention module (i.e., a hierarchical interactive attention module) for vehicle 6DoF pose estimation, accurately model the interaction relationships between targets and between targets and the scene, and improve the pose estimation accuracy.

[0045] S5. Design multi-level, multi-angle loss functions to supervise the training of the joint analysis model, and finally output accurate joint estimation results of shape and pose.

[0046] In this embodiment, S1 includes: extending Mask R-CNN through ReS2Net network to obtain feature maps of the target image; and converting information in pixel space to camera space.

[0047] Specifically, such as Figure 1 , 2 As shown, Mask R-CNN is extended using a ReS2Net backbone network to obtain feature maps of the target image. The bounding box is represented in pixel space as the coordinates of the object's center. and the width of the corresponding bounding box and height Camera intrinsics are defined as follows: ,in , Focal length per unit pixel, ( , The principal point is the center of the image. A fixed scaling factor is introduced. To match the shape of the image, From pixel space to In camera space, as shown in formulas (1)-(4).

[0048] In this embodiment, S2 includes: processing the feature map through RoIAlign to obtain bounding box features; obtaining 2D key points by extending the detection branch of Mask R-CNN; designing an embedded network to regress a key point heatmap; and fusing the bounding box, bounding box features, 2D key points, and key point heatmap to construct target fusion features.

[0049] Specifically, such as Figure 1 As shown, to obtain the appearance features of vehicle instances, RoIAlign is used to process the bounding boxes, thereby obtaining bounding box features. Keypoints not only provide relative position and size information within the instance but also reflect vehicle shape information. This is crucial for shape estimation because it involves the local structure of the vehicle. Considering global information, this invention introduces an embedded network to regress a 2D keypoint heatmap to more accurately represent the visibility of keypoints, helping the network determine whether a local area of ​​the instance is occluded.

[0050] Finally, the corresponding 2D primitive features are processed through an embedding network, and keypoint heatmaps are overlaid onto the corresponding keypoints to enhance instance information. These heatmaps are then stacked with bounding boxes and their corresponding bounding box features through two fully connected layers. Complete target fusion features The construction of the model further enhances its ability to recognize objects and estimate poses, as shown in formula (5). Wherein, It is a heatmap of key points. For 2D key point information, For bounding box information, This refers to the bounding box feature.

[0051] In this embodiment, in S3, a first attention module (i.e., an attention module driven by prior factors) is designed for vehicle 3D shape reconstruction. The shape reconstruction accuracy is significantly improved and the computational cost is reduced by effectively aggregating prior information. This includes: constructing a prior set to obtain prior factors; and completing the vehicle 3D shape reconstruction through the attention module mechanism driven by prior factors.

[0052] Specifically, such as Figure 3 As shown, the prior set consists of three parts: prior features, prior average, and prior bias. Among them, the prior features... Prior factors will be obtained through feature aggregation module. .use =79 prior shapes, each consisting of v=1352 vertices. Therefore, the prior average is expressed as The prior offset is expressed as .

[0053] Target features are fused through linear transformation. Mapped to query vector , prior features Mapped to key vectors respectively value vector Simultaneously, prior features are processed by the feature aggregation module to obtain prior factors. To avoid full computation between the query and the key. As in formulas (6) and (7), where, , These are all intermediate representations of the attention score calculation process. Further combined... Calculate attention score For example (8). To prevent the loss of target feature information during forward propagation, the present invention sets up skip connections to complete the reconstruction of the vehicle shape. For example (9).

[0054] In this embodiment, in S4, a second attention module (i.e., a hierarchical interaction attention module) is designed for vehicle 6DoF pose estimation to accurately model the interaction relationships between targets and between targets and the scene, thereby improving the pose estimation accuracy. This includes: constructing scene features to enrich contextual information; decoupling the interaction relationship between targets and the scene into two categories: object-object interaction and object-scene interaction; and completing vehicle 6DoF pose estimation by designing the second attention module (i.e., a hierarchical interaction attention module).

[0055] Specifically, such as Figure 4 As shown, the input to this module is the target fusion feature. and scene features Unlike the construction method of target fusion features, scene features contain rich contextual information, originating from backbone feature maps and embedded through an embedding network. Scene features are obtained after processing ,in This refers to the dimension of scene features. First, through sub-networks... and To each and To enhance the expressive power of features, processing is performed. Traditional methods typically employ layer normalization to standardize feature distribution in order to improve training stability and optimization efficiency. However, this operation requires calculating the mean and variance of each neuron dimension, resulting in high computational cost. Therefore, a dynamic tanh function is introduced to eliminate the need for statistical calculations. Accordingly, the following definitions exist (Equations 10 and 11). Where, It is a learnable scaling factor used to dynamically adjust the scaling ratio based on the range of the input. and These are learnable scaling and translation parameters, used to fit the true input-output relationship. The final result is the preprocessed output. , .

[0056] For object-to-object interactions, this invention designs an object-level interaction-aware attention mechanism (OLIAA). Similarly, for object-to-scene interactions, a scene-level context-aware attention mechanism (SLCAA) is designed, with the corresponding calculations as follows (Equations 12 and 13). In Equation (12), This is mapped to queries, keys, values, and interaction information between modeling objects. In equation (13) It is mapped to a query, extracting rich contextual information from the scene. Mapped to key, value. For object-level interaction-aware attention scores, Context-aware attention scores are given at the scene level.

[0057] and The output is determined by learnable factors. and The method guides and fuses features at the feature layer, thereby improving the expressive power of pose estimation. Finally, this invention designs a translation regressor. With Rotary Regressor , are used to predict the vehicle's displacement and rotation information respectively (Equations 14 and 15). Where, Represents the three-dimensional translation of the target , Indicates the rotation angle of the target .

[0058] In this embodiment, in S5, a multi-level, multi-angle loss function is designed to supervise the training of the joint analysis model, and finally outputs an accurate shape and pose joint estimation result, including: constructing regression loss and 3D reconstruction loss; constructing target detection loss; constructing world frame-guided shape-pose joint loss to form the final loss function.

[0059] Specifically, regression loss and 3D reconstruction loss are constructed as shown in equations (16, 17). This invention defines the translation loss as follows: Rotational loss is .in, Represents the ground truth value of the rotation vector. This represents the predicted rotation vector. Given that rotation pose regression is essentially a unimodal optimization problem, this invention limits the range of values ​​for each rotation axis to within... Within the range.

[0060] This invention uses L2 loss To monitor the accuracy of 3D shape reconstruction. Among them, Represents the true value of the grid ground.

[0061] Constructing the object detection loss: This invention extends Mask R-CNN. Therefore, it introduces RPN (Region Proposal Network) loss. 2D bounding box loss and 2D keypoint loss The detection loss is defined as the sum of their values, as in equation (19).

[0062] Construct a world-frame-guided shape-pose joint loss to form the final loss function: Let The rotation matrix corresponding to the estimated object rotation. This represents the actual rotation matrix corresponding to the object. The true translation vector corresponding to the object is used to transform the predicted 3D mesh vertices in camera space to world space, and is used to calculate the 3D mesh loss in the world coordinate system, as shown in Equation (20).

[0063] Furthermore, to more clearly decouple the contributions of rotation and translation within the world frame to the final 3D prediction, additional losses in rotation space and translation space are considered, as shown in equations (21, 22). In summary, the world frame-guided shape-pose joint loss is defined as... As shown in (23). The loss function ultimately used to train the model. Defined as equation (24): where, They represent the corresponding The weighting coefficients are used to balance the impact of different losses on model training.

[0064] The main differences between this embodiment and the prior art, as well as its improvements in practical effect, can be demonstrated through testing. For example:

[0065] The proposed method was tested on the public dataset ApolloCar3D and compared with GSNet and BAAM. GSNet: Joint Vehicle Pose and Shape Reconstruction with Geometrical and Scene-aware Supervision (GSNet); BAAM: Monocular 3D Pose and Shape Reconstruction with Bi-contextual Attention Module and Attention-guided Modeling (BAAM). From a methodological design perspective, GSNet mainly relies on PCA shape basis and geometric supervision to complete vehicle shape and pose estimation, but its ability to represent local vehicle details is insufficient, and its modeling of occlusion relationships between targets and interaction relationships between targets and the road scene is inadequate. Although BAAM introduces bi-contextual attention and attention-guided modeling, its shape reconstruction process requires complex correlation calculations between target features and multiple shape priors, resulting in high computational overhead, and its rotation and translation information modeling is relatively separate. In contrast, this invention introduces key point heatmaps, prior factor aggregation, object-level interaction awareness, scene-level context awareness, and dynamic tanh feature enhancement mechanisms, which can supplement the occlusion visibility features, prior compression features, and hierarchical interaction features missing in GSNet and BAAM, resulting in a more complete method structure.

[0066] like Figure 4 As shown, comparative experiments of this invention with GSNet and BAAM under various challenging road conditions are presented. Red boxes represent shape failure cases, yellow boxes represent translation failure cases, white boxes represent rotation failure cases, and blue boxes represent successful cases. The results show that in complex traffic scenarios such as strong light, turning, and dense occlusion, GSNet is prone to insufficient shape detail recovery or translational deviation. BAAM still suffers from rotation estimation failure or unstable vehicle reconstruction due to occlusion in some scenarios. This invention, however, can more stably complete the joint reconstruction of vehicle 3D shape and 6DoF pose. Quantitative results show that the A3DP-Abs mean and A3DP-Rel mean of GSNet are 18.91 and 20.21, respectively, while those of BAAM are 25.19 and 22.85, respectively. This invention achieves 25.66 and 24.28, respectively. The average inference time per image is reduced from 0.45s for GSNet and 0.42s for BAAM to 0.25s, indicating that this invention improves both reconstruction accuracy and inference efficiency. Figure 5 As shown, the present invention further demonstrates the input image, reconstruction result and bird's-eye view 3D output through four typical scenarios: reverse, same direction, dense occlusion and turning, verifying the robustness and efficiency of the present invention in complex real traffic environments.

[0067] In summary, this embodiment transforms the input image into camera space, extracts 2D primitive features, and further constructs target fusion features. A prior-factor-driven attention module is used for vehicle 3D shape reconstruction, effectively aggregating prior information. A hierarchical interactive attention module is used for vehicle 6DoF pose estimation, accurately modeling the interaction relationships between targets and between targets and the scene. Multi-level, multi-angle loss functions supervise the training of the joint analysis model, outputting accurate shape and pose joint estimation results. This solution is particularly suitable for vehicle perception in open and complex traffic environments, supplementing the occlusion visibility features, prior compression features, and hierarchical interactive features missing in GSNet and BAAM.

[0068] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The above descriptions are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for joint analysis of vehicle 3D shape and attitude integrating prior factors and multi-layer interactions, characterized in that, include: Step 1: Extract the feature map of the target image and convert it to camera space; Step 2: Extract 2D primitive features from the feature map and generate target fusion features; Step 3: Using prior factors and the target fusion features, construct the first attention module for vehicle 3D shape reconstruction in the joint analysis model; Step 4: Utilize target fusion features and scene features to construct a second attention module for vehicle 6DoF pose estimation in the joint analysis model; Step 5: Establish a multi-level, multi-angle loss function to supervise the training of the joint analysis model, and then use the trained joint analysis model to perform 3D shape and attitude joint analysis on the vehicle.

2. The method according to claim 1, characterized in that, Step one includes: Feature maps of the target image are obtained by using Mask R-CNN, which is an extension of the Res2Net network. The information of the acquired feature map in pixel space is transformed into camera space.

3. The method according to claim 2, characterized in that, The methods by which the information of the acquired feature maps in pixel space is transformed into camera space include: Will From pixel space to The camera spatial approach is as follows: , , , ,in: The bounding box of the feature map is represented in pixel space as the coordinates of the object's center. , , These are the x and y coordinates of the object's center, corresponding to the width of the bounding box. The height is The camera's internal parameters are , Indicates camera spatial intrinsic parameters. , The values ​​on the x and y axes corresponding to the focal length per unit pixel, , ) represents the principal point of the image center, and z represents the fixed scaling factor used to match the image shape.

4. The method according to claim 1, characterized in that, Step two includes: Extracting 2D primitive features from the feature map; A key point heatmap is constructed using the extracted 2D primitive features, and then target fusion features are generated based on the key point heatmap.

5. The method according to claim 4, characterized in that, The process of generating target fusion features includes: 2D primitive features are processed through an embedded network, where keypoint heatmaps are overlaid on the corresponding keypoints. The 2D primitive features include bounding boxes. Bounding box features 2D key points and key point heatmap , Represents the real number field; The processed 2D primitive features are stacked with the bounding box and its corresponding bounding box features to obtain the target fusion feature. ,in, , , This represents an embedding network that enhances the features of the bounding box. This represents an embedding network that enhances the features of the bounding box. This refers to an embedding network that enhances the features of 2D keypoints. This represents an embedding network that enhances the features of a keypoint heatmap, where n represents the number of detected vehicles and c represents the fused feature dimension. Indicates a fully connected layer. Operators that concatenate variables along their channel dimensions.

6. The method according to claim 4, characterized in that, Step three includes: Construct a prior set and obtain prior factors, wherein the prior set includes: prior features Prior average and prior offset Among them, a total of =79 prior shapes, each consisting of v=1352 vertices, and prior factors are obtained by processing the prior features through a feature aggregation module. Where n represents the number of detected vehicles, and c represents the fused feature dimension. represents the number of prior shapes, and v represents the number of vertices for each prior shape; The 3D shape of the vehicle is reconstructed using an attention module driven by the acquired prior factors.

7. The method according to claim 6, characterized in that, Methods for reconstructing the 3D shape of a vehicle using an attention module driven by acquired prior factors include: Target features are fused through linear transformation. Mapped to query vector , prior features Mapped to key vectors respectively Sum value vector ; Calculate attention score ,in, , ; The reconstructed 3D shape of the vehicle is represented by M: , Operators that represent matrix multiplication.

8. The method according to claim 6, characterized in that, Step four includes: Constructing scene features Preprocessing is then performed, where n represents the number of detected vehicles and d is the dimension of the scene features; the result of the preprocessing is expressed as... and , , tanh() represents the dynamic tanh function; This represents two learnable scaling factors used to dynamically adjust the scaling ratio based on the range of the input. and These represent two learnable scaling parameters and two translation parameters, respectively. The interaction between the target and the scene is decoupled into two categories, including: object-level interaction-aware attention. and context-aware attention at the scene level The object-level interaction-aware attention score is represented as Scene-level context-aware attention scores are represented as In the second attention module, a translation regressor is used. Predicting vehicle displacement via rotational regressor Predict vehicle rotation ;in, , , and For the corresponding and The learnable factor.

9. The method according to claim 1, characterized in that, The multi-level, multi-angle loss functions in step five include: regression loss function, 3D reconstruction loss function, object detection loss function, and world frame-guided shape-pose joint loss function; The translation loss in the regression loss subfunction is: , This represents the ground truth value of the translation vector. This represents the predicted translation vector. This indicates the calculation of the L1 norm; The rotational loss is: , Represents the ground truth value of the rotation vector. Represents the predicted rotation vector; The 3D reconstruction loss function is: , Represents the ground truth of the mesh, where i represents the vertex number of the object mesh, and v represents the number of vertices for each prior shape. This represents the network reconstruction result of the target; The target detection loss function is: , Represents the regional proposal network loss in mask r-cnn. Indicates the 2D bounding box loss. This indicates the loss of 2D key points.

10. The method according to claim 9, characterized in that, The world-frame-guided shape-pose joint loss function is: Among them, the 3D mesh loss in the world coordinate system , Rotational space loss , Translation space loss , The rotation matrix corresponding to the estimated object rotation. This represents the actual rotation matrix corresponding to the object. Represents the actual translation vector corresponding to the object; The multi-level, multi-angle loss function is expressed as follows: ,in, They represent the corresponding The weighting coefficients.

Images