Monocular 3D vehicle reconstruction method, system, device and medium

Abstract

The invention relates to the technical field of computer vision and artificial intelligence, and discloses a monocular 3D vehicle reconstruction method, system and device and a medium, and the method comprises the steps: inputting a single RGB image containing a plurality of target vehicles, and extracting a multi-scale image feature pyramid through a backbone architecture; extracting the primitive information of the target vehicle from the image feature pyramid in parallel based on the detection head, and encoding the primitive information into a feature vector taking the object as the center; based on the image feature pyramid, constructing a multi-scale scene context by using a layered double-context attention module, and fusing with the feature vector to perform pose estimation to obtain a translation estimation feature; inputting the feature vector into a bounded iterative mesh refiner, and gradually recovering the 3D shape of the target vehicle to obtain a reconstructed mesh; and obtaining 3D rotation parameters based on feature vector regression, and outputting corresponding pose information and 3D shape of the target vehicle in combination with the translation estimation feature and the reconstructed grid. According to the invention, high-precision and geometrically consistent vehicle 3D reconstruction can be realized under a monocular view angle.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a monocular 3D vehicle reconstruction method, system, device and medium. Background Technology

[0002] 3D vehicle reconstruction is a fundamental task in computer vision and a key step in achieving comprehensive scene understanding for autonomous driving systems. In autonomous driving, it provides accurate 3D perception information for safe navigation, motion planning, and obstacle avoidance; in intelligent transportation systems, it facilitates fine-grained analysis of traffic flow and vehicle behavior; and in virtual simulation testing, it enables the construction of high-fidelity testing environments by reconstructing real-world vehicle models. While high-precision 3D perception can be achieved using active sensors such as LiDAR or stereo cameras, these solutions face challenges including high hardware costs, complex calibration, and limitations imposed by specific operating conditions (such as sparse point clouds at long distances). Therefore, with the advancement of deep learning technology, developing robust, low-cost, and easily deployable monocular 3D reconstruction methods has become a research hotspot in both academia and industry.

[0003] Monocular 3D vehicle reconstruction aims to recover the translation, rotation, and detailed geometry of a vehicle in three-dimensional space from a two-dimensional image plane. This technology is crucial for constructing three-dimensional world models under passive sensing conditions lacking depth information. However, the monocular 3D reconstruction task itself is an inherently ill-posed problem, facing many inherent challenges: First, a single image lacks explicit depth cues, leading to an infinite number of possible interpretations of the three-dimensional scene when back-projected from a two-dimensional image, resulting in severe geometric ambiguity; second, in autonomous driving scenarios, target vehicles are often far away, occupying very few pixels in the image, resulting in limited and blurry visual cues; third, the 3D shape of a vehicle is complex and variable, and regressing 3D mesh vertices based solely on two-dimensional features is prone to topological errors.

[0004] To address these challenges, existing technologies have explored various solutions. Early methods primarily relied on strong geometric priors, such as DeepMANTA and 3D-RCNN, to recover 3D information by matching observed vehicles with predefined CAD model libraries or PCA shape subspaces. Subsequently, methods like GSNet introduced multi-path feature fusion mechanisms, attempting to jointly infer pose and shape. More recent methods, such as BAAM, utilize dual contextual attention mechanisms, explicitly integrating relationships between objects and scene-level contextual cues, thus improving reconstruction performance to some extent.

[0005] Despite some progress, existing methods still have significant limitations in handling complex scenes, such as distant targets, and in fine-grained shape recovery. In pose estimation, current methods often fail to effectively address the problem of multi-scale feature aggregation. They typically use separate feature streams and lack effective strategies for dynamically fusing fine-grained local features with global scene semantics. This scale-dependent degradation makes it difficult to accurately estimate depth and pose when dealing with distant objects due to the sparsity of visual information. In shape reconstruction, existing models generally employ rigid, one-step deformation strategies, directly mapping image features to shape priors. This approach cannot perform local geometric corrections, and when prediction errors occur, the reconstructed vehicle shape often exhibits severe structural holes, topological artifacts, or geometric inconsistencies, lacking physical realism.

[0006] Therefore, how to achieve high-precision and geometrically consistent 3D reconstruction of vehicles from a monocular perspective is a key problem that urgently needs to be solved in this field. Summary of the Invention

[0007] To address the aforementioned issues, this invention proposes a monocular 3D vehicle reconstruction method, system, device, and medium. It resolves geometric ambiguity through dynamic context adaptation and establishes a progressive refinement mechanism to eliminate shape holes, thereby achieving high-precision, geometrically consistent 3D vehicle reconstruction from a monocular perspective.

[0008] The technical solution adopted in this invention is as follows: A monocular 3D vehicle reconstruction method includes: The input is a single RGB image containing multiple target vehicles. A multi-scale image feature pyramid is extracted through a backbone architecture. Based on the detection head, the primitive information of the target vehicles is extracted from the image feature pyramid in parallel, and the primitive information is encoded into an object-centered feature vector. Based on the image feature pyramid, a multi-scale scene context is constructed using a hierarchical dual-context attention module and fused with the feature vector to obtain translation estimation features; The feature vectors are input into a bounded iterative mesh refiner to progressively recover the 3D shape of the target vehicle and obtain the reconstructed mesh. Based on the feature vector regression, the 3D rotation parameters are obtained, and combined with the translation estimation features and the reconstructed mesh, the corresponding pose information and 3D shape of the target vehicle are output.

[0009] Furthermore, based on the image feature pyramid, a multi-scale scene context is constructed using a hierarchical dual-context attention module, and fused with the feature vector to obtain translation estimation features, including: Multi-level features are obtained from the image feature pyramid. The high-resolution feature map is downsampled to a uniform resolution by bilinear interpolation. After being stitched together in the channel dimension, the channel redundancy is compressed by a 1×1 convolutional layer to generate a uniform scene feature. The unified scene features are processed using a gated dual-path attention mechanism, which splits the unified scene features into local detail paths and global gated paths, and the outputs of the two paths are fused to obtain enhanced scene features. The enhanced scene features are subjected to dual attention interaction and then weighted and fused into the feature vector through residual connections and a learnable scaling matrix to generate translational estimation features.

[0010] Furthermore, the local detail path preserves fine-grained spatial patterns through convolutional blocks; the global gated path extracts scale-invariant global statistical information and generates a dynamic gating signal, which modulates the feature stream, including: Calculate the spatial max-pooling summary and channel variance of the unified scene features; The spatial max-pooling summary and channel variance are processed by a 1×1 convolutional layer, where the weights are learnable parameters. After GELU activation of the convolution result, a dynamic gating signal is obtained through upsampling. The dynamic gating signal modulates the feature flow of the global gating path through element-wise multiplication.

[0011] Furthermore, dual attention interaction is performed on the enhanced scene features, including: Based on inter-object relational attention, spatial dependencies between vehicle instances are modeled through a multi-head self-attention mechanism; Based on scene-object interaction attention, cross-attention calculation is performed using the feature vector as the query and the enhanced scene features as the key and value.

[0012] Furthermore, in the bounded iterative mesh refiner, the 3D shape of the target vehicle is progressively recovered to obtain a reconstructed mesh, including: The feature vector is interacted with a learnable shape concept by passing it through multiple multi-head attention layers to generate enhanced shape features; The basic mesh is generated through the first branch: the enhanced shape features are mapped to deformation coefficients through a fully connected layer, and the deformation basis of the predefined average shape template is weighted and combined to obtain the basic mesh; The second branch performs iterative offset refinement: the coarse offset is initialized, and after T iterations, the enhanced shape feature and the offset of the previous step are received as input in each iteration. The vertex position adjustment is predicted by a multilayer perceptron with shared weights, and the vertex update amplitude is limited by the Tanh bounded correction mechanism to obtain the final offset. The reconstructed mesh is generated by dynamically weighting and fusing the base mesh and the final offset using two learnable scalar weights.

[0013] Furthermore, the adoption of the Tanh bounded correction mechanism to limit the vertex update magnitude includes:

[0014] in, Let be the offset in the t-th iteration. This is the offset for the (t-1)th iteration. Let be the vertex position adjustment amount in the t-th iteration. The tanh activation function limits the vertex update magnitude to the range of (-0.5, 0.5).

[0015] Furthermore, the method employs a multi-task loss function for end-to-end training, with the total loss being a weighted sum of the losses from each subtask:

[0016] in, Total loss; For 2D detection loss, including classification, bounding box regression, and mask segmentation loss; For translational regression loss, the L1 norm is used for XY plane coordinates, and the uncertainty-aware L1 norm is used for depth estimation. For rotational losses, periodic L1 losses are used to handle angular discontinuities; To predict the average L2 distance between grid vertices and real grid vertices; This represents a loss of consistency in 3D space. , , , and These are the weighting coefficients.

[0017] A monocular 3D vehicle reconstruction system, comprising: The preprocessing module is configured to take a single RGB image containing multiple target vehicles as input, extract a multi-scale image feature pyramid through a backbone architecture, extract primitive information of the target vehicles in parallel from the image feature pyramid based on the detection head, and encode the primitive information into object-centered feature vectors. The hierarchical dual-context attention module is configured to construct a multi-scale scene context based on the image feature pyramid and fuse it with the feature vector to obtain translation estimation features; A bounded iterative mesh refiner is configured to progressively recover the 3D shape of the target vehicle based on the eigenvectors to obtain a reconstructed mesh; The 3D reconstruction module is configured to obtain 3D rotation parameters based on the feature vector regression, and combine the translation estimation features with the reconstruction mesh to output the corresponding pose information and 3D shape of the target vehicle.

[0018] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the monocular 3D vehicle reconstruction method.

[0019] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the monocular 3D vehicle reconstruction method.

[0020] The beneficial effects of this invention are as follows: This invention dynamically fuses multi-scale scene semantics and object geometric cues through deep learning networks and employs a progressive optimization strategy to finely reconstruct the 3D mesh. It is particularly suitable for monocular vehicle 3D pose estimation and shape recovery in challenging visual scenarios (such as distant small targets, complex background interference, and occlusion environments). It can be applied to autonomous driving environment perception, intelligent traffic monitoring, autonomous navigation planning, and virtual traffic simulation, etc. The specific description is as follows.

[0021] 1. This invention proposes a hierarchical dual contextual attention (HDCA) module, which achieves dynamic fusion of multi-scale features through a gated dual-path attention (GDA) mechanism. Compared to existing methods that rely solely on coarse feature flows, this invention can adaptively integrate fine-grained local geometric cues with global scene semantics according to the target scale. This significantly solves the problem of depth scale ambiguity and greatly improves the accuracy of 3D translation.

[0022] 2. This invention designs a Bounded Iterative Mesh Refiner (BIMR), which uses the Tanh bounded iterative correction mechanism to replace the rigid single-step deformation strategy in existing technologies. This progressive optimization method can effectively constrain the displacement range of vertices, eliminate structural holes and topological artifacts that often occur in existing methods, thereby generating physically reasonable, smooth, and geometrically consistent 3D vehicle meshes.

[0023] 3. For the highly challenging long-distance targets in autonomous driving scenarios, the HDCA module of this invention compensates for the lack of object features by introducing scene-level context information. Experiments show that the rotation and translation errors of this invention when processing long-distance vehicles are significantly lower than those of existing technologies, demonstrating its robustness in complex long-distance scenarios.

[0024] 4. Experiments demonstrate that this invention achieves superior performance metrics compared to state-of-the-art methods on the large public benchmark dataset ApolloCar3D. Furthermore, compared to baseline models, this invention adds only a minimal number of parameters and a small amount of GPU memory usage, maintaining high-precision reconstruction while sustaining efficient inference speed, thus possessing significant practical application value. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a monocular 3D vehicle reconstruction method according to Embodiment 2 of the present invention.

[0026] Figure 2 This is a schematic diagram of the gated dual-path attention mechanism in Embodiment 2 of the present invention.

[0027] Figure 3 This is a schematic diagram of the bounded iterative mesh refiner of Embodiment 2 of the present invention. Detailed Implementation

[0028] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments are now described. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention; that is, the described embodiments are only a part of the embodiments of the invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0029] Example 1 This embodiment provides a monocular 3D vehicle reconstruction method, including: The input is a single RGB image containing multiple target vehicles. A multi-scale image feature pyramid is extracted through the backbone architecture. Based on the detection head, the primitive information of the target vehicles is extracted from the image feature pyramid in parallel, and the primitive information is encoded into object-centered feature vectors. Based on the image feature pyramid, a multi-scale scene context is constructed using a hierarchical dual-context attention module and fused with the feature vector to obtain translation estimation features; The feature vectors are input into a bounded iterative mesh refiner to progressively recover the 3D shape of the target vehicle and obtain the reconstructed mesh. The 3D rotation parameters are obtained based on eigenvector regression, and combined with translation estimation features and reconstructed mesh, the corresponding pose information and 3D shape of the target vehicle are output.

[0030] It should be noted that this method can achieve end-to-end processing from a single 2D image to 3D vehicle reconstruction. Multi-scale feature extraction ensures the comprehensiveness of information, parallel primitive information extraction improves processing efficiency, and the division of labor and cooperation between the hierarchical dual-context attention module and the bounded iterative mesh refiner not only ensures the accuracy of pose estimation but also achieves fine restoration of 3D shape, thus improving the overall performance of monocular 3D vehicle reconstruction.

[0031] Preferably, in the hierarchical dual-context attention module, a multi-scale scene context is constructed based on the image feature pyramid and fused with the feature vector to obtain translation estimation features. This includes: obtaining multi-level features from the image feature pyramid; downsampling the high-resolution feature map to a uniform resolution using bilinear interpolation; concatenating the features along the channel dimension and compressing channel redundancy through a 1×1 convolutional layer to generate uniform scene features; processing the uniform scene features using a gated dual-path attention mechanism, splitting them into local detail paths and global gated paths, and fusing the outputs of the two paths to obtain enhanced scene features; performing dual attention interaction on the enhanced scene features and weighting and fusing them into the feature vector through residual connections and a learnable scaling matrix to generate translation estimation features.

[0032] Specifically, firstly, feature maps at multiple key levels are selected from the image feature pyramid. For the feature maps with higher resolution, bilinear interpolation is used for downsampling to ensure that all selected feature maps achieve the same resolution. Then, these feature maps of the same resolution are concatenated along the channel dimension to obtain a fusion feature with a large number of channels. The fusion feature is then compressed through a 1×1 convolutional layer to remove redundant information and generate a unified scene feature with appropriate dimensions. Next, a gated dual-path attention mechanism is activated, inputting the unified scene feature into the local detail path and the global gated path respectively. The local path retains fine-grained spatial information in the feature through convolutional blocks, while the global path extracts global statistical information and generates a dynamic gated signal. After modulating the feature stream, the outputs of the two paths are fused to obtain the enhanced scene feature. Subsequently, the enhanced scene feature is subjected to dual attention interaction. On the one hand, the spatial dependency relationship between vehicle instances is modeled, and on the other hand, cross-attention calculation is performed with the object-centered feature vector as the query. Finally, the original feature information is preserved through residual connections, and the interactive feature and the original feature vector are weighted and fused using a learnable scaling matrix to obtain the translation estimation feature.

[0033] It should be noted that bilinear interpolation and 1×1 convolution effectively integrate multi-scale features, avoiding information mismatch caused by features of different resolutions; the gated dual-path attention mechanism takes into account both local details and global context, making the expression of scene features more comprehensive; dual attention interaction further strengthens the correlation information between scene and vehicle, and between vehicles; residual connections and learnable scaling matrices ensure the rationality of feature fusion, ultimately improving the accuracy and robustness of translation estimation.

[0034] More preferably, the local detail path preserves fine-grained spatial patterns through convolutional blocks; the global gated path extracts scale-invariant global statistical information and generates a dynamic gating signal, which modulates the feature flow, including: calculating the spatial max-pooling summary and channel variance of uniform scene features; processing the weighted sum of the spatial max-pooling summary and channel variance through a 1×1 convolutional layer, where the weights are learnable parameters; after GELU activation of the convolutional processing result, the dynamic gating signal is obtained through upsampling; the dynamic gating signal modulates the feature flow of the global gated path through element-wise multiplication.

[0035] Specifically, the local detail path employs a convolutional block composed of convolutional layers, normalization layers, and activation layers to process the input uniform scene features. While performing feature transformation, a well-designed network structure preserves fine-grained spatial patterns within the features, ensuring no loss of local detail information. The global gating path first performs spatial max pooling on the uniform scene features to obtain a max pooling summary reflecting the feature spatial distribution. Simultaneously, it calculates the variance of this feature along the channel dimension to obtain inter-channel differences. Then, it weights and sums the spatial max pooling summary and the channel variance using learnable weight parameters to obtain a feature incorporating global statistical information. This feature undergoes dimensionality adjustment via a 1×1 convolutional layer, followed by a non-linear transformation using the GELU activation function. An upsampling operation restores the feature resolution to match the input uniform scene features, generating a dynamic gating signal. Finally, the dynamic gating signal is element-wise multiplied with the feature stream in the global gating path to achieve dynamic modulation of the feature stream.

[0036] It should be noted that the convolutional block design of the local detail path can accurately preserve fine-grained spatial information, providing rich local detail support for subsequent feature fusion; the global gate path comprehensively captures global statistical information by extracting spatial max pooling summary and channel variance. The generation and modulation process of the dynamic gate signal can adaptively adjust the intensity of the feature stream according to the actual situation of scene features, highlighting important information and suppressing useless information, making the expression of global features more targeted, thereby improving the quality of overall scene features.

[0037] More preferably, dual attention interaction is performed on the enhanced scene features, including: based on inter-object relationship attention, the spatial dependency relationship between vehicle instances is modeled through a multi-head self-attention mechanism; based on scene-object interaction attention, cross-attention calculation is performed with the feature vector as the query and the enhanced scene features as the key and value.

[0038] Specifically, in the inter-object relationship attention calculation process, the vehicle instance features in the enhanced scene features are split, with each vehicle instance feature treated as an independent token, constructing a token sequence. A multi-head self-attention mechanism is then used to process this token sequence, with multiple attention heads calculating the attention weights between each token and all other tokens in parallel. These weights reflect the spatial dependencies between different vehicle instances. The token sequence is then weighted and summed based on these attention weights to obtain features modeling inter-vehicle relationships. In the scene-object interaction attention calculation process, the object-centered feature vector is used as the query vector, and the enhanced scene features are used as the key and value vectors, respectively. The similarity between the query vector and the key vector is calculated to obtain the attention weight, which reflects the degree of association between different parts of the scene features and the target vehicle. The value vector is then weighted and summed based on this attention weight to obtain object features incorporating scene association information. Finally, the features processed by the two attention interactions are integrated to complete the dual attention interaction process.

[0039] It should be noted that the inter-object relationship attention effectively models the spatial positional relationship between multiple vehicle instances through a multi-head self-attention mechanism, solving the problem of mutual influence between instances in multi-vehicle scenarios; while the scene-object interaction attention establishes the association between the target vehicle and the surrounding scene, enabling vehicle features to make full use of scene context information. The combination of the two attention mechanisms strengthens the correlation of features from both the inter-instance and scene-instance dimensions, improving the accuracy of subsequent pose estimation.

[0040] Preferably, in the bounded iterative mesh refiner, the progressive recovery of the 3D shape of the target vehicle to obtain the reconstructed mesh includes: interacting the feature vector with a learnable shape concept embedding through multiple multi-head attention layers to generate enhanced shape features; generating a base mesh through a first branch: mapping the enhanced shape features to deformation coefficients through a fully connected layer, and weighting the deformation basis of a predefined average shape template to obtain a base mesh; performing iterative offset refinement through a second branch: initializing a coarse offset, and after T iterations, receiving the concatenation input of the enhanced shape features and the offset from the previous step in each iteration, predicting the vertex position adjustment amount through a multi-layer perceptron with shared weights, and using a Tanh bounded correction mechanism to limit the vertex update amplitude to obtain the final offset; and dynamically weighting and fusing the base mesh and the final offset through two learnable scalar weights to generate the reconstructed mesh.

[0041] Specifically, firstly, object-centered feature vectors are input into a multi-layer multi-head attention layer. Simultaneously, a learnable shape concept embedding is introduced, which contains general shape information for various types of vehicles. The feature vectors and shape concept embeddings interact multiple times through the multi-head attention layer, fully integrating prior shape knowledge and image feature information to generate enhanced shape features. In the first branch, the enhanced shape features are input into a fully connected layer. A set of deformation coefficients is obtained through mapping by the fully connected layer. A predefined average shape template contains the basic geometric structure of the vehicle, and its deformation basis can represent shape deformations of different directions and degrees. The deformation coefficients and deformation basis are weighted and combined to deform the average shape template, resulting in the base network. In the second branch, a coarse offset is first initialized, followed by an iterative process. In each iteration, the enhanced shape features are concatenated with the offset obtained in the previous iteration to form the input features for that iteration. The input features are fed into a multilayer perceptron with shared weights, which predicts the vertex position adjustment for that iteration. After the vertex position adjustment is constrained by the Tanh bounded correction mechanism, it is added to the offset of the previous iteration to obtain the offset for the current iteration. After T iterations, the final offset is obtained. Finally, two learnable scalar weights are used to weight the base mesh and the final offset respectively, and the weighted results are then fused to generate the final reconstructed mesh.

[0042] It should be noted that the interaction between feature vectors and shape concept embedding introduces prior shape knowledge, making the enhancement of shape features more targeted and laying a good foundation for 3D shape restoration. The generation of the base mesh provides the overall geometric framework of the vehicle, and the iterative offset refinement can gradually correct mesh errors through multiple progressive adjustments. The Tanh bounded correction mechanism effectively avoids mesh distortion caused by excessive vertex update amplitude. Dynamic weighted fusion can adaptively adjust the weights according to the actual quality of the base mesh and offset, so that the reconstructed mesh retains the rationality of the overall structure and has fine local details, improving the accuracy and stability of 3D shape restoration.

[0043] More preferably, a Tanh bounded correction mechanism is used to limit the vertex update magnitude, including:

[0044] in, Let be the offset in the t-th iteration. This is the offset for the (t-1)th iteration. Let be the vertex position adjustment amount in the t-th iteration. The tanh activation function limits the vertex update magnitude to the range of (-0.5, 0.5).

[0045] Specifically, in each iteration of the bounded iterative mesh refiner, the multilayer perceptron predicts the vertex position adjustment amount for that iteration. Then, it is input into the tanh activation function for processing; the output range of the tanh activation function is (-1, 1), and after scaling by a coefficient of 0.5, the effective range of the vertex position adjustment is limited to (-0.5, 0.5); then, the corrected vertex position adjustment is compared with the offset obtained in the previous iteration. Add them together to get the offset of the current t-th iteration. In each subsequent iteration, the above process is repeated until T iterations are completed, and the final offset is obtained.

[0046] It should be noted that the Tanh bounded correction mechanism, through a combination of activation function and coefficient scaling, strictly limits the update range of vertices in each iteration, avoiding problems such as mesh vertex overflow and shape distortion caused by excessive vertex position adjustment. At the same time, this mechanism allows vertices to be fine-tuned within a reasonable range, ensuring the incremental optimization characteristics of the iteration process, enabling the offset to gradually approach the optimal value, thereby improving the stability and accuracy of the reconstructed mesh, and ensuring that the final restored 3D shape conforms to the actual geometric features of the vehicle.

[0047] Preferably, this method uses a multi-task loss function for end-to-end training, and the total loss is the weighted sum of the losses of each subtask, specifically:

[0048] in, Total loss; For 2D detection loss, including classification, bounding box regression, and mask segmentation loss; For translational regression loss, the L1 norm is used for XY plane coordinates, and the uncertainty-aware L1 norm is used for depth estimation. For rotational losses, periodic L1 losses are used to handle angular discontinuities; To predict the average L2 distance between grid vertices and real grid vertices; This represents a loss of consistency in 3D space. , , , and These are the weighting coefficients.

[0049] It should be noted that the multi-task loss function mentioned above integrates the losses of multiple related tasks such as 2D detection, translation regression, rotation estimation, and 3D shape restoration, and achieves synergistic optimization of training for each task through weighted summation. The targeted design of different sub-task losses ensures the effectiveness of training for each task. For example, the depth loss for uncertainty perception improves the accuracy of depth estimation, and the periodic rotation loss solves the problem of angle discontinuity. The introduction of weighting coefficients allows for flexible adjustment of the proportion of each task in training according to task priority, enabling the model to take into account the performance of multiple tasks at the same time, and ultimately improving the overall 3D reconstruction accuracy and comprehensive performance of the model.

[0050] Accordingly, this embodiment provides a monocular 3D vehicle reconstruction system, including: The preprocessing module is configured to take a single RGB image containing multiple target vehicles as input, extract a multi-scale image feature pyramid through the backbone architecture, extract primitive information of the target vehicles in parallel from the image feature pyramid based on the detection head, and encode the primitive information into object-centered feature vectors. The hierarchical dual-context attention module is configured to construct a multi-scale scene context based on the image feature pyramid and fuse it with the feature vector to obtain translation estimation features; A bounded iterative mesh refiner is configured to progressively recover the 3D shape of the target vehicle based on eigenvectors to obtain a reconstructed mesh; The 3D reconstruction module is configured to obtain 3D rotation parameters based on feature vector regression, and combine translation estimation features with the reconstruction mesh to output the corresponding pose information and 3D shape of the target vehicle.

[0051] It should be noted that this system uses a modular design to break down the complex process of monocular 3D vehicle reconstruction into multiple functionally defined modules. These modules work together, reducing system complexity and improving maintainability. Specifically, the preprocessing module provides high-quality basic features for subsequent modules, ensuring the effectiveness of subsequent processing; the hierarchical dual-context attention module and the bounded iterative mesh refiner focus on pose estimation and translation estimation, and 3D shape restoration, respectively, improving the processing accuracy of each core task; the 3D reconstruction module effectively integrates the results from each part, ultimately outputting a complete and accurate 3D reconstruction result, enabling the system to efficiently and accurately complete the entire process from a single 2D image to 3D vehicle reconstruction.

[0052] Example 2 This embodiment provides a monocular 3D vehicle reconstruction method, which reconstructs monocular vehicles based on dynamic context adaptation and progressive geometric refinement, as detailed below.

[0053] Existing monocular reconstruction methods, when dealing with distant objects or objects with drastic scale changes, lack an effective multi-scale feature aggregation mechanism, making it difficult to fuse fine-grained local features with global scene semantics. This leads to severe geometric blurring and a decrease in pose estimation accuracy. To overcome the depth scale uncertainty in monocular vision and improve the robustness of pose estimation for distant targets, this embodiment proposes a hierarchical dual contextual attention (HDCA) module. This module can dynamically fuse object-centered geometric cues and scene-centered semantic information from a multi-resolution feature pyramid through a gated cross-attention mechanism, thereby constructing a scale-invariant enhanced feature representation.

[0054] Existing methods for reconstructing 3D vehicle shapes typically employ rigid, single-step deformation strategies. These methods are prone to structural holes, topological artifacts, and geometric inconsistencies in the reconstructed mesh due to accumulated prediction errors, resulting in a lack of physical realism. To eliminate structural defects in shape reconstruction and ensure the physical plausibility of the geometric topology, this embodiment designs a Bounded Iterative Mesh Refiner (BIMR). This module abandons the traditional single-prediction mode and adopts a progressive optimization strategy. It utilizes multi-head attention combined with a Tanh bounded correction loop to iteratively fine-tune the template shape at local vertices, thereby generating a high-fidelity, smooth-surfaced, and geometrically consistent 3D vehicle model.

[0055] like Figure 1 As shown, the monocular 3D vehicle reconstruction method in this embodiment can be implemented in the following way: First, a given single RGB input image is processed, and a multi-scale image feature pyramid is extracted using a backbone architecture integrating Res2Net-101 and a bidirectional feature pyramid network. Then, a Mask R-CNN-based detection head extracts vehicle primitive information from the image in parallel, including 2D bounding boxes, 66 (reference) 2D keypoints, and their visibility scores. This primitive information is encoded and unified into an object-centric feature vector. This feature encodes the unique appearance and 2D spatial properties of each detected object.

[0056] To address the geometric blurring and shape voids in monocular reconstruction, the feature vectors and image feature pyramids are then fed into two collaborative core modules: a Hierarchical Dual Context Attention (HDCA) module constructs a multi-scale scene context based on the image feature pyramids and fuses it with the feature vectors for pose estimation; and a Bounded Iterative Mesh Refiner (BIMR) module is used for progressively recovering high-fidelity 3D shapes. Finally, the features vectors are combined... The 3D rotation parameters obtained by direct regression are integrated with the predicted translation vector and refined mesh to output the complete 6-DoF pose and 3D shape of each vehicle instance, as detailed below.

[0057] I. Hierarchical Dual Context Attention Module (HDCA) The HDCA module aims to solve the pose estimation problem caused by depth-scale uncertainty in monocular vision. Its core is to build a contextual representation that can dynamically fuse local geometric cues with global scene semantics.

[0058] First, this module extracts multi-level features from the feature pyramid of the backbone network, downsamples the high-resolution feature maps to a uniform resolution using bilinear interpolation, and then concatenates them along the channel dimension. Subsequently, it... Convolutional layers compress channel redundancy, generating unified scene features that encompass everything from local details to global abstractions. .

[0059] To further enhance the expressive power of features, a core gated dual-path attention (GDA) mechanism was introduced. For example... Figure 2 As shown, in GDA, features The computation is split into two parallel paths: the first is the Local Detail Path (LDP), which uses convolutional blocks to preserve fine-grained spatial patterns; the second is the Global Gated Path (GGP), which aims to extract scale-invariant global statistics. The GGP path first computes the spatial max-pooling summary of the input features. and channel variance These two statistics are used to generate dynamic gating signals. .

[0060] Preferably, dynamic gating signal The calculation formula is:

[0061] in, For upsampling operation, and These are learnable parameters. The signal is modulated by element-wise product of the feature stream. This adaptively emphasizes information-rich regions, and the outputs of the two paths are fused to obtain... .

[0062] Building upon this, the module further performs dual-attention interaction: first, it applies inter-object relationship attention, utilizing a multi-head self-attention mechanism to model the spatial dependencies between vehicle instances; second, it applies scenario-object interaction attention, using object features... For querying, based on global scene features Cross-attention computation is performed for both the key and value, placing each object within its environmental context. Finally, the outputs of these two attention branches are weighted and fused into the original object features via residual connections and a learnable scaling matrix to generate the final translation estimation features. :

[0063] in, and For learnable weights, The output of attention is the relationship between objects. Output of scene-object interaction attention; translation estimation features By integrating multi-scale scene semantics and object geometric cues, the robustness of 3D translation estimation is significantly improved.

[0064] II. Bounded Iterative Mesh Refiner (BIMR) The BIMR module aims to eliminate shape and structural holes and topological errors caused by traditional single-step deformation methods. It employs a strategy of first generating a template and then iteratively correcting it. Figure 3 As shown.

[0065] First, object characteristics Enhanced shape features are generated by interacting with a set of learnable shape concept embeddings through multiple layers of multi-head attention (MHA). Subsequently, this feature enters two parallel branches. The first branch performs template-based shape generation, passing the shape through a fully connected layer. Mapped to a set of deformation coefficients for a predefined average shape template The deformation basis is weighted and combined to generate the basic mesh. The second branch performs iterative offset refinement, which is a loop optimization process. The system initializes a coarse offset. In the subsequent T iterations, a shared-weights MLP network is used to predict the adjustment amount of the vertex position. In the t-th iteration, the network receives static features. offset from the previous step The concatenation is used as input, and the adjustment amount for the current step is output. .

[0066] To prevent shape breakage and ensure geometric and physical rationality, this embodiment introduces... The bounded correction mechanism has the following offset update rules:

[0067] in, The activation function forces the vertex update magnitude to be limited to a certain value in each iteration. Within this range, the bounded constraint effectively avoids numerical divergence and unrealistic shape distortion. After T iterations, the final offset is obtained. The final reconstructed mesh From the base grid and final offset Through two learnable scalar weights and It is formed by dynamic weighted fusion:

[0068] III. Loss Function Design This embodiment uses a multi-task loss function for end-to-end training, with a total loss... Defined as the weighted sum of the losses of each subtask:

[0069] in, The loss is for 2D detection, including classification, bounding box regression, and mask segmentation losses; For translational regression loss, the L1 norm is used for XY plane coordinates, and the uncertainty-aware L1 norm is used for depth estimation. For rotational losses, periodic L1 losses are used to handle angular discontinuities; To predict the average L2 distance between grid vertices and real grid vertices; The 3D spatial consistency loss is calculated separately in rotation space, translation space and final world coordinate system to strongly constrain the consistency of geometric topology and ensure that the decoupled pose and shape parameters can still accurately reflect the real 3D structure after combination. , , , and These are the weighting coefficients.

[0070] In summary, the monocular 3D vehicle reconstruction method of this embodiment has the following characteristics: 1. This method proposes a hierarchical dual-context attention mechanism (HDCA), comprising a Local Detail Path (LDP) specifically for capturing fine-grained spatial patterns and a Global Guided Path (GGP) for extracting scale-invariant global statistics. This dual-path architecture can systematically process feature information at different levels, preserving both local geometric details and emphasizing global semantic features. This decoupled design is the core innovation that distinguishes it from existing single-process methods or those lacking fine-grained context modeling.

[0071] 2. Local Detail Path (LDP) employs convolutional operations to highlight key spatial structural information in the input feature map. Through a series of convolutional layers, normalization layers, and activation functions, LDP effectively enhances the ability to express salient features within local regions. Furthermore, LDP incorporates a multi-head self-attention mechanism to further strengthen the understanding of complex relationships between vehicle instances, providing rich feature representations for subsequent tasks.

[0072] 3. The Global Guided Path (GGP) introduces spatial summarization features and channel-level variance features, obtaining global statistics of the input features through max pooling and channel variance calculation. These statistics are concatenated and then passed through a convolutional layer and activation function to generate a dynamically gated signal. This signal is multiplied element-wise with the original features to adaptively enhance semantically salient regions. This process not only improves the model's focus on important features but also suppresses the influence of irrelevant information.

[0073] 4. This method proposes a Bounded Iterative Mesh Optimizer (BIMR) module, which designs a progressive optimization strategy to recover the 3D shape of vehicle instances. First, the target center features are enhanced through a multi-layer multi-head attention mechanism to enrich their semantic information. Then, the BIMR module executes two branches in parallel: a template-based deformation branch uses the enhanced features to perform weighted deformation on the standard template mesh; and an iterative offset optimization branch gradually corrects the vertex positions through multiple iterations, ensuring that the update amplitude is limited each time and avoiding unreasonable geometric distortions. Finally, the results of the two branches are weighted and fused to output a complete 3D vehicle mesh.

[0074] 5. This method, through the collaborative work of the HDCA and BIMR modules, significantly improves the accuracy and stability of 3D vehicle translation, rotation, and shape reconstruction tasks under monocular image conditions. In particular, its overall performance surpasses existing single-path or multi-stage methods for capturing geometric details and preserving physical plausibility in complex scenes, making it especially suitable for monocular 3D vehicle reconstruction in complex scenes.

[0075] Example 3 This embodiment is based on embodiment 2: This embodiment provides a monocular 3D vehicle reconstruction method, built upon a standard 2D object detector, such as Mask R-CNN. First, a hierarchical feature pyramid is generated from the input RGB image using a Res2Net backbone network integrated with BiFPN. Then, key 2D primitive information is extracted for each vehicle instance, including bounding boxes, 2D keypoints, and their visibility scores. This primitive information is processed and integrated into a target-centric feature map, denoted as […]. The feature map encodes the unique appearance features and 2D spatial attributes of each detected target.

[0076] The 3D vehicle representation is decomposed into three parallel branches: translation, rotation, and shape. For 3D translation estimation, a hierarchical dual contextual attention (HDCA) module is employed. This module constructs a comprehensive scene context from pyramid features and seamlessly integrates it with target-centric features, thus achieving robust translation estimation. For 3D rotation, its parameters are directly obtained by regressing from the target-centric feature vector through three fully connected layers. A bounded iterative mesh optimizer (BIMR) module is dedicated to shape recovery: a template mesh is first used, and then its vertex positions are progressively updated through a bounded iterative process to accurately capture fine-grained geometric details. Finally, an accurate six-DOF (6-DoF) pose and a geometrically consistent 3D mesh model are obtained for each vehicle instance, as detailed below.

[0077] I. Hierarchical Dual Context Attention Module (HDCA) The input to the HDCA module includes target center features. The multi-scale feature pyramid output by the backbone network is used. First, the feature maps of each level in the feature pyramid are adjusted to the same spatial resolution using bilinear interpolation and then stitched together along the channel dimension; the stitched result is then processed by a... After processing by the convolutional layer, the output is a feature map. Subsequently, It is copied into two copies, which are sent to the Local Detail Path (LDP) and the Global Boot Path (GGP) respectively.

[0078] In LDP, a copy After processing with one convolutional block, the output is In GGP, another document... Perform max pooling along the spatial dimension to obtain features. Meanwhile, regarding this document The variance of each channel is calculated along the spatial dimension to obtain the features. ;Will and After concatenation, the vector passes through a 1×1 convolutional layer and a GELU activation function, outputting a low-dimensional vector. This vector is then upsampled to its original size. The spatial dimensions form the gating signal. ;Will With this document Element-wise multiplication, the output is Next, and The sums are then processed through a 1×1 convolutional layer, and the output is the enhanced scene features. Then, all vehicle instances in the current image will be... Input a multi-head self-attention layer, output relation features At the same time, the individual vehicles As a query, As keys and values, they are input to a cross-attention layer, and the output is contextual features. .

[0079] Finally, , and Each feature is multiplied channel-by-channel with its corresponding learnable scaling matrix, and then the three are summed to output the enhanced feature. .

[0080] II. Bounded Iterative Mesh Optimizer Module (BIMR) The input to the BIMR module is the target center feature. First, Embedded with a set of learnable shape concepts The first layer of multi-head attention module is input together, where As a query As keys and values; the output of this layer, after being processed by LayerNorm, serves as the query input for the next layer, and this process is repeated several times to obtain the output augmentation features. .

[0081] It is fed into two parallel branches simultaneously: in the template deformation branch. The coefficients are obtained by mapping the weights through a fully connected layer and then normalizing them using the Softmax function. ;Will With predefined deformable shape base Perform a weighted summation and compare it with the standard template mesh. Add them together, and the output is the base grid. .

[0082] In the offset optimization branch, First, a multilayer perceptron (MLP) is input, and the output is the initial offset. Then proceed with T iterations: at the... In the next iteration, and After concatenation, the input is the same MLP, and the output is the adjustment amount. ;Will enter Multiply the function by 0.5 to get a bounded increment; then combine this increment with... Add them together, and the output is the updated offset. ;when At that time, output the final offset. .

[0083] Finally, the base mesh With final offset Multiply by the corresponding learnable scalar weights respectively and Then add them together, and the output is the final mesh. .

[0084] III. Experimental Setup (1) Dataset selection Specifically, the ApolloCar3D dataset can be used, which provides an industrial-grade, real-scale 3D CAD model for each vehicle instance, along with dense semantic keypoint annotations. Following the standard partitioning of existing methods, the training set contains 4077 images, and the validation set contains 200 images.

[0085] (2) Evaluation indicators Specifically, instance-level 3D average accuracy can be used ( As the primary metric, this metric is based on 10 thresholds ranging from lenient to stringent, jointly measuring the three tasks of 3D translation, 3D rotation, and 3D vehicle shape reconstruction. The result under the lenient criterion is denoted as... The result under strict criteria is denoted as 3D translation error was calculated using Euclidean distance, and 3D rotation error was calculated using inverse cosine distance. For 3D shape reconstruction, the predicted mesh was rendered from 100 different viewpoints, and the average intersection-union ratio (IU / U) between it and the ground truth mask was calculated. Furthermore, translation error is evaluated in both absolute and relative forms, corresponding to... and Two versions of the indicator.

[0086] (3) Implementation details All experiments were conducted on two NVIDIA RTX A6000 GPUs, and the model was built based on the Mask R-CNN framework. Training employed a two-stage strategy: the first stage used a pre-trained model provided by the BAAM method to initialize the 2D detection branch; the second stage performed end-to-end fine-tuning of the entire network, optimizing the joint loss of five tasks including 2D detection, 3D translation, 3D rotation, 3D shape, and 3D spatial consistency.

[0087] The weights for each task's loss were set as follows: 2D detection loss weight 1.0, translation loss weight 0.5, rotation loss weight 1.0, shape loss weight 3.0, and 3D spatial loss weight 0.01. The network optimization used the AdamW optimizer, with a global batch size of 4. The initial learning rate was set to... And it decays linearly to [value] in the last 10 epochs of training. .

[0088] Experimental results show that our proposed method outperforms state-of-the-art methods on the large public benchmark dataset ApolloCar3D. Furthermore, compared to the baseline model, our method increases the number of parameters and reduces GPU memory usage significantly, maintaining high-precision reconstruction while ensuring efficient inference speed, thus demonstrating high practical application value.

[0089] Example 4 This embodiment provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement a monocular 3D vehicle reconstruction method according to Embodiment 1 or 2. The computer program can be in the form of source code, object code, executable file, or some intermediate form.

[0090] Example 5 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a monocular 3D vehicle reconstruction method of Embodiment 1 or 2. The computer program can be in the form of source code, object code, executable file, or some intermediate form. The storage medium includes any entity or device capable of carrying computer program code, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0091] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

[0092] It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

Claims

1. A monocular 3D vehicle reconstruction method, characterized in that, include: The input is a single RGB image containing multiple target vehicles. A multi-scale image feature pyramid is extracted through a backbone architecture. Based on the detection head, the primitive information of the target vehicles is extracted from the image feature pyramid in parallel, and the primitive information is encoded into object-centered feature vectors. Based on the image feature pyramid, a multi-scale scene context is constructed using a hierarchical dual-context attention module and fused with the feature vector to obtain translation estimation features; The feature vectors are input into a bounded iterative mesh refiner to progressively recover the 3D shape of the target vehicle and obtain the reconstructed mesh. Based on the feature vector regression, the 3D rotation parameters are obtained, and combined with the translation estimation features and the reconstructed mesh, the corresponding pose information and 3D shape of the target vehicle are output.

2. The monocular 3D vehicle reconstruction method according to claim 1, characterized in that, The process involves constructing a multi-scale scene context using a hierarchical dual-context attention module based on the image feature pyramid, and fusing it with the feature vector to obtain translation estimation features, including: Multi-level features are obtained from the image feature pyramid. The high-resolution feature map is downsampled to a uniform resolution by bilinear interpolation. After being stitched together in the channel dimension, the channel redundancy is compressed by a 1×1 convolutional layer to generate a uniform scene feature. The unified scene features are processed using a gated dual-path attention mechanism, which splits the unified scene features into local detail paths and global gated paths, and then merges the outputs of the two paths to obtain enhanced scene features. The enhanced scene features are subjected to dual attention interaction and then weighted and fused into the feature vector through residual connections and a learnable scaling matrix to generate translational estimation features.

3. The monocular 3D vehicle reconstruction method according to claim 2, characterized in that, The local detail path preserves fine-grained spatial patterns through convolutional blocks; The global gated path extracts scale-invariant global statistical information and generates a dynamic gated signal, which is then used to modulate the feature stream, including: Calculate the spatial max-pooling summary and channel variance of the unified scene features; The spatial max-pooling summary and channel variance are processed by a 1×1 convolutional layer, where the weights are learnable parameters. After GELU activation of the convolution result, a dynamic gating signal is obtained through upsampling. The dynamic gating signal modulates the feature flow of the global gating path through element-wise multiplication.

4. The monocular 3D vehicle reconstruction method according to claim 2, characterized in that, Perform dual attention interaction on the enhanced scene features, including: Based on inter-object relational attention, spatial dependencies between vehicle instances are modeled through a multi-head self-attention mechanism; Based on scene-object interaction attention, cross-attention calculation is performed using the feature vector as the query and the enhanced scene features as the key and value.

5. The monocular 3D vehicle reconstruction method according to claim 1, characterized in that, In the bounded iterative mesh refiner, the 3D shape of the target vehicle is progressively recovered to obtain the reconstructed mesh, including: The feature vector is interacted with a learnable shape concept by passing it through multiple multi-head attention layers to generate enhanced shape features; The basic mesh is generated through the first branch: the enhanced shape features are mapped to deformation coefficients through a fully connected layer, and the deformation basis of the predefined average shape template is weighted and combined to obtain the basic mesh; The second branch performs iterative offset refinement: the coarse offset is initialized, and after T iterations, the enhanced shape feature and the offset of the previous step are received as input in each iteration. The vertex position adjustment is predicted by a multilayer perceptron with shared weights, and the vertex update amplitude is limited by the Tanh bounded correction mechanism to obtain the final offset. The reconstructed mesh is generated by dynamically weighting and fusing the base mesh and the final offset using two learnable scalar weights.

6. The monocular 3D vehicle reconstruction method according to claim 5, characterized in that, The adoption of the Tanh bounded correction mechanism to limit the vertex update magnitude includes: in, Let be the offset in the t-th iteration. This is the offset for the (t-1)th iteration. Let be the vertex position adjustment amount in the t-th iteration. The tanh activation function limits the vertex update magnitude to the range of (-0.5, 0.5).

7. The monocular 3D vehicle reconstruction method according to claim 1, characterized in that, The method employs a multi-task loss function for end-to-end training, with the total loss being a weighted sum of the losses from each sub-task: in, Total loss; For 2D detection loss, including classification, bounding box regression, and mask segmentation loss; For translational regression loss, the L1 norm is used for XY plane coordinates, and the uncertainty-aware L1 norm is used for depth estimation. For rotational losses, periodic L1 losses are used to handle angular discontinuities; To predict the average L2 distance between grid vertices and real grid vertices; This represents a loss of consistency in 3D space. , , , and These are the weighting coefficients.

8. A monocular 3D vehicle reconstruction system, characterized in that, include: The preprocessing module is configured to take a single RGB image containing multiple target vehicles as input, extract a multi-scale image feature pyramid through a backbone architecture, extract primitive information of the target vehicles in parallel from the image feature pyramid based on the detection head, and encode the primitive information into object-centered feature vectors. The hierarchical dual-context attention module is configured to construct a multi-scale scene context based on the image feature pyramid and fuse it with the feature vector to obtain translation estimation features; A bounded iterative mesh refiner is configured to progressively recover the 3D shape of the target vehicle based on the eigenvectors to obtain a reconstructed mesh; The 3D reconstruction module is configured to obtain 3D rotation parameters based on the feature vector regression, and combine the translation estimation features with the reconstruction mesh to output the corresponding pose information and 3D shape of the target vehicle.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the monocular 3D vehicle reconstruction method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the monocular 3D vehicle reconstruction method according to any one of claims 1-7.

Images