Feature extraction method based on double-branch network structure

Abstract

The invention discloses a feature extraction method based on a double-branch network structure. The method comprises the following steps: constructing a double-branch architecture; the auxiliary branch completely multiplexes an original camera branch structure to serve as a'feature library 'to output multi-scale features; a main branch continues to use the same structure, context information of the same layer and a deeper layer in a feature library is called in real time at each level through a self-adaptive up-sampling-splicing-compression mechanism, and the features of the main branch are re-enhanced. According to the method, on the premise of not introducing extra large parameters, the feature quality and the multi-modal fusion effect are remarkably improved, and then comprehensive improvement of automatic driving 3D target detection precision, robustness and stability is achieved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a feature extraction method based on a dual-branch network structure. Background Technology

[0002] In autonomous driving systems, environmental perception is a core module for ensuring driving safety. Its main task is to detect and accurately locate surrounding objects (including vehicles, pedestrians, obstacles, etc.) in real time. 3D object detection, compared to traditional 2D detection, requires additional information about the target's depth, physical dimensions, and direction of motion. To achieve this, modern autonomous driving systems commonly employ multi-sensor fusion solutions: cameras provide rich texture and color information, while LiDAR provides precise distance measurements. To address the issue of inconsistent perspectives from different sensors, the industry uses a bird's-eye view representation, projecting multi-source data onto a top-down coordinate system.

[0003] Existing technology: The BEVFusion framework. Currently, the industry-leading solution is the BEVFusion model, which comprises two main branches: the camera branch employs a multi-level self-attention encoder structure to extract image features, reducing computational complexity through hierarchical attention mechanisms and local window calculations; the LiDAR branch uses a 3D convolutional network to process point cloud data. The features from both branches are fused in the BEV space and then input into a 3D detection head to complete target recognition. This approach outperforms traditional single-modal methods in detection accuracy on publicly available datasets, demonstrating superior recognition capabilities and robustness.

[0004] Existing technical bottlenecks: Through actual road testing and dataset validation, the BEVFusion framework was found to have the following core issues: 3.1 Defects in Multi-Scale Feature Extraction and Integration While the BEVFusion camera branch possesses multi-scale feature extraction capabilities, its integration of features at different scales suffers from serious shortcomings: shallow features retain local details but lack semantic information, while deep features are rich in semantics but lose spatial details, making effective cross-scale feature fusion difficult. Specifically, this manifests as incomplete feature responses for small targets and ambiguity in boundary recognition for large targets, affecting overall detection performance and exposing the fundamental problem of missing information interaction between scales. This deficiency stems from the fact that a single-branch structure cannot simultaneously optimize the quality and coordination of features at different scales.

[0005] 3.2 Inhomogeneous feature quality leads to fusion error Multi-scale feature quality differences directly induce matching failure with lidar point clouds: small targets exhibit significant deviations in feature alignment, exceeding the system's safety tolerance range and weakening fusion accuracy; large targets show obvious edge shifts due to deep feature geometric distortion. Feature quality imbalance further leads to cross-modal alignment instability. Under adverse conditions such as rain and fog, the alignment error of small targets increases sharply, and the contour distortion of large targets intensifies, forming a chain of negative effects: "feature quality decline—fusion accuracy decay—insufficient detection robustness".

[0006] 3.3 Resource consumption and inefficiency Significant cross-modal feature alignment errors exist: perceptible offsets exist in static scenes, and the deviations are further amplified under dynamic high-speed conditions. Resource allocation imbalance is also prominent, with camera branches consuming a large amount of computing resources but failing to generate matching and effective feature outputs. Redundant computation accounts for a high proportion in multi-scale processing, forcing the fusion module to invest additional computing power to compensate for alignment deviations, thus limiting overall efficiency and making it difficult to guarantee real-time performance.

[0007] These technical deficiencies directly lead to adverse impacts on business operations: First, insufficient integration of multi-scale features in camera branches increases the risk of missing small targets, and even small fluctuations in the missed detection rate can significantly amplify potential driving safety hazards; second, inconsistent feature quality causes the system's performance stability to decline under varying lighting conditions, especially in adverse weather conditions such as rain and fog, which significantly reduces detection capability and weakens the overall reliability of autonomous driving; finally, to compensate for the fusion bias caused by the lack of multi-scale features, a large amount of additional computing power is required, resulting in increased deployment costs and limited real-time performance, which in turn affects the practical application in complex traffic scenarios.

[0008] The BEVFusion framework, as a current mainstream multimodal 3D object detection scheme, has the following technical drawbacks in its camera branch: 1. Low efficiency of multi-scale feature integration: The coordination mechanism of features at different scales in camera branches is imperfect, resulting in insufficient cross-scale information interaction and incomplete feature representation. Specifically, this manifests as weak response of small target features, fuzzy boundary recognition of large targets, and increased redundant computation in feature processing.

[0009] 2. Poor feature quality consistency: The quality difference between shallow and deep features is significant, affecting the cross-modal alignment accuracy with the lidar point cloud. Small targets have large alignment errors, while large targets show obvious edge offset, reducing the reliability of fused features.

[0010] 3. Low resource utilization: Camera branches waste computational resources, and redundant multi-scale feature processing requires the fusion module to consume additional resources to compensate for alignment errors, resulting in high overall computational complexity and affecting real-time performance. Summary of the Invention

[0011] To address the shortcomings of existing technologies, the present invention aims to provide a feature extraction method based on a dual-branch network structure, developing a feature enhancement branch mechanism: while maintaining the original BEVFusion camera branch main branch structure, an enhancement branch is introduced to specifically fuse feature information at different scales, supplementing the feature extraction capability of the main branch and improving the overall feature quality and multimodal fusion effect. A multi-scale adaptive scheme is designed: a dynamic multi-scale feature coordination mechanism is constructed to optimize cross-scale feature integration, improve the extraction capability of small targets and edge features, and reduce the impact of feature quality differences on fusion accuracy. An efficient fusion architecture is constructed: while improving feature quality, cross-modal alignment errors are reduced, computational complexity is controlled, and a balance between detection accuracy, robustness, and real-time performance is achieved. To achieve the above-mentioned objectives and other advantages of the present invention, a feature extraction method based on a dual-branch network structure is provided, comprising: Obtain multi-dimensional raw images; The original image is processed using a dual-branch network to obtain a 2D feature map; The 2D feature map is voxelized to obtain the BEV feature map of the image; The image BEV feature map is fused with the lidar BEV feature map to output a prediction; The dual-branch network includes a main branch sub-network and an auxiliary branch sub-network. The main branch sub-network uses a multi-level self-attention mechanism encoder structure to extract basic features and high-level semantic information. The auxiliary branch sub-network employs a densely connected convolutional network to capture local details and low-level features, supplementing key information that the main branch sub-network may have missed.

[0012] By introducing a dual-branch in the camera branch, the following objectives are achieved: Enhance multi-scale feature integration capabilities: Design a dual-branch system to specifically fuse feature information at different scales, serving as a supplement to the main branch (maintaining the original camera branch structure of BEVFusion). This enables efficient coordination of cross-scale features, reduces incomplete feature representation, and improves the overall efficiency of camera branch feature processing.

[0013] Improve feature quality consistency and fusion accuracy: Through a dual-branch adaptive fusion mechanism, feature quality differences are reduced, cross-modal alignment errors are decreased, feature quality of small and large targets is improved, and the matching reliability between camera branch output features and LiDAR point clouds is enhanced.

[0014] Optimize resource utilization and computational efficiency: While maintaining the original structure of the main branch, enhance the targeted design of the branches to improve feature quality while controlling computational complexity, reducing resource redundancy, improving the overall efficiency of the camera branch, and achieving a balance between 3D target detection accuracy and real-time performance.

[0015] Significantly improves the accuracy and stability of BEVFusion-based 3D target detection systems, especially in small target detection in complex scenes and adaptability to harsh environments, while maintaining low computational complexity.

[0016] This invention employs a two-branch architecture where the main branch and auxiliary branches work together, resulting in the following direct technical effects: Parallel extraction of multi-scale features: The main branch focuses on global semantic features, while the auxiliary branch focuses on local detailed features. Working in parallel, they avoid the problem of interference between features of different scales in traditional single-branch architectures. Enhanced feature complementarity: The two branches extract features from different perspectives, forming complementary feature representations, directly improving the completeness and richness of feature expression. The second layer of derivation: Based on multi-scale feature parallel extraction and complementarity enhancement, the further technical effects are: Reduced loss of multi-scale information: When traditional BEVFusion camera branches process multi-scale features in a single branch, shallow details are easily lost during deep processing. However, the dual-branch architecture directly reduces the loss of multi-scale information by using an auxiliary branch to specifically preserve details. Enhanced feature representation capability: Feature fusion between the main branch and the auxiliary branch enables the network to express both global semantics and local details simultaneously, directly improving its feature representation capability.

[0017] Direct technical effects of the adaptive fusion mechanism: First layer derivation: This invention adopts an adaptive fusion mechanism based on attention weights, and the direct technical effects are: Dynamic feature importance assessment: This involves calculating the importance weights of different features using an attention mechanism, enabling dynamic feature selection and weighted fusion. Improved fusion accuracy: Compared to traditional fixed-weight fusion methods, adaptive fusion can dynamically adjust the fusion strategy based on the input content, directly improving fusion accuracy. The second layer of derivation: Based on the dynamic evaluation of feature importance and the improvement of fusion accuracy, the further technical effects are: improved multimodal matching accuracy: the adaptive fusion mechanism can better balance the importance of different modal features, directly improving the matching accuracy of camera features and LiDAR features. Enhanced small target detection capability: by dynamically adjusting feature weights, the network can better focus on the key features of small targets, directly improving the small target detection capability.

[0018] Improvements to address the issue of "low efficiency in multi-scale feature extraction": Direct effect: The dual-branch architecture enables parallel processing of multi-scale features, avoiding the phase of features at different scales in a single branch.

[0019] Mutual interference problem Indirect effects: The effective integration of multi-scale features significantly enhances the network's adaptability to targets of different scales, and the detection performance of both small and large targets is significantly improved, with the overall performance being much better than the original model.

[0020] Final result: It solves the problem of insufficient detection accuracy caused by the lack of multi-scale information in traditional BEVFusion.

[0021] Improvements to address "insufficient feature consistency": Direct effect: The adaptive fusion mechanism achieves an effective balance of features at different levels through dynamic weight adjustment.

[0022] Indirect effect: Improved feature consistency directly improves the matching accuracy between camera features and LiDAR features, and reduces inconsistencies between modes.

[0023] Final results: Small target detection accuracy is significantly improved, large target shape recognition accuracy is greatly enhanced, and overall detection performance is significantly improved. The original model achieved a significant leap, effectively solving the vicious cycle of "feature inconsistency → multimodal matching failure → reduced detection accuracy → insufficient feature extraction".

[0024] Improvements to address "low resource utilization efficiency": Direct effect: Adjusting the feature dimension through 1×1 convolution reduces computational complexity and improves resource utilization through feature reuse mechanism.

[0025] Indirect effect: Improved computational efficiency allows the network to reduce computational resource consumption while maintaining detection accuracy.

[0026] Final results: In terms of resource consumption and inference efficiency, this solution demonstrates superior performance compared to the original model, with higher practicality and deployment friendliness, achieving a balance between detection accuracy, robustness and real-time performance. Attached Figure Description

[0027] Figure 1 The flowchart of the BEVFusion model based on the feature extraction method of the dual-branch network structure according to the present invention is shown below. Figure 2 The flowchart below shows the feature extraction method based on a dual-branch network structure according to the present invention. Figure 3 This is a flowchart illustrating the dual-branch feature extraction method based on a dual-branch network structure according to the present invention. Detailed Implementation The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] This invention relates to feature extraction techniques in the field of computer vision, particularly multi-branch network structures and dense connection mechanisms, which help to understand the innovations of this invention. Feature extraction in computer vision is a core component of image understanding. Early convolutional neural networks extracted image features by stacking convolutional operations layer by layer, but suffered from problems such as feature degradation and gradient vanishing.

[0029] To address this, researchers proposed a multi-branch network architecture to enhance feature representation capabilities through parallel processing paths. Among these innovations, dense connection mechanisms are a key technological advancement, as illustrated in the dense connection model. This mechanism achieves feature reuse and multi-scale information fusion by establishing direct connections between each layer and all subsequent layers in the network. The basic principle of dense connections: Let the output of the Lth layer of the network be x_L. Then the input of the subsequent layers not only includes the output of the previous layer, but also the feature maps of all the previous layers, that is, x_L = H_L([x_0, x_1, ..., x_{L-1}]), where H_L is a composite function and [ ] represents the feature concatenation operation.

[0030] • Advantages of dense connections: promote feature reuse, reduce the number of parameters, enhance feature propagation, alleviate gradient vanishing, and integrate information at different scales.

[0031] • Applications of dense connections: Widely used in tasks such as image classification and object detection, it improves model performance through dense cross-layer connections.

[0032] These well-known techniques provide a theoretical basis for multi-branch feature extraction, helping to understand how this invention applies a dense connection mechanism to optimize camera branch feature extraction based on the BEVFusion framework.

[0033] like Figure 1 As shown, the closest existing technology to this invention is the BEVFusion model, a multimodal fusion framework for 3D object detection in autonomous driving. This invention modifies the BEVFusion camera branch by introducing a mechanism similar to dense connections, forming a dual-branch structure: the main branch maintains the original BEVFusion camera branch structure, while the enhancement branch supplements multi-scale information through dense feature interactions. The structural composition and relationships of the BEVFusion camera branches are as follows: The feature extraction process of the BEVFusion camera branch can be divided into the following steps: 1. Image preprocessing: The input image is resized and normalized to convert it into a format suitable for network input.

[0034] 2. Hierarchical Feature Extraction: A multi-level self-attention encoder structure is adopted, including: • Block segmentation: The input image is divided into non-overlapping pixel blocks, which are used as sequence input.

[0035] • Hierarchical encoding: Features are encoded through multiple self-attention mechanism encoder layers, each layer containing window multi-head self-attention and shift window multi-head self-attention.

[0036] • Downsampling: After each stage, pixel blocks are merged to achieve multi-scale feature representation.

[0037] 3. Cross-scale feature integration: Feature maps of different scales are initially fused through skip connections or attention mechanisms to form feature representations with multi-scale information.

[0038] 4. BEV Projection: Projecting 2D image features onto a bird's-eye view space and fusing them with LiDAR features.

[0039] Structural composition and its relationships: • Backbone network: A multi-level self-attention mechanism encoder acts as a feature extractor, responsible for extracting hierarchical features from the image.

[0040] • Attention mechanisms: Window attention is used for local feature modeling, while shift window attention is used for cross-window information interaction.

[0041] • Multi-scale representation: Captures information from local texture to global semantics through feature maps of different resolutions.

[0042] • Fusion Interface: The output features are aligned with the LiDAR branches in the BEV space to form a unified 3D representation.

[0043] The BEVFusion camera branch design provides a foundation for multi-scale feature extraction, but suffers from insufficient cross-scale feature coordination and feature quality discrepancies. This invention modifies this by introducing a dual-branch mechanism similar to dense connections, achieving higher accuracy on the nuScenes dataset than the original model, but further optimization of feature integration efficiency is still needed.

[0044] Reference Figure 2This invention aims to significantly improve the accuracy and stability of BEVFusion-based 3D object detection systems, particularly in small object detection in complex scenes and adaptability to harsh environments, while maintaining low computational complexity. A feature extraction method based on a dual-branch network structure includes: The core innovation of the dual-branch feature fusion network architecture lies in breaking through the limitations of single-branch feature extraction in the camera branch of bevfusion. It designs a network structure in which the main branch and auxiliary branch work together to achieve comprehensive capture and effective fusion of multi-scale and multi-level feature information.

[0045] Dual-branch parallel processing architecture: The main branch uses a multi-level self-attention mechanism encoder structure to extract basic features and high-level semantic information, while the auxiliary branch focuses on capturing local details and low-level features to supplement key information that the main branch may miss.

[0046] Furthermore, the overall architecture of the main branch: The main branch adopts a multi-level self-attention mechanism encoder as the backbone network, responsible for extracting basic features and high-level semantic information of the image. Its technical implementation process is divided into four core modules: image preprocessing, image patch embedding, multi-level self-attention mechanism encoder module, and feature pyramid network.

[0047] Image patch embedding module implementation: The image patch embedding module segments the input image into non-overlapping image patches and establishes local feature representations.

[0048] Image block embedding image segmentation technology implementation steps: 1. Input image size standardization 2. Image segmentation: dividing an image into multiple non-overlapping image patches of the same pixel size. 3. Feature Embedding: Each image patch is unfolded into a one-dimensional vector, which is then mapped to a high-dimensional feature space through a linear projection layer. 4. Location coding: Add location coding to each image patch to establish spatial location relationships.

[0049] Furthermore, the main branch employs multiple multi-level self-attention mechanism encoder modules for feature extraction. Each block contains a shift window attention layer and a feedforward neural network layer.

[0050] Shift window attention mechanism: Divide the feature map into multiple windows and shift them cyclically between adjacent layers to achieve efficient long-distance dependency modeling.

[0051] Multi-head attention computation: Parallel computation of multiple attention subspaces with 32 attention heads enhances the richness of feature representation.

[0052] Feedforward Neural Network: Two-layer multilayer perceptron performs nonlinear feature transformation, input to hidden layers changes feature dimensions, and the activation function is the GELU function.

[0053] Furthermore, the main branch constructs a feature pyramid network to achieve multi-scale feature fusion.

[0054] Top-down approach: Starting with high-level features, spatial resolution is restored layer by layer through upsampling.

[0055] Lateral connectivity: Upsampled features are fused with corresponding low-level features to establish a connection between high-level semantics and low-level details. Bilinear interpolation upsampling: As shown in the figure, bilinear interpolation is used to restore the feature map size.

[0056] The auxiliary branch focuses on extracting local detail features and employs a densely connected convolutional network, referencing the design principles of densely connected convolutional networks. The output of each layer is connected to all subsequent layers, enabling feature reuse and efficient gradient propagation.

[0057] Dense feature extraction implementation: Multi-scale convolutional kernel design: Parallel convolutional kernels of different sizes capture feature information from different receptive fields. Dense connection mechanism: Feature reuse strategy, with the output of each layer connected to all subsequent layers, ensuring that features are fully propagated throughout the network.

[0058] Dilated convolutional enhancement: Parallel dilation convolutions with different dilation rates (1, 2, 3, 4) are used to capture features at different scales from small to large.

[0059] Feature dimension alignment is achieved by uniformly mapping the output features of auxiliary branches to the same feature dimensions as the main branch, and using 1×1 convolution to adjust the number of channels to ensure that features between branches can be effectively fused.

[0060] Inter-branch fusion mechanism implementation: This invention proposes a dynamic feature fusion method based on attention weights. A schematic diagram of the adaptive feature fusion mechanism is provided. Attention weight calculation: • Calculate the correlation between the main branch features and the auxiliary branch features; • Generate attention weight map; • Weighted fusion.

[0061] Feature concatenation and fusion are achieved through: Channel-level concatenation: directly connecting two features along the channel dimension. Convolutional network fusion: using a 1×1 convolutional network to learn cross-branch feature interactions; the fused feature dimension is twice that of the original feature dimension.

[0062] like Figure 3 As shown, the specific process of dual-branch feature extraction is as follows: The auxiliary branch continues to use the feature extraction method of the BEVFusion camera branch, serving as the "feature library" for the entire dual-branch feature extraction network. It is responsible for extracting and storing multi-scale feature information from the original image. This branch adopts the same network structure as the original BEVFusion camera branch to ensure the consistency and stability of feature extraction.

[0063] The fourth-level feature generation process: The auxiliary branch extracts features at four different levels, generating four different sizes of feature maps, thus forming a rich multi-scale feature library. First-level feature extraction: • After the input image undergoes image block embedding processing, it passes through the first multi-level self-attention mechanism encoder module.

[0064] • This hierarchical feature map has the highest spatial resolution and contains rich details and local features.

[0065] • The feature map obtained at this level provides high-resolution foundational features for subsequent levels. Second-level feature extraction: • Based on the first level, processing is performed through downsampling operations and a second multi-level self-attention mechanism encoder module.

[0066] • The spatial resolution is reduced by half, but the number of channels is doubled, resulting in a more abstract representation of features.

[0067] • This level of feature extraction balances spatial details and semantic information, and is an important component of the feature pyramid: Third-level feature extraction. • Feature extraction continues through downsampling and a third multi-level self-attention mechanism encoder module.

[0068] • Spatial resolution is further reduced, while the number of channels continues to increase, forming higher-level semantic features.

[0069] • This level of features mainly contains information about the overall shape and structure of the target.

[0070] Fourth-level feature extraction: • The final multi-level self-attention encoder module processes the data to generate features at the highest level of abstraction.

[0071] • It has the lowest spatial resolution but the most channels, containing the richest semantic information.

[0072] • This level of features primarily reflects the target's category and global contextual information.

[0073] Feature library storage mechanism: The four sets of feature maps generated by the auxiliary branch are stored in the feature library, sorted according to hierarchical depth. This storage method provides the main branch with a flexible feature access mechanism, allowing the main branch to select features of appropriate levels for fusion as needed.

[0074] Multi-level feature fusion in the main branch: When extracting features for the BEVFusion camera branch, the main branch dynamically combines features of different sizes from the feature library of the auxiliary branch. The fusion strategy follows the "depth-first" principle, meaning that each level of the main branch combines all features from the auxiliary branch that are at or above the depth of the current level.

[0075] First-level feature fusion: Fusion objective: The first-level features of the main branch need to be fused with the features of all four levels of the auxiliary branches.

[0076] Upsampling operation: • The second-level features of the auxiliary branch are upsampled to the size of the first-level features of the main branch through a 2x deconvolution.

[0077] • The third-level features of the auxiliary branch are upsampled to the size of the first-level features of the main branch by a 4x deconvolution.

[0078] • The features of the fourth level of the auxiliary branch are upsampled to the size of the features of the first level of the main branch through an 8x deconvolution.

[0079] • The first-level features of the auxiliary branch can be used directly without the need for upsampling feature fusion: 1. Add the four upsampled auxiliary branch features to the first-level features of the main branch element by element.

[0080] 2. Adjust the number of channels using 1×1 convolutions to ensure consistency in feature dimensions.

[0081] 3. Apply the ReLU activation function to enhance nonlinear expressive power.

[0082] Technical advantages: The first-level fusion fully utilizes all the hierarchical information of the auxiliary branches, providing a rich multi-scale feature foundation for subsequent levels.

[0083] Second-level feature fusion: Fusion target: Second-level features from the main branch, combined with features from the auxiliary branch that are at or above the second-level depth. Upsampling operation: • The third-level features of the auxiliary branch are upsampled to the second-level feature size of the main branch through a 2x deconvolution.

[0084] • The features of the fourth level of the auxiliary branch are upsampled to the size of the features of the second level of the main branch through a 4x deconvolution.

[0085] • Second-level features in auxiliary branches can be used directly.

[0086] Feature fusion process: 1. The features of the three upsampled auxiliary branches are added element-wise to the second-level features of the main branch.

[0087] 2. Channel number adjustment and feature extraction are performed using 1×1 convolution.

[0088] 3. Apply batch normalization to stabilize the training process.

[0089] Technical advantages: The second-level fusion maintains high-level semantic information while reducing interference from low-level detailed information, thus improving the purity of feature representation.

[0090] Third-level feature fusion: Fusion objective: Third-level features in the main branch, combined with features in the auxiliary branch that are at or above the third-level depth.

[0091] Upsampling operation: • The features of the fourth level of the auxiliary branch are upsampled to the size of the features of the third level of the main branch through a 2x deconvolution.

[0092] • The third-level features of the auxiliary branch can be used directly.

[0093] Feature fusion process: 1. The features of the two upsampled auxiliary branches are added element-wise to the third-level features of the main branch.

[0094] 2. Feature integration and dimension adjustment are performed using 1×1 convolution.

[0095] 3. Apply Dropout regularization to prevent overfitting.

[0096] Technical advantages: The third-level fusion focuses on the integration of high-level semantic information, providing a stable abstract feature foundation for the final feature representation.

[0097] Fourth-level feature fusion: Fusion target: Fourth-level features from the main branch, combined with features from the auxiliary branch that are at or above the fourth-level depth. Upsampling operation: • The fourth-level features of the auxiliary branch can be used directly without upsampling.

[0098] Feature fusion process: 1. The fourth-level features of the auxiliary branch are added element-wise to the fourth-level features of the main branch.

[0099] 2. Perform final feature integration using 1×1 convolution.

[0100] 3. Use residual connections to maintain gradient flow.

[0101] Technical advantages: The fourth-level fusion enables the direct integration of features at the highest level of abstraction, providing optimal semantic feature representation for subsequent FPN processing.

[0102] Feature fusion technology details: Deconvolution upsampling technique Deconvolution kernel design: • Core size: 3×3 or 5×5, dynamically adjusted according to the upsampling factor.

[0103] • Step size: equal to the upsampling factor.

[0104] • Padding: Maintains the accuracy of output size calculations.

[0105] Upsampling calculation process: 1. Input feature map preparation: C×H×W dimensions.

[0106] 2. Transposed convolution operation: Y[i,j] = ΣΣ X[is·k, js·l] × W[k,l].

[0107] 3. Activation function application: ReLU or GELU enhance nonlinearity.

[0108] 4. Batch normalization: Stabilize the training process.

[0109] 1×1 Convolution Feature Adjustment: Channel number adjustment: • Number of input channels: The total number of channels of the fused features.

[0110] • Number of output channels: The number of standard channels at the target level.

[0111] • Weight initialization: Xavier or He initialization method.

[0112] Feature extraction process: 1. 1×1 convolution dimensionality reduction: Reduce computational complexity.

[0113] 2. Feature integration: Integrating feature information from different sources.

[0114] 3. Nonlinear activation: Enhances feature representation ability.

[0115] 4. Regularization: Prevents overfitting.

[0116] Analysis of technical advantages: Advantages of multi-scale feature integration: Through a four-level feature fusion scheme, the dual-branch feature extraction network achieves comprehensive integration of multi-scale features. • Preservation of details: The first level of integration preserves rich spatial detail information.

[0117] • Semantic enhancement: Subsequent hierarchical fusion gradually enhances semantic expressive capabilities.

[0118] • Information complementarity: Features at different levels complement each other to form a complete feature representation.

[0119] Computational efficiency optimization: • Layered processing: Avoids redundant calculations and improves processing efficiency.

[0120] • Feature reuse: Auxiliary branch features are used multiple times, improving resource utilization.

[0121] • Memory optimization: Reduce feature dimensions and lower memory usage by using 1×1 convolutions.

[0122] Improved generalization ability: • Multi-scale adaptation: capable of handling targets of different scales.

[0123] • Enhanced robustness: Multi-level fusion improves adaptability to complex scenarios.

[0124] • Feature stability: The stability of feature extraction is ensured through a feature library mechanism.

[0125] This technical solution demonstrates the innovation and technical advantages of the dual-branch feature extraction network in multi-scale feature fusion by describing the dual-branch feature extraction process in detail, and provides powerful feature representation capabilities for 3D object detection tasks.

[0126] In summary, the dual-branch architecture offers unique advantages: comprehensive multi-scale feature extraction: the main branch captures global semantic information, while the auxiliary branch captures local detailed features, achieving comprehensiveness and richness in feature representation.

[0127] Feature complementarity mechanism: The two branches extract features from different angles, complementing each other and effectively avoiding the limitations of a single branch.

[0128] Adaptive fusion optimization: Dynamically adjust the fusion weights based on feature importance to achieve optimized integration of feature information.

[0129] Breakthrough innovation: It is the first time that a dual-branch parallel processing feature extraction architecture has been proposed, providing a new approach to feature extraction in the field of computer vision.

[0130] Highly practical: The architecture is simple and efficient, scalable to other vision tasks, and has broad application prospects. Excellent performance: The effectiveness of the technical solution has been verified on multiple public datasets, providing reliable technical support for practical applications. This invention, through an innovative dual-branch network architecture, effectively overcomes the limitations of traditional single-branch feature extraction, opening up new directions for the development of visual feature extraction technology.

[0131] System-level technical effect derivation: First-level system effect: Based on the aforementioned direct and indirect technical effects, the resulting system-level effects include: Overall Improvement in Detection Accuracy: Through the application of a dual-branch architecture and an adaptive fusion mechanism, the system has significantly improved its detection accuracy on various datasets, especially in terms of the accuracy of multimodal feature fusion.

[0132] Enhanced adaptability across multiple scenarios: The system's performance has improved across different difficulty levels and diverse scenarios, demonstrating strong environmental adaptability and generalization capabilities.

[0133] Second-layer system effect: Based on the improved detection accuracy and enhanced adaptability to multiple scenarios, the resulting system-level effects include: Significantly improved safety: The reduced rate of missed detections of small objects directly decreases the risk of potential accidents, thereby improving the overall safety of the autonomous driving system. The system can more accurately identify small objects on the road, providing more reliable environmental perception information for vehicle decision-making.

[0134] Significantly improved robustness: In complex environments, the system's detection accuracy decreases significantly, enhancing its reliability and stability. Even in challenging scenarios such as poor lighting conditions, severe weather, or target occlusion, the system maintains high detection performance.

[0135] Third-layer system effect: Based on the improvements in security and robustness, the resulting system-level effects include: Engineering Application Value Realization: It solves the problem of limited application of traditional BEVFusion in complex scenarios, providing reliable technical support for the commercial deployment of autonomous driving systems. While maintaining high precision, the system significantly reduces computational complexity, meeting the real-time requirements of practical applications.

[0136] Technological advancements have driven this development, providing new technical approaches for the field of multimodal 3D object detection and propelling the advancement of related technologies. The dual-branch collaborative mechanism and adaptive fusion strategy proposed in this invention offer valuable references and inspiration for subsequent research.

[0137] The essence of the technical effect is manifested in the following: Through the step-by-step derivation of the above-mentioned system-level technical effects, the complete causal chain of this invention from local technical improvements to overall system performance enhancement is clearly demonstrated. This structured method of deducing technical effects not only highlights the innovation of this invention at the technical level, but more importantly, it reflects its practical value in solving real-world engineering problems, laying a solid theoretical and practical foundation for the further development of multimodal 3D target detection technology.

[0138] The core innovation of this invention lies in transforming the traditional serial processing mode of single-branch feature extraction into a parallel processing mode of dual-branch collaboration. This architectural innovation fundamentally solves the problems of information loss and inefficiency in multi-scale feature processing. Architectural Breakthrough: For the first time, a dual-branch collaboration mechanism is introduced into the BEVFusion framework, achieving effective integration of multi-scale features. Algorithmic Breakthrough: An adaptive fusion mechanism is proposed, realizing dynamic evaluation of feature importance and weight adjustment. Engineering Breakthrough: While maintaining detection accuracy, computational complexity is significantly reduced, achieving a balance between accuracy and efficiency.

[0139] The technical advantages demonstrated by this invention through reasoning not only address the specific shortcomings of existing technologies but, more importantly, provide a new technical paradigm for the field of multimodal 3D object detection, possessing significant theoretical and practical value. The step-by-step derivation of its technical advantages clearly demonstrates a complete causal chain from local technical improvements to system performance enhancement, laying a solid foundation for the further development of related technologies.

[0140] The number of devices and processing scale described herein are for simplification of the invention. Applications, modifications, and variations of this invention will be readily apparent to those skilled in the art. Although embodiments of the invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. It can be applied to various fields suitable for this invention, and further modifications can be readily implemented by those skilled in the art. Therefore, without departing from the general concept defined by the claims and their equivalents, this invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A feature extraction method based on a dual-branch network structure, characterized in that, Includes the following steps: Obtain multi-dimensional raw images; The original image is processed using a dual-branch network to obtain a 2D feature map; The 2D feature map is voxelized to obtain the BEV feature map of the image; The image BEV feature map is fused with the lidar BEV feature map to output a prediction; The dual-branch network includes a main branch sub-network and an auxiliary branch sub-network. The main branch sub-network uses a multi-level self-attention mechanism encoder structure to extract basic features and high-level semantic information. The auxiliary branch sub-network employs a densely connected convolutional network to capture local details and low-level features, supplementing key information that the main branch sub-network may have missed.

2. The feature extraction method based on a dual-branch network structure as described in claim 1, characterized in that, The main branch sub-network includes an image preprocessing module, an image patch embedding module, a multi-level self-attention mechanism encoder module, and a feature pyramid network. The image patch embedding module segments the input image into non-overlapping image patches and establishes local feature representations. The feature pyramid network is used for multi-scale feature fusion.

3. The feature extraction method based on a dual-branch network structure as described in claim 2, characterized in that, The main branch subnetwork extracts features through multiple multi-level self-attention mechanism encoder modules, and each block contains a shift window attention layer and a feedforward neural network layer.

4. The feature extraction method based on a dual-branch network structure as described in claim 3, characterized in that, The shifted window attention layer is used to divide the feature map into multiple windows and shift them cyclically between adjacent layers to achieve efficient long-distance dependency modeling. Multi-head attention computation is used to compute multiple attention subspaces in parallel, thereby enhancing the richness of feature representation; Nonlinear feature transformation is performed using a multilayer perceptron with a feedforward neural network. The input hidden layer changes the feature dimension, and the GELU function is used as the activation function.

5. The feature extraction method based on a dual-branch network structure as described in claim 2, characterized in that, The specific process of the feature pyramid network includes: Spatial resolution is restored by upsampling layer by layer, starting from high-level features; By fusing upsampled features with corresponding low-level features, a connection between high-level semantics and low-level details is established. Feature map size recovery is performed using bilinear interpolation.

6. The feature extraction method based on a dual-branch network structure as described in claim 1, characterized in that, The auxiliary branch sub-network acquires 2D feature maps by: generating four different sizes of feature maps through feature extraction at four different levels, forming a rich multi-scale feature library. The four sets of feature maps generated by the auxiliary branch sub-network are stored in the feature library and sorted according to the layer depth. The feature library provides a flexible feature access mechanism for the main branch sub-network, enabling the main branch sub-network to select and fuse features at appropriate levels as needed.

7. The feature extraction method based on a dual-branch network structure as described in claim 6, characterized in that, When performing feature extraction, the main branch sub-network dynamically combines features of different sizes from the feature library for fusion. The fusion strategy follows the principle of "hierarchical depth first", which means that each level of the main branch sub-network will combine all features in the auxiliary branch sub-network that are greater than or equal to the depth of the current level.

Images