Three-dimensional target detection model training method and device, medium and program product

Abstract

The invention discloses a three-dimensional target detection model training method and device, a medium and a program product, and relates to the technical field of computer vision and automatic driving. Local BEV features are uniformly constructed by adopting a distance-independent feature sampling strategy, a virtual view cone is dynamically constructed based on a feature size and a preset depth range, and meanwhile, the depth prediction precision is improved by utilizing a depth diffusion strategy and occupation coding. According to the method, uniform distribution of features in the BEV space is realized from semantic and geometric dimensions, inconsistent information distribution of targets in near and far regions in the BEV space is effectively relieved, and stable and sufficient image features are aggregated for sparse point cloud features. And multi-modal fusion is carried out on an enhanced image BEV feature map formed by splicing the local BEV features and the global image BEV features and the global point cloud feature map for target detection, so that the three-dimensional target detection model obtained according to the invention has higher target detection precision.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and autonomous driving technology, and in particular to a method, device, medium and program product for training a three-dimensional target detection model. Background Technology

[0002] 3D object detection is a core foundational task in the field of autonomous driving. Its core objective is to accurately acquire key information such as the position, size, and attitude of surrounding objects, providing reliable environmental perception support for driving decisions and path planning. Its performance directly determines the safety and reliability of the autonomous driving system. Multimodal fusion solutions, leveraging the precise geometric measurement capabilities of LiDAR and the rich semantic appearance information of cameras, achieve complementarity between geometric structure and semantic features. This has become a mainstream technology for improving the accuracy and robustness of 3D object detection, especially in BEV (Bird's Eye View) spaces, where multimodal feature fusion and detection are widely used due to their superior spatial alignment capabilities.

[0003] However, existing multimodal 3D target detection methods still face key technical bottlenecks when fusing sufficient image information into spatially sparse LiDAR point clouds: Due to the inherent data characteristics of LiDAR and the perspective imaging principle of cameras, multimodal BEV features suffer from uneven distribution in depth and distance. Distant targets occupy fewer pixels in the image and have insufficient feature sampling, while nearby targets have dense feature sampling. This leads to an imbalance in the semantic representation and geometric distribution of BEV features for near and far targets, violating the translation invariance assumption relied upon by mainstream CNN detection heads and affecting detection performance. At the same time, the number of image pixels is far greater than the number of LiDAR point clouds. Only a small number of pixels can obtain accurate depth information through LiDAR point cloud projection. Most areas need to rely on depth completion mechanisms. Existing depth estimation networks are sensitive to image acquisition environments such as camera parameters and lighting conditions, which can easily lead to prediction bias. Furthermore, depth completion-based schemes require dense real label supervision, which is costly. This can cause spatial misalignment when image features are projected into the BEV space, affecting the effectiveness of multimodal fusion.

[0004] To address these issues, a series of technological explorations have been undertaken in related fields: some solutions utilize image features to enhance the performance of LiDAR detectors; recent mainstream solutions such as BEVFusion project two modalities of features into the BEV space to achieve point-by-point fusion; and AutoAlign improves alignment accuracy by modeling spatial and semantic mapping relationships using learnable alignment maps. However, none of these methods effectively solve the core problems of uneven BEV feature distribution and unreliable depth information supplementation. Overall, visual modal methods struggle to meet the requirements of geometric and semantic accuracy, multimodal fusion methods have weak feature capture capabilities for distant targets, and depth supplementation schemes either suffer from poor robustness or high computational overhead. All these approaches fail to efficiently construct a uniform and accurate BEV feature space, thus hindering further improvements in 3D target detection performance. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a method, device, medium, and program product for training a three-dimensional target detection model, thereby improving the target detection accuracy of the three-dimensional target detection model.

[0006] The first aspect discloses a method for training a 3D object detection model. The method includes: acquiring a training dataset, wherein any sample in the training dataset includes a synchronized input image, an input point cloud, and target category labels and 3D bounding boxes; extracting image features from the input image and performing view transformation to obtain a global BEV feature map, and extracting point cloud features from the input point cloud and performing projection along its height to obtain a global point cloud feature map; and obtaining an enhanced image BEV feature map based on the global BEV feature map and multiple local BEV feature maps, wherein the local BEV feature maps are obtained based on foreground target region features and corresponding region depth encoding, and the foreground target... The region feature is obtained by sampling each foreground target region corresponding to the global image feature map according to a preset sampling size to obtain a first preset number of region features. The global image feature map is extracted from the panoramic image features and contains multiple foreground target regions. The enhanced image BEV feature map and the global point cloud feature map are fused in a multimodal manner to obtain a fused BEV feature map. The fused BEV feature map is input into a pre-trained 3D target detection model. The 3D target detection model is trained and optimized according to the matching cost between the target category label, the 3D bounding box and the predicted category and the predicted box corresponding to the current sample until the model converges.

[0007] As one possible implementation, obtaining the enhanced image BEV feature map based on the global image BEV feature map and multiple local BEV feature maps includes: concatenating the local BEV feature maps of all foreground targets with the global image BEV feature map along the channel dimension according to their spatial position correspondence to obtain a joint feature that fuses global and local information; and inputting the joint feature into a preset convolutional layer to obtain the enhanced image BEV feature map.

[0008] As one possible implementation, the local BEV feature map is obtained based on foreground target region features and corresponding region depth encoding, including: obtaining two-dimensional bounding boxes corresponding to multiple foreground targets based on the global image feature map; sampling a first preset number of region features for each image region corresponding to the two-dimensional bounding box according to a preset feature sampling size to obtain the foreground target region features; dynamically constructing a corresponding virtual frustum for each foreground target; obtaining the region depth encoding corresponding to each foreground target image region; and obtaining a local BEV feature map based on the region features of the foreground targets within the virtual frustum range and the corresponding region depth encoding.

[0009] As one possible implementation, the step of dynamically constructing a corresponding virtual view frustum for each foreground target includes: based on the height and width of the preset sampling size, combined with a preset depth range threshold, dynamically constructing a dedicated virtual view frustum for each foreground target through spatial geometric mapping relationships.

[0010] As one possible implementation, obtaining the region depth encoding corresponding to each foreground target image region includes: obtaining sparse depth region features based on the perspective projection result of the input point cloud onto the image plane, wherein the sparse depth region represents pixels with depth information, pixels without corresponding point cloud data after projection are regions without depth information, and the entire image region includes sparse depth regions and regions without depth information; encoding the depth information of the perspective projection result through occupancy coding to obtain the depth encoding result of the sparse depth region; obtaining a global similarity matrix by calculating the dot product similarity matrix between the sparse depth region features and the global image features; calculating the normalized weights for depth propagation based on the global similarity matrix; weighted propagating the depth encoding result of the sparse depth region to the regions without depth information based on the normalized weights to obtain the depth encoding result of the regions without depth information; concatenating and fusing the depth encoding result of the sparse depth region with the depth encoding result of the regions without depth information according to the spatial position correspondence of image pixels to obtain the depth encoding result of the entire image region; and obtaining the foreground region depth encoding corresponding to each foreground target image region based on the depth encoding result of the entire image region.

[0011] As one possible implementation, before obtaining the region depth encoding corresponding to each foreground target image region, the method further includes: dividing the continuous depth space into a second preset number of depth intervals using a uniform discretization method; constructing an occupancy code based on the depth intervals and a preset depth range threshold, wherein the occupancy code can capture the inherent correlation between different depth intervals.

[0012] As one possible implementation, the step of obtaining a global similarity matrix by calculating the dot product similarity matrix between the sparse depth region features and the global image features includes: performing learnable linear transformation and position embedding processing on the sparse depth region features and the global image features respectively; and calculating the dot product similarity matrix between the processed sparse depth region features and the global image features through a broadcast operation to obtain the global similarity matrix.

[0013] The second aspect discloses a computer device including a processor and a memory, the memory storing a computer program that, when the computer program is run, implements a three-dimensional target detection model training method as disclosed in the first aspect or any possible implementation thereof.

[0014] The third aspect discloses a computer-readable storage medium storing a computer program or computer instructions that, when executed, implement the three-dimensional target detection model training method disclosed in the first aspect or any possible implementation thereof.

[0015] The fourth aspect discloses a computer program product that, when run on a computer, causes the computer to execute the three-dimensional target detection model training method disclosed in the first aspect or any possible implementation of the first aspect.

[0016] As can be seen from the above technical solutions, the present invention has the following beneficial effects:

[0017] This invention constructs uniform local BEV features using a distance-independent feature sampling strategy. It samples a fixed number of region features for foreground targets at different distances and dynamically constructs virtual view frustums based on feature size and a preset depth range. Local BEV features are generated by combining the dynamically constructed virtual view frustums with LSS-based virtual view frustum perspective transformation operations. This achieves uniform feature distribution within the BEV space from both semantic and geometric dimensions, satisfying the translation invariance assumption of CNN detection heads. It also effectively alleviates the inconsistency in information distribution between near and far targets in the BEV space, aggregating stable and sufficient image features for sparse point cloud features. Furthermore, the enhanced image BEV feature map, obtained by concatenating local and global image BEV features, is fused with the global point cloud feature map using multimodal methods. The fused BEV feature map is then input into a 3D target detection model for target detection, significantly improving the performance of distant target detection.

[0018] Furthermore, a similarity-guided depth propagation mechanism is designed to construct an efficient and accurate depth densification mechanism. By calculating the similarity matrix between sparse depth regions and global image features, depth encoding is directly propagated to depthless regions based on feature similarity without complex depth prediction. This reduces computational overhead and enhances robustness to changes in illumination and texture, providing reliable depth support for the accurate projection of image features into the BEV space.

[0019] Furthermore, this invention employs an occupancy coding method to accurately capture the correlation between depth intervals, ensuring the accuracy of depth propagation. Through effective fusion of multimodal BEV features and optimized training of the 3D detection head, the accuracy and robustness of 3D target detection are improved, particularly enhancing the detection capability of distant targets. Attached Figure Description

[0020] Figure 1 The flowchart of a three-dimensional target detection model training method provided by the present invention is shown.

[0021] Figure 2 This is a schematic diagram of the network structure of a three-dimensional target detection method provided by the present invention.

[0022] Figure 3 This is a schematic diagram of the network structure for obtaining enhanced image BEV feature maps provided by the present invention.

[0023] Figure 4 The visualization effect diagram provided for this invention.

[0024] Figure 5 This is a schematic diagram of the network structure for obtaining the depth coding result of the entire image region, as provided by the present invention.

[0025] Figure 6 The diagram shows the effect of the depth encoding provided by this invention projected onto the image at different distances. Detailed Implementation

[0026] To make the objectives, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Several embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of the present invention will be thorough and complete.

[0027] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] It should be noted that when an element is referred to as being "fixed to" another element, it can be directly on the other element or there may be an intervening element. When an element is considered to be "connected to" another element, it can be directly connected to the other element or there may be an intervening element. The terms "vertical," "horizontal," "left," "right," "up," "down," and similar expressions used herein are for illustrative purposes only and are not intended to indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0029] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances. The term "and / or" as used herein includes any and all combinations of one or more of the related listed items.

[0030] In one embodiment, the present invention provides a method for training a three-dimensional object detection model, such as... Figure 1 As shown, the specific steps include:

[0031] S101. Obtain the training dataset, wherein any sample in the training dataset includes a synchronized input image, an input point cloud, and target category labels and 3D bounding boxes;

[0032] S102. The input image is processed by image feature extraction and view transformation to obtain a global BEV feature map, and the input point cloud is processed by point cloud feature extraction and projection along the height to obtain a global point cloud feature map.

[0033] Among them, the original image sequence and point cloud sequence collected synchronously were selected from the 3D target detection dataset, as well as the target category label and 3D annotation box corresponding to the foreground object in the corresponding scene;

[0034] like Figure 2 As shown, Figure 2 This diagram illustrates the network structure of a 3D object detection method. First, to meet the requirements of subsequent feature extraction, the original image sequence and point cloud sequence need to undergo preprocessing operations. Specifically, the original image sequence is sequentially normalized to a preset resolution, and distortion correction based on camera intrinsic parameters is performed to eliminate distortion interference during image acquisition, resulting in an input image that meets the feature extraction requirements. Then, the input image is processed through the Swin-T image feature extraction network for multi-scale feature extraction to obtain a global image feature map. Where R represents the real number field, , , These represent the height, width, and number of RGB channels, respectively. This indicates the number of images captured during the panoramic view, which is set according to the dataset. In this embodiment... The value is 6. After undergoing standardized LSS back-projection view transformation, the global image feature map maps the two-dimensional features of the image to the three-dimensional BEV space, generating a global BEV feature map. ,in and These represent the dimensions of the BEV feature map.

[0035] In addition, a statistical filtering algorithm is used to filter discrete noise points and outliers from the point cloud sequence, and then regions of interest are selected according to a preset detection range to remove invalid background point clouds, thus obtaining the input point cloud. Subsequently, the input point cloud is passed through the feature extraction network Voxelnet. According to the voxel size specified in the network, the input point cloud is divided into uniform voxel units, and the point cloud features within each voxel unit are aggregated to obtain the target point cloud. ,in, , , These represent the coordinate information of the point cloud in three spatial dimensions. This represents the number of dimensions of the reflection intensity feature of the point cloud. The target point cloud is projected along the height direction to generate a global point cloud feature map with the same shape as the aforementioned global image BEV feature map. ,in and These represent the dimensions of the BEV feature map.

[0036] Preprocessing steps eliminate distortion interference caused by lens or installation deviations during image acquisition, resulting in an input image that meets feature extraction requirements. Invalid background point clouds exceeding the detection range are removed, yielding a simplified input point cloud. This allows the synchronized input image, input point cloud, target category labels, and 3D bounding boxes to construct a training dataset. This invention introduces multimodal data preprocessing and feature extraction into 3D target detection, designing a simultaneous processing model based on a dual-branch approach of image and point cloud. It proposes a feature extraction method utilizing targeted preprocessing and a dedicated backbone network, and further unifies feature dimensions through multi-scale fusion and spatial mapping, thereby improving the completeness and effectiveness of BEV features and providing high-quality feature support for subsequent depth alignment and multimodal fusion.

[0037] S103. An enhanced image BEV feature map is obtained based on the global image BEV feature map and multiple local BEV feature maps. The local BEV feature maps are obtained by encoding the foreground target region features and the corresponding region depth. The foreground target region features are obtained by sampling each foreground target region corresponding to the global image feature map according to a preset sampling size to obtain a first preset number of region features. The global image feature map is obtained by extracting features from the panoramic image and contains multiple foreground target regions.

[0038] like Figure 3 As shown, Figure 3 A schematic diagram of the network structure for enhancing BEV feature map acquisition in images. Specifically, it includes the following:

[0039] The step of obtaining the enhanced image BEV feature map based on the global image BEV feature map and multiple local BEV feature maps specifically includes:

[0040] The local BEV feature maps of all foreground targets are concatenated with the global image BEV feature map along the channel dimension according to their spatial correspondence, to obtain a joint feature that integrates global and local information.

[0041] The joint features are input into a preset convolutional layer to obtain the enhanced image BEV feature map.

[0042] It should be noted that by inputting the joint features into a pre-defined convolutional layer, convolution operations can achieve feature dimension fusion and semantic enhancement, thereby obtaining an enhanced image BEV feature map. .

[0043] The visualization effect of this embodiment is as follows: Figure 4 As shown in the figure, the highlighted area is the attention heatmap of the BEV feature map. The left side shows the effect of the traditional method, the middle side shows the effect of the present invention, and the rightmost side shows the actual detection effect of the present invention. It can be seen that the present invention results in a higher heatmap value for foreground objects in the BEV features, allowing the model to better focus on the foreground object region and effectively detect foreground objects.

[0044] Preferably, the local BEV feature map is obtained based on the features of the foreground target region and the corresponding region depth encoding, specifically including:

[0045] S1031. Obtain the two-dimensional bounding boxes corresponding to each of the multiple foreground targets based on the global image feature map;

[0046] S1032. For each image region corresponding to the two-dimensional bounding box, a preset number of region features are sampled according to a preset sampling size to obtain the foreground target region features;

[0047] S1033. Dynamically construct a corresponding virtual view frustum for each foreground target;

[0048] S1034. Obtain the region depth encoding corresponding to each foreground target image region;

[0049] S1035. Obtain the local BEV feature map based on the regional features of the foreground target within the virtual view frustum and the corresponding regional depth encoding.

[0050] Specifically, the two-dimensional detection head YOLO-X is applied to the multi-scale image features output by the image feature extraction network Swin-T to identify and locate foreground targets, and predict the two-dimensional bounding boxes corresponding to each of the multiple foreground targets. Where M represents the number of detected foreground targets, and the two-dimensional bounding box is represented by four positioning parameters that describe the spatial position and size of each two-dimensional bounding box. These parameters are the horizontal and vertical coordinates of the center point of the two-dimensional bounding box in the image, as well as the width and height of the two-dimensional bounding box, so as to accurately select the area where the foreground target is located in the image plane in coordinate form.

[0051] Secondly, for each image region corresponding to a two-dimensional bounding box, based on the preset feature sampling size parameters... ,in, and These represent the sampling dimensions of the region features in the height and width directions, respectively. A bilinear interpolation method is used to uniformly sample a first preset number of region features. This first preset number is a fixed number that can be set according to actual needs, ensuring that foreground targets at different distances and with different image proportions obtain region features of consistent quantity and dimension. ,in, The number of channels represents the regional characteristics, and the sampling operation is shown in formula (1):

[0052] (1)

[0053] This invention overcomes the limitation of traditional methods where distant targets suffer from insufficient feature sampling due to their small pixel count by employing a distance-independent design. After locating the foreground target using a 2D detection head, it extracts a fixed number of region features for targets at different distances. When image feature enhancement is insufficient for distant point cloud features, it can provide spatially consistent feature information, thereby constructing uniform BEV spatial features.

[0054] This invention proposes a depth-aligned uniform foreground BEV feature, specifically addressing the core issue of uneven BEV feature distribution. Traditional methods are affected by the sparsity of depth information acquired by LiDAR and the perspective principle of cameras, resulting in a low pixel ratio and insufficient feature sampling for distant targets. This leads to an imbalance in the distribution of near and far target features in the BEV space, violating the assumption of translation invariance of the detection head. This invention employs a local BEV feature uniform construction mechanism and a distance-independent feature sampling strategy to uniformly sample a fixed number of region features for foreground targets at different distances. Based on the feature size and a preset depth range, a virtual frustum is dynamically constructed. Local BEV features are generated through LSS-based virtual frustum perspective transformation. After concatenation and enhancement with global image BEV features, a uniform distribution of BEV features at both semantic and geometric levels is achieved, significantly improving the detection performance of distant targets.

[0055] Furthermore, the specific steps for dynamically constructing the corresponding virtual view frustum for each foreground target include:

[0056] Based on the height and width of the preset sampling size, and combined with the preset depth range threshold, a unique virtual view frustum is dynamically constructed for each foreground target through spatial geometric mapping.

[0057] Specifically, based on the preset height and width dimensions of the feature samples, combined with a preset depth range threshold... A unique virtual view frustum is dynamically constructed for each foreground target through spatial geometric mapping relationships. Specifically, it is expressed as:

[0058] (2)

[0059] Based on foreground feature size and and depth range Constructed virtual view frustum It can fully cover the potential distribution area of ​​the foreground target in three-dimensional space, providing accurate spatial range constraints for subsequent feature projection.

[0060] In one embodiment, the step of obtaining the region depth code corresponding to each foreground target image region specifically includes:

[0061] S10341. Based on the perspective projection result from the input point cloud to the image plane, the sparse depth region features are obtained. The sparse depth region represents pixels with depth information. Pixels without corresponding point cloud data after projection are regions without depth information. The entire image region includes sparse depth regions and regions without depth information.

[0062] S10342. Encode the depth information of the perspective projection result by occupancy coding to obtain the depth coding result of the sparse depth region.

[0063] S10343. The global similarity matrix is ​​obtained by calculating the dot product similarity matrix between the sparse depth region features and the global image features.

[0064] S10344. Calculate the normalized weights for depth propagation based on the global similarity matrix.

[0065] S10345. Based on the normalized weights, the depth encoding results of the sparse depth region are weighted and propagated to the region without depth information to obtain the depth encoding results of the region without depth information.

[0066] S10346. According to the spatial correspondence of image pixels, the depth coding results of sparse depth regions and the depth coding results of regions without depth information are spliced ​​and fused to obtain the depth coding results of the entire image region.

[0067] S10347. Obtain the foreground region depth code corresponding to each foreground target image region based on the full image region depth coding result.

[0068] like Figure 5 As shown, Figure 5 A schematic diagram of the network structure for obtaining the depth coding results of the entire image region is shown below:

[0069] Specifically, based on preset sensor intrinsic and extrinsic parameters, the input point cloud is projected onto the corresponding image plane. Furthermore, based on the presence or absence of point cloud projection results at each pixel location, sparse depth regions are clearly distinguished. Regions without depth information Among them, sparse depth region Medium pixels contain effective depth information Collect sparse depth regions The characteristics of sparse depth regions are obtained.

[0070] In one embodiment, the step of obtaining the global similarity matrix by calculating the dot product similarity matrix of sparse depth region features and global image features specifically includes:

[0071] Learnable linear transformations and position embedding processes are applied to the sparse depth region features and global image features, respectively.

[0072] The global similarity matrix is ​​obtained by calculating the dot product similarity matrix between the processed sparse depth region features and the global image features through broadcast operations.

[0073] Specifically, by performing learnable linear transformations and position embedding processes on the sparse depth region features corresponding to the sparse depth map and the global image features respectively, and then calculating the dot product similarity matrix of the two through a broadcast operation, a global similarity matrix is ​​formed. The calculation method is shown in formula (3):

[0074] (3)

[0075] in, This represents the learnable positional encoding, initialized to 0. and All are learnable weights. and This represents global image features and sparse depth region image features.

[0076] Furthermore, sub-matrices corresponding to sparse depth regions and regions without depth information are extracted from the global similarity matrix. The submatrix is ​​then input into the Sigmoid function for normalization, and the normalized weights for depth propagation are calculated. Based on these normalized weights, the depth encoding results of the sparse depth regions are weighted and propagated to the depth-information-free regions to obtain the depth encoding results for the depth-information-free regions. The calculation method is shown in formula (4):

[0077] (4)

[0078] in, This represents the summation operation. This represents the normalized weight.

[0079] In one embodiment, a reference similarity matrix is ​​constructed based on the true depth values ​​in the sparse depth map:

[0080] (5)

[0081] in, The hyperparameter is a preset value, which can be set according to the specific model; in this embodiment, it is set to 0.001. The reference similarity matrix is ​​used to characterize the relative depth relationship of different pixels in the image plane in three-dimensional space. Using the reference similarity matrix as the supervision target, the submatrices corresponding to the sparse depth regions in the global similarity matrix are analyzed. Supervised training is performed by applying binary cross-entropy loss to ensure that the feature similarity calculation can accurately reflect the deep correlation between pixels. The binary cross-entropy loss function is shown in formula (6):

[0082] (6)

[0083] Furthermore, based on the spatial correspondence of image pixels, the depth coding results of sparse depth regions are concatenated and fused with the depth coding results of regions without depth information to generate the full image region coding result. Finally, based on the full image region depth coding results, the foreground region depth coding corresponding to each foreground target image region is obtained.

[0084] This invention addresses the problems of inaccurate alignment of multimodal information in the BEV space and uneven distribution of near and far target features in multimodal 3D object detection. Through an innovative similarity-guided depth propagation strategy, it overcomes the limitations of traditional depth prediction networks, which rely on complex models and are susceptible to interference from acquisition conditions. This invention leverages the sparse depth encoding results provided by point clouds and global similarity calculations of image features to assign precise propagation weights to image regions without depth coverage. Combined with normalization processing, it achieves dense completion of depth information, enhancing the accuracy of image feature projection into 3D space. This satisfies the requirement for alignment of multimodal feature fusion distribution within the BEV space, effectively alleviating the inherent defects of existing methods when dealing with images lacking depth information and spatially uneven point clouds. It provides reliable depth constraints for the construction of image BEV features while avoiding the introduction of complex depth prediction networks, thus maintaining a lightweight model while enhancing the effectiveness of multimodal feature fusion.

[0085] In one embodiment, before obtaining the region depth encoding corresponding to each foreground target image region, the method further includes:

[0086] The continuous depth space is divided into a second preset number of depth intervals using a uniform discretization method.

[0087] Occupancy codes are constructed based on depth intervals and preset depth range thresholds, wherein the occupancy codes can capture the inherent correlation between different depth intervals.

[0088] Specifically: The continuous depth space is divided into a second preset number of parts using a uniform discretization method. A depth range, The value can be set according to actual needs. Occupation coding is constructed based on the depth coding formula (7):

[0089] (7)

[0090] in, This represents the true depth information of the pixels in the projected image. This represents the interval index of the depth information. The effective depth information in the sparse depth map is encoded using this occupancy code to obtain the depth information of the encoded sparse depth region. The occupancy encoding in this embodiment can effectively capture the inherent correlation between different depth ranges.

[0091] This invention employs a depth occupancy coding method to represent depth information, overcoming the limitations of traditional coding. Traditional one-hot coding cannot capture the correlation between depth intervals, leading to inaccurate depth modeling. This invention divides depth intervals into uniform discretization and uses occupancy coding to map depth values ​​into feature forms that include interval correlation information. This enables more comprehensive and accurate modeling of depth distribution features, ensuring the accuracy of depth propagation and further improving the accuracy of mapping image features to the BEV space.

[0092] The deep coding strategy designed in this invention has significant effectiveness in practical applications, such as... Figure 6 As shown, Figure 6 The images show the effect of depth encoding projected onto the image at different distances. The highlighted areas in the images represent locations with similar depth encoding. As can be seen, this invention can effectively identify regions with consistent depth at different distances.

[0093] Finally, a virtual view frustum transformation operation is used to transform the virtual view frustum. Regional characteristics of foreground targets within the range Corresponding region depth encoding Element-wise product fusion is performed, and foreground BEV features are extracted using a standardized LSS back-projection view transformation operation to obtain local BEV feature maps:

[0094] (8)

[0095] This embodiment can fully integrate semantic features and geometric depth information to generate a local BEV feature map with consistent dimensions and region features. .

[0096] S104. The enhanced image BEV feature map and the global point cloud feature map are fused in a multimodal manner to obtain a fused BEV feature map.

[0097] Specifically, feature concatenation and convolution operations are used to fuse the enhanced image BEV features with the global point cloud feature map along the channel dimension to obtain a fused BEV feature map that integrates image semantic information and point cloud geometric information.

[0098] S105. Input the fused BEV feature map into the pre-trained 3D object detection model, and train and optimize the 3D object detection model according to the matching cost between the target category label and the 3D bounding box corresponding to the current sample and the predicted category and the predicted box, until the model converges.

[0099] Specifically, the fused BEV feature map is input into the pre-trained 3D target detection model, i.e. the preset 3D detection head. The Hungarian algorithm is used to calculate the matching cost between the target category label and the 3D bounding box and the predicted category and the predicted box, so as to complete the accurate matching between the two.

[0100] It should be further noted that this embodiment uses a focus loss function. The loss calculation for the target classification task is expressed as formula (9):

[0101] (9)

[0102] in, This represents the probability that a sample is predicted by the model to be its true class. This represents the class balance coefficient, used to alleviate the problem of imbalanced positive and negative sample numbers. This represents the focusing parameter, used to reduce the impact of easily classified samples on the overall loss.

[0103] use The loss function calculates the loss for the bounding box position, size, and pose regression task, and its expression is shown in formula (10):

[0104] (10)

[0105] in, Indicates the first Predicted values ​​of the bounding box regression parameters. This represents the corresponding actual labeled value. This indicates the number of bounding box parameters involved in the regression.

[0106] The 2D detector head loss function in this example With 3D detection head loss function Both adopt the focus loss function and The loss function combination takes the form shown in formula (11-12):

[0107] (11)

[0108] (12)

[0109] Furthermore, the aforementioned binary cross-entropy loss Used to construct the overall loss function to improve the detection accuracy of the 3D object detection model.

[0110] The model parameters are updated via backpropagation, and the overall loss function is shown in Equation 13:

[0111] (13)

[0112] in, , and The weighting factors are set to 0.1, 0.1, and 1.0 in this embodiment. After multiple rounds of iterative training and optimization, the model outputs 3D target detection results containing target category, location, size, and pose information.

[0113] Feature concatenation and convolution operations are used to fuse the enhanced image BEV feature map with the global point cloud feature map in a multimodal manner, resulting in a fused BEV feature map. This fused BEV feature map is then input into a 3D detection head, where the Hungarian algorithm is used to achieve accurate matching between the target and the predicted bounding box. A focus loss function is employed. Optimize target classification tasks, by The loss function optimizes the prediction of the bounding box's position and size; after iterative training and optimization of the model, it outputs accurate 3D target detection results containing the target category, position, and size.

[0114] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to provide a three-dimensional target detection model training method according to the above method embodiments.

[0115] Furthermore, an electronic device is provided for implementing the method provided in the embodiments of this application. This device can participate in constituting or including the apparatus or system provided in the embodiments of this application. The electronic device may include one or more processors (processors may include, but are not limited to, processing devices such as microprocessors (MCUs) or programmable logic devices (FPGAs), a memory for storing data, and a transmission device for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera.

[0116] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits can be implemented wholly or partially as software, hardware, firmware, or any other combination. Furthermore, the data processing circuits can be a single, independent processing module, or wholly or partially integrated into any other element within a device (or mobile device). As involved in the embodiments of this application, the data processing circuit serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0117] The memory can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method described in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the above-mentioned data processing method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to electronic devices via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0118] The transmission device is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the device's communication provider. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0119] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows users to interact with the user interface of an electronic device (or mobile device).

[0120] This application also provides a computer storage medium storing at least one instruction or at least one program, which is loaded and executed by a processor to implement a three-dimensional target detection model training method provided in the above-described method embodiments.

[0121] Optionally, in this embodiment, the aforementioned computer storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the aforementioned storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0122] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer storage medium. The processor of an electronic device reads the computer instructions from the computer storage medium and executes the computer instructions, causing the electronic device to perform the three-dimensional target detection model training method provided in the above-described method embodiments.

[0123] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0124] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for training a three-dimensional target detection model, characterized in that, The method includes: Obtain a training dataset, wherein any sample in the training dataset includes a synchronized input image, an input point cloud, and target category labels and 3D bounding boxes; The input image is processed by image feature extraction and view transformation to obtain a global image BEV feature map, and the input point cloud is processed by point cloud feature extraction and projection along the height to obtain a global point cloud feature map. An enhanced image BEV feature map is obtained based on the global image BEV feature map and multiple local BEV feature maps. The local BEV feature maps are obtained by encoding the features of the foreground target region and the corresponding region depth. The foreground target region features are obtained by sampling each foreground target region corresponding to the global image feature map according to a preset sampling size to obtain a first preset number of region features. The global image feature map is obtained by extracting features from the panoramic image and contains multiple foreground target regions. The enhanced image BEV feature map is fused with the global point cloud feature map in a multimodal manner to obtain a fused BEV feature map. The fused BEV feature map is input into a pre-trained 3D object detection model. The 3D object detection model is trained and optimized based on the matching cost between the target category label, the 3D bounding box and the predicted category and bounding box corresponding to the current sample, until the model converges.

2. The method according to claim 1, characterized in that, The step of obtaining the enhanced image BEV feature map based on the global image BEV feature map and multiple local BEV feature maps includes: The local BEV feature maps of all foreground targets are concatenated with the global image BEV feature map along the channel dimension according to their spatial correspondence, to obtain a joint feature that integrates global and local information. The joint features are input into a preset convolutional layer to obtain the enhanced image BEV feature map.

3. The method according to claim 1, characterized in that, The local BEV feature map is obtained based on the features of the foreground target region and the corresponding region depth encoding, including: Based on the global image feature map, obtain the two-dimensional bounding boxes corresponding to each of the multiple foreground targets; For each image region corresponding to the two-dimensional bounding box, a first preset number of region features are sampled according to a preset feature sampling size to obtain the foreground target region features; Dynamically construct a corresponding virtual view frustum for each foreground target; Obtain the region depth encoding corresponding to each foreground target image region; The local BEV feature map is obtained by using the regional features of the foreground target within the virtual view frustum and the corresponding regional depth encoding.

4. The method according to claim 1, characterized in that, The process of dynamically constructing a corresponding virtual view frustum for each foreground target includes: Based on the height and width of the preset sampling size, and combined with the preset depth range threshold, a unique virtual view frustum is dynamically constructed for each foreground target through spatial geometric mapping.

5. The method according to claim 1, characterized in that, The step of obtaining the region depth encoding corresponding to each foreground target image region includes: The sparse depth region features are obtained based on the perspective projection results from the input point cloud to the image plane. The sparse depth region represents a pixel with depth information, and a pixel without corresponding point cloud data after projection is a region without depth information. The entire image region includes sparse depth regions and regions without depth information. The depth information obtained by perspective projection is encoded by occupancy coding to obtain the depth coding result of sparse depth region. The global similarity matrix is ​​obtained by calculating the dot product similarity matrix between the sparse depth region features and the global image features; The normalized weights for depth propagation are calculated based on the global similarity matrix. Based on the normalized weights, the depth coding results of the sparse depth region are propagated to the region without depth information in a weighted manner to obtain the depth coding results of the region without depth information. Based on the spatial correspondence of image pixels, the depth coding results of sparse depth regions are concatenated and fused with the depth coding results of regions without depth information to obtain the depth coding results of the entire image region. The foreground region depth code corresponding to each foreground target image region is obtained based on the full image region depth coding result.

6. The method according to claim 1, characterized in that, Before obtaining the region depth encoding corresponding to each foreground target image region, the method further includes: The continuous depth space is divided into a second preset number of depth intervals using a uniform discretization method. Occupancy codes are constructed based on depth intervals and preset depth range thresholds, wherein the occupancy codes can capture the inherent correlation between different depth intervals.

7. The method according to claim 1, characterized in that, The step of calculating the dot product similarity matrix between the sparse depth region features and the global image features to obtain the global similarity matrix includes: Learnable linear transformations and position embedding processes are applied to the sparse depth region features and global image features, respectively. The global similarity matrix is ​​obtained by calculating the dot product similarity matrix between the processed sparse depth region features and the global image features through broadcast operations.

8. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the method as claimed in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, which a processor reads from and executes to implement the method as described in any one of claims 1 to 7.

Images