Three-dimensional target detection method and device based on radar mixed point and image fusion

Abstract

The application provides a three-dimensional target detection method and device based on radar mixed points and image fusion. The method provided by the application comprises: acquiring radar point cloud data and image data collected under the same timestamp; constructing a mixed point cloud based on the radar point cloud data, and extracting a radar bird's-eye view feature based on the mixed point cloud; determining an image bird's-eye view feature based on the radar point cloud data and the image data; fusing the radar bird's-eye view feature and the image bird's-eye view feature based on a channel attention mechanism and a spatial attention mechanism to obtain a comprehensive feature, and performing three-dimensional target detection based on the comprehensive feature. The method and device provided by the application are used to solve the problems of radar sparsity, insufficient image fusion and inaccurate spatial positioning in the prior art, and significantly enhance the overall performance of three-dimensional target detection.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and in particular to a three-dimensional target detection method and apparatus based on radar mixed point fusion and image fusion. Background Technology

[0002] Currently, common 3D target detection methods mainly rely on sparse 3D data such as LiDAR point clouds or millimeter-wave radar point clouds, combined with image data, to spatially locate and identify targets such as vehicles and pedestrians through feature extraction and multimodal fusion techniques. These methods typically convert radar point clouds into bird's-eye views or voxel representations and fuse them with image features to improve detection accuracy.

[0003] However, existing methods have the following shortcomings: First, since millimeter-wave radar point clouds are sparse and noisy, directly using the original point cloud can easily lead to insufficient feature representation capabilities, limiting detection performance; Second, the fusion methods of radar and images are mostly simple stitching or attention mechanisms, ignoring the fine-grained correspondence between point clouds and images in spatial semantics, making it difficult to fully utilize the structural information in the image; Third, some methods fail to fully explore the complementarity between radar and images in depth estimation, resulting in inaccurate spatial location estimation.

[0004] Therefore, there is an urgent need for a method to solve the problems of radar sparsity, insufficient image fusion and inaccurate spatial positioning in existing methods, and to significantly enhance the overall performance of 3D target detection. Summary of the Invention

[0005] In view of this, this application provides a three-dimensional target detection method and apparatus based on radar hybrid point and image fusion to solve the problems of radar sparsity, insufficient image fusion and inaccurate spatial positioning in existing methods, and significantly enhance the overall performance of three-dimensional target detection.

[0006] Specifically, this application is implemented through the following technical solution:

[0007] The first aspect of this application provides a three-dimensional target detection method based on radar hybrid point fusion and image fusion, the method comprising:

[0008] Acquire radar point cloud data and image data collected at the same timestamp;

[0009] A hybrid point cloud is constructed based on the radar point cloud data, and radar bird's-eye view features are extracted based on the hybrid point cloud.

[0010] Based on the radar point cloud data and the image data, the features of the bird's-eye view of the image are determined;

[0011] The radar bird's-eye view features and image bird's-eye view features are fused based on the channel attention mechanism and the spatial attention mechanism to obtain comprehensive features, and three-dimensional target detection is performed based on the comprehensive features.

[0012] A second aspect of this application provides a three-dimensional target detection device based on radar hybrid point fusion and image fusion, the device comprising an acquisition module, an extraction module, a determination module and a fusion module;

[0013] The acquisition module is used to acquire radar point cloud data and image data collected at the same timestamp;

[0014] The extraction module is used to construct a hybrid point cloud based on the radar point cloud data, and extract radar bird's-eye view features based on the hybrid point cloud;

[0015] The determining module is used to determine the bird's-eye view features of the image based on the radar point cloud data and the image data;

[0016] The fusion module is used to fuse the radar bird's-eye view features and the image bird's-eye view features based on the channel attention mechanism and the spatial attention mechanism to obtain comprehensive features, and to perform three-dimensional target detection based on the comprehensive features.

[0017] This application provides a 3D target detection method and apparatus based on radar point cloud data and image fusion. Firstly, it considers that radar point cloud data and image data originate from different types of sensors, each with its own unique advantages and disadvantages. Radar point cloud data provides accurate 3D spatial information, particularly excelling in distance measurement and target position estimation. However, its point cloud data is relatively sparse and lacks rich texture and color information, making it difficult to accurately distinguish targets that appear similar but belong to different categories. Image data, on the other hand, possesses high-resolution texture and color information, providing rich semantic information such as target category, boundaries, and detailed features. However, due to the inherent limitations of 2D imaging, it is difficult to directly obtain accurate depth information, especially under complex lighting or occlusion conditions. This application transforms radar point cloud data and image data into unified bird's-eye view features, and then fuses them based on an attention mechanism. This fully leverages the advantages of both types of data, complementing each other's shortcomings. Specifically, radar bird's-eye view features enhance spatial structure and distance perception capabilities, while image bird's-eye view features provide refined semantic and texture information. The fused integrated features possess richer environmental and target information, effectively improving the accuracy and robustness of 3D target detection, especially in target classification, localization, and shape estimation. Furthermore, multimodal fusion enhances the model's adaptability to harsh environments (such as nighttime, rain, snow, and strong light), improving the overall stability and reliability of the system. Secondly, this application considers that effective fusion of multimodal data relies not only on information complementarity but also on the spatiotemporal synchronization of the data. Radar and cameras, as different sensors, often have asynchronous acquisition frequencies and times. If the acquisition times of the two types of data differ, the target and scene may shift or change, leading to a mismatch in spatial location and semantic information, causing misalignment and errors in data fusion. This temporal deviation can cause the fused features to lose alignment, reducing the accuracy of 3D target detection and even leading to false detections or missed detections. This application ensures complete temporal synchronization of the two types of data by strictly guaranteeing that radar point cloud data and image data are acquired at the same timestamp, avoiding inconsistencies caused by target displacement within the acquisition interval. This temporal synchronization makes the spatial correspondence of projected points more accurate, enabling precise matching between image instance masks and radar point clouds. This, in turn, improves the accuracy of foreground point selection and the quality of virtual point cloud generation. Ultimately, the fused comprehensive features accurately reflect the complete information of the scene at that time, achieving more precise 3D target detection. Attached Figure Description

[0018] Figure 1 A flowchart of a three-dimensional target detection method based on radar hybrid point and image fusion provided in Embodiment 1 of this application;

[0019] Figure 2This is a schematic diagram of the structure of the three-dimensional target detection device based on radar hybrid point and image fusion provided in Embodiment 2 of this application. Detailed Implementation

[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0021] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0022] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0023] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0024] Figure 1 This is a flowchart illustrating the 3D target detection method based on radar point fusion and image fusion provided in Embodiment 1 of this application. Please refer to... Figure 1 The method provided in this embodiment may include:

[0025] S101. Acquire radar point cloud data and image data collected at the same timestamp.

[0026] Specifically, radar point cloud data refers to a spatial data set composed of a large number of discrete points generated after scanning the environment with radar (such as millimeter-wave radar or lidar). Each point typically contains three-dimensional spatial coordinates ((X, Y, Z), representing the position of a point on the surface of a target object), an intensity value (reflection intensity, reflecting the strength of the signal returned from that point), etc. Image data refers to two-dimensional visual information captured by a camera, typically including RGB images or grayscale images.

[0027] It should be noted that radar point cloud data possesses strong 3D spatial perception capabilities, providing information on target distance, orientation, and height, and exhibits strong robustness against lighting conditions and adverse weather. However, its spatial distribution is sparse, and target outlines are blurred, making it difficult to accurately identify detailed features. Image data, on the other hand, boasts high resolution and rich semantic information, clearly displaying target texture, color, and outline, facilitating target identification and classification. However, it lacks depth information and is susceptible to interference from factors such as lighting and occlusion. Therefore, a single modality is insufficient for comprehensive and accurate 3D target detection. Thus, it is necessary to fuse radar point cloud data and image data, leveraging their complementary strengths to improve detection accuracy and robustness. To ensure spatial and temporal consistency between the two types of data and avoid information errors caused by dynamic target movement or differences in viewing angle, radar and image data must be acquired at the same timestamp to achieve precise alignment and effective fusion.

[0028] In practical implementation, a multi-sensor system with time synchronization capabilities is configured, simultaneously connecting millimeter-wave radar sensors and image acquisition devices (such as RGB cameras) to a unified time synchronization module (such as a GPS clock or PPS pulse synchronizer). During system operation, acquisition signals are triggered according to a set frequency, controlling the radar sensor and image acquisition device to complete data acquisition at the same timestamp. The acquired radar point cloud data and image data are saved separately, and the same timestamp is marked in the data frame. The radar point cloud data and image data at the same time are aligned and paired through the timestamp.

[0029] S102. Construct a hybrid point cloud based on the radar point cloud data, and extract radar bird's-eye view features based on the hybrid point cloud.

[0030] Specifically, hybrid point cloud refers to a point cloud collection formed by combining radar point cloud data with image data to generate virtual point cloud data, and then stitching and fusing the two together. Hybrid point cloud not only preserves the spatial accuracy and sparse structure of radar point cloud data but also introduces image-guided generation to complete the point cloud. Radar bird's-eye view features refer to the process of projecting the hybrid point cloud onto a horizontal plane in the radar coordinate system (i.e., BEV: Bird's Eye View), then discretizing it using cylinders and using a feature extraction network to encode the features of each cylinder unit, ultimately obtaining a two-dimensional feature map.

[0031] In specific implementation, the step of constructing a hybrid point cloud based on the radar point cloud data and extracting radar bird's-eye view features based on the hybrid point cloud includes:

[0032] (1) The radar point cloud data is projected onto the image coordinate system using the intrinsic and extrinsic parameter matrices to obtain the projection point set.

[0033] Specifically, the intrinsic parameter matrix describes the internal imaging characteristics of the camera, mainly including information such as focal length, principal point coordinates (image center), and pixel size ratio. The intrinsic parameter matrix is ​​used to project 3D points in the camera coordinate system onto the 2D image plane. The extrinsic parameter matrix describes the spatial geometric relationship between the radar coordinate system and the camera coordinate system, mainly including rotation matrices and translation vectors. The projected point set refers to the set of 2D point coordinates in the image coordinate system (2D image plane) obtained after transforming the 3D points in the radar point cloud data through extrinsic parameter transformation (from radar coordinate system to camera coordinate system) and then through intrinsic parameter matrix transformation (projecting from camera coordinate system to image plane).

[0034] It should be noted that radar point cloud data is usually a set of three-dimensional points represented in the radar coordinate system. In order to achieve data alignment and joint perception between radar and image, this application needs to project the radar point cloud data in the radar coordinate system to the image coordinate system to achieve cross-modal fusion.

[0035] In practice, firstly, the radar point cloud data is transformed from the radar coordinate system to the camera coordinate system using an extrinsic parameter matrix. The transformation formula is as follows:

[0036] ;

[0037] Among them, the The three-dimensional coordinates of radar point cloud data in the radar coordinate system; For rotation matrix; the These are three-dimensional points in the transformed camera coordinate system; It is a translation vector.

[0038] Furthermore, the three-dimensional points in the camera coordinate system are projected onto the image plane using the intrinsic parameter matrix to obtain the two-dimensional coordinates in the image coordinate system. The projection formula is:

[0039] ;

[0040] Among them, the The intrinsic parameter matrix; The coordinates of a three-dimensional point in the camera coordinate system; The projection point is in the image coordinate system.

[0041] For example, in one embodiment, the i-th projection point in the set of projection points can be represented as: , wherein The coordinates in the image space, These are depth and radar signature, respectively.

[0042] (2) Match the set of projection points based on the instance mask information to determine the set of foreground points.

[0043] Specifically, instance mask information refers to the pixel region in the image corresponding to each target instance, extracted from image data using an image instance segmentation algorithm. Instance mask information typically exists in the form of a binary mask image, where each mask image represents the position and shape of a specific target in the image. The foreground point set refers to the set of points in the projection point set that fall within one or more instance mask regions. These points are determined to belong to a valid target in the image, rather than the background or invalid region.

[0044] It should be noted that the foreground point set is a subset of the projection point set. All points in the projection point set are two-dimensional points projected onto the image from radar point cloud data, while the foreground point set consists of points that satisfy the condition of "being located within the instance mask region," representing projection points with actual target semantic meaning.

[0045] In specific implementation, the step of matching the projection point set based on instance mask information to determine the foreground point set includes: performing instance segmentation on the image data to obtain the mask image corresponding to each target instance; performing spatial overlap matching between the projection point set and the mask image to determine whether each projection point falls within the mask area; determining the projection point falling within any mask area as the initial foreground point; and constructing the matching relationship between the image instance and the point cloud based on the correspondence between the initial foreground point and the mask area to form the foreground point set.

[0046] Specifically, an instance segmentation algorithm (such as Mask R-CNN) is used to process the image data to obtain a mask image corresponding to each target instance. Each mask image represents the pixel region of an instance. Each projection point of the radar point cloud data projected onto the image coordinate system is traversed, and it is determined whether it falls within the region represented by any instance mask image. Projection points falling within any mask region are determined as initial foreground points, and the mask instance to which each initial foreground point belongs is recorded. Further, based on the association between the initial foreground points and the corresponding mask instances, a one-to-one correspondence between image instances and radar point cloud data is constructed. Based on this correspondence, all projection points associated with image instances are summarized to obtain the foreground point set. It should be noted that the specific implementation process of the instance segmentation algorithm can be found in relevant technologies and will not be elaborated here.

[0047] (3) The foreground points are filtered based on the projection distance and category information to determine the foreground points.

[0048] Specifically, based on the above description, the foreground point set refers to the preliminary point cloud set obtained by projecting radar point cloud data onto the image coordinate system and matching it with the instance mask region in the image. Although these points spatially overlap with the target area in the image, they may contain noise points or edge points, resulting in limited accuracy and representativeness. Foreground points, on the other hand, are obtained by further filtering based on the foreground point set, incorporating projection distance and target category information, resulting in more accurate, dense, and representative key points. They more reliably correspond to the location of the real target, helping to improve the accuracy of subsequent virtual point cloud generation and target detection. Therefore, foreground points are a subset selected from the foreground point set, representing a more "refined" result that emphasizes accuracy and representativeness.

[0049] In specific implementation, the step of filtering the set of foreground points based on projection distance and category information to determine foreground points includes: for each foreground point in the set of foreground points, determining a reference point for each mask region based on the distance between the foreground point and the origin of the camera coordinate system; for each mask region, determining a distance threshold based on the category information of the instance and using the reference point as a reference, and removing foreground points that exceed the distance threshold; determining the number of foreground points in the filtered set of foreground points, and when the number exceeds a preset filtering ratio, retaining foreground points that are more than a first preset distance from the center reference point and replacing foreground points that are less than a second preset distance from the center reference point, thus obtaining the foreground points.

[0050] Specifically, the reference point refers to the position within each instance mask region that is closest to the camera, calculated based on the distances between each point in the foreground point set and the origin of the camera coordinate system.

[0051] The method provided in this embodiment selects the closest foreground point within the masked area to the camera as the reference point, which can improve the spatial consistency between the image viewpoint and the radar point cloud and effectively suppress interference points caused by occlusion or penetration. Since the image is based on viewpoint projection, objects closer to the camera in the scene are easier to observe completely, while distant or occluded targets have incomplete information or are even invisible in the image. If all foreground points within the masked area are used directly for processing, it is easy to introduce point cloud noise from misjudged points or other occluders behind the same instance, thereby destroying the geometric correspondence between the image and the point cloud. Therefore, by selecting the minimum depth point as the reference, it can be ensured that the retained point cloud is located on the front surface of the visible objects in the image, enhancing the accuracy of multimodal perception. In addition, starting from the depth of this reference point and combining the spatial size prior of the target category, a reasonable threshold range can be constructed in the depth direction to further filter out the effective foreground points that truly belong to the instance. This approach not only simplifies the logic of spatial clipping and avoids complex 3D clustering or boundary calculations, but also improves robustness to occluded objects, making the point cloud and image more accurately aligned in space. This provides a more reliable input basis for subsequent tasks such as object detection, depth estimation, and 3D bounding box fitting.

[0052] In practical implementation, based on the above description, for each foreground point in the set of foreground points, first determine its position in the camera coordinate system, then determine the distance between each foreground point and the origin of the camera coordinate system. Iterate through all calculated distances and identify the closest foreground point as the reference point. Further, determine the instance mask image to which each foreground point belongs and identify its target instance category (e.g., person, vehicle). Further, based on the instance category of the foreground point, call prior category information (e.g., vehicle or pedestrian) to construct a distance threshold. Construct a distance threshold circle or other geometric region centered on the reference point and based on the category information. Foreground points outside the distance threshold are eliminated, and points within a reasonable range are retained as candidate foreground points. The number of foreground points retained after initial screening is counted and compared with a pre-set filtering ratio. If the number of filtered foreground points exceeds the filtering ratio, retain foreground points that are more than a first preset distance from the central reference point. Simultaneously, use the retained foreground points to replace foreground points that are more than a second preset distance from the reference point. Repeat the above process to obtain the filtered foreground points.

[0053] It's important to note that the distance threshold is determined based on the target category information corresponding to each masked region. Specifically, different target categories (such as pedestrians, cars, cyclists, etc.) have different physical dimensions in reality. Therefore, a reasonable spatial range is set for each target category as the basis for filtering foreground points. In determining the distance threshold, the target category of the mask is first identified based on the image instance segmentation results. Then, the typical dimensions (length, width, height, or radius) of that target category in the radar point cloud are obtained by looking up a table or using prior knowledge. Using a reference point as a benchmark, and combining this size information, a threshold region (such as a spherical or ellipsoidal range centered at the center point) is constructed in three-dimensional space. The distance threshold is the maximum radius or boundary distance of this range. For example, if a masked region is identified as belonging to the "pedestrian" category, a radius of 1 meter can be set as the distance threshold; if it's a "car," the threshold can be set to 2 meters. Subsequently, all foreground points whose Euclidean distance from the reference point exceeds this threshold are removed to ensure that the retained foreground points are more likely to actually belong to the target instance, thereby improving the matching accuracy of the point cloud data.

[0054] The method provided in this embodiment determines the reference point for each mask region by calculating the distance between the foreground point and the origin of the camera coordinate system. Then, it dynamically determines the distance threshold by combining instance category information, achieving adaptive filtering for targets of different sizes and shapes. This removes outlier or noise points that are too far from the reference point from the foreground point set, effectively filtering out interference data irrelevant to the target and improving the purity and accuracy of the foreground point set. This method not only considers the geometric relationship of spatial location but also integrates category information, ensuring that the filtering threshold reasonably matches the spatial range of different targets, avoiding erroneous rejection or retention due to a fixed threshold. By setting a preset filtering ratio, when the number of filtered foreground points is too large, a dual threshold strategy is further adopted: retaining foreground points that are more than a first preset distance from the reference point, while replacing foreground points that are less than a second preset distance from the reference point. This achieves a uniform spatial distribution of the point cloud, preventing the point cloud from being too concentrated in the central area of ​​the mask, and improving the spatial representation and detail richness of the point cloud. This process effectively maintains the morphological characteristics of the target, reduces detection blind spots caused by uneven or clustered point cloud density, and improves the quality and realism of virtual point cloud generation. Overall, this screening strategy achieves high-precision screening of foreground point sets by combining geometric distance and category information as dual constraints. It ensures that the screening results have both high relevance and maintain spatial integrity and diversity, providing high-quality input for subsequent virtual point cloud generation and hybrid point cloud construction. This significantly improves the accuracy and robustness of multimodal 3D target detection and enhances the system's ability to identify and adapt to targets in complex environments.

[0055] (4) Within each instance mask area, for each pixel, a corresponding point cloud generation method is matched based on the distance between the pixel and the foreground point, and a virtual point cloud is generated based on the point cloud generation method.

[0056] In specific implementation, within each instance mask region, for each pixel, a point cloud generation method is matched based on the distance between the pixel and the foreground point, and a virtual point cloud is generated based on the point cloud generation method. This includes: for each pixel within the instance mask region, calculating the distance between the pixel and the corresponding foreground point in the image coordinate system; determining the radius based on the category information of the target corresponding to the instance mask and the spatial scale of the mask region; when the distance is less than the radius, generating the virtual point cloud corresponding to the pixel based on a Gaussian distribution strategy; when the distance is not less than the radius, generating the virtual point cloud corresponding to the pixel based on a uniform distribution strategy; and summing up the virtual point clouds generated from all pixels within the instance mask region to obtain a set of virtual point clouds corresponding to the instance mask region.

[0057] Specifically, within each instance mask region, for each pixel within that region, firstly, the Euclidean distance between that pixel and its corresponding foreground point in the image coordinate system is calculated. Then, based on the category information of the target corresponding to the instance mask (such as vehicles, pedestrians, etc.) and the spatial scale of the current mask region in the image, a radius value representing the effective generation range is determined. Next, it is determined whether the distance between each pixel and the foreground point is less than this radius: if it is less than the radius, a Gaussian distribution strategy is used, with the foreground point as the center and the distance as the weight, to generate the virtual point cloud corresponding to that pixel; if it is not less than the radius, a uniform distribution strategy is used to generate the virtual point cloud corresponding to that pixel within the mask region. Finally, the virtual point cloud data generated from all pixels within the mask region are merged to form the complete virtual point cloud set corresponding to that instance mask.

[0058] Optionally, based on the target's category (e.g., pedestrian, bicycle, car, truck, etc.), a typical physical space size is preset (e.g., pedestrian width is approximately 0.6 meters, car width is approximately 1.8 meters, etc.). The size information of this instance mask in the image coordinate system is statistically analyzed, typically using the mask's width, height, or the diagonal length of its smallest bounding rectangle as metrics. The mapping ratio between image pixel coordinates and actual dimensions is calculated using the relationship between image scale and actual size. For example, the actual physical length corresponding to a unit pixel is estimated by the overlap between the LiDAR projection point and the mask area. The actual size of the target is multiplied by the image-physical space scaling ratio to obtain the radius in pixels. For example, for a pedestrian, if the preset half-width is 0.3 meters, and 1 pixel in the image corresponds to 0.01 meters, the radius can be set to 30 pixels; this radius is used to define which pixels are sufficiently close to the foreground point to be considered their neighborhood, thus used to generate a virtual point cloud.

[0059] The method provided in this embodiment employs different point cloud generation strategies based on the distance between pixels and the foreground point, achieving fine-grained control over virtual point cloud generation. When the distance is less than a set radius, a Gaussian distribution strategy is used to generate the virtual point cloud. This strategy can more concentratedly simulate the density distribution of real point clouds in space, enhancing the detail representation of the point cloud in the core area of ​​the target, improving the realism and local accuracy of the virtual point cloud, and facilitating the capture of key features of the target. When the distance is not less than the radius, a uniform distribution strategy is used to generate the point cloud. This effectively avoids overfitting or information redundancy problems caused by excessively dense virtual point clouds, ensuring that the point cloud is evenly distributed at the edges or outer regions, enhancing the spatial coverage and integrity of the point cloud, and reducing information loss in sparse areas. By combining category information and the spatial scale of the mask region to determine the radius, the point cloud generation process can adapt to the size and shape characteristics of different targets, dynamically adjusting the generation strategy and avoiding the limitations of fixed parameters. Overall, this method, by dynamically matching different generation strategies based on distance, balances the richness of local details and the uniformity of overall space in virtual point clouds, improves the quality and expressive power of virtual point clouds, enhances the fusion effect of point clouds and image information, and further improves the accuracy and robustness of multimodal 3D target detection in target recognition.

[0060] Optionally, the specific formula for generating the virtual point cloud corresponding to the pixel based on the Gaussian distribution strategy is as follows:

[0061] ;

[0062] Among them, the For virtual point clouds, the , These are the horizontal and vertical coordinates of the virtual point cloud, respectively; , The standard deviation; , , where x and y are the x and y coordinates of the foreground point, respectively.

[0063] Optionally, the specific formula for generating the virtual point cloud corresponding to the pixel based on the uniform distribution strategy is as follows:

[0064] ;

[0065] Among them, the For virtual point clouds; the This represents the area of ​​the instance region.

[0066] For example, in one embodiment, the generated virtual point cloud can be represented as: , wherein The coordinates in image space; This is to represent the semantic information of the current instance mask using one-hot encoding.

[0067] (5) For each virtual point cloud, the nearest foreground point is matched based on the distance between the virtual point cloud and each foreground point, the virtual point cloud is updated based on the nearest foreground point cloud, and the updated virtual point cloud is spliced ​​with the radar point cloud data to obtain a hybrid point cloud.

[0068] In practice, for each virtual point cloud, the Euclidean distance between it and all foreground points in three-dimensional space is calculated, and the closest foreground point is determined as its matching target. Then, the attribute information of the closest foreground point, including depth value, reflection intensity, velocity information, etc., is extracted and these attributes are assigned or interpolated into the virtual point cloud to update the virtual point cloud. Next, all the updated virtual point clouds are stitched together with the original radar point cloud data in spatial coordinates and attribute dimensions to form a hybrid point cloud data in a unified format.

[0069] For example, in one embodiment, the virtual point cloud can be represented as: , wherein The coordinates in image space; The semantic information of the current instance mask is represented using one-hot encoding; These represent depth and radar features, respectively. Furthermore, the virtual point cloud is back-projected onto the radar space to obtain the virtual point cloud in the radar space: , wherein The coordinates in radar space, the To represent the semantic information of the current instance mask using one-hot encoding, the For radar features. To align with the feature length of the virtual point cloud, the semantic information of the radar point cloud data... Fill with zeros. The final blended point cloud, after being mixed with the virtual point cloud, can be represented as: , wherein The coordinates in radar space; To represent the semantic information of the current instance mask using one-hot encoding, the Radar characteristics; the To distinguish the types of different points using one-hot encoding.

[0070] (6) Discretize the hybrid point cloud in the radar space according to the columnarization strategy, extract features from the processed hybrid point cloud, and obtain radar bird's-eye view features.

[0071] In practice, the process begins by constructing a columnar grid in the radar space, dividing the entire space into multiple fixed-size columnar cells. Then, each point in the hybrid point cloud is assigned to a corresponding columnar cell according to its spatial coordinates. Feature encoding is performed on the points within each columnar cell, such as counting the number of points, maximum height, and average reflection intensity. Furthermore, the features extracted from all columnar cells are uniformly arranged to form a structured feature representation. Finally, this structured feature is input into a feature extraction network, where convolution operations are used to extract spatial geometric information, resulting in a radar bird's-eye view feature representing the spatial distribution characteristics of the target.

[0072] For example, in one embodiment, the obtained radar bird's-eye view features can be represented as: , wherein Features of radar bird's-eye view; the The number of feature channels; , The size of the feature map.

[0073] S103. Based on the radar point cloud data and the image data, determine the bird's-eye view features of the image.

[0074] Specifically, image bird's-eye view features refer to the feature information extracted from image data that spatially represents a scene from a bird's-eye view (i.e., a top-down angle). Image bird's-eye view features are formed by combining image data (such as color, texture, edges, semantic categories, etc.) with projection transformation, geometric calibration and other processing methods to transform it into a spatial coordinate system corresponding to radar point clouds and express it in a top-down manner, thus forming a structured feature representation that can be aligned with radar bird's-eye view features.

[0075] In specific implementation, determining the bird's-eye view features based on the radar point cloud data and the image data includes:

[0076] (1) Based on the image data, use a deep neural network to extract image features, and extract semantic features of the image based on the image features.

[0077] In practice, the image data is first preprocessed, including scaling and normalization. Then, the preprocessed image data is input into a deep neural network, such as using a backbone network like ResNet or Swing Transformer and a neck network to extract multi-scale features from the image data. From the extracted multi-layer image features, semantic information in the image data is further extracted through specific semantic enhancement modules (such as FPN or Transformer encoder) to form a semantic feature map containing high-level semantic information.

[0078] For example, in one embodiment, image features can be represented as: Semantic features can be represented as: , .

[0079] (2) Calculate the image depth probability distribution based on the image features.

[0080] In practice, an image depth network is used to calculate the image depth probability distribution of image features, as shown in the following formula:

[0081] ;

[0082] Among them, the The image depth probability distribution; For image features; the This indicates that the input is the number of channels. The output is the number of channels. 3×3 convolution.

[0083] (3) Calculate the radar depth probability distribution based on the radar point cloud data.

[0084] In practice, the radar point cloud data is first voxelized in pixel space according to the size and depth range of the image data, and then the voxel data is encoded to obtain the projection blending point features. Furthermore, by utilizing radar occupancy networks, the characteristic shape is transformed from... Remodeling To align it with image features, the specific formula is as follows:

[0085] ;

[0086] Among them, the The radar depth probability distribution; Features of projected blending points; Indicates the number of input channels is The number of output channels is 3×3 convolution.

[0087] (4) The image depth probability distribution and the radar depth probability distribution are fused to obtain the fused depth probability distribution.

[0088] In practice, the image depth probability distribution and the radar depth probability distribution are fused based on the following formula: ;

[0089] Among them, the For the fusion depth probability distribution; the The radar depth probability distribution; For the image depth probability distribution; in the function middle, This represents the input query. This represents the input value and key. The formula for calculating neighborhood attention, which incorporates the deep probability distribution, does not have a direct supervisory signal during training.

[0090] (5) Based on the semantic features and the fusion depth probability distribution, the image bird's-eye view features are generated through viewpoint transformation.

[0091] In practice, the bird's-eye view features of the image are generated based on the following perspective transformation formula:

[0092] ;

[0093] Among them, the The features of the image's bird's-eye view; For the fusion depth probability distribution; the For semantic features; the For the voxel pooling module in perspective switching, It represents the outer product.

[0094] S104. Based on the channel attention mechanism and the spatial attention mechanism, the radar bird's-eye view features and the image bird's-eye view features are fused to obtain comprehensive features, and three-dimensional target detection is performed based on the comprehensive features.

[0095] In specific implementation, the fusion of the radar bird's-eye view features and the image bird's-eye view features to obtain comprehensive features includes:

[0096] (1) The radar bird's-eye view features and the image bird's-eye view features are spliced ​​together in the channel dimension to obtain the initial fusion features.

[0097] In practice, the radar bird's-eye view features and the image bird's-eye view features are first aligned to ensure they maintain consistency in spatial dimensions (such as height and width). Then, the feature tensors are concatenated along their channel dimensions, connecting the two sets of features sequentially in the channel direction to form a fused initial feature with more channels. The concatenation process can be represented as:

[0098] ;

[0099] Among them, the To fuse initial features; the Features of radar bird's-eye view; the Features of an image from a bird's-eye view.

[0100] (2) Perform global average pooling and global max pooling on the initial fusion features respectively to obtain the first pooling feature and the second pooling feature.

[0101] In practice, global average pooling and global max pooling operations are performed on the initial features in the spatial dimensions (i.e., height and width). During global average pooling, the feature values ​​at all spatial locations within each channel are averaged to obtain the first pooled feature. During global max pooling, the maximum feature value at all spatial locations within each channel is taken to obtain the second pooled feature.

[0102] (3) Perform multilayer perceptron processing on the first pooling feature and the second pooling feature respectively, add the processing results together and pass them through the activation function to obtain the channel attention weights.

[0103] Specifically, a multilayer perceptron, also known as an MLP, typically consists of two fully connected layers, where the first layer is used for dimensionality reduction and the second layer is used to restore the original dimensionality.

[0104] In practice, the first and second pooling features are first input into a shared multilayer perceptron. The first layer of the multilayer perceptron performs dimensionality reduction on the first and second pooling features, while the second layer restores their original dimensions. After processing the two pooling features through the multilayer perceptron, two intermediate channel attention results are obtained. These two intermediate results are then summed element-wise and processed through a sigmoid activation function to obtain the channel attention weights. The size of the channel attention weights is consistent with the number of channels in the input features. The formula for calculating the channel attention weights is as follows:

[0105] ;

[0106] Among them, the For channel attention weights; the To fuse initial features; the To perform global average pooling on the initial fused features; To perform global max pooling on the initial features of the fusion.

[0107] (4) The channel attention weights are weighted with the initial fusion features one channel at a time to obtain the channel enhancement features.

[0108] In practice, the channel attention weights are first expanded to the same shape as the initial fusion features, making them correspond to each channel in spatial dimension. Then, each channel feature map of the initial fusion features is multiplied channel-by-channel with its corresponding channel attention weight, performing a weighted operation. Finally, the weighted feature maps of each channel are combined to form the channel-enhanced features. The calculation formula for the channel-enhanced features is as follows:

[0109] ;

[0110] Among them, the For channel enhancement features; the For channel attention weights; the To integrate initial features.

[0111] (5) Apply dilated convolution to the channel enhancement features and extract spatial attention weights through convolution operation.

[0112] In practice, firstly, dilated convolution is applied to the channel enhancement features to expand the receptive field and preserve spatial location information, generating an intermediate feature map. Then, standard convolution is performed on the intermediate feature map to extract response values ​​containing spatial location information. Next, the response values ​​are normalized (e.g., using the sigmoid function) to obtain the spatial attention weights. The formula for calculating the spatial attention weights is as follows:

[0113] ;

[0114] Among them, the For spatial attention weights; the Enhance channel features.

[0115] It should be noted that the spatial attention weights are supervised by the bird's-eye view mask generated from the ground truth bounding box using focal loss.

[0116] (6) After weighting the spatial attention weights and the initial fusion features position by position, the fusion features are extracted through convolution operation to obtain the comprehensive features.

[0117] In practice, the spatial attention weights are first multiplied positionally by position of the initial fusion features to achieve spatial attention weighting. Then, the weighted feature map is input into a convolutional layer for convolution to extract higher-level fusion features. The final output features are the comprehensive features, used for subsequent 3D object detection tasks. The formula for calculating the comprehensive features is as follows:

[0118] ;

[0119] Among them, the For comprehensive features; the For spatial attention weights; the To integrate initial features.

[0120] Furthermore, after obtaining the integrated features, the fused integrated features are input into the backbone network of the 3D object detection network. The backbone network extracts multi-scale features through multi-layer convolutional operations. Then, the features output by the backbone network are fed into the detection head, which includes a classification branch and a regression branch. The classification branch processes the features at each spatial location and outputs the object class probability, while the regression branch predicts the bounding box parameters of the 3D object (including position, size, and orientation). Then, all candidate object boxes are sorted according to their classification scores, and a non-maximum suppression algorithm is applied to remove duplicate boxes with high overlap. Finally, the filtered 3D object categories and their corresponding 3D bounding boxes are output, thus achieving 3D object detection.

[0121] The method provided in this embodiment first achieves preliminary fusion of multimodal information by stitching together radar bird's-eye view features and image bird's-eye view features along the channel dimension, ensuring that the features of the two sensor data can be retained in parallel and complement each other. Next, global statistical information is extracted using two different pooling methods: global average pooling and global max pooling, respectively capturing the overall trend and extreme response of the features, enabling the attention mechanism to more comprehensively perceive the importance of the features. The two types of pooled features are nonlinearly transformed and added using a multilayer perceptron, and then channel attention weights are generated through an activation function, achieving dynamic weighting of features from different channels, highlighting channel features more critical to target detection, thereby improving the discriminative ability of feature representation. Subsequently, the weighted channels... By applying dilated convolution to enhance features and extracting spatial attention weights through convolution operations, the receptive field is effectively expanded while maintaining feature resolution, enhancing the ability to capture spatial information. This allows the fused features to accurately focus on the spatial distribution and local details of the target. By weighting the spatial attention weights with the initial fused features position by position and then further fusing them through convolution, the final comprehensive features not only contain rich multimodal information but also undergo optimization through both channel and spatial attention mechanisms, significantly enhancing the expressive power and discriminative power of the features. Overall, this design allows the comprehensive features to not only take into account the advantages of both radar and image processing but also dynamically adjust feature weights and spatial focusing through fine-grained attention mechanisms, improving the accuracy and robustness of 3D target detection.

[0122] The method provided in this embodiment, in its first aspect, avoids target displacement errors caused by asynchronous sensor acquisition by acquiring radar point cloud and image data at the same timestamp, achieving accurate matching between projection points and image instance masks, improving the reliability of foreground point selection and virtual point cloud generation, and ensuring that the fused features truly reflect the spatiotemporal information of the scene, significantly improving the spatiotemporal alignment accuracy of 3D target detection. In its second aspect, by utilizing projection matching, instance mask selection, and dynamic point cloud generation strategies to construct a hybrid point cloud, it retains the spatial accuracy of the radar point cloud while simultaneously using image-guided generation to complete the point cloud and fill sparse areas. Combined with columnar discretization and a feature extraction network, the hybrid point cloud is converted into structured radar bird's-eye view features, strengthening the representation capability of spatial geometric information and providing richer 3D structural clues for subsequent fusion. Thirdly, by fusing image semantic features with depth probability distribution to generate bird's-eye view features, the lack of depth in images is compensated for, achieving semantic-geometric complementarity with radar features. Based on a channel attention mechanism, the channel importance of multimodal features is dynamically adjusted to focus on key information (such as radar distance and image texture). Simultaneously, a spatial attention mechanism combined with dilated convolution is used to accurately locate target regions and suppress background interference. The resulting comprehensive features combine multidimensional information with fine-grained attention optimization, significantly improving target classification, localization accuracy, and robustness in complex scenes. Fourthly, target category information is introduced into point cloud processing and feature generation, adaptively adjusting the screening threshold and point cloud generation strategy to adapt to targets of different sizes and shapes. Through spatial uniformity control and multimodal fusion of hybrid point clouds, the dependence of a single sensor on the environment (such as lighting and occlusion) is reduced, enhancing the model's detection stability under adverse conditions such as nighttime, rain, and snow, and improving the system's generalization ability to diverse targets and complex scenes.

[0123] Corresponding to the aforementioned embodiment of a three-dimensional target detection method based on radar mixed points and image fusion, this application also provides an embodiment of a three-dimensional target detection device based on radar mixed points and image fusion.

[0124] Figure 2 This is a schematic diagram of the three-dimensional target detection device based on radar hybrid point and image fusion provided in Embodiment 2 of this application. Please refer to... Figure 2 The apparatus provided in this embodiment includes an acquisition module 210, an extraction module 220, a determination module 230, and a fusion module 240;

[0125] The acquisition module 210 is used to acquire radar point cloud data and image data collected at the same timestamp;

[0126] The extraction module 220 is used to construct a hybrid point cloud based on the radar point cloud data, and extract radar bird's-eye view features based on the hybrid point cloud.

[0127] The determining module 230 is used to determine the bird's-eye view features of the image based on the radar point cloud data and the image data;

[0128] The fusion module 240 is used to fuse the radar bird's-eye view features and the image bird's-eye view features based on the channel attention mechanism and the spatial attention mechanism to obtain comprehensive features, and to perform three-dimensional target detection based on the comprehensive features.

[0129] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0130] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0131] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0132] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A three-dimensional target detection method based on radar point fusion and image fusion, characterized in that, The method includes: Acquire radar point cloud data and image data collected at the same timestamp; A hybrid point cloud is constructed based on the radar point cloud data, and radar bird's-eye view features are extracted based on the hybrid point cloud. Based on the radar point cloud data and the image data, the features of the bird's-eye view of the image are determined; Based on the fusion of radar bird's-eye view features and image bird's-eye view features using channel attention mechanism and spatial attention mechanism, a comprehensive feature is obtained, and three-dimensional target detection is performed based on the comprehensive feature; The fusion of the radar bird's-eye view features and the image bird's-eye view features yields comprehensive features, including: The radar bird's-eye view features and the image bird's-eye view features are concatenated along the channel dimension to obtain the initial fusion features; The initial fusion features are subjected to global average pooling and global max pooling respectively to obtain the first pooling feature and the second pooling feature; The first pooling feature and the second pooling feature are processed by a multilayer perceptron, and the processing results are added together and then passed through an activation function to obtain the channel attention weights. The channel attention weights are weighted sequentially with the initial fusion features to obtain the channel-enhanced features; Dilated convolution is applied to the channel enhancement features, and spatial attention weights are extracted through the convolution operation; After weighting the spatial attention weights and the initial fusion features position by position, the fusion features are extracted through convolution operations to obtain the comprehensive features.

2. The method according to claim 1, characterized in that, The process of constructing a hybrid point cloud based on the radar point cloud data and extracting radar bird's-eye view features based on the hybrid point cloud includes: The radar point cloud data is projected onto the image coordinate system using intrinsic and extrinsic parameter matrices to obtain a set of projection points; The set of projection points is matched based on the instance mask information to determine the set of foreground points; The foreground points are selected by filtering the set of foreground points based on projection distance and category information; Within each instance mask area, for each pixel, a corresponding point cloud generation method is matched based on the distance between the pixel and the foreground point, and a virtual point cloud is generated based on the point cloud generation method. For each virtual point cloud, the nearest foreground point is matched based on the distance between the virtual point cloud and each foreground point, the virtual point cloud is updated based on the nearest foreground point cloud, and the updated virtual point cloud is stitched together with the radar point cloud data to obtain a hybrid point cloud; The hybrid point cloud is discretized in the radar space according to a columnarization strategy, and features are extracted from the processed hybrid point cloud to obtain radar bird's-eye view features.

3. The method according to claim 2, characterized in that, The step of matching the set of projection points based on instance mask information to determine the set of foreground points includes: The image data is segmented into instances to obtain the mask image corresponding to each target instance; Spatial overlap matching is performed between the set of projection points and the mask image to determine whether each projection point falls within the mask area. Projection points falling within any mask area are determined as initial foreground points. Based on the correspondence between the initial foreground points and the mask area, a matching relationship between image instances and point clouds is constructed to form a set of foreground points.

4. The method according to claim 2, characterized in that, The process of filtering the set of foreground points based on projection distance and category information to determine the foreground points includes: For each foreground point in the set of foreground points, a reference point for each mask region is determined based on the distance between the foreground point and the origin of the camera coordinate system. For each masked region, based on the reference point, a distance threshold is determined according to the instance's category information, and foreground points exceeding the distance threshold are removed. Determine the number of foreground points in the filtered set of foreground points. When the number exceeds a preset filtering ratio, retain foreground points that are more than a first preset distance from the center reference point and replace foreground points that are less than a second preset distance from the center reference point to obtain the foreground points.

5. The method according to claim 2, characterized in that, Within each instance mask region, for each pixel, a point cloud generation method is matched based on the distance between the pixel and a foreground point, and a virtual point cloud is generated based on the point cloud generation method, including: For each pixel within the instance mask region, calculate the distance between the pixel and the corresponding foreground point in the image coordinate system; The radius is determined based on the category information of the target corresponding to the instance mask and the spatial scale of the mask area; When the distance is less than the radius, a virtual point cloud corresponding to the pixel is generated based on a Gaussian distribution strategy; When the distance is not less than the radius, a virtual point cloud corresponding to the pixel is generated based on a uniform distribution strategy; The virtual point clouds generated from all pixels within the instance mask region are aggregated to obtain the virtual point cloud set corresponding to the instance mask region.

6. The method according to claim 1, characterized in that, The step of determining the bird's-eye view features based on the radar point cloud data and the image data includes: Based on the image data, image features are extracted using a deep neural network, and semantic features of the image are extracted based on the image features. Based on the image features, calculate the image depth probability distribution; Based on the radar point cloud data, the radar depth probability distribution is calculated; The image depth probability distribution and the radar depth probability distribution are fused to obtain a fused depth probability distribution; Based on the semantic features and the fusion depth probability distribution, bird's-eye view features of the image are generated through viewpoint transformation.

7. A three-dimensional target detection device based on radar hybrid point fusion and image fusion, characterized in that, The device includes an acquisition module, an extraction module, a determination module, and a fusion module; The acquisition module is used to acquire radar point cloud data and image data collected at the same timestamp; The extraction module is used to construct a hybrid point cloud based on the radar point cloud data, and extract radar bird's-eye view features based on the hybrid point cloud; The determining module is used to determine the bird's-eye view features of the image based on the radar point cloud data and the image data; The fusion module is used to fuse the radar bird's-eye view features and the image bird's-eye view features based on the channel attention mechanism and the spatial attention mechanism to obtain comprehensive features, and to perform three-dimensional target detection based on the comprehensive features. The fusion of the radar bird's-eye view features and the image bird's-eye view features yields comprehensive features, including: The radar bird's-eye view features and the image bird's-eye view features are concatenated along the channel dimension to obtain the initial fusion features; The initial fusion features are subjected to global average pooling and global max pooling respectively to obtain the first pooling feature and the second pooling feature; The first pooling feature and the second pooling feature are processed by a multilayer perceptron, and the processing results are added together and then passed through an activation function to obtain the channel attention weights. The channel attention weights are weighted sequentially with the initial fusion features to obtain the channel-enhanced features; Dilated convolution is applied to the channel enhancement features, and spatial attention weights are extracted through the convolution operation; After weighting the spatial attention weights and the initial fusion features position by position, the fusion features are extracted through convolution operations to obtain the comprehensive features.

Images