Radar-based vehicle body structure region identification method and system based on sparse features
By combining spatiotemporal alignment and sparse feature fusion with LiDAR and camera data, the problems of incomplete information and insufficient robustness in traditional methods are solved, achieving high-precision vehicle structure recognition and improved stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HEBEI PORT GROUP SHULIAN TECHNOLOGY (XIONGAN) CO LTD
- Filing Date
- 2026-02-25
- Publication Date
- 2026-05-29
AI Technical Summary
Traditional methods rely on a single data type, resulting in incomplete information, difficulty in accurately identifying vehicle structural areas, low accuracy in complex environments, insufficient robustness, and significant noise impact.
By spatiotemporally aligning LiDAR point cloud data with camera images, sparse features are extracted, and target detection and semantic segmentation are performed using a deep learning model. Feature variance is calculated to establish dynamic fusion weights, and a temporal filtering update strategy is introduced for weighted processing and geometric fitting to identify vehicle body structure regions.
It achieves higher precision in vehicle structure area recognition, enhances the robustness and stability of the system, reduces false recognition, and improves recognition accuracy, especially in complex backgrounds.
Smart Images
Figure CN122116071A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of structural recognition technology, specifically to a method and system for vehicle structure region recognition based on radar with sparse features. Background Technology
[0002] Currently, traditional methods mainly rely on a single type of data (such as using only LiDAR data or only image data), which leads to incomplete information and may not be able to fully capture the complex features of the target. Moreover, due to the lack of spatiotemporal alignment, traditional methods are prone to errors when dealing with dynamic scenes, resulting in low recognition accuracy, especially in fast-moving or complex environments, where they cannot accurately identify the structural areas of the vehicle.
[0003] Furthermore, traditional methods often use simple geometric feature extraction techniques, which are sensitive to noise and interference and may not be able to effectively distinguish targets in complex backgrounds, leading to misidentification or missed identification. Moreover, traditional methods adopt fixed feature fusion strategies, which are difficult to adapt to changes in the environment, resulting in unstable performance and insufficient robustness in different scenarios. In addition, traditional methods often lack effective noise suppression mechanisms when processing noisy data, which may cause the recognition results to be severely affected by instantaneous noise, thereby reducing the reliability of the entire system. Summary of the Invention
[0004] To achieve the above objectives, the present invention provides the following technical solution: a vehicle structure region identification method based on sparse feature radar, comprising: Acquire point cloud data of radar scanning the vehicle structure and camera images at the corresponding timestamps; use the external parameter calibration matrix of LiDAR and camera to unify the coordinates of point cloud data and camera images, and project the three-dimensional spatial coordinates in the radar coordinate system onto the image plane to obtain spatiotemporally aligned radar point cloud data and image data. Based on the spatiotemporally aligned radar point cloud data, sparse features of the vehicle body structure are extracted. The sparse features include edge feature points, corner feature points and planar feature points of the vehicle body structure. A deep learning vision model is used to perform target detection and semantic segmentation on the spatiotemporally aligned image data to obtain a semantic label map of the vehicle body region. By projecting and mapping sparse features onto semantic label maps, the semantic information of the image is transferred to radar sparse feature points, resulting in a cross-modal sparse feature map with semantic priors. The feature variances of image features and radar point cloud depth data in the cross-modal sparse feature map are calculated separately. Dynamic update fusion weights of image and radar point cloud are established based on the feature variance relationship. The cross-modal sparse feature map is weighted using the dynamic update fusion weights, and a temporal filtering update strategy is introduced to dynamically smooth the weight coefficients. The cross-modal sparse feature map after dynamic smoothing is divided into multiple first image blocks, and the multiple first image blocks are further divided to obtain multiple image block sets corresponding to the vehicle body structure region. Based on the feature intensity and weight coefficients after dynamic smoothing in each image patch set, it is determined whether the region corresponding to each image patch set in the vehicle structure is the target structure region; the boundary point set of the target structure region is obtained, and the boundary point set is geometrically fitted using the weighted least squares method to obtain the vehicle boundary line and the overall geometric contour.
[0005] Preferably, the three-dimensional spatial coordinates in the radar coordinate system are projected onto the image plane to obtain spatiotemporally aligned radar point cloud data and image data, including: Obtain the three-dimensional spatial coordinates in the radar coordinate system and the camera's intrinsic parameter matrix; Based on the extrinsic calibration matrix, intrinsic matrix and three-dimensional spatial coordinates, calculate the two-dimensional pixel coordinates of each three-dimensional point in the radar point cloud after it is projected onto the image plane. By using two-dimensional pixel coordinates, the spatial geometric information of radar point clouds is aligned with the pixel information of images, thus achieving spatiotemporal synchronization of radar and camera data.
[0006] Preferably, the sparse features are projected onto the semantic label map to transfer the image semantic information to the radar sparse feature points, resulting in a cross-modal sparse feature map with semantic priors, including: By using projection mapping, the vehicle semantic labels in the semantic label map are assigned to the corresponding radar sparse feature points; Retain edge feature points, corner feature points, and planar feature points that belong to the vehicle body structure, and remove sparse background feature points that do not belong to the vehicle body structure. The retained radar sparse feature points are jointly encoded with the corresponding image semantic information to form a cross-modal sparse feature map with geometric, texture and semantic feature information.
[0007] Preferably, the feature variances of image features and radar point cloud depth data in the cross-modal sparse feature map are calculated separately, and dynamic update fusion weights for the image and radar point cloud are established based on the feature variance relationship, including: The feature variance is calculated for the image feature channels in the cross-modal sparse feature map, and the feature variance is also calculated for the depth feature channel of the radar point cloud. When the variance of image features is greater than the variance of radar point cloud depth features, reduce the image fusion weight and increase the radar point cloud weight. When the variance of radar point cloud depth features is greater than the variance of image features, reduce the radar point cloud fusion weight and increase the image weight. Establish dynamic weighting coefficients for images and dynamic weighting coefficients for radar point clouds.
[0008] Preferably, the cross-modal sparse feature map is weighted using dynamically updated fusion weights, and a temporal filtering update strategy is introduced to dynamically smooth the weight coefficients, including: The feature vectors in the cross-modal sparse feature map are weighted and summed by dynamically updating the fusion weights at the current time. A time-series filtering update strategy is introduced, which uses the smoothing weight coefficient of the previous time step and the dynamically updated weight of the current time step to perform a weighted average.
[0009] Preferably, the dynamically smoothed cross-modal sparse feature map is divided into multiple first image blocks, and these first image blocks are further divided to obtain multiple image block sets corresponding to the vehicle body structure region, including: The cross-modal sparse feature map after dynamic smoothing is divided into multiple first image blocks according to a preset grid size; Multiple first image blocks belonging to the same vehicle body structure region are aggregated into one image block set; Ensure that each image patch set contains a second image patch from different sparse feature types, with the second image patch corresponding to edge features, corner features, and planar features, respectively.
[0010] Preferably, based on the feature intensity and weight coefficients after dynamic smoothing in each image patch set, it is determined whether the region corresponding to each image patch set in the vehicle structure is the target structure region, including: Calculate the product of the feature intensity of each second image block in each image block set and the corresponding weight coefficient after dynamic smoothing; The product result is calculated as a ratio to the average feature intensity of the first image block to which the second image block belongs; When the ratio exceeds the preset threshold range and the second image block belongs to a high-priority feature type, it is determined that the region corresponding to the second image block belongs to the target structure region. When the number of second image blocks in the image block set that are determined to be the target structure region reaches a preset number, the region corresponding to the image block set is determined to be the target structure region.
[0011] Preferably, the feature intensity of the second image block is the weighted average of the reflection signal intensities of all point clouds within the second image block, and the weights are specifically the dynamic smoothing weight coefficients of the cross-modal fusion stage. The second image patch corresponding to edge features has a higher priority than the second image patch corresponding to corner features; The second image patch corresponding to corner features has a higher priority than the second image patch corresponding to planar features.
[0012] Preferably, the weighted least squares method is used to geometrically fit the boundary point set to obtain the vehicle body boundary line and the overall geometric contour, including: Extract all sparse edge feature points of the target structural region as the boundary point set; The dynamic smoothing weight coefficients of the cross-modal fusion stage are used as the fitting weights for each point in the boundary point set. The slope and intercept of the vehicle body boundary line are obtained by performing least-squares line fitting on the weighted boundary point set. Connecting all boundary lines yields a stable overall geometric profile of the vehicle body, resisting the effects of noise points and anomalies.
[0013] A vehicle structure region recognition system based on sparse feature radar, applicable to the aforementioned vehicle structure region recognition method based on sparse feature radar, includes: The spatiotemporal alignment unit is used to acquire point cloud data of radar scanning the vehicle structure and camera images under the corresponding timestamp; the coordinates of the point cloud data and camera images are unified by using the external parameter calibration matrix of LiDAR and camera, and the three-dimensional spatial coordinates in the radar coordinate system are projected onto the image plane to obtain the spatiotemporally aligned radar point cloud data and image data. The target detection unit is used to extract sparse features of the vehicle structure based on the spatiotemporally aligned radar point cloud data. The sparse features include edge feature points, corner feature points and planar feature points of the vehicle structure. The deep learning vision model is used to perform target detection and semantic segmentation on the spatiotemporally aligned image data to obtain a semantic label map of the vehicle region. The projection mapping unit is used to project and map sparse features to semantic label maps, and to transfer the semantic information of the image to the radar sparse feature points to obtain a cross-modal sparse feature map with semantic prior. The feature weighting unit is used to calculate the feature variance of image features and radar point cloud depth data in the cross-modal sparse feature map respectively, and establish dynamic update fusion weights of image and radar point cloud based on the feature variance relationship; the cross-modal sparse feature map is weighted using the dynamic update fusion weights, and a temporal filtering update strategy is introduced to dynamically smooth the weight coefficients. The feature partitioning unit is used to divide the dynamically smoothed cross-modal sparse feature map into multiple first image blocks, and to perform secondary partitioning on the multiple first image blocks to obtain multiple image block sets corresponding to the vehicle body structure region. The region recognition unit is used to determine whether the region corresponding to each image block set in the vehicle structure is the target structure region based on the feature intensity and weight coefficients after dynamic smoothing in each image block set; to obtain the boundary point set of the target structure region, and to perform geometric fitting on the boundary point set using the weighted least squares method to obtain the vehicle boundary line and the overall geometric contour.
[0014] Compared with the prior art, the beneficial effects of the present invention are: (1) By aligning the lidar point cloud data with the camera image in time and space, the present invention can achieve higher precision vehicle structure area recognition. The time and space synchronization allows the radar and image information to complement each other, improving the accuracy of recognition. Moreover, the extraction of sparse features and the fusion of semantic label information make the recognition process not only dependent on geometric information but also combined with visual information, thus providing a more comprehensive feature representation. (2) This invention establishes a dynamic update fusion weight by calculating the feature variance of image features and radar point cloud depth data. This flexibility allows the system to adaptively adjust the feature fusion strategy according to different scenarios and environmental conditions, thereby enhancing the robustness of the system. Moreover, the introduced temporal filtering update strategy can smooth the weight coefficients, reduce the impact of instantaneous noise on the recognition results, and improve the stability and reliability of the system. Furthermore, by analyzing the feature strength and weight coefficients of the image block set, the target structure region can be effectively determined, ensuring the accuracy of the recognition results, especially in complex backgrounds, reducing the possibility of misidentification. Attached Figure Description
[0015] Figure 1 This is a schematic flowchart of the overall method in one embodiment of the present invention; Figure 2 This is a schematic diagram of the overall system architecture in one embodiment of the present invention.
[0016] In the figure: 1. Spatiotemporal alignment unit; 2. Target detection unit; 3. Projection mapping unit; 4. Feature weighting unit; 5. Feature segmentation unit; 6. Region recognition unit. Detailed Implementation
[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] Example 1, please refer to Figure 1 This invention provides a technical solution: a vehicle structure region identification method based on sparse feature radar, comprising: S1. Acquire point cloud data of radar scanning the vehicle structure and camera image under the corresponding timestamp; use the external parameter calibration matrix of LiDAR and camera to unify the coordinates of point cloud data and camera image, project the three-dimensional spatial coordinates in radar coordinate system onto the image plane, and obtain the spatiotemporally aligned radar point cloud data and image data. S2. Based on the spatiotemporally aligned radar point cloud data, extract sparse features of the vehicle body structure. The sparse features include edge feature points, corner feature points and planar feature points of the vehicle body structure. Use a deep learning visual model to perform target detection and semantic segmentation on the spatiotemporally aligned image data to obtain a semantic label map of the vehicle body region. S3. Project and map the sparse features to the semantic label map to transfer the semantic information of the image to the radar sparse feature points, and obtain a cross-modal sparse feature map with semantic prior. S4. Calculate the feature variance of image features and radar point cloud depth data in the cross-modal sparse feature map respectively. Establish dynamic update fusion weights for image and radar point cloud based on the feature variance relationship. Use the dynamic update fusion weights to perform weighted processing on the cross-modal sparse feature map and introduce a temporal filtering update strategy to dynamically smooth the weight coefficients. S5. Divide the cross-modal sparse feature map after dynamic smoothing into multiple first image blocks, and perform secondary division on multiple first image blocks to obtain multiple image block sets corresponding to the vehicle body structure region. S6. Based on the feature intensity and weight coefficients after dynamic smoothing in each image block set, determine whether the region corresponding to each image block set in the vehicle structure is the target structure region; obtain the boundary point set of the target structure region, and use the weighted least squares method to perform geometric fitting on the boundary point set to obtain the vehicle boundary line and the overall geometric contour.
[0019] It should be noted that the radar scans the vehicle structure to obtain point cloud data, and at the same time, it acquires the camera image at the corresponding timestamp; through the external parameter calibration matrix between the lidar and the camera, the coordinates of the radar's three-dimensional point cloud data and the camera's two-dimensional image data are unified, thereby achieving spatiotemporal alignment; specifically, the three-dimensional spatial coordinates in the radar coordinate system are projected onto the camera's image plane. In the spatiotemporally aligned radar point cloud data, sparse features of the vehicle structure are extracted; these sparse features include edge feature points, corner feature points and surface feature points, which reflect the basic shape and structural information of the vehicle body. Using a deep learning-based visual model, object detection and semantic segmentation are performed on spatiotemporally aligned image data to obtain a semantic label map of the vehicle body region; this step helps to understand the meaning and function of different regions in the image. The extracted sparse features are projected onto the semantic label map, and the semantic information in the image is passed to the sparse feature points of the radar, thus forming a cross-modal sparse feature map with semantic prior. The feature variances of image features and radar point cloud depth data in the cross-modal sparse feature map are calculated, and dynamically updated fusion weights are established based on these variances. The cross-modal sparse feature map is weighted using these weights, and a temporal filtering strategy is introduced to smooth and dynamically adjust the weights. The processed cross-modal sparse feature map is divided into multiple first image blocks, and each image block is further divided to obtain multiple image block sets corresponding to the vehicle body structure region. Based on the feature intensity and dynamically smoothed weight coefficients of each image patch set, we determine which regions are target structure regions and extract the boundary point set of the target structure regions. We then use the weighted least squares method to perform geometric fitting on the boundary point set to obtain the boundary line and overall geometric contour of the vehicle body.
[0020] In an optional embodiment, the three-dimensional spatial coordinates in the radar coordinate system are projected onto the image plane to obtain spatiotemporally aligned radar point cloud data and image data, including: Obtain the three-dimensional spatial coordinates in the radar coordinate system and the camera's intrinsic parameter matrix; Based on the extrinsic calibration matrix, intrinsic matrix and three-dimensional spatial coordinates, calculate the two-dimensional pixel coordinates of each three-dimensional point in the radar point cloud after it is projected onto the image plane. By using two-dimensional pixel coordinates, the spatial geometric information of radar point clouds is aligned with the pixel information of images, thus achieving spatiotemporal synchronization of radar and camera data.
[0021] It should be noted that obtaining three-dimensional spatial coordinates from radar, which are point cloud data represented in the radar coordinate system, usually consists of information reflected back from multiple laser beams; requires the camera's intrinsic matrix, which contains parameters such as the camera's focal length and principal point position, and is used to convert the three-dimensional world coordinates into two-dimensional image coordinates. The extrinsic matrix describes the relationship between the radar coordinate system and the camera coordinate system, and can transform the radar's three-dimensional point cloud coordinates to the camera coordinate system. The three-dimensional spatial coordinates (X, Y, Z) are transformed using an extrinsic parameter matrix to obtain the coordinates (Xc, Yc, Zc) in the camera coordinate system. Then, using the camera's intrinsic parameter matrix, these three-dimensional coordinates are projected onto the image plane to calculate the corresponding two-dimensional pixel coordinates (u, v). After obtaining the two-dimensional pixel coordinates, the spatial geometric information of the radar point cloud can be aligned with the pixel information of the image. Each radar point will have a corresponding pixel coordinate. In this way, the spatiotemporal synchronization of radar and camera data can be achieved. In this way, when analyzing the vehicle structure, the depth information of the radar and the visual information of the camera can be used at the same time.
[0022] In an optional embodiment, a projection mapping is performed between sparse features and a semantic label map to transfer image semantic information to radar sparse feature points, resulting in a cross-modal sparse feature map with semantic priors, including: By using projection mapping, the vehicle semantic labels in the semantic label map are assigned to the corresponding radar sparse feature points; Retain edge feature points, corner feature points, and planar feature points that belong to the vehicle body structure, and remove sparse background feature points that do not belong to the vehicle body structure. The retained radar sparse feature points are jointly encoded with the corresponding image semantic information to form a cross-modal sparse feature map with geometric, texture and semantic feature information.
[0023] It should be noted that, through the previously established projection mapping relationship, semantic labels (such as vehicle body, background, etc.) in the image are assigned to the corresponding radar sparse feature points; this means that for each radar sparse feature point, we can know its semantic meaning in the image, such as whether it belongs to the vehicle body structure. After projection, sparse feature points need to be filtered; retain those edge feature points, corner feature points, and surface feature points that belong to the vehicle body structure; these feature points are important information for identifying the shape and structure of the vehicle body. Sparse background feature points that do not belong to the vehicle body structure are removed to ensure that the final feature information is only related to the vehicle body structure and to avoid interference from background noise. For the retained radar sparse feature points, these feature points are jointly encoded with their corresponding image semantic information. The purpose of this step is to integrate geometric features (such as position and shape), texture features (such as surface details), and semantic features (such as object category). Through this joint encoding, the resulting cross-modal sparse feature map can fully reflect the information of the vehicle structure, including not only geometric shape but also visual semantic understanding.
[0024] In an optional embodiment, the feature variances of image features and radar point cloud depth data in the cross-modal sparse feature map are calculated respectively, and a dynamic update fusion weight for the image and radar point cloud is established based on the feature variance relationship, including: The feature variance is calculated for the image feature channels in the cross-modal sparse feature map, and the feature variance is also calculated for the depth feature channel of the radar point cloud. When the variance of image features is greater than the variance of radar point cloud depth features, reduce the image fusion weight and increase the radar point cloud weight. When the variance of radar point cloud depth features is greater than the variance of image features, reduce the radar point cloud fusion weight and increase the image weight. Establish dynamic weighting coefficients for images and dynamic weighting coefficients for radar point clouds.
[0025] It should be noted that the analysis is performed on different feature channels in the cross-modal sparse feature map; for the image feature channels, their feature variance is calculated; similarly, for the depth feature channels of the radar point cloud, the feature variance is also calculated; this process aims to measure the degree of change and stability of each feature. Once the variances of the two features are calculated, dynamic weight adjustments can be made. If the variance of the image features is greater than the variance of the radar point cloud depth features, it means that the image features have greater variation and may contain more noise or uncertainty. In this case, the fusion weight of the image should be reduced while the weight of the radar point cloud should be increased to enhance the system's dependence on radar data, thereby improving overall reliability. Conversely, if the variance of the depth features of the radar point cloud is greater than the variance of the image features, it indicates that the radar data is highly variable and may not be stable enough. In this case, the fusion weight of the radar point cloud should be reduced and the weight of the image should be increased, relying more on the more stable image information. Based on the above variance relationship, dynamic weight coefficients should be established. These weight coefficients will be adjusted as the variances of the image features and radar point cloud features change.
[0026] In an optional embodiment, the cross-modal sparse feature map is weighted using dynamically updated fusion weights, and a temporal filtering update strategy is introduced to dynamically smooth the weight coefficients, including: The feature vectors in the cross-modal sparse feature map are weighted and summed by dynamically updating the fusion weights at the current time. A time-series filtering update strategy is introduced, which uses the smoothing weight coefficient of the previous time step and the dynamically updated weight of the current time step to perform a weighted average.
[0027] It should be noted that the dynamically updated fusion weights calculated at the current time are used to perform a weighted summation of the feature vectors in the cross-modal sparse feature map. This process can adjust the influence of different features in the final feature map according to their relative importance, thereby better reflecting the information in the real scene. To make the weight coefficients more stable and reduce the impact of noise, a time-series filtering update strategy is introduced. This strategy generates new smooth weight coefficients by weighting the smoothed weight coefficients from the previous time step with the dynamically updated fusion weights from the current time step. This smoothing process helps to eliminate instantaneous fluctuations, making the feature fusion process more stable and reliable. After dynamic smoothing, the cross-modal sparse feature map is divided into multiple first image blocks according to a preset grid size; these image blocks are the basic units for analysis and processing; multiple first image blocks belonging to the same vehicle body structure region are aggregated into a set of image blocks; this step aims to ensure that relevant features are effectively integrated and enhance the overall description of the vehicle body structure. To ensure that each image patch set contains information from different sparse feature types, each image patch set is further divided into second image patches; these second image patches correspond to edge features, corner features, and planar features respectively, ensuring that various features are comprehensively considered.
[0028] In an optional embodiment, the dynamically smoothed cross-modal sparse feature map is divided into multiple first image blocks, and these first image blocks are further divided to obtain multiple image block sets corresponding to the vehicle body structure regions, including: The cross-modal sparse feature map after dynamic smoothing is divided into multiple first image blocks according to a preset grid size; Multiple first image blocks belonging to the same vehicle body structure region are aggregated into one image block set; Ensure that each image patch set contains a second image patch from different sparse feature types, with the second image patch corresponding to edge features, corner features, and planar features, respectively.
[0029] It should be noted that the cross-modal sparse feature map, which has undergone dynamic smoothing, is divided according to a preset grid size. This process decomposes the feature map into multiple small blocks, called the first image blocks. These small blocks can help the system analyze specific regions in the image more meticulously, thereby improving the accuracy of feature extraction. Multiple first image blocks belonging to the same vehicle body structure region are aggregated to form an image block set. The purpose of this aggregation process is to integrate relevant feature information to more comprehensively reflect the feature characteristics of a specific region. Especially when processing vehicle body structures, relevant features often provide more valuable information. After aggregation, each image patch set is further divided to obtain multiple second image patches. These second image patches focus on different types of features, including edge features, corner features, and planar features. By ensuring that each image patch set contains second image patches from different sparse feature types, the diversity of vehicle structure can be effectively captured.
[0030] In an optional embodiment, determining whether the region corresponding to each image patch set in the vehicle structure is a target structure region based on the feature intensity and weight coefficients after dynamic smoothing in each image patch set includes: Calculate the product of the feature intensity of each second image block in each image block set and the corresponding weight coefficient after dynamic smoothing; The product result is calculated as a ratio to the average feature intensity of the first image block to which the second image block belongs; When the ratio exceeds the preset threshold range and the second image block belongs to a high-priority feature type, it is determined that the region corresponding to the second image block belongs to the target structure region. When the number of second image blocks in the image block set that are determined to be the target structure region reaches a preset number, the region corresponding to the image block set is determined to be the target structure region.
[0031] It should be noted that for different second image patches in each image patch set, the product of its feature intensity and the corresponding weight coefficient after dynamic smoothing is first calculated. This calculation process can reflect the importance of each feature in the overall analysis. Taking into account the feature intensity and the corresponding weight, it helps to identify more representative features. The product result is calculated as a ratio to the average feature intensity of the first image patch to which the second image patch belongs. This ratio provides a relative benchmark to assess the importance of the second image patch in a specific region, ensuring that the analysis considers not only the absolute value but also the relationship between the feature and its surrounding environment. If the ratio exceeds the preset threshold range, and the second image patch belongs to a high-priority feature type, then the region corresponding to the second image patch can be determined as the target structure region; this step ensures that only regions with significant and important features are identified as targets. If the number of second image blocks identified as the target structure region in the entire image block set reaches a preset threshold, then the region corresponding to that image block set is confirmed as the target structure region. This design makes the judgment more robust and can avoid erroneous judgments caused by the randomness of individual features.
[0032] In an optional embodiment, the feature intensity of the second image block is the weighted average of the reflection signal intensities of all point clouds within the second image block, and the weights are specifically the dynamic smoothing weight coefficients of the cross-modal fusion stage. The second image patch corresponding to edge features has a higher priority than the second image patch corresponding to corner features; The second image patch corresponding to corner features has a higher priority than the second image patch corresponding to planar features.
[0033] It should be noted that the feature intensity of each second image block is determined by the weighted average of the reflection signal intensities of all point clouds within that block. This means that the influence of different point cloud signals varies when calculating feature intensity. The specific weighting method uses the dynamic smoothing weight coefficients obtained in the cross-modal fusion stage. Such a weighted average can more accurately reflect the overall features of each second image block, ensuring that signals with higher intensity have a greater impact on the final result. Regarding the priority of different feature types, the corresponding second image patch of edge features is given the highest priority; this means that when determining the target region, the presence and importance of edge features will be given priority, because edges can usually provide important information about the shape and structure of an object. The second image patch corresponding to corner features has a higher priority than the second image patch corresponding to planar features; this indicates that corner features also have relatively high importance in the analysis process, but their priority is lower than that of edge features; planar features are located at the lowest priority position, indicating that in feature judgment, the recognition importance of planar features is lower than that of edges and corners.
[0034] In an optional embodiment, the boundary point set is geometrically fitted using the weighted least squares method to obtain the vehicle body boundary line and the overall geometric contour, including: Extract all sparse edge feature points of the target structural region as the boundary point set; The dynamic smoothing weight coefficients of the cross-modal fusion stage are used as the fitting weights for each point in the boundary point set. The slope and intercept of the vehicle body boundary line are obtained by performing least-squares line fitting on the weighted boundary point set. Connecting all boundary lines yields a stable overall geometric profile of the vehicle body, resisting the effects of noise points and anomalies.
[0035] It should be noted that all sparse edge feature points are extracted from the target structural region, and these points constitute the boundary point set; the purpose of this step is to collect key feature points that may represent the vehicle body boundary, providing basic data for subsequent geometric fitting. During the fitting process, the dynamic smoothing weight coefficients calculated in the cross-modal fusion stage are used as the fitting weights for each boundary point. This means that different boundary points will be assigned different weights according to their importance during fitting, so that more critical or reliable points play a greater role in the final fitting result. This method can effectively reduce the influence of noise points and improve the accuracy of the fitting results. The weighted set of boundary points is fitted with a least-squares line. This method allows the slope and intercept of the vehicle body boundary line to be calculated, thus obtaining the linear equation representing the vehicle body boundary. This step allows the system to minimize fitting error while considering the weights of each boundary point, making the boundary line more consistent with the actual situation. Connecting all the fitted boundary lines forms a stable overall geometric profile of the vehicle body. This connection method helps to construct a complete and coherent vehicle body shape, effectively resisting the influence of noise points and outliers, and ensuring that the final geometric profile truly reflects the structural features of the vehicle body.
[0036] Example 2, please refer to Figure 2 This invention provides a technical solution: a vehicle structure region recognition system based on sparse feature radar, applicable to the aforementioned vehicle structure region recognition method based on sparse feature radar, comprising: Spatiotemporal alignment unit 1 is used to acquire point cloud data of radar scanning the vehicle structure and camera images under the corresponding timestamp; the coordinates of point cloud data and camera images are unified by using the external parameter calibration matrix of LiDAR and camera, and the three-dimensional spatial coordinates in the radar coordinate system are projected onto the image plane to obtain the spatiotemporally aligned radar point cloud data and image data. The target detection unit 2 is used to extract sparse features of the vehicle structure based on the spatiotemporally aligned radar point cloud data. The sparse features include edge feature points, corner feature points and planar feature points of the vehicle structure. The deep learning vision model is used to perform target detection and semantic segmentation on the spatiotemporally aligned image data to obtain a semantic label map of the vehicle region. Projection mapping unit 3 is used to project and map sparse features to semantic label map, and to transfer image semantic information to radar sparse feature points to obtain cross-modal sparse feature map with semantic prior. Feature weighting unit 4 is used to calculate the feature variance of image features and radar point cloud depth data in the cross-modal sparse feature map respectively, and establish dynamic update fusion weights of image and radar point cloud based on the feature variance relationship; the cross-modal sparse feature map is weighted using dynamic update fusion weights, and a temporal filtering update strategy is introduced to dynamically smooth the weight coefficients. The feature partitioning unit 5 is used to divide the cross-modal sparse feature map after dynamic smoothing into multiple first image blocks, and to perform secondary partitioning on the multiple first image blocks to obtain multiple image block sets corresponding to the vehicle body structure region. The region recognition unit 6 is used to determine whether the region corresponding to each image block set in the vehicle structure is the target structure region based on the feature intensity and weight coefficient after dynamic smoothing in each image block set; to obtain the boundary point set of the target structure region, and to perform geometric fitting on the boundary point set using the weighted least squares method to obtain the vehicle boundary line and the overall geometric contour.
[0037] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.
Claims
1. A radar-based method for identifying vehicle structure regions based on sparse features, characterized in that, include: Acquire point cloud data of radar scanning the vehicle structure and camera images at the corresponding timestamp; By using the extrinsic parameter calibration matrix of lidar and camera to unify the coordinates of point cloud data and camera image, the three-dimensional spatial coordinates in the radar coordinate system are projected onto the image plane to obtain the spatiotemporally aligned radar point cloud data and image data. Based on the spatiotemporally aligned radar point cloud data, sparse features of the vehicle body structure are extracted. The sparse features include edge feature points, corner feature points and planar feature points of the vehicle body structure. A deep learning vision model is used to perform target detection and semantic segmentation on the spatiotemporally aligned image data to obtain a semantic label map of the vehicle body region. By projecting and mapping sparse features onto semantic label maps, the semantic information of the image is transferred to the radar sparse feature points, resulting in a cross-modal sparse feature map with semantic priors. The feature variances of image features and radar point cloud depth data in the cross-modal sparse feature map are calculated separately. Dynamic update fusion weights of image and radar point cloud are established based on the feature variance relationship. The cross-modal sparse feature map is weighted using the dynamic update fusion weights, and a temporal filtering update strategy is introduced to dynamically smooth the weight coefficients. The cross-modal sparse feature map after dynamic smoothing is divided into multiple first image blocks, and the multiple first image blocks are further divided to obtain multiple image block sets corresponding to the vehicle body structure region. Based on the feature intensity and weight coefficients after dynamic smoothing in each image patch set, it is determined whether the region corresponding to each image patch set in the vehicle structure is the target structure region; the boundary point set of the target structure region is obtained, and the boundary point set is geometrically fitted using the weighted least squares method to obtain the vehicle boundary line and the overall geometric contour.
2. The vehicle structure region identification method based on sparse features of radar according to claim 1, characterized in that, By projecting the three-dimensional spatial coordinates in the radar coordinate system onto the image plane, we obtain spatiotemporally aligned radar point cloud data and image data, including: Obtain the three-dimensional spatial coordinates in the radar coordinate system and the camera's intrinsic parameter matrix; Based on the extrinsic calibration matrix, intrinsic matrix and three-dimensional spatial coordinates, calculate the two-dimensional pixel coordinates of each three-dimensional point in the radar point cloud after it is projected onto the image plane. By using two-dimensional pixel coordinates, the spatial geometric information of the radar point cloud is aligned with the pixel information of the image, thus achieving spatiotemporal synchronization of radar and camera data.
3. The vehicle structure region identification method based on sparse features of radar according to claim 2, characterized in that, By projecting and mapping sparse features onto semantic label maps, the semantic information of the image is transferred to the radar sparse feature points, resulting in a cross-modal sparse feature map with semantic priors, including: By using projection mapping, the vehicle semantic labels in the semantic label map are assigned to the corresponding radar sparse feature points; Retain edge feature points, corner feature points, and planar feature points that belong to the vehicle body structure, and remove sparse background feature points that do not belong to the vehicle body structure. The retained radar sparse feature points are jointly encoded with the corresponding image semantic information to form a cross-modal sparse feature map with geometric, texture and semantic feature information.
4. The vehicle structure region identification method based on sparse features of radar according to claim 3, characterized in that, Calculate the feature variances of image features and radar point cloud depth data in the cross-modal sparse feature map, respectively. Establish dynamic update fusion weights for the image and radar point cloud based on the feature variance relationship, including: The feature variance is calculated for the image feature channels in the cross-modal sparse feature map, and the feature variance is also calculated for the depth feature channel of the radar point cloud. When the variance of image features is greater than the variance of radar point cloud depth features, reduce the image fusion weight and increase the radar point cloud weight. When the variance of radar point cloud depth features is greater than the variance of image features, reduce the radar point cloud fusion weight and increase the image weight. Establish dynamic weighting coefficients for images and dynamic weighting coefficients for radar point clouds.
5. The vehicle structure region identification method based on sparse features of radar according to claim 4, characterized in that, The cross-modal sparse feature map is weighted using dynamically updated fusion weights, and a temporal filtering update strategy is introduced to dynamically smooth the weight coefficients, including: The feature vectors in the cross-modal sparse feature map are weighted and summed by dynamically updating the fusion weights at the current time. A time-series filtering update strategy is introduced, which uses the smoothing weight coefficient of the previous time step and the dynamically updated weight of the current time step to perform a weighted average.
6. The vehicle structure region identification method based on sparse features of radar according to claim 5, characterized in that, The dynamically smoothed cross-modal sparse feature map is divided into multiple first image blocks, and these first image blocks are further divided to obtain multiple image block sets corresponding to the vehicle body structure regions, including: The cross-modal sparse feature map after dynamic smoothing is divided into multiple first image blocks according to a preset grid size; Multiple first image blocks belonging to the same vehicle body structure region are aggregated into one image block set; Ensure that each image patch set contains a second image patch from different sparse feature types, with the second image patch corresponding to edge features, corner features, and planar features, respectively.
7. The vehicle structure region identification method based on sparse features of radar according to claim 6, characterized in that, Based on the feature intensity and weight coefficients after dynamic smoothing in each image patch set, it is determined whether the region corresponding to each image patch set in the vehicle structure is the target structure region, including: Calculate the product of the feature intensity of each second image block in each image block set and the corresponding weight coefficient after dynamic smoothing; The product result is calculated as a ratio to the average feature intensity of the first image block to which the second image block belongs; When the ratio exceeds the preset threshold range and the second image block belongs to a high-priority feature type, it is determined that the region corresponding to the second image block belongs to the target structure region. When the number of second image blocks in the image block set that are determined to be the target structure region reaches a preset number, the region corresponding to the image block set is determined to be the target structure region.
8. The vehicle structure region identification method based on sparse features of radar according to claim 7, characterized in that, The feature intensity of the second image block is the weighted average of the reflection signal intensity of all point clouds within the second image block, and the weight is specifically the dynamic smoothing weight coefficient of the cross-modal fusion stage. The second image patch corresponding to edge features has a higher priority than the second image patch corresponding to corner features; The second image patch corresponding to corner features has a higher priority than the second image patch corresponding to planar features.
9. The vehicle structure region identification method based on sparse features of radar according to claim 8, characterized in that, The weighted least squares method is used to perform geometric fitting on the boundary point set to obtain the vehicle body boundary lines and the overall geometric contour, including: Extract all sparse edge feature points of the target structural region as the boundary point set; The dynamic smoothing weight coefficients of the cross-modal fusion stage are used as the fitting weights for each point in the boundary point set. The slope and intercept of the vehicle body boundary line are obtained by performing least-squares line fitting on the weighted boundary point set. Connecting all boundary lines yields a stable overall geometric profile of the vehicle body, resisting the effects of noise points and anomalies.
10. A vehicle structure region recognition system based on sparse features radar, applicable to the vehicle structure region recognition method based on sparse features radar as described in any one of claims 1-9, characterized in that, include: The spatiotemporal alignment unit is used to acquire point cloud data of radar scanning the vehicle structure and camera images under the corresponding timestamp. By using the extrinsic parameter calibration matrix of lidar and camera to unify the coordinates of point cloud data and camera image, the three-dimensional spatial coordinates in the radar coordinate system are projected onto the image plane to obtain the spatiotemporally aligned radar point cloud data and image data. The target detection unit is used to extract sparse features of the vehicle structure based on the spatiotemporally aligned radar point cloud data. The sparse features include edge feature points, corner feature points and planar feature points of the vehicle structure. The deep learning vision model is used to perform target detection and semantic segmentation on the spatiotemporally aligned image data to obtain a semantic label map of the vehicle region. The projection mapping unit is used to project and map sparse features to semantic label maps, and to transfer the semantic information of the image to the radar sparse feature points to obtain a cross-modal sparse feature map with semantic prior. The feature weighting unit is used to calculate the feature variance of image features and radar point cloud depth data in the cross-modal sparse feature map respectively, and establish dynamic update fusion weights of image and radar point cloud based on the feature variance relationship; the cross-modal sparse feature map is weighted using the dynamic update fusion weights, and a temporal filtering update strategy is introduced to dynamically smooth the weight coefficients. The feature partitioning unit is used to divide the dynamically smoothed cross-modal sparse feature map into multiple first image blocks, and to perform secondary partitioning on the multiple first image blocks to obtain multiple image block sets corresponding to the vehicle body structure region. The region recognition unit is used to determine whether the region corresponding to each image block set in the vehicle structure is the target structure region based on the feature intensity and weight coefficients after dynamic smoothing in each image block set; to obtain the boundary point set of the target structure region, and to perform geometric fitting on the boundary point set using the weighted least squares method to obtain the vehicle boundary line and the overall geometric contour.