A target detection method based on lidar and vision fusion based on spatiotemporal registration

Abstract

The present invention discloses a target detection method based on spatiotemporal registration by integrating lidar and vision. The method relates to the field of lidar and vision fusion technology. The method comprises: activating lidar and vision sensors to collect data and construct a time series dataset; extracting radar and image features and mapping them into a unified high-dimensional embedding space; performing time series comparison to establish a focus on dynamic and static features; performing multi-level feature alignment based on the focus on dynamic and static features, including low-, medium-, and high-level alignment; and finally achieving target detection through center-decay alignment and fusion. This method achieves the technical effects of online monitoring, complete monitoring, and good early warning capabilities.

Description

Technical Field

[0001] The present invention relates to the field of laser radar and vision fusion technology, and in particular to a laser radar and vision fusion target detection method based on spatiotemporal registration. Background Art

[0002] Currently, common object detection methods rely primarily on LiDAR or visual information. LiDAR-based detection methods use point cloud data to construct 3D models to identify targets, offering the advantage of precise distance and spatial structure information. Vision-based detection methods, on the other hand, utilize cameras to capture images and analyze texture and shape features using deep learning algorithms, offering low cost and rich information.

[0003] However, existing target detection methods based on single laser radar or vision have obvious shortcomings. First, when relying solely on laser radar data, although the three-dimensional geometric information of the target can be obtained, it is relatively lacking in texture details and semantic information acquisition, resulting in limited ability to classify and recognize targets in complex scenes, especially when facing targets of similar shapes but different categories. Misjudgment is prone to occur. When relying solely on visual sensors for detection, it is easily affected by factors such as lighting changes and weather conditions. For example, under strong light or shadows, the image quality will be greatly reduced, which in turn affects the accuracy and stability of detection. In addition, the method of simply fusing the two modalities cannot effectively capture the dynamic changing characteristics of the target at different times, and thus performs poorly when dealing with moving targets or scene changes. Summary of the Invention

[0004] The present invention provides a target detection method based on spatiotemporal registration by fusion of lidar and vision to solve the technical problems in the existing technology that lack multi-dimensional fusion, resulting in insufficient detection accuracy and weak environmental adaptability, and to achieve the technical effect of introducing time dimension fusion to improve detection accuracy and environmental adaptability.

[0005] The present invention provides a laser radar and vision fusion target detection method based on spatiotemporal registration, comprising:

[0006] Activate the lidar and vision sensors to perform data collection and establish a time series collection dataset.

[0007] After respectively extracting radar features and image features from the time series acquisition data set, the radar features and the image features are mapped to the same high-dimensional embedding space.

[0008] Perform time series comparison of the time series acquisition data set to establish dynamic and static feature attention of radar features and image features respectively.

[0009] The dynamic and static features are used to focus on performing multi-level feature alignment of the radar features and the image features, where the multi-level feature alignment includes low-level feature alignment, mid-level feature alignment, and high-level feature alignment.

[0010] Center-attenuated alignment and fusion are performed based on the multi-level feature alignment results, and target detection is performed based on the center-attenuated alignment and fusion results.

[0011] In a feasible implementation, performing the time series comparison of the time series acquisition data set to establish dynamic and static feature focus of radar features and image features respectively includes:

[0012] The radar data and image data at the time zero point are used as standard data, and a traversal comparison is performed between the standard data and all time series acquisition data sets, and a first motion and stillness recognition result is established based on the traversal comparison result.

[0013] A neighborhood traversal comparison under a pre-time sequence call is performed with the time-series collected data set, and a second motion-still recognition result is established according to the neighborhood traversal comparison result.

[0014] The first motion-static recognition result and the second motion-static recognition result are subjected to identification fusion to establish global dynamic features, local dynamic features, local static features, and static features.

[0015] The global dynamic features, local dynamic features, local static features and static features are used to generate dynamic and static feature attention.

[0016] In a feasible implementation, the utilizing the static and dynamic features to focus on performing multi-level feature alignment of the radar features and the image features includes:

[0017] A low-level feature extraction channel is used to extract low-level features of the radar features and the image features, and a low-level feature extraction result is established.

[0018] According to the dynamic and static feature attention, dynamic feature attention reconstruction of the low-level feature extraction result is performed, and according to the dynamic feature attention reconstruction result, attention fusion of the low-level feature extraction result is performed to construct the first-level feature alignment result. The dynamic feature attention reconstruction is a suppressed reconstruction of the dynamic feature.

[0019] The first-level feature alignment result is used to perform multi-level feature alignment.

[0020] In a feasible implementation, performing multi-level feature alignment using the first-level feature alignment result includes:

[0021] A mid-level feature extraction channel is used to extract mid-level features of the radar features and the image features, and a mid-level feature extraction result is established.

[0022] The dynamic and static feature balance attention reconstruction of the middle-level feature extraction result is performed according to the dynamic and static feature attention, and the attention fusion of the middle-level feature extraction result is performed according to the dynamic and static feature balance attention reconstruction result to construct the second-level feature alignment result.

[0023] Multi-level feature alignment is performed using the first-level feature alignment result and the second-level feature alignment result.

[0024] In a feasible implementation, performing multi-level feature alignment using the first-level feature alignment result and the second-level feature alignment result includes:

[0025] A high-level feature extraction channel is used to extract mid-level features of the radar features and the image features, and a high-level feature extraction result is established.

[0026] The first-level feature alignment result is read, and the first-level feature alignment result is used to perform static background suppression on the high-level feature extraction result.

[0027] The second-level feature alignment result is read, and a global dynamic trajectory is established using the second-level feature alignment result.

[0028] After adjusting the high-level feature extraction result by using static background suppression, alignment-assisted enhancement is performed based on the global dynamic trajectory to establish a third-level feature alignment result.

[0029] Multi-level feature alignment is performed based on the first-level feature alignment result, the second-level feature alignment result, and the third-level feature alignment result.

[0030] In a feasible implementation, activating the laser radar and the visual sensor to perform data collection and establish a time-series collection data set includes:

[0031] After synchronizing the laser radar and the visual sensor at time zero, the laser radar and the visual sensor are synchronously activated.

[0032] Read the mapping data log of each round of return data of the laser radar and the visual sensor.

[0033] A timestamp error analysis is performed based on the mapping data log, and timestamp error correction is executed.

[0034] A time series acquisition data set is established based on the timestamp error correction results.

[0035] In a feasible implementation, extracting radar features and image features from the time series acquisition data set respectively includes:

[0036] Obtain point cloud data in the time series acquisition data set, and perform filtering and denoising processing on the point cloud data.

[0037] Perform feature extraction of geometric features, distance features, and object boundary features to establish radar signatures.

[0038] In a feasible implementation, extracting radar features and image features from the time series acquisition data set separately further includes:

[0039] Image data in the time-series acquisition data set is acquired.

[0040] After grayscale processing is performed on the image data, image data denoising is performed.

[0041] Edge detection is used to identify the edges of the denoised image data and establish edge recognition features.

[0042] The denoised image data is subjected to texture extraction to establish texture features.

[0043] The denoised image data is subjected to color restoration under the constraints of the original image, and color features are extracted based on the color restored image.

[0044] Perform feature extraction of denoised image data based on convolutional neural network and establish feature extraction results.

[0045] Image features are established based on edge recognition features, texture features, color features, and feature extraction results.

[0046] In a feasible implementation, the utilizing the static and dynamic features to focus on performing multi-level feature alignment of the radar features and the image features includes:

[0047] Determine whether the radar feature and the image feature have a zero point identifier for a time series zero point, where the zero point identifier is a user-selected identifier.

[0048] If there is a zero-point marker, a marker neighborhood focus is constructed based on the zero-point marker.

[0049] The identification neighborhood is utilized to focus on the alignment compensation of the multi-level feature alignment, and a multi-level feature alignment result is reconstructed.

[0050] In a feasible implementation, performing center-attenuation alignment fusion according to the multi-level feature alignment result and performing target detection based on the center-attenuation alignment fusion result includes:

[0051] Get the alignment center point of each feature in the multi-level feature alignment result and obtain the feature alignment strength value.

[0052] Taking the alignment center point as the starting point, performing starting point influence attenuation fitting according to the alignment strength value, and using the starting point influence attenuation fitting result to complete the center attenuation alignment fusion.

[0053] The present invention discloses a target detection method based on spatiotemporal registration of laser radar and vision fusion, comprising: activating laser radar and vision sensors, executing time series acquisition of environmental data, and establishing a time series acquisition data set containing radar data and image data; extracting features from the radar data and image data in the time series acquisition data set respectively, and mapping the extracted radar features and image features to the same high-dimensional embedding space to achieve a unified representation of features; establishing dynamic and static feature attention models for the radar features and image features mapped to the high-dimensional space according to time series information, so as to distinguish dynamic targets from static background features; executing radar feature attention based on dynamic and static feature attention, and Multi-level feature alignment processing with image features, which includes low-level feature alignment, mid-level feature alignment and high-level feature alignment, to improve the matching accuracy of cross-modal features; according to the multi-level feature alignment results, the center attenuation alignment fusion method is adopted for fusion to generate a fused feature representation, and target detection is performed based on the fused feature. The present invention discloses a lidar and vision fusion target detection method based on spatiotemporal registration, which solves the technical problem of lack of multi-dimensional fusion, resulting in insufficient detection accuracy and weak environmental adaptability, and realizes the technical effect of introducing time dimension fusion to improve detection accuracy and environmental adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of a flow chart of a target detection method based on spatiotemporal registration of laser radar and vision fusion in the present invention;

[0055] Figure 2 The figure is a flow chart of multi-level feature alignment in a target detection method based on spatiotemporal registration of lidar and vision fusion. DETAILED DESCRIPTION

[0056] The above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods of the specification to better understand the above technical solution. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention. It should be understood that the present invention is not limited to the example embodiments used only to explain the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, it should be noted that, for the convenience of description, only the parts related to the present invention, rather than all, are shown in the drawings.

[0057] Example, Figure 1The present invention is a flow chart of a method for target detection based on spatiotemporal registration of laser radar and vision fusion, wherein the method for target detection based on spatiotemporal registration of laser radar and vision fusion includes:

[0058] S100: Activate the lidar and vision sensor to perform data collection and establish a time series collection data set.

[0059] Specifically, target-oriented data collection is carried out through preset lidar and visual sensors. The lidar is responsible for obtaining three-dimensional point cloud information of the scene, and the visual sensor obtains two-dimensional image information, that is, the lidar data and visual sensor data containing target information are obtained respectively, thereby providing monitoring objects for subsequent target recognition and detection.

[0060] Specifically, a time-series dataset is a structured dataset composed of lidar point cloud data and image data collected synchronously by timestamp, organized as a frame sequence. This dataset typically includes information such as timestamps, sensor poses, image frames, and point cloud frames, and is used for subsequent spatiotemporal registration and object detection training or inference.

[0061] Optionally, ensure time alignment across sensors by synchronizing system time or using a hardware timestamp synchronization module.

[0062] In some embodiments, activating the laser radar and the visual sensor to perform data collection and establish a time-series collection data set includes:

[0063] After synchronizing the laser radar and the visual sensor at time zero, the laser radar and the visual sensor are synchronously activated; the mapping data log of each round of return data of the laser radar and the visual sensor is read; timestamp error analysis is performed based on the mapping data log, and timestamp error correction is executed; and a time series acquisition data set is established based on the timestamp error correction result.

[0064] Specifically, zero-time synchronization refers to the process of unifying the time bases of the lidar and vision sensors before system startup, aligning their internal clocks to the same reference time point (such as GPS time or the main control system time) to eliminate initial time deviations. Furthermore, after zero-time alignment, both sensors are controlled to begin data collection at the same time, ensuring consistent data collection starting points (i.e., synchronized activation).

[0065] Specifically, the mapping data log is metadata recorded after each frame of data is collected. It typically includes timestamps, frame numbers, sensor identifiers, and pose information, which are used for subsequent data alignment and analysis. Due to factors such as varying sensor sampling frequencies, data processing delays, and network transmission, the actual recorded timestamps may have slight deviations (e.g., milliseconds). This requires analyzing the time differences in the log data and performing interpolation, alignment, or offset correction to analyze and correct timestamp errors.

[0066] Specifically, first, a time synchronization instruction is sent to the lidar and vision sensor through a main control device (such as an embedded controller or industrial PC), or the internal clocks of the two types of sensors are aligned through hardware-level synchronization. After the zero point of time is aligned, a unified activation command is issued to ensure that the lidar and camera start collecting data from the same system time.

[0067] Specifically, after each frame of data is collected, the corresponding log information is recorded. This log serves as the index and registration basis for the data frame and is subsequently used for timestamp error analysis. For example, the log information is as follows:

[0068] {

[0069] "sensor_type": "LiDAR",

[0070] "frame_id": 0123,

[0071] "timestamp": 1714523123.248,

[0072] "data_path": "lidar / 0123.bin"

[0073] }

[0074] {

[0075] "sensor_type": "Camera",

[0076] "frame_id": 0456,

[0077] "timestamp": 1714523123.240,

[0078] "data_path": "image / 0456.jpg"

[0079] }

[0080] Furthermore, due to the different frame rates of LiDAR and cameras, as well as the varying processing delays of different sensors, timestamp errors may occur. For example, the timestamp difference between two frames of data collected at the same moment is 8ms, or some frames experience sudden time drift (such as a 20ms offset caused by network latency). In these cases, timestamp error analysis and correction are necessary. For example, linear interpolation is used to interpolate low-frequency data (such as LiDAR) onto a high-frequency time axis; sliding window matching is used to find the image frame closest in time to a point cloud frame among adjacent image frames; and dynamic time warping (DTW) is used to address nonlinear time drift.

[0081] Finally, all time-corrected point cloud frames and image frames are sorted by time to create a time-series acquisition dataset. Optionally, auxiliary information such as pose, acceleration, and position positioning can be added to the dataset to form a multimodal time-series dataset, providing a foundation for subsequent spatiotemporal registration and object detection.

[0082] S200: After extracting radar features and image features from the time series acquisition data set, the radar features and the image features are mapped to the same high-dimensional embedding space.

[0083] Specifically, representative and discriminative features are identified and extracted from the raw data to reflect the core attributes of the data. For radar data, these features can include information such as point cloud density, reflection intensity, and speed; for image data, these features can include texture, edges, shape, and color histograms.

[0084] Specifically, the high-dimensional embedding space is a multi-dimensional vector space used to map features of different modalities into the same space, so that features from different sources can be compared and fused on the same dimension.

[0085] By mapping radar and image features into the same high-dimensional embedding space, the limitations of single-modality detection methods can be effectively addressed. On the one hand, the fused features can comprehensively leverage the precise spatial information of the lidar and the rich texture information of the visual image, enhancing the recognition of target objects, especially those with similar shapes but different textures. On the other hand, this mapping method enables features to capture more complex relationships at a higher dimension, thereby improving the feature representation and discrimination. Furthermore, the unified feature space facilitates subsequent feature comparison and fusion operations, improving the accuracy and robustness of the entire detection system and enabling better performance in complex environments and dynamic scenes.

[0086] In some embodiments, extracting radar features and image features from the time-series acquired data set respectively includes:

[0087] Acquire point cloud data in the time series acquisition data set, perform filtering and denoising on the point cloud data; perform feature extraction of geometric features, distance features, and object boundary features to establish radar features.

[0088] Specifically, each frame of point cloud data is first extracted from the time-series acquisition data set. The format can be .pcd (Point Cloud Data) or .bin (Binary) format. Each point includes the corresponding three-dimensional coordinates and reflection intensity. Then, the point cloud data is preprocessed through methods such as statistical filtering (removing outliers with large mean deviations), voxel downsampling (reducing the number of point clouds and improving processing efficiency), and radius filtering (eliminating noise points in sparse areas).

[0089] Furthermore, feature extraction operations are performed on the filtered point cloud, such as:

[0090] Geometric feature extraction: Calculate the normal vector and curvature of each point to characterize the local surface shape and calculate the local point density. Distance feature extraction: Calculate the Euclidean distance from each point to the radar origin; calculate the mean and standard deviation of the distance across the full-frame point cloud as the overall distance feature. Boundary feature extraction: Identify boundary points using the rate of change of the normal vector or an edge detection algorithm (such as one based on Gaussian curvature); extract metrics such as the number of boundary points, boundary point density, and boundary linearity.

[0091] Finally, the above features are spliced ​​and fused to form the final radar feature vector, where each indicator can be expanded or tailored according to specific application requirements.

[0092] Through the above process, the following technical effects can be achieved: outliers and noise in the point cloud can be significantly reduced through filtering and denoising, thereby improving the accuracy of subsequent feature extraction; geometric, distance and boundary features are integrated to construct a multi-dimensional radar feature vector, which can comprehensively characterize the spatial structure and distribution characteristics of the target object; the extracted radar features have strong robustness and discriminability, providing a structured and high-quality input basis for subsequent modules such as image feature fusion and time series dynamic modeling.

[0093] In some implementations, extracting radar features and image features from the time-series acquired data set separately further includes:

[0094] Acquire image data in the time-series acquisition data set; perform image data denoising after grayscale processing on the image data; use edge detection to identify edges of the denoised image data and establish edge recognition features; perform texture extraction on the denoised image data and establish texture features; perform color restoration on the denoised image data under the constraints of the original image and extract color features based on the color restored image; perform feature extraction on the denoised image data based on a convolutional neural network and establish feature extraction results; establish image features based on edge recognition features, texture features, color features, and feature extraction results.

[0095] Specifically, first, the image data frame is extracted from the time series acquisition data set. The format can be .jpg, .png or .bmp, and aligned with the radar data by time stamp. Then, grayscale processing is performed to simplify the complexity of subsequent processing. Grayscale processing is the process of converting color images (RGB) into grayscale images. Common methods include weighted averaging, i.e., I gray =0.299R+0.587G+0.114B, where I gray is a grayscale image, and R, G, and B are the red, green, and blue color channels of a color image, respectively.

[0096] Specifically, a Gaussian filter (e.g., kernel size 5×5, standard deviation σ = 1.0) or median filter is then applied to the grayscale image to remove high-frequency noise and improve the robustness of edge detection and texture analysis. Furthermore, an edge detection algorithm such as Canny is used to identify image edges, extract edge images, and calculate characteristic parameters such as edge density and edge direction distribution to form edge recognition features.

[0097] Furthermore, the gray-level co-occurrence matrix is ​​used to calculate the texture features of the image, including energy, contrast, homogeneity, and entropy; or, local texture patterns are extracted based on local binary patterns (LBP) to form a texture histogram as a texture feature.

[0098] Furthermore, the denoised image is subjected to color restoration processing under the constraints of the original image to ensure that denoising does not lose color information, and then the color features of the restored image are extracted, such as RGB or HSV color histogram (each channel is divided into 8 or 16 bins) and main color extraction (such as K-means clustering to extract the first 3 main colors).

[0099] Furthermore, the denoised image is input into a convolutional neural network (such as ResNet-18), and the output feature vector dimension is such as 512 or 1024. The edge recognition features, texture features, color features, and feature extraction results obtained above are then spliced ​​to form the image feature vector. :

[0100]

[0101] in, Characterize edge recognition features (such as edge density, edge direction histogram); Characterize texture features (such as GLCM statistics); Characterize color features (such as HSV histogram); Represents the feature extraction results (such as the output of the CNN intermediate layer).

[0102] Through the above process, feature information from image data can be mined from multiple perspectives, enhancing the expressiveness and distinguishability of features. By performing a series of operations on the image, including grayscale conversion, denoising, edge detection, texture extraction, color restoration, and CNN feature extraction, comprehensive image features, including edge, texture, color, and depth semantics, can be obtained. Compared with single feature extraction methods, this method can more comprehensively describe the visual characteristics of the target object. This helps improve the accuracy of object detection, especially in complex scenes, and can better identify and distinguish targets with similar shapes but different textures, colors, or semantic characteristics.

[0103] S300: performing time series comparison of the time series acquisition data set, and establishing dynamic and static feature focus of radar features and image features respectively.

[0104] In some embodiments, performing the time series comparison of the time series acquisition data set to establish dynamic and static feature focus of radar features and image features respectively includes:

[0105] Taking the radar data and image data at the time zero point as the standard data, perform a traversal comparison between the standard data and all the time series acquisition data sets, and establish a first motion and static recognition result based on the traversal comparison result; perform a neighborhood traversal comparison under the pre-time series call on the time series acquisition data set, and establish a second motion and static recognition result based on the neighborhood traversal comparison result; perform identification fusion on the first motion and static recognition result and the second motion and static recognition result to establish global dynamic features, local dynamic features, local static features, and static features; and use the global dynamic features, local dynamic features, local static features, and static features to generate motion and static feature attention.

[0106] Specifically, the time zero point refers to the reference time point selected in the time series acquisition data set, usually the initial frame or reference frame. The corresponding image and radar data are used as "standard data" for subsequent comparison.

[0107] Specifically, traversal comparison refers to the process of comparing standard data frame by frame with the remaining frames in the time-series data set to identify areas or targets that change over time. Neighborhood traversal comparison involves performing local comparisons within the adjacent frames before and after the current frame to identify short-term dynamic changes and enhance sensitivity to small movements or local disturbances. Static and dynamic feature attention constructs feature attention areas based on the static and dynamic recognition results, used to enhance focus on dynamic areas or the ability to stably model static areas in subsequent analysis.

[0108] Specifically, the construction process of the dynamic and static feature focus includes:

[0109] First, standard data is selected, namely the image and radar point cloud corresponding to the time series zero point. Then, the time series image data and time series radar data are traversed and compared with the standard data of the corresponding category. For example, the image change area is calculated using a structural similarity index, difference map, or optical flow method to extract the image change mask. The point cloud change is calculated using ICP (Iterative Closest Point) or voxel overlap ratio to extract the point cloud change mask. Then, the image and point cloud comparison results are combined to generate the first dynamic and static recognition results.

[0110] Next, the same indicators or methods as the above-mentioned traversal comparison are used to perform neighborhood traversal comparison under the preceding time sequence call, that is, traversal and comparison of each frame of image data, radar data and its adjacent previous frame data to generate the second motion and static recognition result, thereby capturing the dynamic change characteristics within the local time range.

[0111] Furthermore, the first motion-static recognition result and the second motion-static recognition result are identified and fused, that is, the two comparison results are integrated through specific rules or algorithms to distinguish global dynamic features, local dynamic features, local static features and static features, and generate motion-static feature attention to provide guidance for subsequent multi-level feature alignment, ensuring that the alignment process focuses on the key feature changes and stable properties of the target object.

[0112] Exemplarily, the first motion-static recognition result R1 and the second motion-static recognition result R2 are fused and analyzed: if a certain area is a changing area in both R1 and R2, it is marked as a global dynamic feature; if it is a changing area only in R2, it is marked as a local dynamic feature; if it is a static area only in R1, it is marked as a static feature; if it is static in R2 but was dynamic in R1, it is marked as a local static feature (for example, a target that stopped for a short time).

[0113] In the above steps, the generation of static and dynamic feature attention directly guides the key areas of multi-level feature alignment, improving the efficiency and accuracy of feature alignment. This feature attention approach, which combines static and dynamic features and takes both global and local considerations into account, not only enhances the adaptability and accuracy of target detection in dynamic environments, but also improves the system's noise immunity and robustness, enabling the target detection system to operate more stably in complex and changing scenes and accurately identify target objects.

[0114] S400: Utilizing the static and dynamic features to focus on executing multi-level feature alignment of the radar features and the image features, the multi-level feature alignment includes low-level feature alignment, mid-level feature alignment, and high-level feature alignment.

[0115] Specifically, low-level features refer to basic features extracted from images or radar data by shallow networks or primary processing modules, such as edges, textures, intensity, reflectivity, etc., which usually have high resolution and low semantic characteristics.

[0116] Specifically, multi-level feature alignment refers to the process of aligning image and radar features layer by layer at the low, mid, and high semantic feature levels, usually using a pyramid structure or feature pyramid network (FPN).

[0117] Through multi-level feature alignment, low-level alignment ensures spatial and temporal consistency of basic features, reducing errors caused by inter-sensor differences; mid-level alignment strengthens the correlation between features, improving target recognition accuracy; and high-level alignment leverages semantic information to further enhance the ability to understand complex scenes. Furthermore, the application of a dynamic and static feature attention mechanism enables the alignment process to focus on key features and ignore irrelevant or noisy features, thereby improving the efficiency and reliability of feature alignment.

[0118] In some embodiments, the utilizing the static and dynamic features to focus on performing multi-level feature alignment of the radar features and the image features includes:

[0119] The radar features and the image features are extracted at low levels using a low-level feature extraction channel to establish a low-level feature extraction result. Dynamic feature attention reconstruction is performed on the low-level feature extraction result based on the static and dynamic feature attention. Attention fusion of the low-level feature extraction result is performed based on the dynamic feature attention reconstruction result to construct a first-level feature alignment result, wherein the dynamic feature attention reconstruction is a suppressed reconstruction of the dynamic feature. The first-level feature alignment result is used to perform multi-level feature alignment.

[0120] Specifically, dynamic feature attention reconstruction involves suppressing or masking the features of dynamic regions based on the dynamic and static feature attention areas to reduce their interference with alignment accuracy. Guided by dynamic and static weights, low-level features of the image and radar can be weightedly fused or spatially aligned to enhance the multimodal feature consistency of static regions. The first-level feature alignment result refers to the preliminary alignment results completed at the low-level feature level, providing the foundation for subsequent alignment at higher semantic levels.

[0121] Specifically, first, the input image enters a shallow convolutional network (such as the first two layers of ResNet) to extract low-level features such as edges, textures, and color gradients to obtain a feature map; after the input point cloud data is projected by BEV or encoded by RangeView, basic features such as reflectivity, density, and height difference are extracted to obtain a feature map. The above two feature maps are summed to obtain the low-level feature extraction result.

[0122] Specifically, then, a dynamic suppression mask is constructed based on the dynamic and static feature attention map generated above, such as reducing the weight of the dynamic feature or adjusting its expression to reduce the impact of dynamic changes on feature alignment. For example, the dynamic suppression mask Characterized by:

[0123]

[0124] in, Characterize the dynamic and static feature attention map; 、 、 、 They represent global dynamic features, local dynamic features, static features, and local static features respectively.

[0125] Specifically, further, based on the dynamic feature focus reconstruction results (as in the above formula ) Dynamically suppress and reconstruct low-level features to ensure that dynamic area features are suppressed during alignment and static areas remain intact, thereby providing a stable basis for alignment.

[0126] Specifically, feature alignment operations are performed on the reconstructed image and radar low-level features, such as channel alignment (1x1 convolution to unify dimensions), spatial alignment (using calibration matrix or attention mechanism), and feature splicing to obtain the first-level feature alignment results.

[0127] Furthermore, the first-level alignment results are input into the subsequent network to extract higher-level semantic features layer by layer, and perform mid-level and high-level feature alignment. For example, mid-level alignment (structural feature layer) uses a shared convolutional network to extract features such as shape and spatial layout; high-level alignment (semantic feature layer) extracts semantic information such as target category and behavioral status, performs cross-modal attention alignment or Transformer fusion, and finally outputs a multi-level aligned feature set.

[0128] The above steps improve the accuracy and robustness of low-level feature alignment. Dynamic feature-focused reconstruction suppresses the influence of dynamic features, helping to reduce feature fluctuations caused by target motion or environmental changes, thus making feature alignment more stable. The focus fusion process organically combines the reconstructed dynamic features with static features, preserving the key static properties of the target while appropriately accounting for dynamic changes, thereby improving the overall performance of feature alignment.

[0129] In some implementations, such as Figure 2 As shown, the multi-level feature alignment using the first-level feature alignment result includes:

[0130] The radar features and the image features are extracted from the mid-level feature extraction channel to establish a mid-level feature extraction result. The dynamic and static feature balance attention reconstruction of the mid-level feature extraction result is performed based on the dynamic and static feature attention. The attention fusion of the mid-level feature extraction result is performed based on the dynamic and static feature balance attention reconstruction result to construct a second-level feature alignment result. The first-level feature alignment result and the second-level feature alignment result are used to perform multi-level feature alignment.

[0131] Specifically, mid-level features refer to structural features extracted from image or radar data by the mid-level network, such as outlines, boundaries, shapes, and spatial layout. These features possess a certain degree of semantics and possess moderate resolution. The mid-level feature extraction pipeline is a network module used to extract these mid-level features, typically in the middle layers of the backbone network (such as layers 2 and 3 of ResNet or the local geometry encoding layer of PointNet). The mid-level feature extraction results are the feature tensors extracted from the image and radar data in the mid-level pipeline, which are used for subsequent fusion and alignment.

[0132] Specifically, based on the focus on dynamic and static features, we no longer simply suppress dynamic features, but instead implement a balanced reconstruction strategy of dynamic information retention + static information enhancement through attention mechanism or feature reweighting.

[0133] Specifically, the input image passes through the middle layer of the backbone network (such as the third layer of ResNet) to extract middle-level semantic features such as structure, boundary, and local contour. At the same time, the radar point cloud is input and the spatial structure and geometric relationship are extracted through the middle-level processing module (such as local voxel encoding and RangeNet++ middle layer). After integration, the middle-level feature extraction results are obtained.

[0134] Specifically, the dynamic and static feature attention maps are then combined to construct a balanced weight for each pixel / voxel location (i.e., balanced attention to dynamic and static features). This weighting is then adjusted to adjust the focus on global dynamic features, local dynamic features, static features, and local static features, rather than simply suppressing them. Furthermore, the balanced weights are used to reconstruct the image and radar mid-level features separately. This process preserves dynamic information while moderately suppressing its interference and enhancing the contribution of static region alignment, thus achieving a "dynamic and static balance."

[0135] Furthermore, the image and radar mid-level features are spatially aligned (such as using perspective transformation and BEV mapping), the channel dimensions are unified (such as through 1×1 convolution or linear mapping), and a dual-channel attention mechanism (such as CBAM and SE module) is used to combine the dynamic and static features to balance the attention and fusion of the two modal features to obtain the second-level feature alignment result.

[0136] The above process can achieve the following technical advantages by introducing mid-level feature extraction and dynamic-static balance mechanism: enhanced structural semantic alignment, with mid-level features paying more attention to target contours, boundaries, and spatial distribution, which helps improve the alignment accuracy of images and radar at the structural level; retaining dynamic target information, with the dynamic-static balance mechanism avoiding simple suppression of dynamic targets; achieving inter-layer collaborative fusion, with the first level providing geometric and texture foundations, and the second level providing structural and semantic supplements, forming a stable and robust multimodal expression; and providing stable intermediate layer support for the subsequent introduction of high-level semantic labels (such as target categories and behavioral states).

[0137] In some implementations, performing multi-level feature alignment using the first-level feature alignment result and the second-level feature alignment result includes:

[0138] A high-level feature extraction channel is used to extract mid-level features of the radar features and the image features to establish a high-level feature extraction result; the first-level feature alignment result is read, and the first-level feature alignment result is used to suppress static background of the high-level feature extraction result; the second-level feature alignment result is read, and the second-level feature alignment result is used to establish a global dynamic trajectory; after adjusting the high-level feature extraction result using static background suppression, alignment-assisted enhancement is performed based on the global dynamic trajectory to establish a third-level feature alignment result; and multi-level feature alignment is performed based on the first-level feature alignment result, the second-level feature alignment result, and the third-level feature alignment result.

[0139] Specifically, first, high-level feature extraction channels (such as deep convolutional layers, semantic segmentation networks, Transformer encoders, etc.) are used to extract high-level semantic information from the original image features and radar features respectively, and obtain high-level feature extraction results, which contain semantic information such as target category, behavior status, and spatial relationship; then, the first-level feature alignment results are read and the static area mask is extracted to construct a static background suppression weight map, which is used to suppress the response of the background area in the high-level features, where the weight value corresponding to the static area (background) is lower.

[0140] Next, the spatial-temporal evolution information of the dynamic area is extracted from the second-level feature alignment results to construct a global dynamic trajectory. For example, the cross-frame trajectory of the dynamic target is extracted by using a temporal attention mechanism or a motion vector field to combine the inter-frame difference of the image and the change of radar reflectivity, or the cross-frame target ID is tracked through a lightweight target detector to obtain a trajectory tensor (i.e., a global dynamic trajectory) that describes the evolution path of the dynamic target in the feature space.

[0141] Furthermore, the high-level features after background suppression in the above step are fused with the global dynamic trajectory to perform semantic enhancement and alignment correction of the dynamic target. That is, the pre-built global dynamic trajectory is used to provide reference information of motion trends for feature alignment, thereby better calibrating the dynamic features of the target object during the feature alignment process and obtaining the third-level feature alignment result.

[0142] Finally, the first-level, second-level, and third-level feature alignment results are fused to construct a complete multi-level fusion feature set. The fusion method can be feature pyramid (FPN) structure, multi-scale attention fusion, Transformer fusion encoder, or weighted fusion based on dynamic weights.

[0143] The multi-step feature alignment mechanism described above, combined with static and dynamic feature suppression and dynamic trajectory enhancement, can avoid high semantic misleading in background areas and improve the semantic consistency of foreground targets. It can also achieve accurate modeling of dynamic targets, enhance the depth and accuracy of multimodal fusion, and provide high-quality feature input for subsequent target detection.

[0144] In some embodiments, the utilizing the static and dynamic features to focus on performing multi-level feature alignment of the radar features and the image features includes:

[0145] Determine whether the radar feature and the image feature have a zero-point marker for the time series zero point, where the zero-point marker is a marker selected by the user; if a zero-point marker exists, construct a marker neighborhood attention based on the zero-point marker; use the marker neighborhood attention to compensate for the alignment of the multi-level feature alignment, and reconstruct the multi-level feature alignment result.

[0146] Specifically, the zero-point marker is the marker at the time zero point, which is determined by the user's selection. It provides a known reference point. Marker neighborhood attention focuses on the area surrounding the zero-point marker, using the features within that area as a reference for feature alignment compensation. Alignment compensation adjusts the multi-level feature alignment results to correct for any discrepancies and ensure accurate feature alignment.

[0147] The specific operation first determines whether there is a zero-point marker between the radar and image features. If so, a marker neighborhood attention is constructed based on the zero-point marker. This involves determining a neighborhood centered around the zero-point marker and focusing on the features within this neighborhood. Next, the marker neighborhood attention is used to perform alignment compensation on the multi-level feature alignment. This involves comparing and adjusting the feature alignment results with the features within the marker neighborhood to reconstruct the multi-level feature alignment results.

[0148] By introducing the zero point mark selected by the user, a reliable reference basis can be provided for feature alignment, thereby improving the accuracy and reliability of feature alignment.

[0149] In the above method steps, the user's selection of the zero-point marker introduces prior information, helping the system more accurately determine the starting point for feature alignment. Secondly, the marker neighborhood attention mechanism enables the system to make fine adjustments within a local area, improving the accuracy of feature alignment. Finally, the reconstructed multi-level feature alignment results better reflect the true characteristics of the target object, reducing errors caused by time series or environmental changes, thereby improving the accuracy and robustness of target detection.

[0150] S500: Perform center-attenuation alignment fusion according to the multi-level feature alignment result, and perform target detection based on the center-attenuation alignment fusion result.

[0151] In some embodiments, performing center-attenuation alignment fusion according to the multi-level feature alignment results and performing target detection based on the center-attenuation alignment fusion results includes:

[0152] Obtain the alignment center point of each feature in the multi-level feature alignment result and obtain the feature alignment strength value; take the alignment center point as the starting point, perform starting point influence attenuation fitting according to the alignment strength value, and use the starting point influence attenuation fitting result to complete the center attenuation alignment fusion.

[0153] Specifically, for each feature region or candidate target region after multi-level fusion, its alignment center point and alignment strength value are extracted. The alignment center point represents the spatial position where the image features and radar features in the region are optimally aligned; the alignment strength value represents the consistency score of the multimodal feature alignment in the region. A higher score indicates a more accurate alignment.

[0154] Optionally, the acquisition method includes: using a feature similarity matrix (such as cosine similarity) to find the maximum value point as the center, calculating the weighted centroid by aligning the attention map, or performing regression prediction based on the alignment confidence network.

[0155] Specifically, taking each alignment center point as the starting point, a spatial influence attenuation function is constructed to perform weighted fusion on the surrounding features, wherein the attenuation function can be in the form of a Gaussian, exponential or custom kernel function.

[0156] Furthermore, each position in the fused feature map is weightedly fused according to the attenuation function of all center points, thereby retaining the influence of areas with high alignment quality and suppressing edge or low confidence areas, which helps to improve target positioning accuracy, suppress background false detection, and enhance the detection robustness of dynamic targets and boundary targets.

[0157] By introducing the center-attenuated alignment fusion mechanism, the overall performance of multimodal alignment and detection can be significantly improved. Feature weighting is performed with the local alignment optimal point as the center, which improves the spatial consistency of feature expression. At the same time, the weights of each region can be dynamically adjusted during the fusion process to avoid interference from low-quality alignment areas and enhance feature clarity and positioning accuracy.

[0158] In summary, the laser radar and vision fusion target detection method based on spatiotemporal registration provided by the present invention has the following technical effects:

[0159] By activating the lidar and visual sensors, time-series acquisition of environmental data is performed, and a time-series acquisition data set containing radar data and image data is established; feature extraction is performed on the radar data and image data in the time-series acquisition data set respectively, and the extracted radar features and image features are mapped to the same high-dimensional embedding space to achieve unified feature representation; for the radar features and image features mapped to the high-dimensional space, their dynamic and static feature attention models are established respectively according to the time-series information to distinguish dynamic targets from static background features; based on the dynamic and static feature attention, multi-level feature alignment processing of radar features and image features is performed, and the multi-level feature alignment includes low-level feature alignment, mid-level feature alignment and high-level feature alignment to improve the matching accuracy of cross-modal features; according to the multi-level feature alignment results, the center attenuation alignment fusion method is adopted for fusion to generate a fused feature representation, and target detection is performed based on the fused feature, thereby achieving the technical effect of introducing time dimension fusion and improving detection accuracy and environmental adaptability.

[0160] It should be understood that the embodiments disclosed in the present invention and the above description can enable those skilled in the art to use the present invention to implement the present invention. At the same time, the present invention is not limited to the embodiments mentioned above. It should be understood that those skilled in the art can still modify the technical solutions described in the above embodiments or replace some of the technical features therein with equivalents; and such modifications or replacements do not deviate from the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention and are all included in the scope of protection of the present invention.

Claims

1. A target detection method based on spatiotemporal registration of laser radar and vision fusion, characterized in that: include: Activate the lidar and vision sensors to perform data collection and establish a time series collection data set; After extracting radar features and image features from the time series acquisition data set, the radar features and the image features are mapped to the same high-dimensional embedding space; Performing time series comparison of the time series acquisition data set to establish dynamic and static feature focus of radar features and image features respectively; Utilizing the static and dynamic features to focus on performing multi-level feature alignment of the radar features and the image features, the multi-level feature alignment comprising low-level feature alignment, mid-level feature alignment, and high-level feature alignment; Perform center-attenuation alignment fusion based on the multi-level feature alignment results, and perform target detection based on the center-attenuation alignment fusion results; The performing of the time series comparison of the time series acquisition data set to establish the dynamic and static feature focus of radar features and image features respectively includes: Using the radar data and image data at time zero as standard data, performing a traversal comparison between the standard data and all time series acquisition data sets, and establishing a first motion and stillness recognition result based on the traversal comparison results; Performing a neighborhood traversal comparison under a pre-time sequence call with the time series acquisition data set, and establishing a second motion and stillness recognition result according to the neighborhood traversal comparison result; Performing identity fusion on the first motion-static recognition result and the second motion-static recognition result to establish global dynamic features, local dynamic features, local static features, and static features; The global dynamic features, local dynamic features, local static features and static features are used to generate dynamic and static feature attention.

2. The target detection method based on spatiotemporal registration of laser radar and vision fusion according to claim 1, characterized in that: The utilizing the static and dynamic features to focus on performing multi-level feature alignment of the radar features and the image features includes: Performing low-level feature extraction of the radar features and the image features using a low-level feature extraction channel, and establishing a low-level feature extraction result; Perform dynamic feature focus reconstruction of low-level feature extraction results based on the dynamic and static feature focus, perform focus fusion of low-level feature extraction results based on the dynamic feature focus reconstruction results, and construct a first-level feature alignment result, wherein the dynamic feature focus reconstruction is a dynamic feature suppression reconstruction; The first-level feature alignment result is used to perform multi-level feature alignment.

3. The target detection method based on spatiotemporal registration of laser radar and vision fusion according to claim 2, characterized in that: The performing multi-level feature alignment using the first-level feature alignment result includes: Performing mid-level feature extraction of the radar features and the image features using a mid-level feature extraction channel to establish a mid-level feature extraction result; Focusing on the dynamic and static features, performing a balanced attention reconstruction of the dynamic and static features on the mid-level feature extraction results, and performing attention fusion of the mid-level feature extraction results based on the dynamic and static feature balanced attention reconstruction results to construct a second-level feature alignment result; Multi-level feature alignment is performed using the first-level feature alignment result and the second-level feature alignment result.

4. The target detection method based on spatiotemporal registration of laser radar and vision fusion according to claim 3, characterized in that: The performing multi-level feature alignment using the first-level feature alignment result and the second-level feature alignment result includes: Using a high-level feature extraction channel to extract mid-level features of the radar features and the image features, and establishing a high-level feature extraction result; Reading the first-level feature alignment result, and using the first-level feature alignment result to perform static background suppression on the high-level feature extraction result; Reading the second-level feature alignment result, and establishing a global dynamic trajectory using the second-level feature alignment result; After adjusting the high-level feature extraction result by using static background suppression, performing alignment-assisted enhancement based on the global dynamic trajectory to establish a third-level feature alignment result; Multi-level feature alignment is performed based on the first-level feature alignment result, the second-level feature alignment result, and the third-level feature alignment result.

5. The target detection method based on spatiotemporal registration of laser radar and vision fusion according to claim 1, characterized in that: The step of activating the laser radar and the visual sensor to perform data collection and establish a time series collection data set includes: After synchronizing the laser radar and the visual sensor at time zero, synchronously activating the laser radar and the visual sensor; Reading the mapping data log of each round of return data of the laser radar and the visual sensor; Performing timestamp error analysis based on the mapping data log and performing timestamp error correction; A time series acquisition data set is established based on the timestamp error correction results.

6. The target detection method based on spatiotemporal registration of laser radar and vision fusion according to claim 1, characterized in that: The extracting radar features and image features from the time series acquisition data set respectively includes: Acquire point cloud data in the time series acquisition data set, and perform filtering and denoising processing on the point cloud data; Perform feature extraction of geometric features, distance features, and object boundary features to establish radar signatures.

7. The target detection method based on spatiotemporal registration of laser radar and vision fusion according to claim 6, characterized in that: The extracting of radar features and image features from the time series acquisition data set also includes: Acquiring image data in the time series acquisition data set; After grayscale processing is performed on the image data, image data denoising is performed; Use edge detection to identify the edges of the denoised image data and establish edge recognition features; Perform texture extraction on the denoised image data and establish texture features; Perform color restoration on the denoised image data under the constraints of the original image, and extract color features based on the color restored image; Perform feature extraction on the denoised image data based on a convolutional neural network and establish feature extraction results; Image features are established based on edge recognition features, texture features, color features, and feature extraction results.

8. The target detection method based on spatiotemporal registration of laser radar and vision fusion according to claim 1, characterized in that: The utilizing the static and dynamic features to focus on performing multi-level feature alignment of the radar features and the image features includes: Determine whether the radar feature and the image feature have a zero point identifier of a time sequence zero point, where the zero point identifier is a user-selected identifier; If there is a zero-point marker, constructing a marker neighborhood focus based on the zero-point marker; The identification neighborhood is utilized to focus on the alignment compensation of the multi-level feature alignment, and a multi-level feature alignment result is reconstructed.

9. The target detection method based on spatiotemporal registration of laser radar and vision fusion according to claim 1, characterized in that: The method of performing center attenuation alignment fusion according to the multi-level feature alignment result and performing target detection based on the center attenuation alignment fusion result includes: Get the alignment center point of each feature in the multi-level feature alignment result and obtain the feature alignment strength value; Taking the alignment center point as the starting point, performing starting point influence attenuation fitting according to the alignment strength value, and using the starting point influence attenuation fitting result to complete the center attenuation alignment fusion.

Images