Method and system for tracking moving objects by fusing laser point cloud and visible light image

Abstract

The application provides a kind of laser point cloud and visible light image fusion motion target tracking method and system.The method includes obtaining laser point cloud data and visible light image data, and carries out time synchronization and space alignment.Extract the three-dimensional shape features of the target from the laser point cloud data after space-time alignment, and extract the two-dimensional appearance features of the target from the visible light image data.Coding three-dimensional shape features and two-dimensional appearance features into first feature vector and second feature vector respectively, calculate the similarity of two feature vectors, combine the similarity and the geometric position constraint of the target in space, establish the cross-modal feature association between the target in laser point cloud data and the target in visible light image data.Input the cross-modal feature association result into the tracking filter, estimate the state of the moving target and update the position information of the target.Feedback the updated position information to the feature extraction stage to guide the target search and feature extraction of the next frame, and realize adaptive tracking.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for tracking moving targets by fusing laser point clouds and visible light images. Background Technology

[0002] In fields such as autonomous driving, robot navigation, and security monitoring, accurate tracking of moving targets is one of the core technologies for achieving environmental perception and intelligent decision-making. The fusion of LiDAR and visible light cameras has attracted much attention due to the high complementarity of their data. LiDAR can provide accurate three-dimensional spatial geometric information and is unaffected by lighting conditions, but its point cloud data lacks color and texture information, and data loss occurs when distant targets are sparse or occluded. Visible light cameras can provide high-resolution color images rich in semantic information such as color, texture, and edges, but they cannot acquire spatial depth information, and their imaging quality is severely affected by changes in lighting, rain, fog, and complex background interference.

[0003] To overcome the inherent limitations of single sensors, those skilled in the art have proposed a fusion tracking method combining laser point clouds and visible light images. However, existing solutions generally employ data association strategies at the bounding box level, such as the method disclosed in Chinese patent CN116258744B. This method fuses visible light and infrared images, then detects and extracts targets from the fused image and LiDAR point cloud data, and finally projects the point cloud targets onto the image plane for matching. Bounding box-level association is highly dependent on the accuracy of each modal detection algorithm. When the sparseness of the point cloud at long distances causes bounding box drift, or when changes in illumination lead to image detection failure, the association performance drops sharply. Chinese patent CN114881906A directly uses algorithms such as SIFT to perform two-dimensional feature point matching between the dimensionality-reduced point cloud and the image. This direct matching inevitably results in the loss of spatial geometric information, and single-type features are insufficient to overcome the instability problem caused by the sparseness of the point cloud.

[0004] Existing methods mostly remain at the level of simple data-level or decision-level fusion, failing to effectively explore the heterogeneous complementarity between laser point clouds and visible light images. Furthermore, in terms of system architecture, existing methods are mostly open-loop structures of "detection-association-tracking," and the tracking results are not fed back to the front-end feature extraction and association stages. When a frame experiences a detection loss or association error, the error continues to accumulate and propagate, leading to frequent problems such as trajectory fragmentation and identity switching, making it difficult to guarantee the continuity and robustness of tracking. Summary of the Invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a method and system for tracking moving targets by fusing laser point clouds and visible light images.

[0006] To achieve the above objectives, a first aspect of the present invention provides a moving target tracking method that fuses laser point clouds with visible light images, comprising: Acquire laser point cloud data collected by lidar and visible light image data collected by a visible light camera; Perform time synchronization and spatial alignment of laser point cloud data and visible light image data; The three-dimensional shape features of the target are extracted from the spatiotemporally aligned laser point cloud data, and the two-dimensional appearance features of the target are extracted from the visible light image data. The three-dimensional shape features and two-dimensional appearance features are encoded into a first feature vector and a second feature vector, respectively. The similarity between the first feature vector and the second feature vector is calculated. Based on spatiotemporal alignment, and combined with the similarity and the geometric position constraints of the target in space, a cross-modal feature association is established between the target in the laser point cloud data and the target in the visible light image data. The cross-modal feature association results are input into the tracking filter to estimate the state of the moving target; The target's position information is updated based on the state estimation results, and the updated position information is fed back to the feature extraction stage to guide the target search and feature extraction in the next frame, thereby achieving adaptive tracking based on state feedback.

[0007] According to an embodiment of a first aspect of the present invention, extracting three-dimensional shape features of a target from spatiotemporally aligned laser point cloud data includes: Clustering is performed on the spatiotemporally aligned laser point cloud data to obtain multiple clusters; For each point in the laser point cloud data p i Search for a set of neighborhood points within a preset neighborhood radius and construct the covariance matrix of that point: ; in, k The number of neighboring points. Let be the centroid of the neighborhood point set; For the covariance matrix Σ i Perform eigenvalue decomposition to obtain eigenvalues. λ 1≥ λ 2≥ λ 3; Calculation points p i curvature value C i = λ 3 / ( λ 1+ λ 2+ λ3) Mark points with curvature values ​​greater than a preset curvature threshold as edge feature points, and mark points with curvature values ​​less than or equal to a preset curvature threshold as planar feature points.

[0008] According to an embodiment of the first aspect of the present invention, encoding three-dimensional shape features and two-dimensional appearance features into a first feature vector and a second feature vector respectively includes: Based on the edge feature points and planar feature points determined by curvature feature extraction, each edge feature point is projected onto the main direction of its cluster, and the projection values ​​are statistically encoded using histograms to generate the first sub-feature; for the planar feature points, the distribution of the normal vector direction of each planar feature point in three-dimensional space is statistically encoded using histograms to generate the second sub-feature; the first sub-feature and the second sub-feature are weighted and fused to generate the first feature vector. The color histogram and orientation gradient histogram features of the target are extracted from the visible light image data. The color histogram features and orientation gradient histogram features are then concatenated and fused to generate a second feature vector.

[0009] According to an embodiment of a first aspect of the present invention, the two-dimensional appearance features include texture features, and the extraction step includes: Convert visible light image data into grayscale images; The gray-level co-occurrence matrix is ​​used to calculate the texture features of the gray-level image in four directions: 0 degrees, 45 degrees, 90 degrees, and 135 degrees. The average value of each texture feature in the four directions is used as the texture feature vector of the image.

[0010] According to an embodiment of the first aspect of the present invention, a cross-modal feature association is established between a target in laser point cloud data and a target in visible light image data, by combining similarity and geometric position constraints of the target in space, including: For each target in the laser point cloud data, the target with the highest similarity and that satisfies the spatial geometric position constraints is retrieved in the visible light image data, and a candidate matching pair is constructed. Perform bidirectional consistency verification on candidate matching pairs: For a matching pair (A, B), if the nearest neighbor of target B in the visible light image data is target A in the laser point cloud data, and the similarity between the two is greater than the preset similarity matching threshold, then the matching pair is determined to be valid. Valid matching pairs are used as cross-modal feature association results.

[0011] According to an embodiment of the first aspect of the present invention, before calculating the similarity between the first feature vector and the second feature vector, the method further includes: Based on the radial distance between each target and the lidar in the laser point cloud data, a preset similarity matching threshold for the cross-modal features of each target is adaptively determined; for targets whose radial distance is greater than the preset distance threshold, a first similarity matching threshold is used; for targets whose radial distance is less than or equal to the preset distance threshold, a second similarity matching threshold is used; wherein, the first similarity matching threshold is less than the second similarity matching threshold.

[0012] According to an embodiment of the first aspect of the present invention, feeding back the updated location information to the feature extraction stage to guide target search and feature extraction in the next frame includes: The search area range of the next frame is dynamically adjusted based on the trace of the error covariance matrix of the tracking filter: when the trace is less than the first threshold, it is determined that the current tracking confidence is higher than the first confidence level, and the target search area in the visible light image and laser point cloud data is reduced by a first preset ratio; when the trace is greater than the second threshold, it is determined that the current tracking confidence is lower than the second confidence level, and the target search area is expanded by a second preset ratio; wherein, the second threshold is greater than the first threshold, the first preset ratio is less than 1, and the second preset ratio is greater than 1; Extract the two-dimensional appearance features and three-dimensional shape features of the next frame within the adjusted search area.

[0013] According to an embodiment of the first aspect of the present invention, the method further includes a re-detection and identity recovery step in the state of target loss: When the trace of the error covariance matrix of the tracking filter is greater than the loss threshold, the target is determined to be in a lost state. In the case of a lost state, the state estimation update is paused, and cross-modal feature association is re-performed within the target search area expanded by a second preset ratio. If re-association is successful, target tracking will resume and the target's identity will be inherited from before it was lost; if re-association fails for multiple consecutive frames, the target trajectory will be terminated.

[0014] According to an embodiment of the first aspect of the present invention, before performing clustering processing on the laser point cloud data, the method further includes performing hierarchical voxel downsampling on the laser point cloud data, the step of which includes: Calculate the radial distance from each point in the laser point cloud data to the origin of the lidar. , ( x， y, z ) represent the coordinates of each point in the laser point cloud data; Based on a preset distance threshold r th The point cloud is divided into near-field and far-field regions; First voxel size for near regions d near Voxel downsampling is performed, and a second voxel size is used for distant regions.d far Voxel downsampling, first voxel size d near Smaller than the second voxel size d far .

[0015] A second aspect of this invention provides a moving target tracking system that fuses laser point cloud data with visible light images. The system includes a data acquisition module, a preprocessing module, a feature extraction module, an association module, a prediction module, and a closed-loop update module. The data acquisition module acquires laser point cloud data from a lidar sensor and visible light image data from a visible light camera. The preprocessing module performs time synchronization and spatial alignment on the laser point cloud data and the visible light image data. The feature extraction module extracts the three-dimensional shape features of the target from the spatiotemporally aligned laser point cloud data and extracts the two-dimensional appearance features of the target from the visible light image data. The association module encodes the three-dimensional shape features and the two-dimensional appearance features into a first feature vector and a second feature vector, respectively, calculates the similarity between the first and second feature vectors, and establishes a cross-modal feature association between the target in the laser point cloud data and the target in the visible light image data based on the spatiotemporal alignment, combined with the similarity and the target's geometric position constraints in space. The prediction module inputs the cross-modal feature association results into a tracking filter to estimate the state of the moving target and updates the target's position information based on the state estimation results. The closed-loop update module feeds back the updated position information to the feature extraction stage to guide the target search and feature extraction in the next frame, thereby achieving adaptive tracking based on state feedback.

[0016] In summary, the moving target tracking method fusion of laser point cloud and visible light image provided by this invention extracts three-dimensional shape features from the spatially aligned laser point cloud and two-dimensional appearance features from the visible light image. After encoding both into feature vectors, similarity calculation is performed, and cross-modal feature association is established by combining spatial geometric position constraints. This achieves a leap from "detection box-level association" to "feature-level association," fundamentally eliminating the dependence on the accuracy of the target detection algorithm and maintaining stable association performance even when the point cloud is sparse or the image quality degrades. Furthermore, this invention ensures the temporal consistency of the laser point cloud and the visible light image through time synchronization and ensures their spatial geometric correspondence through spatial alignment, providing spatiotemporally complete input data for cross-modal feature association and further guaranteeing the overall performance of the tracking system. In addition, this invention feeds back the state estimation results of the tracking filter to the feature extraction stage to guide the target search and feature extraction in the next frame, realizing adaptive tracking based on state feedback and overcoming the error accumulation defects of open-loop architectures.

[0017] To make the above and other objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] Figure 1 The diagram shown is a flowchart illustrating a moving target tracking method that fuses laser point clouds and visible light images according to an embodiment of the present invention.

[0019] Figure 2 The flowchart shown is a process for extracting three-dimensional shape features from laser point cloud data in step S30.

[0020] Figure 3 The flowchart shown is for extracting two-dimensional appearance features from visible light image data in step S30.

[0021] Figure 4 The diagram shown is a schematic diagram of a moving target tracking system that fuses laser point clouds and visible light images according to an embodiment of the present invention. Detailed Implementation

[0022] Existing methods for fusing and tracking laser point clouds and visible light images mostly employ simple fusion at the detection box level or feature point matching. This fusion approach fails to effectively exploit the heterogeneous complementarity between laser point clouds and visible light images, and is unable to solve the problem of description instability caused by sparse point clouds.

[0023] In view of this, this embodiment provides a moving target tracking method that deeply fuses laser point cloud and visible light image data. The method includes: acquiring laser point cloud data collected by a lidar and visible light image data collected by a visible light camera (step S10). Performing temporal synchronization and spatial alignment on the laser point cloud data and visible light image data (step S20). Extracting the target's three-dimensional shape features from the spatiotemporally aligned laser point cloud data and extracting the target's two-dimensional appearance features from the visible light image data (step S30). Encoding the three-dimensional shape features and two-dimensional appearance features into a first feature vector and a second feature vector, respectively, calculating the similarity between the first and second feature vectors, and establishing a cross-modal feature association between the target in the laser point cloud data and the target in the visible light image data based on the spatiotemporal alignment, combined with the similarity and the target's geometric position constraints in space (step S40). Inputting the cross-modal feature association result into a tracking filter to perform state estimation on the moving target, and updating the target's position information based on the state estimation result (step S50). The updated position information is fed back to the feature extraction stage to guide the target search and feature extraction in the next frame, thereby achieving adaptive tracking based on state feedback (step S60).

[0024] The moving target tracking method based on the fusion of laser point clouds and visible light images provided in this embodiment can be applied to scenarios such as autonomous vehicles, inspection robots, or security monitoring systems. The system includes a lidar, a visible light camera, and a moving target tracking system based on the fusion of laser point clouds and visible light images. The lidar is used to collect laser point cloud data of the target scene, the visible light camera is used to simultaneously collect visible light image data of the target scene, and the moving target tracking system based on the fusion of laser point clouds and visible light images is used to execute the method steps provided in this embodiment. The following will combine... Figure 1 The method provided in this embodiment will be described in detail.

[0025] Step S10: Data Acquisition In one possible implementation, in step S10, the lidar acquires raw point cloud data of the surrounding environment at a frequency of 10Hz, with each point containing three-dimensional spatial coordinates. (x,y,z) Including reflection intensity information, the visible light camera acquires high-resolution color images at a frequency of 30Hz.

[0026] Then, step S20 is performed: time synchronization and spatial alignment of the two types of data. The raw point cloud data and visible light image data are preprocessed to achieve time synchronization and spatial alignment. Specifically, the laser point cloud data and visible light image data are time-synchronized, and the lidar and visible light camera are jointly calibrated to obtain spatial transformation relationships. Based on these relationships, the laser point cloud data and visible light image data are spatially aligned. After time synchronization and spatial alignment, the laser point cloud data and visible light image data achieve a spatiotemporal correspondence, meaning they correspond not only to the same moment in time but also to the same coordinate system in space. For ease of description, this specification refers to the data after time synchronization and spatial alignment as "spatiotemporally aligned data."

[0027] In one possible implementation, time synchronization is achieved through a hardware trigger signal: the lidar and the visible light camera share a clock source. Each time the lidar completes a frame scan, it outputs a trigger pulse signal to the visible light camera. The visible light camera responds to the trigger pulse signal to acquire an image, ensuring strict time synchronization between the lidar point cloud data and the visible light image data. Through this time synchronization method, each frame of data acquired by the lidar and the visible light camera forms a one-to-one correspondence in time. However, this invention does not impose any limitations on this. In other embodiments, synchronization can also be achieved through software interpolation. For example, using the lidar sampling time... t L Based on, retrieval and t L Adjacent consecutive frames I t-1 and I t+1Virtual frames are generated using linear interpolation. I tL :

[0028]

[0029] Among them, the time synchronization interpolation coefficients λ From sampling time t L Within the time interval of the two frames [ t t-1 , t t+1 The relative position within [ ] determines the [ ].

[0030] In one possible implementation, spatial alignment is achieved through joint calibration of the LiDAR and visible light camera: a checkerboard calibration board is placed within the shared field of view of both the LiDAR and the visible light camera, and multiple sets of calibration board point cloud data and image data are acquired under different poses; corner features of the calibration board are extracted from the calibration board image data to obtain the two-dimensional coordinates of the corner points in the image coordinate system; the point cloud of the calibration board plane is extracted from the calibration board point cloud data, and the plane equation of the calibration board plane in the LiDAR coordinate system is fitted to obtain the normal vector of the calibration board plane and the distance from the origin to the plane; based on the normal vector and corner features under multiple poses, a rotation matrix R is constructed. calib The solution model and translation vector T calib The solution model is to solve the rotation matrix R through singular value decomposition. calib The analytical solution is obtained by solving for the translation vector T using the least squares method. calib The optimal solution.

[0031] For any point in the laser point cloud data Its corresponding point in the camera coordinate system Calculated using the following formula:

[0032] However, this invention does not limit this in any way, and other methods that can spatially align laser point cloud data with visible light image data can also be applied to this invention.

[0033] In one possible implementation, after acquiring the laser point cloud data and before joint calibration, outlier removal and hierarchical voxel downsampling are performed on the laser point cloud data. The specific method for outlier removal is as follows: for each point in the original point cloud, a KD-Tree search is used to find its neighborhood. K The nearest neighbors (e.g.) K=50), calculate the average distance from the point to all neighboring points; after traversing all points, calculate the mean and standard deviation of the global average distance. When the average distance of a point is greater than the sum of the global average distance and the preset multiple of the standard deviation, the point is identified as an outlier and removed.

[0034] The specific method for layered voxel downsampling is as follows: Calculate the radial distance from each point in the denoised point cloud to the origin of the lidar:

[0035] ( x, y, z ) represent the coordinates of each point in the laser point cloud data.

[0036] Set distance threshold r th (Preferred) r th =30m), dividing the point cloud into near-field regions ( r ≤ r th ) and distant areas ( r > r th The first voxel size is used for the near region. d near (Preferred) d near Voxel downsampling was performed at 0.15m, and a second voxel size was used for distant regions. d far (Preferred) d far Voxel downsampling is performed at a depth of 0.6m. The specific method of voxel downsampling is as follows: the 3D space is divided into regular small cubic meshes according to the voxel size. For each non-empty mesh, the centroid of all points in the mesh is used to replace multiple original points. Through the above layering strategy, sufficient structural details of nearby obstacles are preserved, and the sparse point cloud at a distant distance is avoided from being lost due to oversampling.

[0037] Then proceed to step S30: Feature Extraction This step extracts the target's three-dimensional shape features from the spatiotemporally aligned laser point cloud data and extracts the target's two-dimensional appearance features from the visible light image data.

[0038] In one possible implementation, such as Figure 2 As shown, extracting 3D shape features from laser point cloud data includes extracting curvature features. The specific steps are as follows: Step S311: Perform clustering processing on the spatiotemporally aligned laser point cloud data to obtain multiple clusters and determine the cluster to which each point in the laser point cloud data belongs. Preferably, the DBSCAN algorithm is used for clustering, and the cluster neighborhood radius is... eps =0.5m, minimum number of points minPts =3.

[0039] Step S312: For each point in the laser point cloud data p i Within a preset neighborhood radius R curv (Preferred) R curv Search for a neighborhood set of points within a radius of 0.3m and construct the covariance matrix of that point: ; in, k The number of neighboring points. Let be the centroid of the neighborhood point set.

[0040] Step S313: For the covariance matrix Σ i Perform eigenvalue decomposition to obtain eigenvalues. λ 1≥ λ 2≥ λ 3.

[0041] Step S314, Calculate points p i curvature value C i = λ 3 / ( λ 1+ λ 2+ λ 3); Curvature C i The value range is [0, 1 / 3]. When the point is on a smooth plane, the curvature approaches 0; when the point is on an edge or corner, the curvature is larger. A curvature threshold is set. T c ( T c =0.1), if C i > T c If the point is marked as an edge feature point, it is marked as a planar feature point.

[0042] In one possible implementation, such as Figure 3 As shown, extracting two-dimensional appearance features from visible light image data includes extracting texture features. The specific steps are as follows: Step S321: Convert the visible light image data into a grayscale image.

[0043] Step S322: Using the gray-level co-occurrence matrix method, calculate the texture features of the gray-level image in four directions: 0 degrees, 45 degrees, 90 degrees, and 135 degrees. The texture features include contrast, energy, correlation, and entropy.

[0044] Step S323: Take the average value of each texture feature quantity in the four directions as the texture feature vector of the image.

[0045] Three-dimensional shape features and two-dimensional appearance features can be executed simultaneously or sequentially; this invention does not impose any limitations on this.

[0046] Then, step S40 is performed: feature encoding and cross-modal feature association. This step includes step S41, encoding the three-dimensional shape features and the two-dimensional appearance features into a first feature vector and a second feature vector, respectively.

[0047] In one possible implementation, based on the edge feature points and planar feature points determined by the curvature feature extraction in step S314, the three-dimensional shape features in the laser point cloud data are encoded into a first feature vector. This includes: for edge feature points: projecting each edge feature point onto the main direction of its cluster, performing histogram statistical encoding on the projected values ​​to generate a first sub-feature; the main direction of the cluster refers to the eigenvector corresponding to the largest eigenvalue obtained by decomposing the covariance matrix of each cluster. For planar feature points, performing histogram statistical encoding on the distribution of the normal vector directions of each planar feature point in three-dimensional space to generate a second sub-feature. Using preset fixed weight coefficients (preferably, the weight of the first sub-feature is 0.6 and the weight of the second sub-feature is 0.4), the first sub-feature and the second sub-feature are weighted and summed to generate the first feature vector.

[0048] This setup employs different encoding strategies for edge feature points and planar feature points based on their varying geometric attributes. Edge feature points utilize histogram encoding based on principal direction projection to highlight their directional characteristics; planar feature points employ 3D spatial encoding based on normal vector distribution to emphasize their planar structural features. Finally, the two encoding results are weighted and fused to form a comprehensive feature descriptor. This differentiated encoding strategy effectively improves the stability and discriminative power of feature descriptions in sparse point cloud scenarios.

[0049] In one possible implementation, encoding the two-dimensional appearance features into a second feature vector includes: extracting the target's color histogram features and orientation gradient histogram features from visible light image data, and concatenating and fusing the color histogram features and the orientation gradient histogram features to generate the second feature vector.

[0050] Step S42: Adaptively determine the preset similarity matching threshold for cross-modal features of each target based on the radial distance between each target and the lidar in the laser point cloud data. Specifically, for targets with a radial distance greater than the distance threshold (preferably 30m), the point cloud is relatively sparse, and the reliability of feature description is low; therefore, a first similarity matching threshold (relatively lenient, preferably 0.5) is used. For targets with a radial distance less than or equal to the distance threshold, the point cloud is relatively dense, and the reliability of feature description is high; therefore, a second similarity matching threshold (more stringent, preferably 0.7) is used, where the first similarity matching threshold is less than the second similarity matching threshold.

[0051] Step S43: Calculate the similarity between the first feature vector and the second feature vector. Specifically, Euclidean distance or cosine similarity can be used to calculate the similarity between the first feature vector and the second feature vector.

[0052] Step S44: Compare the similarity with the preset similarity matching threshold. Based on spatiotemporal alignment, combine the similarity and the geometric position constraints of the target in space to establish a cross-modal feature association between the target in the laser point cloud data and the target in the visible light image data.

[0053] The geometric position constraint includes: the reprojection error between the target's 3D spatial position in the camera coordinate system and its projected position in the visible light image is less than a preset error threshold. Specifically, the 3D position of the target in the laser point cloud data is obtained by joint calibration using the rotation matrix R. calib Translation vector T calib After being projected onto the image plane, the Euclidean distance between the projection point and the corresponding target's image coordinates in the visible light image should be less than a preset error threshold (preferably 5 pixels), thereby ensuring that the targets involved in the association have consistent spatial positions.

[0054] In one possible implementation, the specific method for establishing cross-modal feature associations based on similarity is as follows: For each target in the laser point cloud data, the target with the highest similarity and that satisfies the above spatial geometric position constraints is retrieved in the visible light image data, and a candidate matching pair is constructed. Determine whether the similarity between the first feature vector and the second feature vector within the candidate matching pair is greater than the preset similarity matching threshold determined for the target in step S42. If not, refuse to establish the association between the point cloud target and the visible light image target, and the target is considered unmatched in the current frame. If yes, proceed to bidirectional consistency verification.

[0055] This invention introduces a bidirectional consistency verification mechanism to pre-set similarity matching thresholds: for a candidate matching pair (A, B), it is required that not only is B the nearest neighbor of A in the image domain, but also that A is the nearest neighbor of B in the point cloud domain, and the bidirectional similarity is greater than the preset threshold for the match to be considered valid. All valid matching pairs are then used as the cross-modal feature association results. This bidirectional constraint significantly improves the reliability of cross-modal association and effectively solves the matching ambiguity problem in multi-object scenarios, especially in scenarios where multiple visually similar objects coexist, where unidirectional matching is prone to "one-to-many" or "many-to-one" mismatches.

[0056] Then, step S50 is performed to estimate the state of the moving target and perform adaptive tracking based on cross-modal features.

[0057] This step inputs the cross-modal feature association results into the tracking filter, performs state estimation on the moving target, and updates the target's position information based on the state estimation results.

[0058] In this embodiment, a Kalman filter is used as the tracking filter. However, the present invention does not limit this in any way. In other embodiments, the tracking filter may also be a particle filter.

[0059] The state equation and observation equation of the Kalman filter are as follows: Equations of state: ; Observation equation: ; in, x k Let F be the target state vector (including position, velocity, and acceleration) at time k. k Let w be the state transition matrix. k-1 For process noise, z k for k The observation vector at time step (i.e., the cross-modal feature association result), H k Let v be the observation matrix. k To observe noise.

[0060] The Kalman filter executes alternately through a prediction phase and an update phase: Prediction phase:

[0061]

[0062] Update phase:

[0063]

[0064]

[0065] in, P Let be the error covariance matrix. Q The process noise covariance matrix is... R obs To observe the noise covariance matrix, K k For Kalman gain, I It is the identity matrix. Let be the prior prediction value of the target state vector at time k. This is the posterior estimate.

[0066] Step S60: Feed the updated location information back to the feature extraction stage to guide the target search in the next frame. Feature extraction enables adaptive tracking based on state feedback.

[0067] Specifically, the updated three-dimensional position information of the target in the camera coordinate system P C Through camera intrinsic parameter matrix M int Projecting this onto the visible light image plane yields the target's two-dimensional position information on the image plane: p img =M int ⋅P C ; The search area for the next frame is dynamically adjusted based on the trace of the error covariance matrix of the tracking filter. When the trace is less than a first threshold, the current tracking confidence is determined to be high, and the target search area in the visible light image and laser point cloud data is reduced by a first preset ratio (e.g., 0.5). When the trace is greater than a second threshold, the current tracking confidence is determined to be low, and the target search area is expanded by a second preset ratio (e.g., 2.0). The second threshold is greater than the first threshold, the first preset ratio is less than 1, and the second preset ratio is greater than 1. Within the adjusted search area, the two-dimensional appearance features and three-dimensional shape features of the next frame are extracted respectively.

[0068] This invention dynamically adjusts the search region range of the next frame based on the trace of the tracking filter error covariance matrix. When the tracking confidence is high, the search region shrinks to reduce computational load and false detection rate; when the tracking confidence is low, the search region expands to prevent target loss. This state-aware adaptive search mechanism significantly improves the continuity and robustness of tracking.

[0069] Finally, the tracking results of the moving target are output, including the target's position, velocity, acceleration and identification information in each frame.

[0070] In one possible implementation, the moving target tracking method fusion of laser point cloud and visible light image provided by this invention further includes a re-detection and identity recovery step in the target loss state. Specifically, this includes: determining that the target is in a lost state when the trace of the tracking filter's error covariance matrix is ​​greater than a loss threshold (the loss threshold is greater than a second threshold); pausing state estimation updates in the lost state and re-performing cross-modal feature association within the target search area expanded by a second preset ratio; if re-association is successful, resuming target tracking and inheriting the target identity identifier from before the loss; and terminating the target trajectory if re-association fails for multiple consecutive frames (e.g., 3 consecutive frames). This setting further introduces a lost state determination and re-detection recovery mechanism based on the closed-loop tracking architecture: determining that the target is in a lost state when the trace of the tracking filter's error covariance matrix is ​​greater than a loss threshold, pausing state estimation updates, and re-performing cross-modal feature association within the expanded search area; if re-association is successful, resuming target tracking and inheriting the target identity identifier from before the loss. This mechanism effectively solves the identity recovery problem after target occlusion or temporary loss of view, ensuring the long-term continuity of the target trajectory.

[0071] Corresponding to the aforementioned method for tracking moving targets by fusing laser point clouds and visible light images, this invention also provides a system for tracking moving targets by fusing laser point clouds and visible light images. For example... Figure 4 As shown, it includes a data acquisition module 10, a preprocessing module 20, a feature extraction module 30, an association module 40, a prediction module 50, and a closed-loop update module 60. The data acquisition module 10 acquires laser point cloud data collected by a lidar and visible light image data collected by a visible light camera. The preprocessing module 20 performs time synchronization and spatial alignment on the laser point cloud data and visible light image data. The feature extraction module 30 extracts the target's three-dimensional shape features from the spatiotemporally aligned laser point cloud data and extracts the target's two-dimensional appearance features from the visible light image data. The association module 40 encodes the three-dimensional shape features and two-dimensional appearance features into a first feature vector and a second feature vector, respectively, calculates the similarity between the first and second feature vectors, and establishes a cross-modal feature association between the target in the laser point cloud data and the target in the visible light image data based on the spatiotemporal alignment, combined with the similarity and the target's geometric position constraints in space. The prediction module 50 inputs the cross-modal feature association results into a tracking filter, performs state estimation on the moving target, and updates the target's position information based on the state estimation results. The closed-loop update module 60 feeds back the updated position information to the feature extraction stage to guide the target search and feature extraction in the next frame, thereby achieving adaptive tracking based on state feedback.

[0072] Since the moving target tracking system that fuses laser point clouds and visible light images provided in this embodiment is different from the one described in the above embodiments... Figure 1The system corresponding to the moving target tracking method that fuses laser point clouds and visible light images, the execution steps of each module in the system are based on the same concept as the above method embodiments. Therefore, steps S10-60 above can be referred to, and will not be repeated here.

[0073] The above modules can be embedded in the processor of the computer device in hardware form or independent of it, or they can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of the above modules.

[0074] In summary, the moving target tracking method fusion of laser point cloud and visible light image provided by this invention extracts three-dimensional shape features from the spatially aligned laser point cloud and two-dimensional appearance features from the visible light image. After encoding both into feature vectors, similarity calculation is performed, and cross-modal feature association is established by combining spatial geometric position constraints. This achieves a leap from "detection box-level association" to "feature-level association," fundamentally eliminating the dependence on the accuracy of the target detection algorithm and maintaining stable association performance even when the point cloud is sparse or the image quality degrades. Furthermore, this invention ensures the temporal consistency of the laser point cloud and the visible light image through time synchronization and ensures their spatial geometric correspondence through spatial alignment, providing spatiotemporally complete input data for cross-modal feature association and further guaranteeing the overall performance of the tracking system. In addition, this invention feeds back the state estimation results of the tracking filter to the feature extraction stage to guide the target search and feature extraction in the next frame, realizing adaptive tracking based on state feedback and overcoming the error accumulation defects of open-loop architectures.

[0075] Although the present invention has been disclosed above by way of preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art may make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the scope of protection claimed in the claims.

Claims

1. A method for tracking moving targets by fusing laser point clouds and visible light images, characterized in that, include: Acquire laser point cloud data collected by lidar and visible light image data collected by a visible light camera; The laser point cloud data and visible light image data are synchronized in time and aligned in space. The three-dimensional shape features of the target are extracted from the spatiotemporally aligned laser point cloud data, and the two-dimensional appearance features of the target are extracted from the visible light image data. The three-dimensional shape features and two-dimensional appearance features are encoded into a first feature vector and a second feature vector, respectively. The similarity between the first feature vector and the second feature vector is calculated. Based on the spatiotemporal alignment, and combined with the similarity and the geometric position constraints of the target in space, a cross-modal feature association is established between the target in the laser point cloud data and the target in the visible light image data. The cross-modal feature association results are input into the tracking filter to estimate the state of the moving target and update the target's position information based on the state estimation results. The updated position information is fed back to the feature extraction stage to guide the target search and feature extraction in the next frame, thus achieving adaptive tracking based on state feedback.

2. The moving target tracking method based on the fusion of laser point cloud and visible light image according to claim 1, characterized in that, The extraction of the target's three-dimensional shape features from the spatiotemporally aligned laser point cloud data includes: Clustering is performed on the spatiotemporally aligned laser point cloud data to obtain multiple clusters; For each point in the laser point cloud data p i Search for a set of neighborhood points within a preset neighborhood radius and construct the covariance matrix of that point: ； in, k The number of neighboring points. Let be the centroid of the neighborhood point set; For the covariance matrix Σ i Perform eigenvalue decomposition to obtain eigenvalues. λ 1≥ λ 2≥ λ 3; Calculation points p i curvature value C i = λ 3 / ( λ 1+ λ 2+ λ 3) Mark points with curvature values ​​greater than a preset curvature threshold as edge feature points, and mark points with curvature values ​​less than or equal to the preset curvature threshold as planar feature points.

3. The moving target tracking method based on the fusion of laser point cloud and visible light image according to claim 2, characterized in that, The step of encoding the three-dimensional shape features and the two-dimensional appearance features into a first feature vector and a second feature vector, respectively, includes: Based on the edge feature points and planar feature points determined by curvature feature extraction, each edge feature point is projected onto the main direction of its cluster, and the projection value is statistically encoded using a histogram to generate the first sub-feature; for the planar feature points, the distribution of the normal vector direction of each planar feature point in three-dimensional space is statistically encoded using a histogram to generate the second sub-feature; the first sub-feature and the second sub-feature are weighted and fused to generate the first feature vector. The color histogram features and orientation gradient histogram features of the target are extracted from the visible light image data. The color histogram features and the orientation gradient histogram features are then fused together to generate a second feature vector.

4. The moving target tracking method based on the fusion of laser point cloud and visible light image according to claim 1, characterized in that, The two-dimensional appearance features include texture features. The extraction steps include: Convert the visible light image data into a grayscale image; The gray-level co-occurrence matrix is ​​used to calculate the texture feature quantities of the gray-level image in four directions: 0 degrees, 45 degrees, 90 degrees, and 135 degrees. The average value of each texture feature in the four directions is used as the texture feature vector of the image.

5. The moving target tracking method based on the fusion of laser point cloud and visible light image according to claim 1, characterized in that, The method of establishing cross-modal feature association between targets in laser point cloud data and targets in visible light image data by combining similarity and spatial geometric position constraints includes: For each target in the laser point cloud data, the target with the highest similarity and that satisfies the spatial geometric position constraints is retrieved in the visible light image data, and a candidate matching pair is constructed. A bidirectional consistency check is performed on the candidate matching pairs: For a matching pair (A,B), if the nearest neighbor of target B in the visible light image data is target A in the laser point cloud data, and the similarity between the two is greater than the preset similarity matching threshold, then the matching pair is determined to be valid. The valid matching pairs are used as the cross-modal feature association results.

6. The moving target tracking method based on the fusion of laser point cloud and visible light image according to claim 5, characterized in that, Before calculating the similarity between the first feature vector and the second feature vector, the method further includes: Based on the radial distance between each target and the lidar in the laser point cloud data, a preset similarity matching threshold for the cross-modal features of each target is adaptively determined; for targets whose radial distance is greater than the preset distance threshold, a first similarity matching threshold is used; for targets whose radial distance is less than or equal to the preset distance threshold, a second similarity matching threshold is used; wherein, the first similarity matching threshold is less than the second similarity matching threshold.

7. The moving target tracking method based on the fusion of laser point cloud and visible light image according to claim 1, characterized in that, The step of feeding back the updated location information to the feature extraction stage to guide the target search and feature extraction in the next frame includes: The search area range of the next frame is dynamically adjusted based on the trace of the error covariance matrix of the tracking filter: when the trace is less than a first threshold, it is determined that the current tracking confidence is higher than the first confidence level, and the target search area in the visible light image and laser point cloud data is reduced by a first preset ratio; when the trace is greater than a second threshold, it is determined that the current tracking confidence is lower than the second confidence level, and the target search area is expanded by a second preset ratio; wherein, the second threshold is greater than the first threshold, the first preset ratio is less than 1, and the second preset ratio is greater than 1; Extract the two-dimensional appearance features and three-dimensional shape features of the next frame within the adjusted search area.

8. The moving target tracking method based on the fusion of laser point cloud and visible light image according to claim 7, characterized in that, The method also includes re-detection and identity recovery steps in the case of target loss: When the trace of the error covariance matrix of the tracking filter is greater than the loss threshold, the target is determined to be in a lost state. In the lost state, the state estimation update is paused, and cross-modal feature association is re-performed within the target search area expanded by the second preset ratio; If re-association is successful, target tracking will be restored and the target's identity will be inherited from the point of loss. If re-association fails for multiple consecutive frames, the target trajectory is terminated.

9. The moving target tracking method based on the fusion of laser point cloud and visible light image according to claim 1, characterized in that, Before performing clustering processing on the laser point cloud data, the method further includes performing hierarchical voxel downsampling on the laser point cloud data, which includes: Calculate the radial distance from each point in the laser point cloud data to the origin of the lidar. , ( x, y, z ) represent the coordinates of each point in the laser point cloud data; Based on a preset distance threshold r th The point cloud is divided into near-field and far-field regions; The near region is sized using the first voxel size. d near Voxel downsampling is performed, and a second voxel size is used for the distant region. d far Voxel downsampling is performed, the size of the first voxel is... d near Smaller than the second voxel size d far .

10. A moving target tracking system that fuses laser point clouds with visible light images, characterized in that, include: The data acquisition module acquires laser point cloud data collected by the lidar and visible light image data collected by the visible light camera. The preprocessing module performs time synchronization and spatial alignment on the laser point cloud data and visible light image data; The feature extraction module extracts the three-dimensional shape features of the target from the spatiotemporally aligned laser point cloud data and the two-dimensional appearance features of the target from the visible light image data. The association module encodes the three-dimensional shape features and two-dimensional appearance features into a first feature vector and a second feature vector, respectively, calculates the similarity between the first feature vector and the second feature vector, and, based on the spatiotemporal alignment, combines the similarity and the geometric position constraints of the target in space to establish a cross-modal feature association between the target in the laser point cloud data and the target in the visible light image data. The prediction module inputs the cross-modal feature association results into the tracking filter to estimate the state of the moving target and updates the target's position information based on the state estimation results. The closed-loop update module feeds back the updated position information to the feature extraction stage to guide the target search and feature extraction in the next frame, thereby achieving adaptive tracking based on state feedback.

Images