Radar and visual information fused target tracking method and tracking system

Abstract

The invention provides a radar and visual information fusion target tracking method and tracking system, and relates to the technical field of target tracking, and the method comprises the steps: obtaining radar data collected by a radar module, visual data collected by a visual module, and environment parameters collected by an environment module, and carrying out the data preprocessing; respectively extracting motion features and appearance features of the target from the radar data and the visual data; based on the environmental parameters, performing adaptive weighted fusion on the motion features and the appearance features to obtain fusion features; a pre-trained machine learning model is utilized to perform target association according to the fusion features, wherein the machine learning model is trained through historical data containing different environmental parameters; according to the fusion features and the target correlation result, a state estimation method is used for carrying out trajectory estimation and updating on the target, target tracking is achieved, the limitation of a single sensor system in the prior art is overcome, and the target tracking precision and adaptability are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking, and in particular to a target tracking method and a tracking system that fuses radar and visual information. Background Art

[0002] In the field of target tracking technology, traditional single sensor systems (such as those that rely only on radar or vision systems) often face many challenges in complex and changing environmental conditions. Although radar systems have a long detection range and strong penetration capabilities, they are limited in target recognition, especially in distinguishing similar objects; and although vision systems can provide rich appearance information, their performance will be significantly reduced in bad weather (such as rain, fog, snow) or environments with large changes in lighting conditions. Therefore, how to effectively integrate radar and visual information to overcome the shortcomings of a single sensor and improve the accuracy and robustness of target tracking has become a hot topic in current research.

[0003] In the prior art, there have been some attempts to fuse radar and visual information, but most of these methods are based on simple data splicing or fixed weight distribution strategies, and fail to fully consider the impact of environmental parameters on sensor performance, resulting in unsatisfactory tracking effects in complex and changing environments.

[0004] Therefore, it is necessary to provide a target tracking method and tracking system that fuses radar and visual information to solve the above technical problems. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a target tracking method and tracking system integrating radar and visual information, aiming to overcome the limitations of the single sensor system in the prior art and improve the accuracy and adaptability of target tracking.

[0006] The present invention provides a target tracking method by fusing radar and visual information, the tracking method comprising the following steps:

[0007] Acquire radar data collected by the radar module, visual data collected by the visual module, and environmental parameters collected by the environmental module, and perform data preprocessing;

[0008] Extracting motion features and appearance features of the target from the radar data and the visual data respectively;

[0009] Based on the environmental parameters, adaptively weighted fusion is performed on the motion features and the appearance features to obtain fusion features;

[0010] Associating targets based on the fused features using a pre-trained machine learning model, wherein the machine learning model has been trained using historical data containing different environmental parameters;

[0011] According to the fusion features and the target association results, the target trajectory is estimated and updated using a state estimation method to achieve target tracking.

[0012] Preferably, before data preprocessing, the following steps are further included:

[0013] The radar data and the visual data are adjusted by timestamp matching and coordinate system unification so that the radar data and the visual data correspond to the same spatiotemporal reference frame.

[0014] Preferably, extracting the motion features and appearance features of the target from the radar data and the visual data respectively includes:

[0015] Extracting the motion characteristics of the target from the radar data using a radar signal processing algorithm, wherein the motion characteristics include velocity, acceleration and direction angle;

[0016] The appearance features of the target are extracted from the visual data using an image processing method, wherein the appearance features include color histogram, edge contour and texture features.

[0017] Preferably, the adaptively weighted fusion of the motion features and the appearance features based on the environmental parameters to obtain the fused features includes:

[0018] Determining a first weight of the radar data and a second weight of the visual data according to the environmental parameter;

[0019] The motion feature and the appearance feature are weightedly fused using the first weight and the second weight to obtain a fused feature, wherein the mathematical expression of the fused feature is:

[0020] F=α·F1+β·F2

[0021] Wherein, F represents the fusion feature, F1 and F2 represent the motion feature and the appearance feature respectively, α and β represent the adaptive first weight and the second weight respectively, and α+β=1.

[0022] Preferably, the calculation formula of the first weight is:

[0023]

[0024] The calculation formula of the second weight is:

[0025]

[0026] Among them, f(E) represents the environmental parameter evaluation function, θ represents a positive constant used to adjust the reference point of weight distribution, and E represents the environmental parameter vector.

[0027] Preferably, the target association is performed according to the fusion feature using a pre-trained machine learning model, wherein the machine learning model has been trained using historical data containing different environmental parameters, including:

[0028] Using the fused features and current environmental parameters as inputs to a pre-trained machine learning model;

[0029] Use pre-trained machine learning models to perform reasoning and infer the associations between targets;

[0030] Output a target association result, wherein the target association result represents a dynamic relationship between the target in the radar data and the target in the visual data under current environmental parameters.

[0031] Preferably, the method of using a state estimation method to estimate and update the trajectory of the target based on the fusion feature and the target association result to achieve target tracking includes:

[0032] The fusion feature and the target association result are used as the measurement value of the state estimation method and input, wherein the state estimation method adopts a Kalman filter;

[0033] Use a Kalman filter to estimate and update the trajectory of the target, and output a trajectory estimation result, wherein the trajectory estimation result represents the motion trajectory of the target under the current environmental parameters;

[0034] The Kalman filter dynamically adjusts the target state according to the fusion feature and the target association result to achieve continuous tracking of the target.

[0035] Preferably, the Kalman filter updating step comprises:

[0036] Predict the predicted state and predicted covariance matrix of the target at the current moment, wherein the expression of the predicted state is:

[0037]

[0038] in, represents the predicted state at time k, A represents the state transfer matrix, B represents the control input matrix, μ k represents the control input, Indicates the state of the previous moment k-1;

[0039] The expression of the prediction covariance matrix is:

[0040] P k|k-1 =A·P k-1|k-1 ·A T +Q

[0041] Among them, Pk|k-1 represents the current covariance matrix at prediction time k, P k-1|k-1 represents the covariance matrix of the previous moment k-1, Q represents the process noise covariance matrix, and T represents the transpose operation of the matrix A;

[0042] The state and covariance matrix are updated based on the measurement values ​​including the fusion feature and the target association result, wherein the state update expression is:

[0043]

[0044] in, represents the updated state at time k, K k represents the Kalman gain, z k represents the measurement value at time k, and H represents the measurement matrix;

[0045] The update expression of the covariance matrix is:

[0046] P k|k =(IK k ·H)·P k|k-1

[0047] Among them, P k|k represents the updated covariance matrix at time k, and I represents the identity matrix;

[0048] The expression of the Kalman gain is:

[0049] K k =P k|k-1 ·H T (H.P k|k-1 ·H T +R) -1

[0050] Among them, H T represents the transpose of the measurement matrix H, and R represents the measurement noise covariance matrix.

[0051] The present invention also provides a target tracking system integrating radar and visual information, which is used to execute the target tracking method integrating radar and visual information. The tracking system comprises:

[0052] A data acquisition module is used to acquire radar data collected by the radar module, visual data collected by the visual module, and environmental parameters collected by the environmental module, and perform data preprocessing;

[0053] A feature extraction module, used to extract the motion features and appearance features of the target from the radar data and the visual data respectively;

[0054] A weighted fusion module, used for adaptively weighting and fusing the motion features and the appearance features based on the environmental parameters to obtain fused features;

[0055] a target association module, configured to perform target association based on the fused features using a pre-trained machine learning model, wherein the machine learning model has been trained using historical data containing different environmental parameters;

[0056] The target tracking module is used to estimate and update the trajectory of the target using a state estimation method according to the fusion features and the target association result, so as to achieve tracking of the target.

[0057] Compared with the related art, the target tracking method and tracking system provided by the present invention by integrating radar and visual information have the following beneficial effects:

[0058] The present invention obtains radar data collected by the radar module, visual data collected by the visual module, and environmental parameters collected by the environmental module, and performs data preprocessing and feature extraction; then adaptively weighted fusion of motion features and appearance features is performed based on environmental parameters to obtain fusion features; then the pre-trained machine learning model is used for target association; finally, according to the fusion features and target association results, the state estimation method is used to estimate and update the trajectory of the target. This method not only makes full use of the complementary advantages of radar and visual information, but also realizes adaptive adjustment of the fusion strategy through the introduction of environmental parameters, thereby improving the accuracy and robustness of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A flow chart of a target tracking method that integrates radar and visual information provided by the present invention;

[0060] Figure 2 A module structure diagram of a target tracking system that integrates radar and visual information provided by the present invention. DETAILED DESCRIPTION

[0061] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0062] It should also be noted that, for ease of description, only the part relevant to the present invention but not all content is shown in the accompanying drawings. It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processing or methods depicted as flow charts. Although the flow chart describes each operation (or step) as sequential processing, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. When its operation is completed, the processing can be terminated, but it can also have additional steps not included in the accompanying drawings. The processing can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0063] Embodiment 1

[0064] The present invention provides a target tracking method by fusing radar and visual information, the tracking method comprising the following steps:

[0065] S1: Acquire radar data collected by the radar module, visual data collected by the visual module, and environmental parameters collected by the environmental module, and perform data preprocessing.

[0066] Before data preprocessing, it also includes:

[0067] The radar data and the visual data are adjusted by timestamp matching and coordinate system unification so that the radar data and the visual data correspond to the same spatiotemporal reference frame.

[0068] Step S1 is specifically as follows:

[0069] Get radar data: The radar module collects the target's radar echo signals, which contain the target's distance, speed and direction angle.

[0070] Obtain visual data: The visual module collects the target's image data, which includes the target's color histogram, edge contour, and texture features.

[0071] Obtain environmental parameters: Collect the parameters of the current environment through the environmental module, including light intensity, weather conditions (sunny, cloudy, rainy, foggy), and background complexity.

[0072] Radar data preprocessing: Filter and denoise the radar echo signal to extract clear target features.

[0073] Visual data preprocessing: Perform color correction, lighting compensation and other processing on image data to ensure image quality.

[0074] Timestamp matching and coordinate system normalization: Timestamp matching is used to ensure the time synchronization of radar data and visual data; coordinate system normalization is used to convert radar data and visual data into the same reference coordinate system so that the two can be compared and integrated in the same spatial framework.

[0075] This step ensures the time synchronization of radar data and visual data through timestamp matching, avoiding mismatching due to time deviation; at the same time, through the standardization of coordinate system adjustment, it ensures that radar data and visual data are in the same reference coordinate system, improving the consistency and comparability of the data.

[0076] In addition, by collecting environmental parameters, necessary information is provided for the subsequent adaptive weighted fusion, so that the system can adjust the fusion strategy according to different environmental conditions and improve the robustness and adaptability of the system.

[0077] S2: Extracting motion features and appearance features of the target from the radar data and the visual data respectively.

[0078] Specifically, step S2 includes the following steps:

[0079] S21: Extracting the motion characteristics of the target from the radar data using a radar signal processing algorithm, wherein the motion characteristics include velocity, acceleration and direction angle.

[0080] In this embodiment, in radar data processing, it is necessary to extract the target's motion features from the radar echo signal, which are crucial for target tracking. The motion features include velocity, acceleration, and direction angle, which can help the system accurately estimate the position and motion state of the target.

[0081] Specifically: the Doppler effect is used to analyze the frequency shift in the radar echo signal to calculate the target's speed; the acceleration of the target is calculated by differentiating the speed at multiple consecutive time points; the directional characteristics of the radar antenna or the phase difference of multiple antennas are used to estimate the target's direction angle.

[0082] S22: Extracting appearance features of the target from the visual data using an image processing method, wherein the appearance features include a color histogram, an edge profile, and a texture feature.

[0083] In this embodiment, in visual data processing, it is necessary to extract the appearance features of the target from the image, which are helpful to distinguish different targets and perform target recognition. The appearance features include color histogram, edge contour and texture features, which can provide rich visual information.

[0084] Specifically: convert the image from RGB color space to HSV or Lab color space to facilitate the extraction of color features, calculate the color histogram of the image, and count the frequency of occurrence of each color interval; use the Canny edge detection method to extract the edges in the image, perform connected component analysis on the edges, and extract the contour of the target; use the local binary pattern LBP to extract the texture features of the image, construct a texture feature vector, and represent the texture information of the image.

[0085] S3: Based on the environmental parameters, adaptively weighted fuse the motion features and appearance features to obtain fused features.

[0086] Specifically, step S3 includes the following steps:

[0087] S31: Determine a first weight of the radar data and a second weight of the visual data according to the environmental parameters.

[0088] In the target tracking method of radar and visual information fusion, environmental parameters have an important impact on the quality of radar data and visual data. In order to improve the robustness and adaptability of the system, it is necessary to dynamically adjust the weights of radar data and visual data according to environmental parameters. Specifically, when the environmental conditions are poor (such as insufficient light, foggy weather, etc.), the reliability of radar data is relatively high, while the reliability of visual data is low; vice versa. Therefore, it is necessary to adaptively adjust the weights of radar data and visual data according to environmental parameters to ensure the reliability and accuracy of fused features.

[0089] Define the environmental parameter evaluation function, which is used to evaluate the impact of current environmental parameters on radar data and visual data.

[0090] Specifically include:

[0091] Definition of radar data evaluation function: The quality of radar data under different environmental conditions is mainly affected by weather conditions, especially in conditions with low visibility (such as foggy days), the quality of radar data is relatively good. In addition, background complexity has little effect on radar data.

[0092] The radar data evaluation function can be defined as: considering the impact of weather conditions on radar data quality and assigning a certain weight; considering the impact of background complexity on radar data quality and assigning a certain weight.

[0093] Specifically, the radar data evaluation function combines weather conditions and background complexity to evaluate the quality of radar data in the current environment through a given weight coefficient.

[0094] Define the visual data evaluation function: The quality of visual data is mainly affected by light intensity and weather conditions. When the light intensity is high and the weather conditions are good, the visual data quality is good; on the contrary, when the light intensity is low or the weather conditions are bad, the visual data quality is poor. In addition, the complexity of the background will also affect the quality of the visual data. Complex backgrounds make target recognition more difficult.

[0095] The visual data evaluation function can be defined as:

[0096] Consider the impact of light intensity on the quality of visual data and assign a certain weight.

[0097] Consider the impact of weather conditions on the quality of visual data and assign a certain weight.

[0098] Consider the impact of background complexity on the quality of visual data and assign a certain weight.

[0099] Specifically, the visual data evaluation function combines three factors: light intensity, weather conditions, and background complexity, and evaluates the quality of visual data in the current environment through a given weight coefficient.

[0100] S32: Using the first weight and the second weight to perform weighted fusion on the motion feature and the appearance feature to obtain a fusion feature, wherein the mathematical expression of the fusion feature is:

[0101] F=α·F1+β·F2

[0102] Wherein, F represents the fusion feature, F1 and F2 represent the motion feature and the appearance feature respectively, α and β represent the adaptive first weight and the second weight respectively, and α+β=1.

[0103] Specifically, the calculation formula of the first weight is:

[0104]

[0105] The calculation formula of the second weight is:

[0106]

[0107] Among them, f(E) represents the environmental parameter evaluation function, θ represents a positive constant used to adjust the reference point of weight distribution, and E represents the environmental parameter vector.

[0108] In this embodiment, after obtaining the first weight of radar data and the second weight of visual data, the two weights need to be applied to the weighted fusion of motion features and appearance features to obtain the final fused features. This ensures that the fused features still have high reliability and accuracy under different environmental conditions.

[0109] S4: Utilize a pre-trained machine learning model to perform target association based on the fusion features, wherein the machine learning model has been trained using historical data containing different environmental parameters.

[0110] Specifically, step S4 includes the following steps:

[0111] S41: Using the fused features and current environmental parameters as input of a pre-trained machine learning model.

[0112] In this embodiment, during the target tracking process, the fusion feature already contains the weighted fusion result of radar data and visual data, and the environmental parameters provide information about the current environmental conditions. The pre-trained machine learning model learns the target association pattern under different environmental parameters through historical data, so the fusion feature and the current environmental parameters are needed as input for target association.

[0113] S42: Use the pre-trained machine learning model for reasoning and infer the association between targets.

[0114] In this embodiment, the pre-trained machine learning model has learned the target association patterns under different environmental conditions through a large amount of historical data. Therefore, under the current environmental conditions, the model can be used for reasoning to infer the association relationship between the targets in the radar data and the targets in the visual data.

[0115] Perform inference using the input fused features and environment parameters.

[0116] Machine learning models perform internal calculations based on input data and infer the associations between targets.

[0117] Output the association relationship between targets, that is, determine which targets detected by radar and which targets identified by vision are the same target.

[0118] S43: Outputting a target association result, wherein the target association result represents a dynamic relationship between the target in the radar data and the target in the visual data under current environmental parameters.

[0119] In this embodiment, the target association result needs to clarify the correspondence between the target in the radar data and the target in the visual data, which is helpful for subsequent state estimation and trajectory tracking.

[0120] According to the target association relationship inferred in step S42, a target association result is formed.

[0121] The association results should clearly indicate the correspondence between the targets in the radar data and the targets in the visual data.

[0122] Output the target association results for use in subsequent steps.

[0123] The association result should contain the ID of each target and the corresponding feature information in the radar data and visual data.

[0124] S5: According to the fusion features and the target association results, a state estimation method is used to estimate and update the trajectory of the target to achieve tracking of the target.

[0125] Specifically, step S5 includes the following steps:

[0126] S51: Using the fusion feature and the target association result as the measurement value of the state estimation method and inputting it, wherein the state estimation method adopts a Kalman filter.

[0127] In this embodiment, state estimation is a key step in the target tracking process, which requires updating the state of the target based on the current observation data (i.e., fusion features and target association results) and the previous estimated state. Kalman filter is a widely used state estimation method, which is particularly suitable for processing noisy measurement data and state estimation of dynamic systems.

[0128] S52: Use the Kalman filter to estimate and update the trajectory of the target, and output a trajectory estimation result, where the trajectory estimation result represents the motion trajectory of the target under the current environmental parameters.

[0129] The Kalman filter dynamically adjusts the target state according to the fusion feature and the target association result to achieve continuous tracking of the target.

[0130] In this embodiment, the Kalman filter estimates the state of the target through two main steps: prediction and update. The prediction step predicts the state at the current moment based on the state estimate and control input at the previous moment; the update step uses the measurement value at the current moment to correct the predicted state to obtain a more accurate state estimate.

[0131] Specifically, the updating step of the Kalman filter includes:

[0132] Predict the predicted state and predicted covariance matrix of the target at the current moment, wherein the expression of the predicted state is:

[0133]

[0134] in, represents the predicted state at time k, A represents the state transfer matrix, B represents the control input matrix, μ k represents the control input, Indicates the state of the previous moment k-1;

[0135] The expression of the prediction covariance matrix is:

[0136] P k|k-1 =A·P k-1|k-1 ·A T +Q

[0137] Among them, P k|k-1 represents the current covariance matrix at prediction time k, P k-1|k-1 represents the covariance matrix of the previous moment k-1, Q represents the process noise covariance matrix, and T represents the transpose operation of the matrix A;

[0138] The state and covariance matrix are updated based on the measurement values ​​including the fusion feature and the target association result, wherein the state update expression is:

[0139]

[0140] in, represents the updated state at time k, K k represents the Kalman gain, z k represents the measurement value at time k, and H represents the measurement matrix;

[0141] The update expression of the covariance matrix is:

[0142] P k|k =(IK k ·H)·P k|k-1

[0143] Among them, P k|k represents the updated covariance matrix at time k, and I represents the identity matrix;

[0144] The expression of the Kalman gain is:

[0145] K k =P k|k-1 ·H T (H.P k|k-1 ·H T +R) -1

[0146] Among them, H T represents the transpose of the measurement matrix H, and R represents the measurement noise covariance matrix.

[0147] The working principle of the target tracking method provided by the present invention by integrating radar and visual information is as follows:

[0148] First, data is collected from the radar module, vision module and environment module respectively. The radar data provides the target's motion information such as distance, speed, acceleration, etc., while the vision data provides the target's appearance information such as color, edge contour, texture, etc. The environment module collects environmental parameters such as light intensity and weather conditions. These parameters have an important impact on the accuracy and reliability of radar and vision data.

[0149] In the data preprocessing stage, radar data and visual data are mapped to the same spatiotemporal reference frame through timestamp matching and coordinate system normalization to ensure data consistency and comparability.

[0150] Next, the motion features and appearance features of the target are extracted from the preprocessed radar data and visual data respectively. The radar data uses the radar signal processing algorithm to extract the motion features of the target such as speed, acceleration and direction angle; while the visual data uses the image processing method to extract the appearance features of the target such as color histogram, edge contour and texture features.

[0151] Based on environmental parameters, the extracted motion features and appearance features are adaptively weighted fused because the performance of radar and visual sensors varies in different environments.

[0152] The weights of radar data and visual data are determined according to environmental parameters, and then these weights are used to perform weighted fusion of motion features and appearance features to obtain fused features.

[0153] Target association is performed based on fused features using a pre-trained machine learning model that has been trained using historical data containing different environmental parameters and can accurately infer the dynamic relationship between targets in radar data and targets in visual data.

[0154] In this step, the fused features and current environmental parameters are used as input to the machine learning model, the association relationship between targets is obtained through model reasoning, and the target association results are output.

[0155] Finally, according to the fusion features and target association results, the state estimation method is used to estimate and update the target trajectory. The Kalman filter is used as the state estimation method here because it can effectively process noisy measurement data and update the target state in real time.

[0156] The Kalman filter first predicts the predicted state and predicted covariance matrix at the current moment based on the state and process noise covariance matrix at the previous moment. Then, based on the measurement values ​​including fusion features and target association results, the Kalman filter updates the state and covariance matrix. By continuously iterating this process, the Kalman filter can dynamically adjust the target state, achieve continuous tracking of the target, and output the target's motion trajectory.

[0157] Embodiment 2

[0158] The present invention also provides a target tracking system integrating radar and visual information, which is used to execute the target tracking method integrating radar and visual information. The tracking system comprises:

[0159] The data acquisition module 100 is used to acquire radar data collected by the radar module, visual data collected by the visual module, and environmental parameters collected by the environmental module, and perform data preprocessing.

[0160] The feature extraction module 200 is used to extract the motion features and appearance features of the target from the radar data and the visual data respectively.

[0161] The weighted fusion module 300 is used to perform adaptive weighted fusion on the motion features and appearance features based on the environmental parameters to obtain fused features.

[0162] The target association module 400 is used to use a pre-trained machine learning model to perform target association according to the fusion feature, wherein the machine learning model has been trained with historical data containing different environmental parameters.

[0163] The target tracking module 500 is used to estimate and update the trajectory of the target using a state estimation method according to the fusion feature and the target association result, so as to track the target.

[0164] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, the storage medium including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0166] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A target tracking method integrating radar and visual information, characterized in that: The tracking method comprises the following steps: Acquire radar data collected by the radar module, visual data collected by the visual module, and environmental parameters collected by the environmental module, and perform data preprocessing; Extracting motion features and appearance features of the target from the radar data and the visual data respectively; Based on the environmental parameters, adaptively weighted fusion is performed on the motion features and the appearance features to obtain fusion features; Associating targets based on the fused features using a pre-trained machine learning model, wherein the machine learning model has been trained using historical data containing different environmental parameters; According to the fusion features and the target association results, the target trajectory is estimated and updated using a state estimation method to achieve target tracking.

2. The target tracking method of claim 1, characterized in that: Before data preprocessing, it also includes: The radar data and the visual data are adjusted by timestamp matching and coordinate system unification so that the radar data and the visual data correspond to the same spatiotemporal reference frame.

3. The target tracking method of radar and visual information fusion according to claim 2 is characterized in that: The extracting the motion features and the appearance features of the target from the radar data and the visual data respectively includes: Extracting the motion characteristics of the target from the radar data using a radar signal processing algorithm, wherein the motion characteristics include velocity, acceleration and direction angle; The appearance features of the target are extracted from the visual data using an image processing method, wherein the appearance features include color histogram, edge contour and texture features.

4. The target tracking method of claim 3, characterized in that: The step of adaptively weighting and fusing the motion features and the appearance features based on the environmental parameters to obtain fused features includes: Determining a first weight of the radar data and a second weight of the visual data according to the environmental parameter; The motion feature and the appearance feature are weightedly fused using the first weight and the second weight to obtain a fused feature, wherein the mathematical expression of the fused feature is: F=α·F1+β·F2 Wherein, F represents the fusion feature, F1 and F2 represent the motion feature and the appearance feature respectively, α and β represent the adaptive first weight and the second weight respectively, and α+β=1.

5. The target tracking method of radar and visual information fusion according to claim 4 is characterized in that: The calculation formula of the first weight is: The calculation formula of the second weight is: Among them, f(E) represents the environmental parameter evaluation function, θ represents a positive constant used to adjust the reference point of weight distribution, and E represents the environmental parameter vector.

6. The target tracking method of radar and visual information fusion according to claim 5 is characterized in that: The using of a pre-trained machine learning model to perform target association according to the fusion feature, wherein the machine learning model has been trained by historical data containing different environmental parameters, comprises: Using the fused features and current environmental parameters as inputs to a pre-trained machine learning model; Use pre-trained machine learning models to perform reasoning and infer the associations between targets; Output a target association result, wherein the target association result represents a dynamic relationship between the target in the radar data and the target in the visual data under current environmental parameters.

7. The target tracking method of claim 6, characterized in that: The method of using a state estimation method to estimate and update the trajectory of the target based on the fusion feature and the target association result to track the target includes: The fusion feature and the target association result are used as the measurement value of the state estimation method and input, wherein the state estimation method adopts a Kalman filter; Use a Kalman filter to estimate and update the trajectory of the target, and output a trajectory estimation result, wherein the trajectory estimation result represents the motion trajectory of the target under the current environmental parameters; The Kalman filter dynamically adjusts the target state according to the fusion feature and the target association result to achieve continuous tracking of the target.

8. The target tracking method of claim 7, characterized in that: The updating step of the Kalman filter comprises: Predict the predicted state and predicted covariance matrix of the target at the current moment, wherein the expression of the predicted state is: in, represents the predicted state at time k, A represents the state transfer matrix, B represents the control input matrix, μ k represents the control input, Indicates the state of the previous moment k-1; The expression of the prediction covariance matrix is: P k|k-1 =A·P k-1|k-1 ·A T +Q Among them, P k|k-1 represents the current covariance matrix at prediction time k, P k-1|k-1 represents the covariance matrix of the previous moment k-1, Q represents the process noise covariance matrix, and T represents the transpose operation of the matrix A; The state and covariance matrix are updated based on the measurement values ​​including the fusion feature and the target association result, wherein the state update expression is: in, represents the updated state at time k, K k represents the Kalman gain, z k represents the measurement value at time k, and H represents the measurement matrix; The update expression of the covariance matrix is: P k|k =(I-K k ·H)·P k|k-1 Among them, P k|k represents the updated covariance matrix at time k, and I represents the identity matrix; The expression of the Kalman gain is: K k =P k|k-1 ·H T (H·P k|k-1 ·H T +R) -1 Among them, H T represents the transpose of the measurement matrix H, and R represents the measurement noise covariance matrix.

9. A target tracking system integrating radar and visual information, used to execute a target tracking method integrating radar and visual information as claimed in any one of claims 1 to 8, characterized in that: The tracking system includes: A data acquisition module is used to acquire radar data collected by the radar module, visual data collected by the visual module, and environmental parameters collected by the environmental module, and perform data preprocessing; A feature extraction module, used to extract the motion features and appearance features of the target from the radar data and the visual data respectively; A weighted fusion module, used for adaptively weighting and fusing the motion features and the appearance features based on the environmental parameters to obtain fused features; a target association module, configured to perform target association based on the fused features using a pre-trained machine learning model, wherein the machine learning model has been trained using historical data containing different environmental parameters; The target tracking module is used to estimate and update the trajectory of the target using a state estimation method according to the fusion features and the target association result, so as to achieve tracking of the target.

Images