An infrared and radar target fusion technology based on an interactive multi-model

Abstract

This invention provides an infrared and radar target fusion method based on interactive multi-model, which uses infrared sensors and radar to acquire data from M targets from the t0th second to the tth second. B The location information is given in seconds, b = 1, 2...B, m = 1, 2...M; after acquiring the location information, it is analyzed through multiple steps to finally generate the system trajectory; the system trajectory is the trajectory of the 1st to the Mth targets, with each target from second t0 to second t... B The overall trajectory within seconds. This invention has the advantage of improving the accuracy of acquiring the target trajectory.

Description

Technical Field

[0001] This invention belongs to the field of multi-source heterogeneous data fusion and target tracking, and in particular relates to an infrared and radar target fusion technology based on interactive multi-model. Background Technology

[0002] To address the demand for stable target detection and tracking in the digital and intelligent construction of road traffic and security monitoring, visual, radar perception, and information fusion technologies are typically employed to solve industry challenges such as low false alarm and low missed alarm detection in complex environments and continuous tracking of dense targets, thereby improving the reliability of monitoring systems and providing accurate information for urban governance. Trajectory tracking and multi-sensor-based vehicle trajectory fitting have become research hotspots. However, insufficient trajectory continuity and poor trajectory rationality lead to low accuracy in digital twin simulation and difficulty in improving traffic management efficiency, which are industry pain points. Microwave and millimeter-wave radar have advantages such as high measurement accuracy, immunity to lighting conditions, and insensitivity to weather, but their target recognition capabilities are lacking. Traditional target tracking only uses the motion state measurement information of individual targets to feed into the filtering algorithm, ignoring the obstacle avoidance characteristics in actual road scenarios. Summary of the Invention

[0003] The purpose of this invention is to provide an infrared and radar target fusion technology based on interactive multi-model, which has the advantage of improving the accuracy of target trajectory acquisition.

[0004] To achieve the above objectives, the present invention provides an infrared and radar target fusion method based on an interactive multi-model approach. The method includes: step S1, acquiring data from the t0th second to the tth second of a total of M targets using infrared sensors and radar respectively. B The infrared and radar position information of each second is used to determine the correspondence between the infrared and radar position information, resulting in the values ​​from second t0 to second t for a total of M targets. B Infrared and radar data for seconds, b = 1, 2...B, m = 1, 2...M; Step S2, using the IMM algorithm, sequentially analyzes the infrared data from t0 to t1 of a total of M targets. B The target trajectory S is obtained by processing infrared data and radar data in seconds. f,1 ~S f,M Among them, S f,m This represents the m-th target from second t0 to second t. B The fusion target trajectory in seconds; Step S3, according to the pre-set teaching trajectory S a,1 ~S a,M and the fusion target trajectory S f,1 ~S f,M The local trajectories f1 to f2 are obtained. M Then, the DMP algorithm is used to obtain the values ​​from the t0th second to the tth second for a total of M targets. B Uniform local trajectory S in secondsg,1 ~S g,M Step S4: Analyze and process the uniform local trajectories S of the M targets. g,1 ~S g,M This generates the system trajectory output.

[0005] Preferably, in step S1, the infrared position information is represented as the infrared image coordinate system obtained by the infrared sensor; the radar position information is represented as the radar coordinate system acquired by the radar; spatial alignment is performed on the infrared image coordinate system and the radar coordinate system acquired by the radar for corresponding calibration; through corresponding calibration, the time from t0 to tt of a total of M targets is obtained. B Infrared and radar data in seconds.

[0006] Preferably, in step S1, the PnP algorithm is used to perform spatial alignment operation between the infrared image coordinate system and the radar coordinate system acquired by the radar for corresponding calibration.

[0007] Preferably, step S2 includes: step S21, based on the uniform motion model, according to the time from t0 to t1 of the M targets. B In step S22, the infrared data motion state model and radar data motion state model are constructed based on the infrared data and radar data of the specified seconds. Then, in step S23, the infrared data motion state model is input into an interaction function and subjected to Kalman filtering to obtain the infrared data trajectory estimation result. Finally, in step S24, the infrared data trajectory estimation result and the radar data trajectory estimation result are weighted and summed to obtain the trajectory estimation results for M targets from second t0 to second t1. B The fusion target trajectory S in seconds f,1 ~S f,M The m-th target from the t0th second to the t-th second... B The fusion target trajectory S in seconds f,m The forms of expression are as follows:

[0008]

[0009] in, Indicates the m-th target at time t b-1 To t b The fusion target trajectory during this time period, kb represents t b-1 To t b The number of data points fused to the target trajectory within this time period; t f,m This only represents the m-th target from second t0 to second t. B The fusion target trajectory S in seconds f,m Timestamp; for a single It is expressed as follows:

[0010]

[0011] in, The fused target trajectory of the m-th target at t b-1 To t b The set of x-coordinates within this time period The fused target trajectory t for the m-th target b-1 To t b The set of ordinates within this time period The fused target trajectory t of the m-th target b-1 To t b The set of elevation coordinates within this time period; as well as Each data point is represented as a 3×1 vector, with the three dimensions being the spatial position in terms of x-coordinate / y-coordinate / height coordinate, the target's current velocity, and the target's current acceleration, respectively.

[0012] Preferably, in step S23, when performing Kalman filtering, the likelihood function is used to update the Kalman filter.

[0013] Preferably, step S3 includes: step 31, pre-setting the time from t0 to tt of M targets. B Teaching trajectory S in seconds a,1 ~S a,M S a,m The teaching trajectory represents the m-th target; the teaching trajectory S for a single target a,m for,

[0014]

[0015] in, Represents the m-th target at t b-1 To t b The teaching trajectory during this time period, kb represents t. b-1 To t b The number of data points fused to the target trajectory within this time period; t a,m This only represents the m-th target from second t0 to second t. B Teaching trajectory S in seconds a,m Timestamp; for a single It is expressed as follows:

[0016]

[0017] in, The teaching trajectory for the m-th target at t b-1 To t b The set of x-coordinates within this time period The teaching trajectory for the m-th target at t b-1 To t b The set of ordinates within this time period The teaching trajectory of the m-th target is at t b-1 To t b The set of elevation coordinates within this time period; as well as Each data point is represented as a 3×1 vector, with the three dimensions being the spatial position in terms of x-axis / y-axis / height coordinates, the target's current velocity, and the target's current acceleration, respectively; Step S32, based on the teaching trajectory S a,1 ~S a,M With the fusion target trajectory S f,1 ~S f,M Step S33: Calculate the distance e(m) between the trajectories of each of the m targets; Step S33: Adjust the teaching trajectory according to the distance e(m) between the trajectories to obtain the optimal teaching trajectory (best) for each target. a,1 ~(best)S a,M ;(best)S a,m Represents the optimal teaching trajectory for the m-th target; Step S34, proceed step by step according to the optimal teaching trajectory (best) for each target. a,m Obtain the corresponding local trajectories f1 to f2. M ; where f m Represent the local trajectory of the m-th target; Step S35, according to f 1～ f M The uniform local trajectories S of M targets are obtained using the DMP algorithm. g,1 ~S g,M ;

[0018] S g,m =[x g,m,1 ,x g,m,2 ,......x g,m,t ......x g,m,T ]

[0019] x g,m,t =[x g,m,t，1 x g,m,t，2 x g,m,t，3 ]

[0020] t = 1, 2, ..., T

[0021] T = t B

[0022] Where, x g,m,t Let x be a set, representing the state of the m-th target at second t. g,m,t，1 Let x be the x-coordinate of the uniform local trajectory of the m-th target at second t. g,m,t，2Let x be the ordinate of the uniform local trajectory of the m-th target at second t. g,m,t，3 The height coordinates of the uniform local trajectory of the m-th target at second t.

[0023] Preferably, in step S32, the distance e(m) between the trajectories of the m-th target is...

[0024]

[0025] K = k1 + k2 + ... + kB

[0026] k = 1, 2...K

[0027] Where K represents the total number of data points for the m-th target, as mentioned above, kb represents t. b-1 To t b The number of data points fused to the target trajectory within this time period is easily understood; the total number of data points is obtained by adding the number of data points across all time periods. f,1，(m，k) The x-coordinate represents the x-coordinate of the k-th data point of the m-th target under the fused target trajectory; a,1,(m，k) x represents the x-coordinate of the k-th data point of the m-th target under the teaching trajectory; f,2,(m，k) The x-coordinate represents the ordinate of the k-th data point of the m-th target under the fused target trajectory; a,2,(m，k) x represents the ordinate of the k-th data point of the m-th target under the teaching trajectory; f,3,(m，k) x represents the height coordinate of the k-th data point of the m-th target under the fused target trajectory; a,3,(m，k) C1, C2, and C3 represent the height coordinates of the k-th data point of the m-th target under the teaching trajectory; C1, C2, and C3 are constants that can be selected according to the actual situation.

[0028] Preferably, step S34 includes: sequentially processing the optimal teaching trajectory (best) S a,m Input a spring-damped system model and use a local weighted regression algorithm to obtain the corresponding f. m The details are as follows:

[0029]

[0030] ξ(k)=s(x f,m,k -x a,m,k )

[0031] Where, x a The speed at which the teaching trajectory is fitted to the target. The fitting speed in the target model, f(ω) represents the fitting acceleration in the target model; g represents the endpoint where the target converges; τ is the time scaling factor; m ,s) is a nonlinear function, representing the trajectory shape learner; αx For regularization parameters, α y Let β be the model parameter of the first spring-damped system. y Here are the model parameters for the second spring-damped system; f(ω) m (s) is the trajectory shape learning function, where s is a preset variable, and its relationship with the running time is the function s = exp(-α). x t / τ), where exp is the natural exponential function and t is the running time. Let g be the derivative of the preset variable, g be the initial value of the target trajectory, and ω be the derivative of the preset variable. m For the weights of each basis function, Ψ m (z) is a radial basis function. Let c be the variance of the basis functions. m Γ is the center of the basis functions. m Let f be the radial basis function incidence matrix. m Let be the local trajectory represented by the target data points, 'a' be the motion difference matrix between the taught trajectory and the fused target trajectory, and 'x' be the local trajectory. f,m,k x is the motion fit value of the fused target trajectory of the target. a,m,k The motion fitting value of the teaching trajectory for the target.

[0032] Preferably, step S4 includes: step S41, let m = 1, t = 1; step S42, input the uniform local trajectory of the m-th target from the (t-1)th second to the tth second into the blank judgment model, and detect whether there is a collision with the existing uniform local trajectory.

[0033]

[0034] Where j = 1, 2, ..., M, j ≠ m, ε(m→j, 1) represents the distance between the m-th target and other targets at second t; if ε(m→j, 1) is greater than a preset threshold, a collision is determined to exist, and the process proceeds to step S43; otherwise, no collision is determined to exist, and the process proceeds to step S44; in step S43, a direction within 0 to 180° is randomly selected from the collision point, and the distance is extended with the endpoint position at second t as the destination. The trajectory of each extension is called the extension trajectory; if the extension trajectory intersects with other trajectories during the extension, that direction is discarded, until the distance between the randomly extended nodes is less than the threshold. The value is recorded as a uniform local trajectory from the (t-1)th second to the tth second; and t = T is checked. If it is, proceed to step S45; otherwise, let t = t + 1 and return to step S42; Step S44, the value is recorded as a uniform local trajectory from the (t-1)th second to the tth second; and t = T is checked. If it is, proceed to step S45; otherwise, let t = t + 1 and return to step S42; Step S45, the uniform local trajectory of the mth target from the 0th second to the Tth second is output; and m = M is checked. If it is, all system trajectories are sorted and output. If not, let m = m + 1, t = 1, and return to step S42.

[0035] Preferably, step S43 uses the RRT algorithm and the segmented DMP algorithm to obtain the target trajectory of the expanded trajectory.

[0036] In summary, compared with the prior art, the infrared and radar target fusion technology based on interactive multi-model provided by the present invention has the following beneficial effects:

[0037] First, by adopting an obstacle avoidance algorithm, the trajectory fitting problem can be transformed into obstacle avoidance trajectory learning when target tracking is discontinuous. This solves the problem that traditional target tracking only uses the motion state measurement information of the target individual to feed into the filtering algorithm, ignoring the obstacle avoidance characteristics in the actual road scene, and improves the stability of the system trajectory output.

[0038] Second, by comprehensively utilizing the position and speed of other traffic participants in the surrounding area, as well as target measurement information, and by incorporating the target's motion characteristics in the scene and using historical trajectory data, the target's motion state can be predicted, which is more in line with target kinematics and improves trajectory accuracy. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0040] The following will be combined with the appendix in the embodiments of the present invention. Figure 1 The technical solutions, structural features, objectives and effects achieved in the embodiments of the present invention will be described in detail.

[0041] It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions. They are only used to facilitate and clarify the purpose of illustrating the embodiments of the present invention, and are not intended to limit the implementation conditions of the present invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportional relationship, or adjustments to the size should still fall within the scope of the technical content disclosed in the present invention, provided that they do not affect the effects and objectives that the present invention can produce.

[0042] It should be noted that, in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only the expressly listed elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0043] This invention provides an infrared and radar target fusion method based on interactive multi-model, which uses infrared sensors and radar to acquire data from M targets from the t0th second to the tth second. B The system acquires position information in seconds (specifically including infrared and radar position information, detailed below), b = 1, 2...B, m = 1, 2...M; after acquiring the position information, it analyzes the position information through multiple steps, and finally generates the system trajectory; the system trajectory is the trajectory of the 1st to the Mth targets, for each target from t0 seconds to t... B The overall trajectory in seconds.

[0044] Here, t0 represents the initial time at which target acquisition begins. t0 does not necessarily start from 0 seconds when the target is still, meaning that moving targets can be acquired in real time. Furthermore, for any two adjacent t0 values... b-1 With t b The time intervals between them do not necessarily follow a fixed value; that is, t0, t1, t2...t B The key is to express a point in time, rather than expressing it in a fixed way with an interval of 1 second or 2 seconds.

[0045] like Figure 1 As shown, the method includes:

[0046] Step S1: Acquire data from the t0th second to the tth second of a total of M targets using infrared sensors and radar respectively. B The infrared and radar position information of each second is used to determine the correspondence between the infrared and radar position information, resulting in the values ​​from second t0 to second t for a total of M targets.B Infrared and radar data in seconds, b = 1, 2...B, m = 1, 2...M;

[0047] Specifically, the infrared position information is represented as an infrared image coordinate system obtained by the infrared sensor; the radar position information is represented as a radar coordinate system obtained by the radar.

[0048] The corresponding calibration refers to the spatial alignment operation between the infrared image coordinate system obtained by the infrared sensor and the radar coordinate system obtained by the radar. Since the infrared image coordinate system and the radar coordinate system are obtained from two different devices, their time axes, captured spatial ranges, etc. are inconsistent. If the corresponding calibration method is not used, there will be no unified standard between the two, and subsequent analysis will be meaningless.

[0049] Specifically, the PnP (Perspective-n-Point, an algorithm for estimating camera pose from feature points in an image) algorithm can be used for correspondence calibration; through correspondence calibration, the values ​​from t0 to tt of a total of M targets can be obtained. B Infrared and radar data in seconds.

[0050] In step S2, the IMM (Interacting Multiple Model) algorithm is used to process each of the M targets from the t0th second to the tth second. B The target trajectory S is obtained by processing infrared data and radar data in seconds. f,1 ~S f,M Among them, S f,m This represents the m-th target from second t0 to second t. B The fusion target trajectory in seconds;

[0051] The IMM algorithm includes tools such as a uniform-speed computation model and interaction functions for analyzing infrared and radar data.

[0052] Step S3, according to the pre-set teaching trajectory S a,1 ~S a,M And the fused target trajectory S calculated in S2 f,1 ~S f,M The local trajectories f1 to f2 are obtained. M Then, the DMP (Dynamic Movement Primitives) algorithm is used to obtain the values ​​of M targets from the t0th second to the tth second. B Uniform local trajectory S in seconds g,1 ~S g,M ; where f m This represents the m-th target from second t0 to second t. BLocal trajectory in seconds, S a,m This represents the m-th target from second t0 to second t. B The teaching trajectory in seconds, S g,m This represents the m-th target from second t0 to second t. B A uniform local trajectory in seconds.

[0053] The teaching trajectory represents the historical data of the target or the general behavior of similar targets, and has the same representation as the fused target trajectory; for example, it can be obtained by extracting the historical motion features of a specified target and then reproducing those features in a similar scene. However, compared to the fused target trajectory, the teaching trajectory reproduces the target's motion features using only one or a few motion features. It is crucial to note that the teaching trajectory must be pre-set strictly according to the representation of the fused target trajectory, i.e., it needs to be accurate to the corresponding t. b-1 To t b During this period, the two components are completely identical; otherwise, the DMP algorithm cannot be used for subsequent processing.

[0054] Step S4: Analyze and process the uniform local trajectories S of the M targets. g,1 ~S g,M Generate system trajectory output;

[0055] The concept of system trajectory has been described above and will not be repeated here. The purpose of step S4 is to check whether there is a collision in the system trajectory. If there is a collision, it means that there is a problem in the trajectory fusion process and correction is needed.

[0056] Specifically, step S2 includes:

[0057] Step S21, based on the uniform motion model, according to the time from t0 to t1 of the M targets. B Using second-level infrared and radar data, construct corresponding infrared data motion state models and radar data motion state models;

[0058] Step S22: Input the infrared data motion state model into the interaction function and perform Kalman filtering to obtain the infrared data trajectory estimation result;

[0059] Step S23: Input the radar data motion state model into the interaction function and perform Kalman filtering to obtain the radar data trajectory estimation result;

[0060] Step S24: Perform a weighted summation of the infrared data trajectory estimation results and the radar data trajectory estimation results to obtain the values ​​from the t0th second to the tth second for a total of M targets. B The fusion target trajectory S in seconds f,1 ~S f,M ;

[0061] In the Kalman filtering process, a likelihood function is used to update the Kalman filter; the likelihood function plays a role in predicting the calculation results, thereby enhancing the accuracy of the Kalman filtering process.

[0062] The m-th target from second t0 to second t B The fusion target trajectory S in seconds f,m The forms of expression are as follows:

[0063]

[0064] in, Indicates the m-th target at time t b-1 To t b The fusion target trajectory during this time period, kb represents t b-1 To t b The number of data points for the fused target trajectory within this time period is easy to understand, as there must be multiple and continuous data points within a given time period. Using kb as the total number of data points is merely for informational purposes and has no computational significance. Furthermore, regarding t... f,m This only represents the m-th target from second t0 to second t. B The fusion target trajectory S in seconds f,m The timestamp is only used for information representation and has no computational meaning.

[0065] For a single It is expressed as follows:

[0066]

[0067] in, The fused target trajectory of the m-th target at t b-1 To t b The set of x-coordinates within this time period The fused target trajectory t for the m-th target b-1 To t b The set of ordinates within this time period The fused target trajectory t of the m-th target b-1 To t b The set of elevation coordinates within this time period; additionally, here... as well as Each data point is represented as a 3×1 vector, with the three dimensions being the spatial position in terms of x-axis, y-axis, and altitude, the target's current velocity, and the target's current acceleration, respectively.

[0068] For example, the fusion trajectory of the first target during the time period from t0 to t1: Represented by a 3×1 vector, the target's spatial position under the horizontal coordinate, the target's current velocity, and the target's current acceleration are respectively; the rest are similar.

[0069] Specifically, step S3 includes:

[0070] Step 31, pre-set the time from t0 to tt for M targets. B Teaching trajectory S in seconds a,1 ~S a,M S a,m This represents the teaching trajectory of the m-th target;

[0071] The method for obtaining the teaching trajectory has already been explained above and will not be repeated here.

[0072] The teaching trajectory must be pre-set strictly according to the representation of the fused target trajectory, so the teaching trajectory S of a single target a,m for,

[0073]

[0074] Similarly, Represents the m-th target at t b-1 To t b The teaching trajectory during this time period, kb represents t. b-1 To t b The number of data points fused to the target trajectory within this time period is only used for information representation and has no computational significance; t a,m This only represents the m-th target from second t0 to second t. B Teaching trajectory S in seconds a,m The timestamp is only used for informational purposes and has no computational meaning. It is evident that the teaching trajectory and the fused target trajectory have the same representation and structure.

[0075] Furthermore, for a single It is expressed as follows:

[0076]

[0077] in, The teaching trajectory for the m-th target at t b-1 To t b The set of x-coordinates within this time period The teaching trajectory for the m-th target at t b-1 To t b The set of ordinates within this time period The teaching trajectory of the m-th target is at t b-1 To t b The set of elevation coordinates within this time period; additionally, here... as well as Each data point is represented as a 3×1 vector, with the three dimensions being the spatial position (horizontal / vertical / height coordinates), the target's current velocity, and the target's current acceleration, respectively. Clearly, the teaching trajectory and the fused target trajectory share the same representation and structure.

[0078] Step S32, based on the teaching trajectory S a,1 ~S a,M With the fusion target trajectory S f,1 ~S f,M Calculate the distance e(m) between the trajectories of each of the m targets;

[0079] Specifically, the distance e(m) between the trajectories of the m-th target.

[0080]

[0081] K = k1 + k2 + ... + kB

[0082] k = 1, 2...K

[0083] Where K represents the total number of data points for the m-th target, as mentioned above, kb represents t. b-1 To t b The number of data points fused to the target trajectory within this time period is easily understood; the total number of data points is obtained by adding the number of data points across all time periods. f,1，(m，k) The x-coordinate represents the x-coordinate of the k-th data point of the m-th target under the fused target trajectory; a,1,(m，k) x represents the x-coordinate of the k-th data point of the m-th target under the teaching trajectory; f,2,(m，k) The x-coordinate represents the ordinate of the k-th data point of the m-th target under the fused target trajectory; a,2,(m，k) x represents the ordinate of the k-th data point of the m-th target under the teaching trajectory; f,3,(m，k) x represents the height coordinate of the k-th data point of the m-th target under the fused target trajectory; a,3,(m，k) Let C1, C2, and C3 represent the height coordinates of the k-th data point of the m-th target under the teaching trajectory; C1, C2, and C3 are constants that can be selected according to the actual situation. For example, in a road traffic scenario, C1 = 2~4, C2 = 1~2, and C3 = 0 are used, i.e., C3 is omitted. Here, e(m) can be used to determine the correlation between the teaching trajectory and the fusion target trajectory. For example, in the aforementioned road traffic scenario, when e(m) ≤ η and η is 0.5 to 1.5, the trajectory is predicted to be correlated.

[0084] Step S33: Adjust the teaching trajectory according to the distance e (m) between trajectories to obtain the optimal teaching trajectory (best) S for each target. a,1 ~(best)S a,M;(best)S a,m Let represent the optimal teaching trajectory for the m-th target;

[0085] As mentioned earlier, there may be cases where e(m)≤η does not satisfy the condition. In such cases, it is necessary to regenerate the teaching trajectory, or even generate multiple teaching trajectories, and finally select the optimal teaching trajectory. The purpose of this step is to ensure that all the obtained teaching trajectories are teaching trajectories that can be used for subsequent analysis.

[0086] Step S34, proceed step by step according to the optimal teaching trajectory (best) S for each target. a,m Obtain the corresponding local trajectories f1 to f2. M ; where f m This represents the local trajectory of the m-th target;

[0087] Specifically, the optimal teaching trajectory (best) S is assigned one by one. a,m Input a spring-damped system model and use a local weighted regression algorithm to obtain the corresponding f. m The details are as follows:

[0088]

[0089]

[0090] ξ(k)=s(x f,m,k -x a,m,k )

[0091] In the formula, x a The speed at which the teaching trajectory is fitted to the target. The fitting speed in the target model, f(ω) represents the fitting acceleration in the target model; g represents the endpoint where the target converges; τ is the time scaling factor; m ,s) is a nonlinear function, representing the trajectory shape learner; α x For regularization parameters, α y Let β be the model parameter of the first spring-damped system. y Here are the model parameters for the second spring-damped system; f(ω) m (s) is the trajectory shape learning function, where s is a preset variable, and its relationship with the running time is the function s = exp(-α). x t / τ), where exp is the natural exponential function and t is the running time. The derivative of the preset variable is given by ω, where g is the initial value of the target trajectory (here, the x-coordinate), and ω is the x-coordinate. m For the weights of each basis function, Ψ m (z) is a radial basis function. Let c be the variance of the basis functions. m Γ is the center of the basis functions.m Let f be the radial basis function incidence matrix. m Let be the local trajectory represented by the target data points, 'a' be the motion difference matrix between the taught trajectory and the fused target trajectory, and 'x' be the local trajectory. f,m,k x is the motion fit value of the fused target trajectory of the target. a,m,k The motion fitting value of the teaching trajectory for the target.

[0092] In step S35, according to f 1～ f M The uniform local trajectories S of M targets are obtained using the DMP algorithm. g,1 ~S g,M ;

[0093] S g,m =[x g,m,1 ,x g,m,2 ,......x g,m,t ......x g,m,T ]

[0094] x g,m,t =[x g,m,t，1 x g,m,t，2 x g,m,t，3 ]

[0095] t = 1, 2, ..., T

[0096] T = t B

[0097] Where, x g,m,t Let x be a set, representing the state of the m-th target at second t. g,m,t，1 Let x be the x-coordinate of the uniform local trajectory of the m-th target at second t. g,m,t，2 Let x be the ordinate of the uniform local trajectory of the m-th target at second t. g,m,t，3 The height coordinates of the uniform local trajectory of the m-th target at second t; Furthermore, since the trajectory calculations are performed on the targets over a certain time period, T = t B The only difference here is in the form of expression; there is no difference in substance.

[0098] Specifically, step S4 includes:

[0099] Step S41, let m = 1, t = 1;

[0100] Step S42: Input the uniform local trajectory of the m-th target from the (t-1)th second to the tth second into the blank judgment model and detect whether there is a collision with the existing uniform local trajectory.

[0101]

[0102] Where j = 1, 2, ..., M, j ≠ m, ε(m→j, 1) represents the distance between the m-th target and other targets in the t-th second; if ε(m→j, 1) is greater than a preset threshold, it is determined that there is a collision and proceeds to step S43; otherwise, it is determined that there is no collision and proceeds to step S44.

[0103] When m=1, there are no other uniform local trajectories, so the uniform local trajectory of the first input to the blank judgment model can be directly input at once and proceed to step S45.

[0104] Step S43: Randomly select a direction within 0 to 180° from the collision point, and extend the distance with the endpoint position at second t as the destination. The trajectory of each extension is called the extension trajectory. If the extension trajectory intersects with other trajectories during the extension, discard that direction until the random extension node satisfies the position distance is less than the threshold. Record it as a uniform local trajectory from second t-1 to second t. Check if t = T. If yes, proceed to step S45; otherwise, let t = t + 1 and return to step S42.

[0105] Step S44: Record it as a uniform local trajectory from second t-1 to second t; and check if t = T. If yes, proceed to step S45; otherwise, let t = t+1 and return to step S42.

[0106] Step S45: Output the uniform local trajectory of the m-th target from second 0 to second T, and determine if m = M. If so, organize and output all system trajectories; otherwise, let m = m + 1, t = 1, and return to step S42.

[0107] Furthermore, for step S43, the RRT (Rapidly-Exploring Random Trees) algorithm and the segmented DMP algorithm can be used to obtain the corrected (expanded) target trajectory, which solves the problem of poor trajectory prediction accuracy caused by reduced data rate due to occlusion in multi-target scenes.

[0108] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A method for fusion of infrared and radar targets based on interactive multi-model, characterized in that, The method includes: Step S1: Acquire data from the t0th second to the tth second of a total of M targets using infrared sensors and radar respectively. B The infrared and radar position information of each second is used to determine the correspondence between the infrared and radar position information, resulting in the values ​​from second t0 to second t for a total of M targets. B Infrared and radar data in seconds, b=1, 2...B, m=1, 2...M; Step S2: Use the IMM algorithm to process each of the M targets from the t0th second to the tth second. B The target trajectory S is obtained by processing infrared data and radar data in seconds. f,1 ~S f,M Among them, S f,m This represents the m-th target from second t0 to second t. B The fusion target trajectory in seconds; Step S3, according to the pre-set teaching trajectory S a,1 ~S a,M and the fusion target trajectory S f,1 ~S f,M The local trajectories f1 to f2 are obtained. M Then, the DMP algorithm is used to obtain the values ​​from the t0th second to the tth second for a total of M targets. B Uniform local trajectory S in seconds g,1 ~S g,M ; Step S4: Analyze and process the uniform local trajectories S of the M targets. g,1 ~S g,M This generates the system trajectory output.

2. The infrared and radar target fusion method based on interactive multi-model according to claim 1, characterized in that, In step S1, the infrared position information is represented as the infrared image coordinate system obtained by the infrared sensor; the radar position information is represented as the radar coordinate system obtained by the radar; and spatial alignment is performed between the infrared image coordinate system and the radar coordinate system obtained by the radar for corresponding calibration. Through corresponding calibration, the values ​​from t0 to tt of a total of M targets are obtained. B Infrared and radar data in seconds.

3. The infrared and radar target fusion method based on interactive multi-model according to claim 2, characterized in that, In step S1, the PnP algorithm is used to perform spatial alignment operation between the infrared image coordinate system and the radar coordinate system acquired by the radar for corresponding calibration.

4. The infrared and radar target fusion method based on interactive multi-model according to claim 3, characterized in that, Step S2 includes: Step S21, based on the uniform motion model, according to the time from t0 to t1 of the M targets. B Using second-level infrared and radar data, construct corresponding infrared data motion state models and radar data motion state models; Step S22: Input the infrared data motion state model into the interaction function and perform Kalman filtering to obtain the infrared data trajectory estimation result; Step S23: Input the radar data motion state model into the interaction function and perform Kalman filtering to obtain the radar data trajectory estimation result; Step S24: Perform a weighted summation of the infrared data trajectory estimation results and the radar data trajectory estimation results to obtain the values ​​from the t0th second to the tth second for a total of M targets. B The fusion target trajectory S in seconds f,1 ~S f,M ; The m-th target from second t0 to second t B The fusion target trajectory S in seconds f,m The forms of expression are as follows: in, Indicates the m-th target at time t b-1 To t b The fusion target trajectory during this time period, kb represents t b-1 To t b The number of data points fused to the target trajectory within this time period; t f,m This only represents the m-th target from second t0 to second t. B The fusion target trajectory S in seconds f,m timestamp; For a single It is expressed as follows: in, The fused target trajectory of the m-th target at t b-1 To t b The set of x-coordinates within this time period The fused target trajectory t for the m-th target b-1 To t b The set of ordinates within this time period The fused target trajectory t of the m-th target b-1 To t b The set of elevation coordinates within this time period; as well as Each data point is represented as a 3×1 vector, with the three dimensions being the spatial position in terms of x-coordinate / y-coordinate / height coordinate, the target's current velocity, and the target's current acceleration, respectively.

5. The infrared and radar target fusion method based on interactive multi-model according to claim 4, characterized in that, In step S23, when performing Kalman filtering, the likelihood function is used to update the Kalman filter.

6. The infrared and radar target fusion method based on interactive multi-model according to claim 5, characterized in that, Step S3 includes: Step 31, pre-set the time from t0 to tt for M targets. B Teaching trajectory S in seconds a,1 ~S a,M S a,m This represents the teaching trajectory of the m-th target; Teaching trajectory S of a single target a,m for, in, Represents the m-th target at t b-1 To t b The teaching trajectory during this time period, kb represents t. b-1 To t b The number of data points fused to the target trajectory within this time period; t a,m This only represents the m-th target from second t0 to second t. B Teaching trajectory S in seconds a,m timestamp; For a single It is expressed as follows: in, The teaching trajectory for the m-th target at t b-1 To t b The set of x-coordinates within this time period The teaching trajectory for the m-th target at t b-1 To t b The set of ordinates within this time period The teaching trajectory of the m-th target is at t b-1 To t b The set of elevation coordinates within this time period; as well as Each data point is represented as a 3×1 vector, with the three dimensions being the spatial position in terms of x-axis / y-axis / height coordinates, the target's current velocity, and the target's current acceleration, respectively. Step S32, based on the teaching trajectory S a,1 ~S a,M With the fusion target trajectory S f,1 ~S f,M Calculate the distance e(m) between the trajectories of each of the m targets; Step S33: Adjust the teaching trajectory according to the distance e (m) between trajectories to obtain the optimal teaching trajectory (best) S for each target. a,1 ~(best)S a,M ;(best)S a,m Let represent the optimal teaching trajectory for the m-th target; Step S34, proceed step by step according to the optimal teaching trajectory (best) S for each target. a,m Obtain the corresponding local trajectories f1 to f2. M ; where f m This represents the local trajectory of the m-th target; Step S35, according to f 1～ f M The uniform local trajectories S of M targets are obtained using the DMP algorithm. g,1 ~S g,M ; S g,m =[x g,m,1 ,x g,m,2 ,......x g,m,t ......x g,m,T ] x g,m,t =[x g,m,t，1 ,x g,m,t，2 ,x g,m,t，3 ] t = 1, 2, ..., T T=t B Where, x g,m,t Let x be a set, representing the state of the m-th target at second t. g,m,t，1 Let x be the x-coordinate of the uniform local trajectory of the m-th target at second t. g,m,t，2 Let x be the ordinate of the uniform local trajectory of the m-th target at second t. g,m,t，3 The height coordinates of the uniform local trajectory of the m-th target at second t.

7. The infrared and radar target fusion method based on interactive multi-model according to claim 6, characterized in that, In step S32, the distance e(m) between the trajectories of the m-th target is... K = k1 + k2 + ... + kB k = 1, 2...K Where K represents the total number of data points for the m-th target; x f,1，(m，k) The x-coordinate represents the x-coordinate of the k-th data point of the m-th target under the fused target trajectory; a,1,(m，k) x represents the x-coordinate of the k-th data point of the m-th target under the teaching trajectory; f,2,(m，k) The x-coordinate represents the ordinate of the k-th data point of the m-th target under the fused target trajectory; a,2,(m，k) x represents the ordinate of the k-th data point of the m-th target under the teaching trajectory; f,3, (m, k) ) x represents the height coordinate of the k-th data point of the m-th target under the fused target trajectory; a,3, (m, k) ) C1, C2, and C3 represent the height coordinates of the k-th data point of the m-th target under the teaching trajectory; C1, C2, and C3 are constants that can be selected according to the actual situation.

8. The infrared and radar target fusion method based on interactive multi-model according to claim 7, characterized in that, Step S34 includes: sequentially processing the optimal teaching trajectory (best) S a,m Input a spring-damped system model and use a local weighted regression algorithm to obtain the corresponding f. m The details are as follows: ξ(k)=s(x f,m,k -x a,m,k ) Where, x a The speed at which the teaching trajectory is fitted to the target. The fitting speed in the target model, f(ω) represents the fitting acceleration in the target model; g represents the endpoint where the target converges; τ is the time scaling factor; m ,s) is a nonlinear function, representing the trajectory shape learner; α x For regularization parameters, α y Let β be the model parameter of the first spring-damped system. y These are the model parameters for the second spring-damped system; f(ω m (s) is the trajectory shape learning function, where s is a preset variable, and its relationship with the running time is the function s = exp(-α). x t / τ), where exp is the natural exponential function and t is the running time. Let g be the derivative of the preset variable, g be the initial value of the target trajectory, and ω be the derivative of the preset variable. m For the weights of each basis function, Ψ m (z) is a radial basis function. Let c be the variance of the basis functions. m Γ is the center of the basis functions. m Let f be the radial basis function incidence matrix. m Let be the local trajectory represented by the target data points, 'a' be the motion difference matrix between the taught trajectory and the fused target trajectory, and 'x' be the local trajectory. f,m,k x is the motion fit value of the fused target trajectory of the target. a,m,k The motion fitting value of the teaching trajectory for the target.

9. The infrared and radar target fusion method based on interactive multi-model according to claim 8, characterized in that, Step S4 includes: Step S41, let m = 1, t = 1; Step S42: Input the uniform local trajectory of the m-th target from the (t-1)-th second to the t-th second into the blank judgment model and detect whether there is a collision with the existing uniform local trajectory. Where j = 1, 2, ..., M, j ≠ m, ε(m→j, 1) represents the distance between the m-th target and other targets in the t-th second; if ε(m→j, 1) is greater than a preset threshold, it is determined that there is a collision and proceeds to step S43; otherwise, it is determined that there is no collision and proceeds to step S44. Step S43: Randomly select a direction within 0 to 180° from the collision point, and extend the distance with the endpoint position at second t as the destination. The trajectory of each extension is called the extension trajectory. If the extension trajectory intersects with other trajectories during the extension, discard that direction until the random extension node satisfies the position distance is less than the threshold. Record it as a uniform local trajectory from second t-1 to second t. Check if t = T. If yes, proceed to step S45; otherwise, let t = t + 1 and return to step S42. Step S44: Record it as a uniform local trajectory from second t-1 to second t; and check if t = T. If yes, proceed to step S45; otherwise, let t = t+1 and return to step S42. Step S45: Output the uniform local trajectory of the m-th target from second 0 to second T; and determine if m = M. If so, organize and output all system trajectories; otherwise, let m = m + 1, t = 1, and return to step S42.

10. The infrared and radar target fusion method based on interactive multi-model according to claim 9, characterized in that, Step S43 uses the RRT algorithm and the segmented DMP algorithm to obtain the target trajectory of the expanded trajectory.

Images