Infrared and radar target fusion technology based on interactive multiple models

Abstract

The invention provides an infrared and radar target fusion method based on interactive multiple models, and the method comprises the steps: obtaining the position information of M targets from the t0th second to the tBth second based on an infrared sensor and a radar, b = 1, 2... B, and m = 1, 2... M; after position information is obtained, the position information is analyzed through multiple steps, and finally a system track is generated. The system track is the overall moving track of each target from the first target to the Mth target and from the t0th second to the tBth second. The method has the advantage of improving the accuracy of obtaining the target trajectory.

Description

Technical Field

[0001] The invention belongs to the field of multi-source heterogeneous data fusion and target tracking, and in particular relates to an infrared and radar target fusion technology based on interactive multi-models. Background Art

[0002] In response to the demand for stable target detection and tracking in the digital and intelligent construction of road traffic and security monitoring, vision, radar perception and information fusion technology are usually used to solve industry problems such as low false alarm and low missed alarm detection of targets in complex environments and continuous tracking of dense targets, improve the perception reliability of the monitoring system, and provide accurate information for urban governance. Trajectory tracking and multi-sensor based vehicle trajectory fitting have become research hotspots. However, there are industry pain points such as insufficient trajectory continuity and poor trajectory rationality, which lead to low accuracy of digital twin simulation and deduction, and difficulty in improving traffic management efficiency. Microwave millimeter wave radar has the advantages of high measurement accuracy, no influence of light, and insensitivity to weather, but lacks target recognition function. Traditional target tracking only uses the motion state measurement information of the target monomer to inject it into the filtering algorithm, ignoring the obstacle avoidance characteristics in the actual road scene. Summary of the invention

[0003] The purpose of the present invention is to provide an infrared and radar target fusion technology based on interactive multi-model, which has the advantage of improving the accuracy of obtaining target trajectory.

[0004] To achieve the above object, the present invention provides an infrared and radar target fusion method based on interactive multi-model, the method comprising: step S1, respectively obtaining the infrared sensor and the radar from the t0th second to the tth second of a total of M targets B The infrared position information and radar position information of the M targets from t0 to t0 seconds are calibrated accordingly. B Infrared data and radar data of seconds, b = 1, 2 ... B, m = 1, 2 ... M; Step S2, using the IMM algorithm to analyze the total M targets one by one from the t0th second to the t B The infrared data of 10 seconds and the radar data are processed to obtain the fused target trajectory S f,1 ~S f,M ; Among them, S f,m Indicates the mth target from t0th second to tth second B Step S3, according to the preset teaching trajectory S a,1 ~S a,M And the fusion target trajectory S f,1 ~S f,M , and obtain the local trajectory f1~f M , and then use the DMP algorithm to obtain the M targets from the t0th second to the tth second one by one. B The uniform local trajectory S of secondsg,1 ~S g,M ; Step S4, analyzing and processing the uniform local trajectory S of M targets g,1 ~S g,M , generating system trajectory output.

[0005] Preferably, in step S1, the infrared position information is expressed in the form of an infrared image coordinate system obtained by the infrared sensor; the radar position information is expressed in the form of a radar coordinate system obtained by the radar; a spatial alignment operation is performed on the infrared image coordinate system and the radar coordinate system obtained by the radar to perform corresponding calibration; through corresponding calibration, the coordinates of the M targets from the t0th second to the tth second are obtained. B Seconds of infrared and radar data.

[0006] Preferably, in step S1, a PnP algorithm is used to perform a spatial alignment operation on the infrared image coordinate system and the radar coordinate system acquired by the radar for corresponding calibration.

[0007] Preferably, the step S2 comprises: step S21, based on the uniform motion model, according to the time from t0 second to t1 second of a total of M targets B In step S22, the infrared data motion state model is input into the interaction function and subjected to Kalman filtering to obtain the infrared data trajectory estimation result; in step S23, the radar data motion state model is input into the interaction function and subjected to Kalman filtering to obtain the radar data trajectory estimation result; in step S24, the infrared data trajectory estimation result and the radar data trajectory estimation result are weightedly summed to obtain the trajectory estimation result from the t0th second to the tth second for a total of M targets. B The fused target trajectory S of seconds f,1 ~S f,M ; The mth target from t0th second to tth second B The fused target trajectory S of seconds f,m The expression is:

[0008]

[0009] in, Indicates that the mth target is at t b-1 to b The fused target trajectory within this period of time, kb represents t b-1 to b The number of data points of the fused target trajectory within this period; t f,m Only represents the mth target from t0th second to tth second B The fused target trajectory S of seconds f,m timestamp; for a single It is expressed as follows:

[0010]

[0011] in, is the fused target trajectory of the mth target at t b-1 to b The horizontal axis set within this time period is is the fused target trajectory t of the mth target b-1 to b The vertical coordinate set within this time period is The fused target trajectory t of the mth target b-1 to b The set of height coordinates within this time period; as well as Each data in is represented by a vector with a dimension of 3×1, and the three dimensions are the spatial position in the horizontal coordinate / vertical coordinate / altitude coordinate orientation, the current speed of the target, and the current acceleration of the target.

[0012] Preferably, in step S23, when performing Kalman filter processing, a likelihood function is used to update the Kalman filter.

[0013] Preferably, the step S3 comprises: step 31, presetting the time from the t0th second to the tth second of M targets B Teaching trajectory S in seconds a,1 ~S a,M ; S a,m represents the teaching trajectory of the mth target; the teaching trajectory of a single target S a,m for,

[0014]

[0015] in, represents the mth target at t b-1 to b The teaching trajectory during this period of time, kb represents t b-1 to b The number of data points of the fused target trajectory within this period; t a,m Only represents the mth target from t0th second to tth second B Teaching trajectory S in seconds a,m timestamp; for a single It is expressed as follows:

[0016]

[0017] in, is the teaching trajectory of the mth target at t b-1 to b The horizontal axis set within this time period is is the teaching trajectory of the mth target at t b-1 to b The vertical coordinate set within this time period is The teaching trajectory of the mth target is at t b-1 to b The set of height coordinates within this time period; as well as Each data in is represented by a vector with a dimension of 3×1, and the three dimensions are the spatial position under the horizontal coordinate / vertical coordinate / altitude coordinate orientation, the current speed of the target, and the current acceleration of the target; Step S32, according to the teaching trajectory S a,1 ~S a,M and the fused target trajectory S f,1 ~S f,M , calculate the distance e(m) between the trajectories of the m targets one by one; step S33, adjust the teaching trajectory according to the distance e(m) between the trajectories, and obtain the optimal teaching trajectory (best) S for each target a,1 ~(best)S a,M ; (best) S a,m represents the optimal teaching trajectory of the mth target; step S34, one by one according to the optimal teaching trajectory (best) S of each target a,m , get the corresponding local trajectory f1~f M ; Among them, f m represents the local trajectory of the mth target; step S35, according to f 1～ f M , the uniform local trajectory S of a total of M targets is obtained through the DMP algorithm g,1 ~S g,M ;

[0018] S g,m =[x g,m,1 ,x g,m,2 ,......x g,m,t ......x g,m,T ]

[0019] x g,m,t =[x g,m,t，1 , x g,m,t，2 , x g,m,t，3 ]

[0020] t=1、2……T

[0021] T=t B

[0022] Among them, x g,m,t is a set representation, representing the state of the mth target at the tth second, x g,m,t，1 is the horizontal coordinate of the uniform local trajectory of the mth target at the tth second, x g,m,t，2is the ordinate of the uniform local trajectory of the mth target at the tth second, x g,m,t，3 Height coordinate of the uniform local trajectory of the mth target at the tth second.

[0023] Preferably, in step S32, the distance between the tracks of the mth target e(m)

[0024]

[0025] K=k1+k2+......+kB

[0026] k=1、2……K

[0027] Among them, K represents the total number of data points of the mth target, because kb represents t b-1 to b The number of data points of the fused target trajectory in this time period is not difficult to understand. The total number of data points is obtained by adding the number of data points in all time periods; f,1，(m，k) represents the horizontal coordinate of the kth data point of the mth target under the fusion target trajectory; x a,1,(m，k) represents the horizontal coordinate of the kth data point of the mth target under the teaching trajectory; x f,2,(m，k) represents the ordinate of the kth data point of the mth target under the fusion target trajectory; x a,2,(m，k) represents the ordinate of the kth data point of the mth target under the teaching trajectory; x f,3,(m，k) represents the height coordinate of the kth data point of the mth target under the fused target trajectory; x a,3,(m，k) Represents the height coordinate of the kth data point of the mth target under the teaching trajectory; C1, C2, C3 are constants and can be selected according to actual conditions.

[0028] Preferably, the step S34 comprises: setting the optimal teaching trajectory (best) S one by one a,m Input the spring damping system model and use the local weighted regression algorithm to obtain the corresponding f m , as follows:

[0029]

[0030] ξ(k)=s(x f,m,k -x a,m,k )

[0031] Among them, x a is the target teaching trajectory fitting speed, is the fitting speed in the target model, is the fitted acceleration in the target model; g is the end position to which the target converges, τ is the time expansion factor; f(ω m ,s) is a nonlinear function, representing the trajectory shape learner; αx is the regularization parameter, α y is the first spring damping system model parameter, β y is the model parameter of the second spring damping system; f(ω m ,s) is the trajectory shape learning function, s is the preset variable, and it forms a function with the running time s = exp(-α x t / τ), exp is the natural exponential function, t is the running time, is the derivative of the preset variable, g is the initial value of the target moving trajectory, ω m is the weight value of each basis function, Ψ m (z) is the radial basis function, is the variance of the basis function, c m is the center of the basis function, Γ m is the radial basis function correlation matrix, f m is the local trajectory represented by the target data point, a is the motion difference matrix between the target teaching trajectory and the fused target trajectory, x f,m,k is the motion fitting value of the fused target trajectory, x a,m,k is the motion fitting value of the target's teaching trajectory.

[0032] Preferably, the step S4 comprises: step S41, setting m=1, t=1; step S42, inputting the uniform local trajectory of the mth target from the t-1th second to the tth second into a blank judgment model to detect whether there is a collision with an existing uniform local trajectory;

[0033]

[0034] Wherein, j=1, 2...M, j≠m, ε(m→j, 1) represents the distance between the mth target and other targets at the tth second; if ε(m→j, 1) is greater than a preset threshold, it is determined that there is a collision and the process goes to step S43; if not, it is determined that there is no collision and the process goes to step S44; in step S43, a direction within 0 to 180° is randomly selected for the collision point, and the end position of the tth second is used as the destination to perform distance extension, and each extended trajectory is an extended trajectory; if the extended trajectory intersects with other trajectories during the extension, the direction is discarded until the random extended node meets the position distance less than the threshold. value, record it as the uniform local trajectory from the t-1th second to the tth second; and detect t=T, if so, enter step S45; if not, set t=t+1 and return to step S42; step S44, record it as the uniform local trajectory from the t-1th second to the tth second; and detect t=T, if so, enter step S45; if not, set t=t+1 and return to step S42; step S45, output the uniform local trajectory of the mth target from the 0th second to the Tth second; and judge m=M, if so, organize and output all system trajectories, if not, set m=m+1, t=1 and return to step S42.

[0035] Preferably, the step S43 uses an RRT algorithm and a segmented DMP algorithm to obtain the target trajectory of the expanded trajectory.

[0036] In summary, compared with the prior art, the infrared and radar target fusion technology based on interactive multi-model provided by the present invention has the following beneficial effects:

[0037] First, the obstacle avoidance algorithm can transform the trajectory fitting problem into obstacle avoidance trajectory learning when target tracking is discontinuous. It solves the problem that traditional target tracking only uses the motion state measurement information of the target monomer to input into the filtering algorithm, ignoring the obstacle avoidance characteristics in the actual road scene, and improves the stability of the system trajectory output.

[0038] Second, comprehensively utilize the position, speed, and target measurement information of other traffic participants around, introduce the characteristics of the target motion law in the scene, and use historical trajectory data to predict the target's motion state, which is more in line with the target kinematics and improves trajectory accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the attached embodiment of the present invention Figure 1 , the technical solutions, structural features, objectives achieved and effects in the embodiments of the present invention are described in detail.

[0041] It should be noted that the drawings are in a very simplified form and use non-precise proportions. They are only used to conveniently and clearly assist in explaining the embodiments of the present invention, and are not used to limit the conditions for the implementation of the present invention. Therefore, they have no substantive technical significance. Any structural modification, change in proportional relationship or adjustment of size, without affecting the effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical content disclosed by the present invention.

[0042] It should be noted that, in the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only the elements explicitly listed, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0043] The present invention provides an infrared and radar target fusion method based on interactive multi-model, which is based on the infrared sensor and radar acquiring a total of M targets from the t0th second to the tth second. B After obtaining the position information, the position information is analyzed through multiple steps to finally generate a system trajectory; the system trajectory is the position information from the 1st target to the Mth target, and each target from t0th second to t0th second is B The overall running trajectory of seconds.

[0044] Wherein, t0 represents the initial time of starting to capture the target, and t0 does not necessarily start from 0 seconds when the target is not moving, which means that the moving target can be captured in real time; in addition, for any two adjacent t b-1 With t b The time intervals between them do not necessarily follow certain fixed values; i.e., t0, t1, t2, ... t B The emphasis is on expressing at a single point in time, rather than expressing it at fixed intervals of 1 second or 2 seconds.

[0045] like Figure 1 As shown, the method includes:

[0046] Step S1, respectively obtain the time from t0th second to tth second of a total of M targets through infrared sensors and radars B The infrared position information and radar position information of the M targets from t0 to t0 seconds are calibrated accordingly.B Seconds of infrared data and radar data, b = 1, 2...B, m = 1, 2...M;

[0047] Specifically, the infrared position information is expressed in the form of an infrared image coordinate system obtained by the infrared sensor; the radar position information is expressed in the form of a radar coordinate system obtained by the radar;

[0048] The corresponding calibration specifies the spatial alignment of the infrared image coordinate system obtained by the infrared sensor and the radar coordinate system obtained by the radar. Because the infrared image coordinate system and the radar coordinate system are obtained based on two different devices, the time axis, the captured spatial range, etc. are inconsistent. If the corresponding calibration method is not adopted, there will be no unified standard for the two, and subsequent analysis will be meaningless.

[0049] Specifically, the PnP (Perspective-n-Point, an algorithm for estimating the camera posture from feature points in an image) algorithm can be used for corresponding calibration; through corresponding calibration, the image data from the t0th second to the tth second of a total of M targets are obtained. B Seconds of infrared and radar data.

[0050] In step S2, the IMM (Interacting Multiple Model) algorithm is used to analyze the M targets one by one from the t0th second to the tth second. B The infrared data of 10 seconds and the radar data are processed to obtain the fused target trajectory S f,1 ~S f,M ; Among them, S f,m Indicates the mth target from t0th second to tth second B fused target trajectory of seconds;

[0051] The IMM algorithm includes a uniform speed operation model, an interactive function and other tools for analyzing infrared data and radar data.

[0052] Step S3, according to the preset teaching trajectory S a,1 ~S a,M And the fused target trajectory S calculated in S2 f,1 ~S f,M , and obtain the local trajectory f1~f M , and then use the DMP (Dynamic Movement Primitives) algorithm to obtain the time from t0 to t1 of a total of M targets. B The uniform local trajectory S of seconds g,1 ~S g,M ; Among them, f m Indicates the mth target from t0th second to tth second BSeconds of local trajectory, S a,m Indicates the mth target from t0th second to tth second B Seconds of teaching trajectory, S g,m Indicates the mth target from t0th second to tth second B Uniform local trajectory of seconds.

[0053] The teaching trajectory is the historical data of the target or the general behavior of the same type of target, and has the same form of expression as the fused target trajectory; for example, it can be obtained by extracting the historical motion features of the specified target and then reproducing the features in the same scene. However, compared to the fused target trajectory, the teaching trajectory is reproduced twice through a single or a small number of motion features. It is important to note here that the teaching trajectory must be pre-set in strict accordance with the form of expression of the fused target trajectory, that is, it needs to be accurate to the corresponding t b-1 to b During this period of time, the composition properties of the two are exactly the same, otherwise the subsequent DMP algorithm processing cannot be performed.

[0054] Step S4, analyzing and processing the uniform local trajectories S of the M targets g,1 ~S g,M , generate system trajectory output;

[0055] The concept of system trajectory has been described above and will not be repeated here. The purpose of step S4 is to check whether there is a collision in the system trajectory. If there is a collision, it means that there is a problem in the trajectory fusion process and correction is required.

[0056] Specifically, the step S2 includes:

[0057] Step S21, based on the uniform motion model, according to the M targets from t0 to t B 10 seconds of infrared data and radar data, and construct corresponding infrared data motion state model and radar data motion state model;

[0058] Step S22, inputting the infrared data motion state model into the interaction function and performing Kalman filtering to obtain the infrared data trajectory estimation result;

[0059] Step S23, inputting the radar data motion state model into the interaction function and performing Kalman filtering to obtain the radar data trajectory estimation result;

[0060] Step S24, weighted summing the infrared data trajectory estimation results and the radar data trajectory estimation results is performed to obtain the trajectory from t0 to t1 of a total of M targets. B The fused target trajectory S of seconds f,1 ~S f,M ;

[0061] When performing Kalman filter processing, a likelihood function is used to update the Kalman filter; the likelihood function plays a role in predicting the operation results and enhancing the accuracy of the Kalman filter processing.

[0062] The mth target from t0 seconds to t B The fused target trajectory S of seconds f,m The expression is:

[0063]

[0064] in, Indicates that the mth target is at t b-1 to b The fused target trajectory within this period of time, kb represents t b-1 to b The number of data points of the fused target trajectory in this time period is not difficult to understand here, because the data in a time period must be multiple and continuous. kb is used as the total number of data points, which is only used as information representation and has no computational significance. In addition, for t f,m Only represents the mth target from t0th second to tth second B The fused target trajectory S of seconds f,m The timestamp is only used for information representation and has no computational significance.

[0065] For a single It is expressed as follows:

[0066]

[0067] in, is the fused target trajectory of the mth target at t b-1 to b The horizontal axis set within this time period is is the fused target trajectory t of the mth target b-1 to b The vertical coordinate set within this time period is The fused target trajectory t of the mth target b-1 to b The height coordinates set within this time period; in addition, here as well as Each data in is represented by a vector with a dimension of 3×1, and the three dimensions are the spatial position in the horizontal coordinate / vertical coordinate / altitude coordinate orientation, the current speed of the target, and the current acceleration of the target.

[0068] For example, the fused target trajectory of the first target in the time period from t0 to t1 is: Represented by a vector with a dimension of 3×1, they are the spatial position under the horizontal coordinate, the current speed of the target, and the current acceleration of the target; the rest are similar.

[0069] Specifically, step S3 includes:

[0070] Step 31: pre-set the time from t0th second to tth second for M targets. B Teaching trajectory S in seconds a,1 ~S a,M ; S a,m represents the teaching trajectory of the mth target;

[0071] How to obtain the teaching trajectory has been explained above and will not be repeated here;

[0072] The teaching trajectory must be pre-set strictly according to the expression form of the fusion target trajectory, so the teaching trajectory S of a single target a,m for,

[0073]

[0074] Similarly, represents the mth target at t b-1 to b The teaching trajectory during this period of time, kb represents t b-1 to b The number of data points of the fused target trajectory during this period is only used as information representation and has no computational significance; t a,m Only represents the mth target from t0th second to tth second B Teaching trajectory S in seconds a,m The timestamp is only used as information representation and has no computational significance. Obviously, the representation and structure of the teaching trajectory and the fused target trajectory are the same.

[0075] Furthermore, for a single It is expressed as follows:

[0076]

[0077] in, is the teaching trajectory of the mth target at t b-1 to b The horizontal axis set within this time period is is the teaching trajectory of the mth target at t b-1 to b The vertical coordinate set within this time period is The teaching trajectory of the mth target is at t b-1 to b The height coordinates set within this time period; in addition, here as well as Each data in is represented by a vector with a dimension of 3×1, and the three dimensions are the spatial position in the horizontal coordinate / vertical coordinate / altitude coordinate orientation, the current speed of the target, and the current acceleration of the target. Obviously, the representation and structure of the teaching trajectory and the fused target trajectory are the same.

[0078] Step S32, according to the teaching trajectory S a,1 ~S a,M and the fused target trajectory S f,1 ~S f,M , calculate the distance e(m) between the trajectories of m targets one by one;

[0079] Specifically, the distance between the trajectories of the mth target is e(m)

[0080]

[0081] K=k1+k2+......+kB

[0082] k=1、2……K

[0083] Among them, K represents the total number of data points of the mth target, because kb represents t b-1 to b The number of data points of the fused target trajectory in this time period is not difficult to understand. The total number of data points is obtained by adding the number of data points in all time periods; f,1，(m，k) represents the horizontal coordinate of the kth data point of the mth target under the fusion target trajectory; x a,1,(m，k) represents the horizontal coordinate of the kth data point of the mth target under the teaching trajectory; x f,2,(m，k) represents the ordinate of the kth data point of the mth target under the fusion target trajectory; x a,2,(m，k) represents the ordinate of the kth data point of the mth target under the teaching trajectory; x f,3,(m，k) represents the height coordinate of the kth data point of the mth target under the fused target trajectory; x a,3,(m，k) Represents the height coordinate of the kth data point of the mth target in the teaching trajectory; C1, C2, C3 are constants and can be selected according to the actual situation. For example, in the road traffic scene, use C1 = 2 ~ 4, C2 = 1 ~ 2, C3 = 0, that is, omit Here, e(m) can be used to judge the correlation between the teaching trajectory and the fusion target trajectory. For example, in the aforementioned road traffic scenario, e(m)≤η, when η is 0.5-1.5, the trajectory is predicted to be correlated.

[0084] Step S33, adjusting the teaching trajectory according to the distance between trajectories e(m) to obtain the optimal teaching trajectory (best) S for each target a,1 ~(best)S a,M; (best) S a,m represents the optimal teaching trajectory of the mth target;

[0085] As mentioned above, there may be a situation where e(m)≤η is not satisfied. In this case, it is necessary to regenerate the teaching trajectory or even generate multiple teaching trajectories, and finally select the optimal teaching trajectory. The purpose of this step is to ensure that all the teaching trajectories obtained are teaching trajectories that can be used for subsequent analysis.

[0086] Step S34, one by one according to the optimal teaching trajectory (best) S a,m , get the corresponding local trajectory f1~f M ; Among them, f m represents the local trajectory of the mth target;

[0087] Specifically, the optimal teaching trajectory (best) S a,m Input the spring damping system model and use the local weighted regression algorithm to obtain the corresponding f m , as follows:

[0088]

[0089]

[0090] ξ(k)=s(x f,m,k -x a,m,k )

[0091] In the formula, x a is the target teaching trajectory fitting speed, is the fitting speed in the target model, is the fitted acceleration in the target model; g is the end position to which the target converges, τ is the time expansion factor; f(ω m ,s) is a nonlinear function, representing the trajectory shape learner; α x is the regularization parameter, α y is the first spring damping system model parameter, β y is the model parameter of the second spring damping system; f(ω m ,s) is the trajectory shape learning function, s is the preset variable, and it forms a function with the running time s = exp(-α x t / τ), exp is the natural exponential function, t is the running time, is the derivative of the preset variable, g is the initial value of the target moving trajectory (here is the horizontal coordinate), ω m is the weight value of each basis function, Ψ m (z) is the radial basis function, is the variance of the basis function, c m is the center of the basis function, Γm is the radial basis function correlation matrix, f m is the local trajectory represented by the target data point, a is the motion difference matrix between the target teaching trajectory and the fused target trajectory, x f,m,k is the motion fitting value of the fused target trajectory, x a,m,k is the motion fitting value of the target's teaching trajectory.

[0092] In step S35, according to f 1～ f M , the uniform local trajectory S of a total of M targets is obtained through the DMP algorithm g,1 ~S g,M ;

[0093] S g,m =[x g,m,1 ,x g,m,2 ,......x g,m,t ......x g,m,T ]

[0094] x g,m,t =[x g,m,t，1 , x g,m,t，2 , x g,m,t，3 ]

[0095] t=1、2……T

[0096] T=t B

[0097] Among them, x g,m,t is a set representation, representing the state of the mth target at the tth second, x g,m,t，1 is the horizontal coordinate of the uniform local trajectory of the mth target at the tth second, x g,m,t，2 is the ordinate of the uniform local trajectory of the mth target at the tth second, x g,m,t，3 The height coordinate of the uniform local trajectory of the mth target at the tth second; In addition, because the trajectory of the target is calculated within a certain period of time, T = t B , here only the form of expression is different, there is no difference in essence.

[0098] Specifically, step S4 includes:

[0099] Step S41, set m=1, t=1;

[0100] Step S42, inputting the uniform local trajectory of the mth target from the t-1th second to the tth second into a blank judgment model to detect whether there is a collision with an existing uniform local trajectory;

[0101]

[0102] Wherein, j=1, 2...M, j≠m, ε(m→j, 1) represents the distance between the mth target and other targets at the tth second; if ε(m→j, 1) is greater than a preset threshold, it is determined that there is a collision, and the process goes to step S43; if not, it is determined that there is no collision, and the process goes to step S44;

[0103] When m=1, there are no other uniform local trajectories, so the uniform local trajectories input to the blank judgment model for the first time can be directly input once and then proceed to step S45.

[0104] Step S43, randomly select a direction within 0 to 180 degrees for the collision point, take the end position of the t-th second as the destination, and perform distance extension. The trajectory of each extension is the extended trajectory; if the extended trajectory intersects with the other trajectories during the extension, the direction is discarded until the random extended node meets the position distance less than the threshold, and it is recorded as a uniform local trajectory from the t-1 second to the t-th second; and detect t=T, if so, enter step S45; if not, set t=t+1 and return to step S42;

[0105] Step S44, record it as a uniform local trajectory from the t-1th second to the tth second; and detect t=T, if yes, go to step S45; if no, set t=t+1, and return to step S42;

[0106] Step S45, output the uniform local trajectory of the mth target from the 0th second to the Tth second, and judge whether m=M. If so, sort out and output all system trajectories. If not, set m=m+1, t=1, and return to step S42.

[0107] In addition, for step S43, the RRT (Rapidly-Exploring Random Trees) algorithm and the segmented DMP algorithm can be used to obtain the corrected (expanded) target trajectory to solve the problem of poor trajectory prediction accuracy caused by reduced data rate due to occlusion in multi-target scenes.

[0108] Although the content of the present invention has been described in detail through the above preferred embodiments, it should be appreciated that the above description should not be considered as a limitation of the present invention. After reading the above content, it will be apparent to those skilled in the art that various modifications and substitutions of the present invention will occur. Therefore, the protection scope of the present invention should be limited by the appended claims.

Claims

1. A method for infrared and radar target fusion based on interactive multi-model, characterized in that: The method comprises: Step S1, respectively obtain the time from t0th second to tth second of a total of M targets through infrared sensors and radars B The infrared position information and radar position information of the M targets from t0 to t0 seconds are calibrated accordingly. B seconds of infrared and radar data, b=1, 2...B, m=1, 2...M; In step S2, the IMM algorithm is used to analyze the M targets one by one from the t0th second to the tth second. B The infrared data of 10 seconds and the radar data are processed to obtain the fused target trajectory S f,1 ~S f,M ; Among them, S f,m Indicates the mth target from t0th second to tth second B fused target trajectory of seconds; Step S3, according to the preset teaching trajectory S a,1 ~S a,M And the fusion target trajectory S f,1 ~S f,M , and obtain the local trajectory f1~f M , and then use the DMP algorithm to obtain the M targets from the t0th second to the tth second one by one. B The uniform local trajectory S of seconds g,1 ~S g,M ; Step S4, analyzing and processing the uniform local trajectories S of the M targets g,1 ~S g,M , generating system trajectory output.

2. The infrared and radar target fusion method based on interactive multi-model according to claim 1 is characterized in that: In the step S1, the infrared position information is expressed in the form of an infrared image coordinate system obtained by the infrared sensor; the radar position information is expressed in the form of a radar coordinate system obtained by the radar; and a spatial alignment operation is performed on the infrared image coordinate system and the radar coordinate system obtained by the radar for corresponding calibration; Through corresponding calibration, we obtain the M targets from the t0th second to the tth second. B Seconds of infrared and radar data.

3. The infrared and radar target fusion method based on interactive multiple models according to claim 2 is characterized in that: In the step S1, a PnP algorithm is used to perform a spatial alignment operation on the infrared image coordinate system and the radar coordinate system acquired by the radar for corresponding calibration.

4. The infrared and radar target fusion method based on interactive multiple models according to claim 3 is characterized in that: The step S2 comprises: Step S21, based on the uniform motion model, according to the M targets from t0 to t B 10 seconds of infrared data and radar data, and construct corresponding infrared data motion state model and radar data motion state model; Step S22, inputting the infrared data motion state model into the interaction function and performing Kalman filtering to obtain the infrared data trajectory estimation result; Step S23, inputting the radar data motion state model into the interaction function and performing Kalman filtering to obtain the radar data trajectory estimation result; Step S24, weighted summing the infrared data trajectory estimation results and the radar data trajectory estimation results is performed to obtain the trajectory from the t0th second to the tth second of a total of M targets. B The fused target trajectory S of seconds f,1 ~S f,M ; The mth target from t0 seconds to t B The fused target trajectory S of seconds f,m The expression is: in, Indicates that the mth target is at t b-1 to b The fused target trajectory within this period of time, kb represents t b-1 to b The number of data points of the fused target trajectory within this period; t f,m Only represents the mth target from t0th second to tth second B The fused target trajectory S of seconds f,m timestamp; For a single It is expressed as follows: in, is the fused target trajectory of the mth target at t b-1 to b The horizontal axis set within this time period is is the fused target trajectory t of the mth target b-1 to b The vertical coordinate set within this time period is The fused target trajectory t of the mth target b-1 to b The set of height coordinates within this time period; as well as Each data in is represented by a vector with a dimension of 3×1, and the three dimensions are the spatial position in the horizontal coordinate / vertical coordinate / altitude coordinate orientation, the current speed of the target, and the current acceleration of the target.

5. The infrared and radar target fusion method based on interactive multiple models according to claim 4 is characterized in that: In step S23, when performing Kalman filter processing, a likelihood function is used to update the Kalman filter.

6. The infrared and radar target fusion method based on interactive multiple models according to claim 5 is characterized in that: The step S3 comprises: Step 31: pre-set the time from t0th second to tth second for M targets. B Teaching trajectory S in seconds a,1 ~S a,M ; S a,m represents the teaching trajectory of the mth target; Teaching trajectory S for a single target a,m for, in, represents the mth target at t b-1 to b The teaching trajectory during this period of time, kb represents t b-1 to b The number of data points of the fused target trajectory within this period; t a,m Only represents the mth target from t0th second to tth second B Teaching trajectory S in seconds a,m timestamp; For a single It is expressed as follows: in, is the teaching trajectory of the mth target at t b-1 to b The horizontal axis set within this time period is is the teaching trajectory of the mth target at t b-1 to b The vertical coordinate set within this time period is The teaching trajectory of the mth target is at t b-1 to b The set of height coordinates within this time period; as well as Each data in is represented by a vector with a dimension of 3×1, and the three dimensions are the spatial position in the horizontal coordinate / vertical coordinate / altitude coordinate orientation, the current speed of the target, and the current acceleration of the target; Step S32, according to the teaching trajectory S a,1 ~S a,M and the fused target trajectory S f,1 ~S f,M , calculate the distance e(m) between the trajectories of m targets one by one; Step S33, adjusting the teaching trajectory according to the distance between trajectories e(m) to obtain the optimal teaching trajectory (best) S for each target a,1 ~(best)S a,M ; (best) S a,m represents the optimal teaching trajectory of the mth target; Step S34, one by one according to the optimal teaching trajectory (best) S a,m , get the corresponding local trajectory f1~f M ; Among them, f m represents the local trajectory of the mth target; Step S35, according to f 1～ f M , the uniform local trajectory S of a total of M targets is obtained through the DMP algorithm g,1 ~S g,M ; S g,m =[x g,m,1 ,x g,m,2 ,......x g,m,t ......x g,m,T ] x g,m,t =[x g,m,t，1 ,x g,m,t，2 ,x g,m,t，3 ] t=1、2……T T=t B Among them, x g,m,t is a set representation, representing the state of the mth target at the tth second, x g,m,t，1 is the horizontal coordinate of the uniform local trajectory of the mth target at the tth second, x g,m,t，2 is the ordinate of the uniform local trajectory of the mth target at the tth second, x g,m,t，3 Height coordinate of the uniform local trajectory of the mth target at the tth second.

7. The infrared and radar target fusion method based on interactive multi-model according to claim 6 is characterized in that: In step S32, the distance between the tracks of the mth target is e(m) K=k1+k2+......+kB k=1、2……K Among them, K represents the total number of data points of the mth target, because kb represents t b-1 to b The number of data points of the fused target trajectory in this time period is not difficult to understand. The total number of data points is obtained by adding the number of data points in all time periods; f,1，(m，k) represents the horizontal coordinate of the kth data point of the mth target under the fusion target trajectory; x a,1,(m，k) represents the horizontal coordinate of the kth data point of the mth target under the teaching trajectory; x f,2,(m，k) represents the ordinate of the kth data point of the mth target under the fusion target trajectory; x a,2,(m，k) represents the ordinate of the kth data point of the mth target under the teaching trajectory; x f,3,(m，k) represents the height coordinate of the kth data point of the mth target under the fused target trajectory; x a,3,(m，k) Represents the height coordinate of the kth data point of the mth target under the teaching trajectory; C1, C2, C3 are constants and can be selected according to actual conditions.

8. The infrared and radar target fusion method based on interactive multi-model according to claim 7 is characterized in that: The step S34 includes: setting the optimal teaching trajectory (best) S one by one a,m Input the spring damping system model and use the local weighted regression algorithm to obtain the corresponding f m , as follows: ξ(k)=s(x f,m,k -x a,m,k ) Among them, x a is the target teaching trajectory fitting speed, is the fitting speed in the target model, is the fitted acceleration in the target model; g is the end position to which the target converges, τ is the time expansion factor; f(ω m ,s) is a nonlinear function, representing the trajectory shape learner; α x is the regularization parameter, α y is the first spring damping system model parameter, β y is the model parameter of the second spring damping system; f(ω m ,s) is the trajectory shape learning function, s is the preset variable, and it forms a function with the running time s = exp(-α x t / τ), exp is the natural exponential function, t is the running time, is the derivative of the preset variable, g is the initial value of the target moving trajectory, ωm is the weight value of each basis function, Ψ m (z) is the radial basis function, is the variance of the basis function, c m is the center of the basis function, Γ m is the radial basis function correlation matrix, f m is the local trajectory represented by the target data point, a is the motion difference matrix between the target teaching trajectory and the fused target trajectory, x f,m,k is the motion fitting value of the fused target trajectory, x a,m,k is the motion fitting value of the target's teaching trajectory.

9. The infrared and radar target fusion method based on interactive multi-model according to claim 8, characterized in that: The step S4 comprises: Step S41, set m=1, t=1; Step S42, inputting the uniform local trajectory of the mth target from the t-1th second to the tth second into a blank judgment model to detect whether there is a collision with an existing uniform local trajectory; Wherein, j=1, 2...M, j≠m, ε(m→j, 1) represents the distance between the mth target and other targets at the tth second; if ε(m→j, 1) is greater than a preset threshold, it is determined that there is a collision, and the process goes to step S43; if not, it is determined that there is no collision, and the process goes to step S44; Step S43, randomly select a direction within 0 to 180 degrees for the collision point, take the end position of the t-th second as the destination, and perform distance extension. The trajectory of each extension is the extended trajectory; if the extended trajectory intersects with the other trajectories during the extension, the direction is discarded until the random extended node meets the position distance less than the threshold, and it is recorded as a uniform local trajectory from the t-1 second to the t-th second; and detect t=T, if so, enter step S45; if not, set t=t+1 and return to step S42; Step S44, record it as a uniform local trajectory from the t-1th second to the tth second; and detect t=T, if yes, go to step S45; if no, set t=t+1, and return to step S42; Step S45, output the uniform local trajectory of the mth target from the 0th second to the Tth second; and determine whether m=M. If so, organize and output all system trajectories. If not, set m=m+1, t=1, and return to step S42.

10. The infrared and radar target fusion method based on interactive multiple models according to claim 9, characterized in that: The step S43 uses the RRT algorithm and the segmented DMP algorithm to obtain the target trajectory of the expanded trajectory.

Images