Multi-target data association method, system and device based on infrared fisheye system

Abstract

The present invention provides a multi-target data association method, system, and device based on an infrared fisheye system. The multi-target data association method based on an infrared fisheye system is used to track multiple targets. The multi-target data association method includes the following steps: Step S1, obtaining measurement values ​​of each target at a first moment and a second moment; Step S2, calculating the direction difference of each target between the first moment and the second moment based on the measurement values ​​in Step S1; and Step S3, fusing the distance and direction differences of each target using a Gaussian weighting method to obtain a covariance matrix of each target's trajectory. The multi-target data association method based on an infrared fisheye system according to the present invention improves the tracking accuracy of the tracked targets by fusing the distance and direction of the tracked targets, thereby achieving good performance.

Description

Technical Field

[0001] The present invention relates to the field of target tracking technology, and more particularly to a storage method for a mobile storage device. The present invention also relates to an infrared fisheye system utilizing the multi-target data association method. Furthermore, the present invention also relates to an infrared fisheye device utilizing the infrared fisheye system. Background Art

[0002] The infrared fisheye warning system not only has the advantages of passive detection, being not susceptible to interference, and being highly concealed, but it can also perceive threats from different directions in the hemispheric airspace in real time. It will surely have broad application prospects in future high-tech wars.

[0003] Multi-target tracking is one of the primary functions of the infrared fisheye warning system. Multi-target tracking involves the use of various observation and computational methods to model, estimate, and track the state of a moving object of interest. This technology enables timely early warning or tracking of moving targets on land, sea, air, or space. It detects and locks onto tracked targets, estimates and analyzes their motion state, and provides essential information for fire control, threat assessment, situational assessment, and even decision-making at all levels of command and control systems.

[0004] The core issue in multi-target tracking is how to effectively perform data association. Data association involves comparing candidate measurements with known target trajectories and ultimately determining the correct observation / trajectory pairing. The Joint Probabilistic Data Association (JPDA) algorithm is specifically designed for multi-target tracking.

[0005] Due to its excellent performance in tracking multiple targets in dense clutter, this method has attracted great attention since its introduction and is considered the most effective and reliable method for tracking multiple targets. However, as the number of targets and clutter increases, the number of joint association hypotheses required for this method increases dramatically, resulting in a combinatorial explosion in the computational complexity, which limits its widespread practical application.

[0006] In recent years, many researchers have analyzed the characteristics of the JPDA algorithm, drawing on the parallel nature of computational intelligence methods. They have viewed the JPDA algorithm as a combinatorial optimization problem similar to the Traveling Salesman Problem (TSP), which has computational complexity of NP-complete. Therefore, they have proposed a new joint probabilistic data association method, the Neural Joint Probabilistic Data Association (NJPDA), which simulates the Hopfield (neural network) solution to the TSP.

[0007] The NJPDA algorithm uses distance information as the sole criterion for data association, selecting the observation closest to the track for association. However, it is important to note that the observation closest to the predicted position may not be the best matching observation for the target in dense clutter environments. Therefore, the NJPDA algorithm's correct association rate is significantly reduced in high clutter environments. Summary of the Invention

[0008] In view of this, the present invention aims to propose a multi-target data association method based on an infrared fisheye system to improve the tracking accuracy of multiple targets.

[0009] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0010] A multi-target data association method based on an infrared fisheye system is used to track multiple targets. The multi-target data association method comprises the following steps:

[0011] Step S1, respectively obtaining the measurement value of each target at the first moment and the second moment;

[0012] Step S2: Calculate the direction difference of each target between the first moment and the second moment based on the measured value in step S1;

[0013] Step S3: Use Gaussian weighting method to fuse the distance and direction differences of each target from the first moment to the second moment to obtain the covariance matrix of each target trajectory.

[0014] Furthermore, in step S2, the direction difference is calculated as: Δ in =|γ in -γ n |,i=1,2…n

[0015] γ in represents the angle between the vector direction of the i-th tracking target moving from the first moment to the second moment and the horizontal direction, γ n It represents the angle between the vector direction of the predicted value moving from the first moment to the second moment on the predicted target trajectory and the horizontal direction.

[0016] Furthermore, in step S3, the direction difference information is weighted using the Gaussian function exp(-Δ in P -1 Δ in / 2) to modify the likelihood function of the calculated measurement i (i≠0) corresponding to the target n (n≠0).

[0017] Furthermore, the modified likelihood function is:

[0018]

[0019] Among them, P φ (k) is the covariance matrix of the target heading.

[0020] Furthermore, the error covariance matrix P of the target heading is φ for:

[0021]

[0022] Among them, the four-dimensional state vector of a tracking target is defined as: x and y are the positions of the tracking target on the x-axis and y-axis, and The velocity components of the tracking target on the x-axis and y-axis.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] The multi-target data association method based on the infrared fisheye system of the present invention is beneficial to improving the tracking accuracy of the tracked target by fusing the distance and direction difference of the tracked target, and has a good use effect.

[0025] In addition, another object of the present invention is to propose an infrared fisheye system, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the multi-target data association method as described above when executing the computer program.

[0026] The infrared fisheye system of the present invention, by executing the above-mentioned multi-target data association method based on the infrared fisheye system, is conducive to improving the tracking accuracy of the infrared fisheye system for the target and has good practicality.

[0027] In addition, another object of the present invention is to provide an infrared fisheye device, in which the infrared fisheye system as described above is applied.

[0028] The infrared fisheye device of the present invention, by applying the infrared fisheye system as described above, has high tracking accuracy and usage effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which constitute part of the present invention, are provided to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are provided to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0030] Figure 1 This is a flow chart of a multi-target data association method based on an infrared fisheye system according to an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of the direction difference according to an embodiment of the present invention;

[0032] Figure 3 A motion trajectory diagram of two tracking targets according to an embodiment of the present invention;

[0033] Figure 4 RMSE curves of the X-direction position coordinates of the tracking target 1 obtained by the three algorithms described in the embodiment of the present invention;

[0034] Figure 5 RMSE curves of the Y-direction position coordinates of the tracking target 1 obtained by the three algorithms described in the embodiment of the present invention;

[0035] Figure 6 It is a partial motion trajectory curve of three targets that intersect in the infrared field of view according to an embodiment of the present invention;

[0036] Figure 7 A tracking graph when the tracking targets do not intersect according to an embodiment of the present invention;

[0037] Figure 8 The tracking image of the tracking target in the 620th frame according to the embodiment of the present invention;

[0038] Figure 9 The tracking image of the tracking target in the 630th frame according to the embodiment of the present invention;

[0039] Figure 10 The tracking image of the tracking target in the 640th frame according to the embodiment of the present invention;

[0040] Figure 11 The tracking image of the tracking target in the 650th frame according to the embodiment of the present invention;

[0041] Figure 12 The tracking image of the tracking target in the 660th frame according to the embodiment of the present invention;

[0042] Figure 13 The tracking image of the tracked target in the 670th frame according to the embodiment of the present invention;

[0043] Figure 14 The tracking image of the tracking target in the 680th frame according to the embodiment of the present invention;

[0044] Figure 15 The tracking image of the tracked target in the 690th frame according to the embodiment of the present invention;

[0045] Figure 16 The tracking image of the tracking target in the 700th frame according to the embodiment of the present invention;

[0046] Figure 17The tracking image of the tracking target in the 710th frame according to the embodiment of the present invention;

[0047] Figure 18 The tracking image of the tracking target in the 720th frame according to the embodiment of the present invention; DETAILED DESCRIPTION

[0048] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0049] In the description of the present invention, it should be noted that the terms "upper," "lower," "inner," and "back" and other terms indicating orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0050] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0051] This embodiment relates to a multi-target data association method based on an infrared fisheye system for tracking multiple targets. Figure 1 As shown in , the multi-target data association method includes the following steps: step S1, respectively obtaining the measurement value of each target at the first moment and the second moment; step S2, calculating the direction difference of each target between the first moment and the second moment based on the measurement value in step S1; step S3, using the Gaussian weighting method to fuse the distance and direction difference of each target from the first moment to the second moment to obtain the covariance matrix of each target trajectory.

[0052] As a preferred implementation, in this embodiment, the calculation formula of the direction difference is: Δ in =|γ in -γ n |,i=1,2…n. γ in represents the angle between the vector direction of the i-th tracking target moving from the first moment to the second moment and the horizontal direction, γ n It represents the angle between the vector direction of the predicted value moving from the first moment to the second moment on the predicted target trajectory and the horizontal direction.

[0053] For details, refer to Figure 2 As shown in , assuming that the trajectory of target n has been established, the state prediction value of target n at time k+1 is Assume there are two measurements Z1 and Z2 in the tracking gate, and the state of target n at time k is estimated to be Use γ 1nIndicates the connection between measurement Z1 and The vector direction, γ 2n Indicates the connection measurement Z2 and The vector direction, γ n Represents the connection prediction value and estimated values The vector direction of .

[0054] Considering that the target moves relatively slowly on the image plane in an infrared fisheye system and the system sampling frequency is relatively high, the target's motion can be considered as uniform linear motion within two adjacent sampling periods. In other words, the direction and speed of the target's motion trajectory change very little, and no sudden changes occur.

[0055] In this embodiment, in order to improve the data association effect between the direction difference and the target trajectory, according to Δ in The size of the direction difference is given a certain direction weight. In step S3, the direction difference information is weighted by using the Gaussian function exp(-Δ in P -1 Δ in / 2), and correct the likelihood function of the calculated measurement i (i≠0) corresponding to the target n (n≠0).

[0056] Among them, the likelihood function after correction in step S22 is:

[0057]

[0058] Among them, P φ (k) is the covariance matrix of the target heading.

[0059] The error covariance matrix P of the target heading φ for:

[0060] Among them, the four-dimensional state vector of a tracking target is defined as: x and y are the positions of the tracking target on the x-axis and y-axis, and The velocity components of the tracking target on the x-axis and y-axis.

[0061] In order to verify the effectiveness of the multi-target data association method described in this implementation, a Kalman filter was used to first conduct a tracking experiment on the simulated trajectory, and the tracking performance was compared and analyzed with that of the JPDA algorithm and the NJPDA algorithm; then a tracking experiment was carried out on the actual image data of a domestically produced trainer aircraft to prove the effectiveness of the improved algorithm.

[0062] First, the specific content of the experimental analysis based on the simulation trajectory is as follows:

[0063] Assume that the sampling frequency of the infrared fisheye system is 50 frames / s. There are two targets in the field of view of the infrared fisheye system: target 1 and target 2. Assume that target 1 and target 2 are both moving on a plane, and their initial positions are (100, 80) and (100, 120) respectively. The two targets move in a uniform linear motion at a speed of 0.46 pixels / frame and 0.32 pixels / frame respectively. The movement time of the two targets is 200 cycles. In the two-dimensional rectangular coordinate system, the measurement noise on the X-axis and Y-axis is a Gaussian white noise sequence with zero mean and an error of 1 pixel. The observation standard deviation is 0.5 pixels. The detection probability of the target is P D =1, the gate probability is 0.99, the clutter obeys the Poisson distribution and takes λ=14×10 -6 / m 2 . Figure 3 are the motion trajectory diagrams of the two targets. Let the state vector of the target be The initial states of the two targets are:

[0064]

[0065] Table 1. Comparison of tracking performance of three algorithms

[0066]

[0067] As shown in Table 1, through tracking experiments and analysis of the simulated trajectory, compared with the JPDA algorithm and the NJPDA algorithm, the algorithm of this scheme improves the tracking accuracy while maintaining a small calculation time.

[0068] In order to verify the improvement effect of the multi-target data association method on the data association performance of multi-target tracking in an actual system, the improved algorithm is used in this embodiment to process infrared image data of a domestically produced trainer aircraft in flight taken by a medium-wave infrared fisheye system.

[0069] Since the target is not very far away, it appears as a few-pixel spot on the infrared image. It should be noted that in reality, capturing multiple targets simultaneously in the air is rare, especially when two targets are crossing each other. Therefore, this paper uses video overlay to obtain multi-target motion data for three targets in a 1000-frame image sequence.

[0070] The specific method is: prepare three groups of infrared images of targets with different flight trajectories in the same airspace, each group of 1000 frames. In the MATLAB environment, take the first group of infrared image data as the main one, extract the target data from the other two groups of infrared images, and superimpose them on the first group of infrared image data accordingly, so as to obtain infrared images containing three targets in the same scene.

[0071] Figure 6 Partial trajectory curves of three targets appearing in the infrared field of view, showing a target intersection. Using the multi-target data association method of this embodiment, the CV model is used as the target motion model, and a Kalman filter is employed to estimate the future state and track these three targets. Prior information about the target's motion on the image plane (position, velocity, etc.) is provided by the track initiation process.

[0072] Figures 7 to 18 The figure shows the whole process of tracking three targets using the improved algorithm. In order to distinguish the position changes of the two targets when they intersect, the tracking window of target 3 is represented by a black frame, the tracking window of target 1 is represented by a white frame on the right, and the tracking window of target 2 is represented by a white frame on the left. Figure 7 The figure shows the trajectories of the three targets after the track starts and before they intersect. At this time, the movements of the three targets do not affect each other, and each target is within its own tracking window, maintaining normal tracking.

[0073] When the targets are close to each other, their tracks will cross. In this process, the tracking windows of the crossing targets go through five processes: approach, intersection, overlap, intersection, and separation. In order to clearly illustrate the changes in these five processes, in the tracking process, when the targets are about to cross, every 10 frames of experimental images are compared. The experimental images obtained are as follows: Figures 8 to 18 shown.

[0074] in, Figure 8 It shows that the two targets are very close to each other. The targets have not yet crossed, but their tracking windows begin to intersect. As time goes by, the two targets get closer and their tracking windows further intersect, but the targets are still within the tracking windows. Figures 9 to 11 As shown in . Figure 12 and Figure 13 In the example, the tracking windows of the two targets almost completely overlap; Figure 14 At the beginning, the two targets began to separate Figure 15 、 Figure 16 Until Figure 17 , the tracking windows of the two are completely separated.

[0075] In summary, the entire experimental image series of target crossing motion demonstrates that the tracking window never loses contact with the target, demonstrating similar tracking characteristics to a single target. It should be noted that during target crossing, the algorithm not only utilizes distance information for association but also fully utilizes the direction of target motion to attribute the measurements of the overlapping tracking windows. This reduces the probability of false associations, allowing for the separation and continuity of crossing target trajectories, thereby improving target tracking accuracy.

[0076] During the aforementioned experiment, when the targets did not intersect, the improved algorithm took approximately 2.6ms to complete data association for the three targets. However, when targets intersect, the algorithm needs to adjust each parameter of the target trajectory for which actual measurements are missing, including recording the number of missed measurements and reallocating the measurements. This increases execution time.

[0077] The algorithm takes the longest to execute, but only takes about 3.3 ms. This leaves plenty of room for computation given the 20 ms execution time. Furthermore, because the improved algorithm uses a neural network, the multi-target data association method has stronger data parallel processing capabilities. Therefore, compared to the JPDA algorithm, the execution time of the multi-target data association method in this embodiment does not increase significantly even when dealing with data association problems involving a large number of targets. This is another advantage of the improved algorithm.

[0078] In summary, the multi-target data association method in this embodiment is beneficial to improving the tracking accuracy of the tracked target by fusing the distance and direction of the tracked target, and has a good use effect.

[0079] In addition, this embodiment also relates to an infrared fisheye system, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the multi-target data association method described in the first embodiment is implemented.

[0080] The infrared fisheye system described in this embodiment, by executing the above-mentioned multi-target data association method based on the infrared fisheye system, is conducive to improving the tracking accuracy of the infrared fisheye system for the target and has good practicality.

[0081] In addition, this embodiment also relates to an infrared fisheye device, which is applied with the infrared fisheye system described above. By applying the infrared fisheye system, the tracking accuracy and use effect of the entire device are improved.

[0082] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-target data association method based on an infrared fisheye system for tracking multiple targets, characterized in that: The multi-target data association method comprises the following steps: Step S1, respectively obtaining the measurement value of each target at the first moment and the second moment; Step S2: Calculate the direction difference of each target between the first moment and the second moment based on the measured value in step S1; Step S3: Using Gaussian weighting method to fuse the distance and direction differences of each target from the first moment to the second moment, to obtain the covariance matrix of each target trajectory; In step S2, the direction difference is calculated as: Δ in =|γ in - γ n |,i=1,2…n γ in represents the angle between the vector direction of the i-th tracking target moving from the first moment to the second moment and the horizontal direction, γ n Indicates the angle between the vector direction of the predicted value moving from the first moment to the second moment on the predicted target trajectory and the horizontal direction; In step S3, the direction difference is weighted by using Gaussian function exp(-Δ in P -1 Δ in / 2) to modify the likelihood function of the calculated measurement i, i≠0 corresponding to the target n, n≠0; The corrected likelihood function is: in, is the covariance matrix of the target heading; The error covariance matrix of the target heading for: Among them, the four-dimensional state vector of a target is defined as: x and y are the positions of the target on the x-axis and y-axis, and are the velocity components of the target on the x-axis and y-axis.

2. An infrared fisheye system, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the multi-target data association method according to claim 1 when executing the computer program.

3. An infrared fisheye device, characterized in that: The infrared fisheye device is applied with the infrared fisheye system according to claim 2.

Images