A target tracking method based on multispectral decision fusion

Abstract

This invention discloses a target tracking method with multi-spectral decision fusion, which relates to the field of computer vision. The present invention proposes a multi-spectral target tracking method with multi-expert decision fusion, which uses a visual target tracker independent of the spectral distribution to perceive data of different spectra and fuses the tracking results obtained for each spectral image. This fusion result makes full use of the features of different modalities. At the same time, it belongs to decision fusion, avoiding the complex and tedious cross-modal feature alignment. It only needs to process the alignment transformation of the target single point coordinates, thereby improving the final accuracy and efficiency of the multi-spectral target and the anti-interference ability of misaligned multi-spectral data.

Description

Technical Field

[0001] The present invention relates to the field of computer vision. Specifically, it relates to a target tracking method using multispectral decision fusion. This method explicitly establishes intra-modal and inter-modal predictive correlation models and uses the correlation of target tracking trajectories to train a visual target tracker, which is of great significance for improving the accuracy of multispectral target tracking. Background Art

[0002] Target tracking methods based on multispectral data fusion can significantly improve the performance of target tracking tasks in adverse working environments. Traditional multispectral target tracking methods, such as Jointly modeling motion and appearancecues for robust RGB-T tracking, use intermediate feature fusion methods to achieve complementary fusion of multispectral information. This requires strict alignment of the image feature space. Datasets such as LasHeR, Li C, Xue W, Jia Y, et al. LasHeR: A large-scale high-diversity benchmark for RGBT tracking[J]. IEEE Transactions on Image Processing, 2021, 31: 392-404, rely on complex and cumbersome image pre-registration operations. In addition, due to the effectiveness differences between different spectral images, direct feature fusion will cause invalid spectral features to interfere with valid modal features, resulting in poor target tracking performance. Traditional methods such as patents CN101252677A and CN110349185A require the use of complex fusion mechanisms to calculate the feature weights of different modalities, increasing the complexity of the algorithm. Summary of the Invention

[0003] To address the low accuracy, poor interference immunity, and low efficiency of multispectral target tracking caused by interference from failed features and high data registration requirements, a multispectral decision fusion target tracking method is proposed. Independent visual target trackers are used to track targets in different spectral images, and a correction model for intra-spectral and inter-spectral target tracking results is established. Based on the tracking results from different spectra, a trajectory-correlated loss function is used to optimize the visual target tracker, improving the final target tracking accuracy.

[0004] The specific technical solutions of the present invention are as follows:

[0005] A target tracking method based on multispectral decision fusion, the method comprising:

[0006] Step 1: Create state variables :

[0007] ;

[0008] in 、 、 Respectively represent the position, velocity, and acceleration of the target in the image coordinate system, and are all vectors;

[0009] Step 2: Calculate the velocity and acceleration sequences of each modal trajectory. For time j, the tracking results of the target under each spectrum are recorded as , where M represents the total number of spectra, then the speed and acceleration The update formula is as follows:

[0010] ;

[0011] ;

[0012] in 、 and are the target position, velocity and acceleration calculated for spectral component m at time j, is the time difference between two frames;

[0013] Step 3: Update the factor, speed, and acceleration using the moving average:

[0014] ;

[0015] ;

[0016] in, represents the speed movement update factor, represents the speed moving average update factor, represents the acceleration movement update factor, represents the acceleration moving average update factor, is the smoothing factor;

[0017] Step 4: Construct the state transfer equation A:

[0018] ;

[0019] in, ;

[0020] ;

[0021] represents the identity matrix, Indicates the time difference between two frames;

[0022] Using previous target tracking results and motion equations to predict the target state at the next moment :

[0023] ;

[0024] in,

[0025] , ;

[0026] Step 5: At time j+1, use independent visual target tracking to process the image of spectral component m to obtain the spatial probability distribution map of the target position , the position of its maximum value is the observation position of target tracking, recorded as , update the observed velocity, and observed acceleration;

[0027] Step 6: Update the prediction error metric :

[0028] ;

[0029] in, is the error metric of the target tracking result, To predict noise, the superscript T indicates transposition;

[0030] use Calculate the update matrix :

[0031] ;

[0032] in, Represents a probability distribution graph The observation noise; then the j+1 frame, the target tracking result is obtained by using the information correction in the spectrum m. for:

[0033] ;

[0034] in, The tracking result of the predicted spectral component m at time j+1, Indicates the measurement results, is the update matrix of spectral component m at time j+1;

[0035] Systematic error metrics for:

[0036] ;

[0037] Step 7: Construct the state observation matrix from m spectrum to n spectrum components :

[0038] ;

[0039] ;

[0040] in, 、 as well as They represent the internal parameters of the spectrum n camera, the internal parameters of the spectrum m camera, and the rotation matrix from spectrum m to spectrum n, respectively. represents the inverse matrix, and They represent the x and y components of the translation vector between the spectrum n and spectrum m cameras, respectively. Indicates that time j+1 is the distance between the target and the observation point;

[0041] Step 8: Use the n-spectrum observation results to correct the m-spectrum target tracking results :

[0042] ;

[0043] ;

[0044] ;

[0045] in represents the observation noise of spectrum n, is the update matrix from spectrum m to spectrum n at time j+1;

[0046] Step 9: Repeat steps 7 to 8 until all other modes’ observations are updated to the final state of mode m. and error metrics In; realize target tracking.

[0047] Furthermore, the correlation between the trajectories obtained from different spectral components is used as the optimization target, and the method of updating the observed velocity and observed acceleration in step 5 is optimized using the following loss function:

[0048] ;

[0049] in, represents the loss function, 、 It represents the position sequence composed of tracking positions at different moments under spectrum m and spectrum n, expressed in the form of a vector. Indicates spectrum m, spectrum n Tracking position at all times.

[0050] Furthermore, in step 1 、 、 It is of the following form, where the subscripts x and y represent the x-direction and the y-direction respectively;

[0051] , , .

[0052] Furthermore, the updating method of step 5 is:

[0053] ;

[0054] ;

[0055] And fit the probability distribution graph through the Gaussian probability model , and get its variance, recorded as observation noise .

[0056] This paper proposes a multispectral target tracking method based on multi-expert decision fusion. This method uses a spectrally independent visual target tracker to perceive data from different spectra and fuses the tracking results from each spectral image. This fusion method fully utilizes the characteristics of different modalities. Furthermore, as a decision fusion method, it avoids the complex and tedious cross-modal feature alignment and only requires the alignment transformation of a single target point. This improves the ultimate accuracy and efficiency of multispectral target tracking and its ability to resist interference from misaligned multispectral data. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flowchart of the visible light and infrared multispectral target tracking example of this patent. DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with examples and drawings. The exemplary embodiments of the present invention and their descriptions are only used to explain the present invention and are not intended to limit the present invention.

[0059] like Figure 1 As shown in the figure, unlike the traditional multispectral target tracking that uses feature fusion, this scheme tracks the target independently for each spectral component and uses the intra-spectral and inter-spectral tracking results to calibrate and verify each other, thereby improving the model's ability to resist interference from misaligned data and improving the accuracy and computational efficiency of multispectral target tracking. The specific steps are as follows:

[0060] Step 1: Build state variables ;

[0061] ;

[0062] Where s, v, and a are the position, velocity, and acceleration of the target in the image coordinate system, respectively, and are all vectors with the following form, where the subscripts x and y represent the x-direction and y-direction, respectively.

[0063] ;

[0064] ;

[0065] ;

[0066] For the tracking task, the initial value of s is obtained by the center position of the target selected by the user, and the initial values ​​of v and a are 0;

[0067] Step 2: For time j, the target tracking result is , where M represents the total number of spectra. Taking the visible-infrared multispectral camera as an example, M=2. Calculate the velocity and acceleration sequences of each modality based on the trajectory:

[0068] ;

[0069] ;

[0070] in 、 and are the target position, velocity and acceleration calculated for spectral component m at time j, is the time difference between two frames, which can be recorded as 1 in digital discrete systems;

[0071] Step 3: Use moving average to update the speed and acceleration to ensure that the speed does not cause large jitter:

[0072] ;

[0073] ;

[0074] in is the smoothing factor, which can be taken as 0.9;

[0075] Step 4: Construct the state transfer equation A and take :

[0076] ;

[0077] ;

[0078] ;

[0079] Using previous target tracking results and motion equations to predict the target state at the next moment :

[0080] ;

[0081] in , m={1,2}, subscript 1 represents infrared and 2 represents visible light;

[0082] Step 5: At time j+1, use MDNet to process the input spectral component m and output the spatial probability distribution map of the target position. , the position where the maximum value is located is the target position, recorded as the observation result , update the observed velocity, and observed acceleration:

[0083] ;

[0084] ;

[0085] And fit the probability distribution graph through the Gaussian probability model , and get its variance , denoted as observation noise, a fitting method is as follows,

[0086] ;

[0087] Step 6: Update the prediction error metric :

[0088] ;

[0089] in To predict noise, set it to 0.01I, where I is the unit matrix, . use Calculate the update matrix:

[0090] ;

[0091] Then, in the j+1 frame, the target tracking result obtained by correcting the information in the spectrum m is:

[0092] ;

[0093] Systematic error metrics for:

[0094] ;

[0095] Step 7: Construct the state transition matrix from spectrum m to spectrum n :

[0096] ;

[0097] ;

[0098] in 、 They represent the internal parameters of the spectrum n and spectrum m cameras respectively, Represents the rotation matrix from m to n and They represent the translation between the spectrum n and spectrum m cameras, 、 、 、 and It can be obtained by pre-calibration. For example, assuming that the focal length of the visible light camera and the infrared camera are both 15mm, the resolution of the infrared camera is 640×512, the resolution of the visible light camera is 1920×1080, the pixel size of the infrared camera is 12um, and the pixel size of the visible light camera is 2um, and there is no rotation between the infrared and visible light cameras, and the two camera lenses are well installed, then 、 for:

[0099] ;

[0100] ;

[0101] ;

[0102] Indicates the distance of the target from the observation point at time j+1. In a short period of time, z can be considered a fixed value and can be obtained by the position deviation of the target between the initial position of the visible light camera and the infrared camera.

[0103] Step 8: Use the n-spectrum observation results to correct the m-spectrum target tracking results:

[0104] ;

[0105] ;

[0106] ;

[0107] in The observation noise of spectrum n can be obtained by Gaussian fitting of the probability distribution graph of spectrum n. One fitting method is as follows:

[0108] ;

[0109] Step 9. Repeat steps 7 to 8 until all other modes’ observations are updated to the final state of mode m. and error metrics middle.

[0110] Step 10: Perform the same operation on all spectra and calculate the correlation between the trajectories obtained from different spectral components as the optimization target to optimize MDNet:

[0111] ;

[0112] The multispectral target tracking method of the present invention utilizes the constraints within and between the spectra of the multispectral camera and realizes mutual verification and calibration of target tracking results through decision fusion, which can effectively improve the accuracy of target tracking.

Claims

1. A target tracking method based on multispectral decision fusion, characterized in that: The method includes: Step 1: Create state variables : ； in 、 、 Respectively represent the position, velocity, and acceleration of the target in the image coordinate system, and are all vectors; Step 2: Calculate the velocity and acceleration sequences of each modal trajectory. For time j, the tracking results of the target under each spectrum are recorded as , where M represents the total number of spectra, and the update speed and acceleration ; Step 3: Update the factor, speed, and acceleration using the moving average update: ； ； in, represents the speed movement update factor, represents the speed moving average update factor, represents the acceleration movement update factor, represents the acceleration moving average update factor, is the smoothing factor; Step 4: Construct the state transfer equation A: ； in, ; ； represents the identity matrix, Indicates the time difference between two frames; Using previous target tracking results and motion equations to predict the target state at the next moment : ； in, , ; Step 5: At time j+1, use independent visual target tracking to process the image of spectral component m to obtain the spatial probability distribution map of the target position , the position of its maximum value is the observation position of target tracking, recorded as , update the observed velocity, and observed acceleration; Step 6: Update the prediction error metric : ； in, is the error metric of the target tracking result, To predict noise, the superscript T indicates transposition; use Calculate the update matrix : ； in, Represents a probability distribution graph The observation noise; then the j+1 frame, the target tracking result is obtained by using the information correction in the spectrum m. for: ； in, The tracking result of the predicted spectral component m at time j+1, Indicates the measurement results, is the update matrix of spectral component m at time j+1; Systematic error metrics for: ； Step 7: Construct the state observation matrix from m spectrum to n spectrum components : ； ； in, 、 as well as They represent the internal parameters of the spectrum n camera, the internal parameters of the spectrum m camera, and the rotation matrix from spectrum m to spectrum n, respectively. represents the inverse matrix, and They represent the x and y components of the translation vector between the spectrum n and spectrum m cameras, respectively. Indicates that time j+1 is the distance between the target and the observation point; Step 8: Use the n-spectrum observation results to correct the m-spectrum target tracking results : ； ； ； in represents the observation noise of spectrum n, is the update matrix from spectrum m to spectrum n at time j+1; Step 9: Repeat steps 7 to 8 until all other modes’ observations are updated to the final state of mode m. and error metrics In; realize target tracking.

2. The target tracking method of multispectral decision fusion according to claim 1, characterized in that: Speed ​​in step 2 and acceleration The update formula is as follows: ； ； in 、 and are the target position, velocity and acceleration calculated for spectral component m at time j, is the time difference between two frames.

3. The target tracking method of multispectral decision fusion according to claim 1, characterized in that: The correlation between the trajectories obtained from different spectral components is used as the optimization target. The method of updating the observed velocity and observed acceleration in step 5 is optimized using the following loss function: ； in, represents the loss function, 、 It represents the position sequence composed of tracking positions at different moments under spectrum m and spectrum n, expressed in the form of a vector. Indicates spectrum m, spectrum n Tracking position at all times.

4. The target tracking method of multispectral decision fusion according to claim 1, characterized in that: In step 1 、 , is in the following form, where the subscripts x and y represent the x-direction and the y-direction respectively; ， ， 。 5. The target tracking method of multispectral decision fusion according to claim 1, characterized in that: The updating method of step 5 is: ； ； And fit the probability distribution graph through the Gaussian probability model , and obtain its variance, which is recorded as observation noise .

Images