Method and system for spatial moving target track association management based on multiple frames of optical images

Abstract

The application provides a multi-frame optical image-based spatial moving target track correlation management method and system, which comprises the following steps: target pre-detection is performed on a single-frame optical image to obtain a pre-detection result; it is judged whether there is a fine detection result of a current frame; the detection result of the current frame is correlated with a track library formed by correlation based on two-dimensional distribution; a track of the fine detection result serving as an initial target point is deleted; it is judged and recorded that there is a target missing detection in multi-target track tracking; a single-frame optical image slice sequence is cropped according to missing detection information under the condition that a set fine detection algorithm intervention condition is met; the cropped slice sequence is input into the fine detection algorithm to obtain initial target position information; the initial target position information output by the fine detection algorithm is converted into coordinates in a corresponding single-frame optical image; the pre-detection and fine detection initial target position information in the same frame are de-redundant, and a missing detection start frame number is returned to be re-correlated.

Description

Technical Field

[0001] This invention relates to the field of spatial target recognition technology, and more specifically, to a method and system for managing the trajectory association of moving spatial targets based on multi-frame optical images. Background Technology

[0002] The detection and tracking of moving targets in weak points in space under complex backgrounds has always been a hot topic in aerospace research. The presence of cloud clutter and other background clutter, system noise and other inherent noise often submerges the target signal in noise, increasing the difficulty of weak target detection; at the same time, due to the long imaging distance and the lack of obvious features of the target on the image plane, it is impossible to identify space targets based on information such as brightness or shape.

[0003] Patent document CN103810499A (application number: CN201410062826.0) discloses an application of infrared weak target detection and tracking in complex backgrounds. Its features include the following steps: Step 1: Suppressing clutter and preserving the image's topological structure, constructing a biomimetic visual weighted entropy model based on spatial proximity and direction priority, transforming the image from grayscale mode to entropy mode; Step 2: Analyzing the motion state of weak targets with sudden or stable characteristics, employing nonlinear diffusion smoothing of entropy flow and adaptive local constraint criteria, constructing an adaptive entropy flow target motion estimation model that conforms to the maneuvering characteristics of weak targets, achieving an estimated velocity that approximates the true motion state of the weak target; Step 3: Studying a weak target tracking method based on multi-feature fusion and measurement, constructing a sequential filtering model for multi-feature fusion, achieving accurate, robust, and real-time identification of weak targets.

[0004] Patent document CN105184814A (application number: CN201510445611.1) discloses a moving target detection and tracking method based on multi-frame radar images. The method comprises the following steps: Step 1: When reading the first to fifth frames, average these five frames to obtain the first background image, and then perform dilation followed by erosion processing on the first background image; Step 2: When reading the sixth to tenth frames, average these five frames to obtain the second background image, and then perform dilation followed by erosion processing on the second background image; Step 3: When reading the eleventh to fifteenth frames, use the first background image for background subtraction. During this process, target merging based on connected regions is also performed to complete the tracking of the detected moving target.

[0005] Currently, when dealing with target recognition in real space-based environments, a common approach is to detect targets in a single frame and then perform multi-frame correlation. This method relies on the continuity of target motion and the consistency of trajectories in the sequence of images. However, in reality, target occlusion, intersection, separation, and loss often occur, resulting in multiple consecutive missed detections in the sequence of images. Therefore, continuous and stable detection and tracking cannot be achieved. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method and system for managing the trajectory association of spatial moving targets based on multi-frame optical images.

[0007] According to the present invention, a method and system for spatial moving target trajectory association management based on multi-frame optical images are provided, the scheme of which is as follows:

[0008] A method for spatial moving target trajectory association management based on multi-frame optical images, the method comprising:

[0009] Step S1: Perform target pre-detection on a single frame of optical image and obtain the pre-detection results;

[0010] Step S2: Determine whether there is a fine detection result for the current frame. If there is, read the fusion result of the pre-detection and fine detection. Otherwise, use the pre-detection result of the current frame to perform trajectory association and form a trajectory library.

[0011] Step S3: Perform data association between the detection results of the current frame and the already associated trajectory library based on two-dimensional allocation;

[0012] Step S4: Delete the trajectory of the fine detection result used as the initial target point;

[0013] Step S5: Determine and record any missed target detections in multi-target trajectory tracking;

[0014] Step S6: Based on the missed detection information that meets the set conditions for the intervention of the fine detection algorithm, crop the single-frame optical image slice sequence;

[0015] Step S7: Input the cropped slice sequence into the fine detection algorithm to obtain the initial target position information;

[0016] Step S8: Convert the initial target position information output by the fine detection algorithm into coordinates in the corresponding single-frame optical image;

[0017] Step S9: Remove redundancy from the initial target position information of the pre-detection and fine detection in the same frame, and return the missed detection start frame number to re-associate.

[0018] Preferably, step S1 requires pre-detection processing based on the current frame, including:

[0019] Step S1.1: Using three concentric windows of different sizes, scan the entire single-frame optical image in a step-by-step manner to process the image blocks and obtain a three-layer model for each image block;

[0020] Step S1.2: Based on the Gaussian function characteristics of small targets in a single frame optical image and the three-layer model of each image patch, intensity comparison is performed to enhance the potential target region;

[0021] Step S1.3: Using the mean difference of variance between the inner and outer layers in a single frame optical image as the standard for distinguishing between the target region and the clutter edge region, calculate the enhanced image of the target;

[0022] Step S1.4: Perform an erosion morphological operation on the enhanced image to further filter out false alarm targets;

[0023] Step S1.5: Using the connected component processing method, obtain the target connected components of the enhanced image and output the bounding box positions of the candidate targets.

[0024] Preferably, in step S2, redundancy removal is performed based on the target location information output by the pre-detection and fine-detection algorithms in the current frame. If the intersection-union ratio of the detection boxes in the pre-detection and fine-detection exceeds a set threshold, they are considered to overlap and represent the same target detection result. The target location information in the fine-detection is retained, and the corresponding result in the pre-detection is deleted. The calculation formula is as follows:

[0025]

[0026] In this context, A and B are the detection boxes for the overlapping regions of the sub-images.

[0027] Preferably, step S3 includes:

[0028] Step S3.1: Based on the motion model, predict the target state corresponding to the trajectory library at the previous moment, and determine the estimate that the current moment falls within each trajectory prediction gate according to the tracking threshold;

[0029] Step S3.2: Construct the allocation matrix, perform standard two-dimensional allocation association under the uniqueness constraint of the association between each trajectory and the target measurement within the predicted gate, and solve it using the auction algorithm;

[0030] Step S3.3: Traverse the trajectory of the target candidate detection point associated with the current time, and use the state as the estimate of the trajectory at the current time to realize the continuation of the trajectory;

[0031] Step S3.4: If all trajectories are associated with the estimated position, the association between the current trajectory and the measurement is completed; otherwise, the association matrix with respect to the estimated set is reconstructed, and the two-dimensional allocation is performed iteratively.

[0032] Preferably, the determination and recording of target misses in multi-target trajectory tracking in step S5 includes the following three situations:

[0033] 1) For the currently associated trajectory library, if a single trajectory stores a target point that appeared in the previous frame but did not appear in the current frame, start recording the missed detection information;

[0034] 2) For the currently associated trajectory library, if a single trajectory storage target point did not appear in the previous frame but appears in the current frame, determine whether the conditions for the notification fine detection algorithm to intervene are met;

[0035] 3) For the currently associated trajectory library, if a single trajectory storage target point does not appear in the previous frame and does not appear in the current frame, determine whether the conditions for notification fine detection algorithm intervention are met.

[0036] Preferably, in step S6, the condition for notifying the fine detection algorithm to intervene is that a single trajectory storage target point has five consecutive frames of missed detection; the single frame optical image provided by the fine detection algorithm is cropped with the missed detection location information as the center point coordinates and the width and height of the cropping dynamic block set with a threshold.

[0037] Preferably, the fine detection algorithm module in step S7 includes the following steps:

[0038] Step S7.1: Perform numerical normalization on the single-frame optical slice;

[0039] Step S7.2: Input the normalized single-frame optical image sequence into the ConvLSTM network for processing;

[0040] Step S7.3: Perform threshold segmentation to obtain a binarized image;

[0041] Step S7.4: Extract the suspected target region and generate connected component output of the target's sub-pixel position and the target's local raw data;

[0042] Step S7.5: Output the location information of the detected target.

[0043] Preferably, the coordinates of the target position information output by the fine detection algorithm in the corresponding single-frame optical image in step S8 are calculated according to the following formula:

[0044]

[0045] Where X represents the x-coordinate of the missed detection position provided by the slice cropping module, Y represents the y-coordinate of the missed detection position provided by the slice cropping module, x represents the x-coordinate of the suspected target position output by the fine detection module, y represents the y-coordinate of the suspected target position output by the fine detection module, w represents the width of the suspected target detection box output by the fine detection module, and h represents the height of the suspected target detection box output by the fine detection module.

[0046] Preferably, step S9 requires returning to the missed detection frame and re-associating the trajectory based on the fused target location information of the coarse detection and fine detection.

[0047] Secondly, a spatial moving target trajectory association management system based on multi-frame optical images is provided, the system comprising:

[0048] Module M1: Performs target pre-detection on a single frame of optical image and obtains the pre-detection results;

[0049] Module M2: Determines whether there is a fine detection result for the current frame. If so, it reads the fusion result of the pre-detection and fine detection. Otherwise, it uses the pre-detection result of the current frame to perform trajectory association and form a trajectory library.

[0050] Module M3: Performs two-dimensional data association between the detection results of the current frame and the already associated trajectory library;

[0051] Module M4: Deletes the trajectory of the fine detection result used as the initial target point;

[0052] Module M5: Identifies and records instances of missed target detection during multi-target trajectory tracking;

[0053] Module M6: Based on the missed detection information under the set conditions of fine detection algorithm intervention, crop the single-frame optical image slice sequence;

[0054] Module M7: Inputs the cropped slice sequence into the fine detection algorithm to obtain the initial target position information;

[0055] Module M8: Converts the initial target position information output by the fine detection algorithm into coordinates in the corresponding single-frame optical image;

[0056] Module M9: Removes redundancy from the initial target position information of the pre-detection and fine detection in the same frame, and returns the frame number where the missed detection started to re-associate it.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] This invention records the missed detection information of each trajectory and processes it according to three different missed detection situations. It can accurately perform local re-detection of the target loss position, adapt to situations where moving targets are continuously occluded, lost, or blurred in complex backgrounds, and can effectively solve the trajectory interruption problem caused by lost targets, thus achieving continuous and stable trajectory association.

[0059] Other beneficial effects of the present invention will be explained in detail through the introduction of specific technical features and technical solutions in specific embodiments. Those skilled in the art should be able to understand the beneficial technical effects brought about by these technical features and technical solutions through the introduction of these technical features and technical solutions. Attached Figure Description

[0060] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0061] Figure 1 This is a flowchart of the overall algorithm of the present invention;

[0062] Figure 2 Here is a flowchart of the pre-detection algorithm;

[0063] Figure 3 Here is a flowchart of the precision detection algorithm;

[0064] Figure 4 a-4d is a diagram showing the results of the single-frame missed detection trajectory correlation processing;

[0065] Figure 5 This is a schematic diagram illustrating the target detection process before precision detection intervention.

[0066] Figure 6 This is a schematic diagram illustrating the target detection process after precision detection intervention.

[0067] Figure 7 The result is a diagram showing the trajectory associated with the target. Detailed Implementation

[0068] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0069] This invention provides a method for spatial moving target trajectory association management based on multi-frame optical images, referring to... Figure 1 As shown, the method specifically includes:

[0070] Step S1: Perform target pre-detection on a single frame of optical image and obtain the pre-detection results.

[0071] In step S1, pre-detection processing is required based on the current frame. The algorithm flow is as follows: Figure 2 As shown, it includes:

[0072] Step S1.1: Using three concentric windows of different sizes, scan the entire single-frame optical image in a step-by-step manner to process the image blocks and obtain a three-layer model for each image block;

[0073] Step S1.2: Based on the Gaussian function characteristics of small targets in a single frame optical image and the three-layer model of each image patch, intensity comparison is performed to enhance the potential target region;

[0074] Step S1.3: Using the mean difference of variance between the inner and outer layers in a single frame optical image as the standard for distinguishing between the target region and the clutter edge region, calculate the enhanced image of the target;

[0075] Step S1.4: Perform an erosion morphological operation on the enhanced image to further filter out false alarm targets;

[0076] Step S1.5: Using the connected component processing method, obtain the target connected components of the enhanced image and output the bounding box positions of the candidate targets.

[0077] Step S2: Determine whether there is a fine detection result for the current frame. If there is, read the fusion result of the pre-detection and fine detection. Otherwise, use the pre-detection result of the current frame to perform trajectory association and form a trajectory library, which is the result of trajectory tracking.

[0078] In step S2, redundancy removal is performed based on the target location information output by the pre-detection and fine-detection algorithms in the current frame. If the intersection-union ratio of the detection boxes in the pre-detection and fine-detection exceeds a set threshold, they are considered to overlap and represent the same target detection result. The target location information in the fine-detection is retained, and the corresponding result in the pre-detection is deleted. The calculation formula is as follows:

[0079]

[0080] In this context, A and B are the detection boxes for the overlapping regions of the sub-images.

[0081] Step S3: Perform multi-target trajectory tracking, which involves data association between the detection results of the current frame and the already associated trajectory library based on two-dimensional allocation.

[0082] Step S3 includes:

[0083] Step S3.1: Based on the motion model, predict the target state corresponding to the trajectory library at the previous moment, and determine the estimate that the current moment falls within each trajectory prediction gate according to the tracking threshold;

[0084] Step S3.2: Construct the allocation matrix, perform standard two-dimensional allocation association under the uniqueness constraint of the association between each trajectory and the target measurement within the predicted gate, and solve it using the auction algorithm;

[0085] Step S3.3: Traverse the trajectory of the target candidate detection point associated with the current time, and use the state as the estimate of the trajectory at the current time to realize the continuation of the trajectory;

[0086] Step S3.4: If all trajectories are associated with the estimated position, the association between the current trajectory and the measurement is completed; otherwise, the association matrix with respect to the estimated set is reconstructed, and the two-dimensional allocation is performed iteratively.

[0087] Step S4: Deleting the fine detection results as the trajectory of the initial target point is to avoid the interference of points falsely detected by the fine detection on subsequent target tracking.

[0088] Step S5: Determine and record any missed targets in multi-target trajectory tracking.

[0089] In step S5, the missed target detections in multi-target trajectory tracking are identified and recorded, including the following three situations:

[0090] 1) For the currently associated trajectory library, if the target point of a single trajectory appears in the previous frame but does not appear in the current frame, the missed detection information will be recorded. The target point of a single trajectory is the location information of each track that has been tracked and formed.

[0091] 2) For the currently associated trajectory library, if a single trajectory storage target point did not appear in the previous frame but appears in the current frame, determine whether the conditions for the notification fine detection algorithm to intervene are met;

[0092] 3) For the currently associated trajectory library, if a single trajectory storage target point does not appear in the previous frame and does not appear in the current frame, determine whether the conditions for notification fine detection algorithm intervention are met.

[0093] Step S6: Based on the missed detection information that meets the set conditions for the intervention of the fine detection algorithm, crop the single-frame optical image slice sequence.

[0094] In step S6, the condition for notifying the fine detection algorithm to intervene is that there are 5 consecutive frames of missed detections of the target point in a single trajectory storage; the single-frame optical image provided by the fine detection algorithm is cropped with the missed detection location information as the center point coordinates and the width and height of the cropping dynamic block set with a threshold.

[0095] Step S7: Input the cropped slice sequence into the fine detection algorithm module of the system to obtain the initial target position information.

[0096] The fine detection algorithm module in step S7 has the following steps: Figure 3 As shown, it includes the following steps:

[0097] Step S7.1: Perform numerical normalization on the single-frame optical slice;

[0098] Step S7.2: Input the normalized single-frame optical image sequence into the ConvLSTM network for processing;

[0099] Step S7.3: Perform threshold segmentation to obtain a binarized image;

[0100] Step S7.4: Extract the suspected target region and generate connected component output of the target's sub-pixel position and the target's local raw data;

[0101] Step S7.5: Output the location information of the detected target.

[0102] Step S8: Convert the initial target position information output by the fine detection algorithm module into coordinates in the corresponding single-frame optical image.

[0103] In step S8, the coordinates of the target position information output by the fine detection algorithm in the corresponding single-frame optical image are calculated using the following formula:

[0104]

[0105] Where X represents the x-coordinate of the missed detection position provided by the slice cropping module, Y represents the y-coordinate of the missed detection position provided by the slice cropping module, x represents the x-coordinate of the suspected target position output by the fine detection module, y represents the y-coordinate of the suspected target position output by the fine detection module, w represents the width of the suspected target detection box output by the fine detection module, and h represents the height of the suspected target detection box output by the fine detection module.

[0106] Step S9: Remove redundancy from the initial target position information of the pre-detection and fine detection in the same frame, return the missed detection frame and re-associate the trajectory based on the fused target position information of the coarse detection and fine detection.

[0107] This invention also provides a spatial moving target trajectory association management system based on multi-frame optical images. The spatial moving target trajectory association management based on multi-frame optical images can be implemented by executing the process steps of the spatial moving target trajectory association management method based on multi-frame optical images. That is, those skilled in the art can understand the spatial moving target trajectory association management method based on multi-frame optical images as a preferred embodiment of the spatial moving target trajectory association management system based on multi-frame optical images. The system specifically includes the following:

[0108] Module M1: Performs target pre-detection on a single frame of optical image and obtains the pre-detection results.

[0109] Module M1 requires pre-detection processing based on the current frame, including:

[0110] Module M1.1: Uses three concentric windows of different sizes to scan the entire single-frame optical image in a step-by-step manner, extracts image blocks, and obtains a three-layer model for each image block;

[0111] Module M1.2: Enhances the potential target region by comparing the intensity of small targets in a single frame optical image and the three-layer model of each image patch;

[0112] Module M1.3: Using the mean difference of variance between the inner and outer layers in a single frame optical image as the standard to distinguish between the target region and the clutter edge region, the enhanced image of the target is calculated;

[0113] Module M1.4: Performs erosion morphological operations on the enhanced image to further filter out false alarm targets;

[0114] Module M1.5: Utilizes connected component processing methods to obtain the target connected regions of the enhanced image and outputs the bounding box positions of candidate targets.

[0115] Module M2: Determines whether there is a fine detection result for the current frame. If so, it reads the fusion result of the pre-detection and fine detection. Otherwise, it uses the pre-detection result of the current frame to perform trajectory association and forms a trajectory library, which is the result of trajectory tracking.

[0116] In module M2, redundancy removal is performed based on the target location information output by the pre-detection and fine-detection algorithms in the current frame. When the intersection-union ratio of the detection boxes in the pre-detection and fine-detection exceeds a set threshold, they are considered to overlap and represent the same target detection result. The target location information in the fine-detection is retained, and the corresponding result in the pre-detection is deleted. The calculation formula is as follows:

[0117]

[0118] In this context, A and B are the detection boxes for the overlapping regions of the sub-images.

[0119] Module M3: Performs multi-target trajectory tracking, which involves data association between the detection results of the current frame and the already associated trajectory library based on two-dimensional allocation.

[0120] This module M3 includes:

[0121] Module M3.1: Based on the motion model, predicts the target state corresponding to the trajectory library at the previous moment, and determines the estimate of the current moment falling within each trajectory prediction gate based on the tracking threshold;

[0122] Module M3.2: Constructs an allocation matrix, performs standard two-dimensional allocation association under the constraint of uniqueness of the association between each trajectory and the target measurement within the predicted gate, and solves the problem using an auction algorithm;

[0123] Module M3.3: Traverses the trajectory of the target candidate detection points associated with the current time, uses the state as an estimate of the trajectory at the current time, and realizes the continuation of the trajectory;

[0124] Module M3.4: If all trajectories are associated with the estimated position, the association between the current trajectory and the measurement is completed; otherwise, the association matrix with respect to the estimated set is reconstructed, and the two-dimensional assignment is performed iteratively.

[0125] Module M4: Deleting the trajectory of the initial target point from the fine detection result is to avoid interference from points falsely detected by the fine detection in subsequent target tracking.

[0126] Module M5: Determines and records instances of missed target detection during multi-target trajectory tracking.

[0127] This module M5 identifies and records target misses during multi-target trajectory tracking, including the following three scenarios:

[0128] 1) For the currently associated trajectory library, if the target point of a single trajectory appears in the previous frame but does not appear in the current frame, the missed detection information will be recorded. The target point of a single trajectory is the location information of each track that has been tracked and formed.

[0129] 2) For the currently associated trajectory library, if a single trajectory storage target point did not appear in the previous frame but appears in the current frame, determine whether the conditions for the notification fine detection algorithm to intervene are met;

[0130] 3) For the currently associated trajectory library, if a single trajectory storage target point does not appear in the previous frame and does not appear in the current frame, determine whether the conditions for notification fine detection algorithm intervention are met.

[0131] Module M6: Based on the missed detection information under the set fine detection algorithm intervention conditions, crop the single-frame optical image slice sequence.

[0132] In module M6, the condition for the fine detection algorithm to intervene is that there are 5 consecutive frames of missed detections of the target point in a single trajectory storage; the single-frame optical image provided by the fine detection algorithm is cropped with the missed detection location information as the center point coordinates and the width and height of the cropping dynamic block set with a threshold.

[0133] Module M7: Inputs the cropped slice sequence into the precision detection algorithm module in the system to obtain the initial target position information.

[0134] The fine detection algorithm module in module M7 includes the following steps:

[0135] Module M7.1: Performs numerical normalization on single-frame optical slices;

[0136] Module M7.2: Inputs the normalized single-frame optical image sequence into the ConvLSTM network for processing;

[0137] Module M7.3: Performs threshold segmentation to obtain a binarized image;

[0138] Module M7.4: Extracts suspected target regions and generates connected components, outputting the sub-pixel positions of the target and the original local data of the target;

[0139] Module M7.5: Outputs the location information of the detected target.

[0140] Module M8: Converts the initial target position information output by the fine detection algorithm module into coordinates in the corresponding single-frame optical image.

[0141] The M8 module calculates the coordinates of the target position information output by the fine detection algorithm in the corresponding single-frame optical image using the following formula:

[0142]

[0143] Where X represents the x-coordinate of the missed detection position provided by the slice cropping module, Y represents the y-coordinate of the missed detection position provided by the slice cropping module, x represents the x-coordinate of the suspected target position output by the fine detection module, y represents the y-coordinate of the suspected target position output by the fine detection module, w represents the width of the suspected target detection box output by the fine detection module, and h represents the height of the suspected target detection box output by the fine detection module.

[0144] Module M9: Removes redundancy from the initial target position information of the pre-detection and fine detection in the same frame, returns the missed detection frame, and re-associates the trajectory based on the fused target position information of the coarse detection and fine detection.

[0145] The present invention will now be described in more detail.

[0146] In real-world space-based environments, targets exhibit phenomena such as occlusion, intersection, separation, and loss. Traditional detection and tracking algorithms rely on the continuity of target motion and the consistency of trajectory within a sequence of images, making it impossible to achieve continuous and stable detection and tracking even with multiple consecutive missed detections. This invention provides a spatial moving target trajectory association management method based on multi-frame optical images, such as... Figure 1 The diagram shows a flowchart of a spatial moving target trajectory association management method based on multi-frame optical images provided in an embodiment. The specific steps are as follows:

[0147] Step S1: Perform target pre-detection on a single frame of optical image to obtain the location information of suspected target points;

[0148] Step S1 includes the following steps:

[0149] Step S1.1: Using three concentric windows of different sizes, scan the entire image in a step-by-step manner to process the image blocks and obtain a three-layer model for each image block.

[0150] Step S1.2: Based on the Gaussian function features of small targets in the image and the three-layer model of each image patch, intensity comparison is performed to enhance the potential target region. The calculation formula is as follows:

[0151]

[0152] Among them, D MM This indicates the strength difference between the inner and middle layers; P rem The map represents the possible region of the target; (x0, y0) represents the position of the center pixel of the image patch.

[0153] Step S1.3: Using the difference in the mean variance between the inner and outer layers of the image as the standard for distinguishing between the target region and the clutter edge region, calculate the enhanced image of the target according to the following formula:

[0154]

[0155]

[0156] VAED = V i -V e

[0157] Among them, D i V represents the size of the inner window; VAED is the difference between the strength variances of the inner and outer layers; V i and V e M represents the variance of the intensity of the inner and outer layers of the image patch, respectively. VARD This represents the mean of VAED;

[0158] Step S1.4: Perform erosion morphological operations on the enhanced image to further filter out false alarm targets;

[0159] Step S1.5: Using the connected component processing method, obtain the target connected components of the enhanced image and output the bounding box positions of the candidate targets.

[0160] Step S2: Redundancy removal is performed based on the target location information output by the pre-detection and fine-detection algorithms in the current frame. If the intersection-union ratio of the detection boxes in the pre-detection and fine-detection exceeds a set threshold, they are considered to overlap and represent the same target detection result. The target location information in the fine-detection is retained, and the corresponding result in the pre-detection is deleted. The calculation formula is as follows:

[0161]

[0162] In this context, A and B are the detection boxes for the overlapping regions of the sub-images.

[0163] Step S3: Perform two-dimensional data association between the detection results of the current frame and the already associated trajectory library, specifically including the following steps:

[0164] Step S3.1: Predict the target state corresponding to the trajectory library at the previous time step based on the motion model, and determine the estimate of whether the target falls within the prediction gate of each trajectory at the current time step according to the tracking threshold, based on the following formula:

[0165]

[0166]

[0167]

[0168] in, This represents the state prediction vector from time k-1 to time k; TN represents the state vector at time k-1; i represents the target number at time k-1; k-1 This represents the number of targets at time k-1; Let represent the estimation error covariance matrix from time k-1 to time k; F represents the posterior estimation error covariance matrix at time k-1; F is the state transition matrix; F T Represents the transpose of the state transition matrix; Q is the error covariance matrix corresponding to the process noise; c i，j This represents the predicted value of the target state; This represents the state vector at time k; σ represents the inverse of the error covariance matrix; T To track and predict the gate; j represents the target number at time k; This represents the number of targets at time k;

[0169] Step S3.2: Construct the allocation matrix, perform standard two-dimensional allocation association under the uniqueness constraint of the association between each trajectory and the target measurement within the predicted gate, and solve it using the auction algorithm;

[0170] Step S3.3: Traverse the trajectories of the target candidate detection points associated with the current time, and use this state as the estimate of the trajectory at the current time to achieve trajectory continuation, based on the following formula:

[0171]

[0172] in, To link The trajectory This represents the estimate of the trajectory at time k. This indicates the number of estimated trajectories associated with this two-dimensional assignment;

[0173] Step S3.4: If all trajectories are associated with the estimated position, the association between the current trajectory and the measurement is completed; otherwise, the association matrix with respect to the estimated set is reconstructed, and the two-dimensional allocation is performed iteratively.

[0174] Step S4: Delete the trajectory that uses the output of the fine detection algorithm as the trajectory starting point to avoid the interference of points falsely detected by the fine detection algorithm on subsequent target tracking;

[0175] Step S5: Traverse all trajectories associated with the current frame, determine if there are any missed detections, and store and process them.

[0176] Case 1: For the currently associated trajectory set, if the target point of a single trajectory appears in the previous frame but does not appear in the current frame, start recording the missed detection information. The target point of a single trajectory is the target point stored for each position information in the track that has been tracked.

[0177] Case 2: For the currently associated trajectory set, if a single trajectory stores a target point that did not appear in the previous frame but appears in the current frame, determine whether the conditions for the notification fine detection algorithm to intervene are met.

[0178] Case 3: For the currently associated trajectory set, if a single trajectory storage target point did not appear in the previous frame and does not appear in the current frame, determine whether the conditions for the notification fine detection algorithm to intervene are met.

[0179] Step S5: Traverse the recorded missed detection trajectories and store the missed detection trajectory information that meets the conditions of the notification fine detection algorithm module;

[0180] Step S6: Crop the image slice sequence based on the missed detection information that meets the set conditions for the intervention of the fine detection algorithm;

[0181] Step S7: Input the cropped slice sequence into the fine detection algorithm module to obtain the suspected target location information, which specifically includes the following steps:

[0182] Step S7.1: Perform numerical normalization on the slices of a single frame image;

[0183] Step S7.2: Input the normalized image sequence into the ConvLSTM network for processing;

[0184] Step S7.3: Perform threshold segmentation to obtain a binarized image, according to the following formula:

[0185] Th=μ+k t ×σ

[0186]

[0187] Where μ and σ are the mean and standard deviation of the gray level of the residual image I, respectively, and k t This is an empirical value, typically taken as k. t ≥3, T is the binary segmentation image.

[0188] Step S7.4: Extract the suspected target region and generate connected component output of the target's sub-pixel position and the target's local raw data.

[0189] Step S7.5: Output the location information of the detected target.

[0190] Step S8: Convert the target location information output by the fine detection algorithm into coordinates in the corresponding original image, according to the following formula:

[0191]

[0192] Where X represents the x-coordinate of the missed detection position provided by the slice cropping module, Y represents the y-coordinate of the missed detection position provided by the slice cropping module, x represents the x-coordinate of the suspected target position output by the fine detection module, y represents the y-coordinate of the suspected target position output by the fine detection module, w represents the width of the suspected target detection box output by the fine detection module, and h represents the height of the suspected target detection box output by the fine detection module.

[0193] Step S9: Remove redundancy from the pre-detection and fine-detection target location information in the same frame, and return the missed detection start frame number to re-associate them.

[0194] The effects of this invention can be further illustrated by the following simulation results.

[0195] 1. Simulation data:

[0196] The optical image sequence of the space target was used, and the data processing was completed through simulation on the PyCharm 2017 platform.

[0197] 2. Simulation content and results:

[0198] To verify the effectiveness of the spatial moving target trajectory association management method based on multi-frame optical images provided in this invention, a sequence of 2000 spatial target optical images was selected for experimental testing. Two sets of representative original experimental images and processing results at each stage are compared below. Figure 4 This diagram shows the results of the single-frame missed detection trajectory correlation processing. Figure 4 (a) shows a schematic diagram of the last detection before the target was missed in frame 267. Figure 4 (b) shows a schematic diagram of target misses in frame 268. Figure 4 (c) shows a schematic diagram of target detection again in frame 269. Figure 4 (d) shows the effect diagram of the target associated trajectory.

[0199] Figures 5-7 This diagram represents the results of correlation processing of missed detection trajectories across multiple consecutive frames. Figure 5 This diagram illustrates the target detection process before precision detection intervention. Figure 5 (a) shows a schematic diagram of the last detection before the target was missed in frame 370. Figure 5 (b) indicates that the target began to be missed in frame 371. Figure 5 (c) indicates that the target missed detection ended at frame 375.

[0200] Figure 6 This diagram illustrates the target detection process after precision detection intervention. Figure 6 (a) indicates the target fine detection in 371 frames. Figure 6 (b) indicates target precision detection at frame 373. Figure 6 (c) indicates target precision detection at 376 frames. Figure 7 This is a diagram showing the effect of the target's associated trajectory.

[0201] This invention provides a spatial moving target trajectory association management method and system based on multi-frame optical images. It can accurately re-detect the location of lost targets, adapt to situations where moving targets are continuously occluded, lost, or blurred in complex backgrounds, and effectively solve the trajectory interruption problem caused by lost targets, thus achieving continuous and stable trajectory association.

[0202] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0203] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for spatial moving target trajectory association management based on multi-frame optical images, characterized in that, include: Step S1: Perform target pre-detection on a single frame of optical image and obtain the pre-detection results; Step S2: Determine whether there is a fine detection result for the current frame. If there is, read the fusion result of the pre-detection and fine detection. Otherwise, use the pre-detection result of the current frame to perform trajectory association and form a trajectory library. Step S3: Perform multi-target trajectory tracking, which involves data association between the detection results of the current frame and the already associated trajectory library based on two-dimensional allocation; Step S4: Delete the trajectory of the fine detection result used as the initial target point; Step S5: Determine and record any missed target detections in multi-target trajectory tracking; Step S6: Based on the missed detection information that meets the set conditions for the intervention of the fine detection algorithm, crop the single-frame optical image slice sequence; Step S7: Input the cropped slice sequence into the fine detection algorithm to obtain the initial target position information; Step S8: Convert the initial target position information output by the fine detection algorithm into coordinates in the corresponding single-frame optical image; Step S9: Remove redundancy from the initial target position information of the pre-detection and fine detection in the same frame, and return the missed detection start frame number to re-associate; Step S1 requires pre-detection processing based on the current frame, including: Step S1.1: Using three concentric windows of different sizes, scan the entire single-frame optical image in a step-by-step manner to process the image blocks and obtain a three-layer model for each image block; Step S1.2: Based on the Gaussian function characteristics of small targets in a single frame optical image and the three-layer model of each image patch, intensity comparison is performed to enhance the potential target region; Step S1.3: Using the mean difference of variance between the inner and outer layers in a single frame optical image as the standard for distinguishing between the target region and the clutter edge region, calculate the enhanced image of the target; Step S1.4: Perform an erosion morphological operation on the enhanced image to further filter out false alarm targets; Step S1.5: Using the connected component processing method, obtain the target connected components of the enhanced image and output the bounding box positions of the candidate targets; The fine detection algorithm in step S7 includes the following steps: Step S7.1: Perform numerical normalization on the single-frame optical slice; Step S7.2: Input the normalized single-frame optical image sequence into the ConvLSTM network for processing; Step S7.3: Perform threshold segmentation to obtain a binarized image; Step S7.4: Extract the suspected target region and generate connected component output of the target's sub-pixel position and the target's local raw data; Step S7.5: Output the location information of the detected target.

2. The spatial moving target trajectory association management method based on multi-frame optical images according to claim 1, characterized in that, In step S2, redundancy needs to be removed based on the target location information output by the pre-detection and fine-detection algorithms in the current frame. If the intersection-union ratio of the detection boxes in the pre-detection and fine-detection exceeds a set threshold, they are considered to overlap and represent the same target detection result. The target location information in the fine-detection is retained, and the corresponding result in the pre-detection is deleted. The calculation formula is as follows: in, and These are the detection boxes for the overlapping areas of the sub-images.

3. The spatial moving target trajectory association management method based on multi-frame optical images according to claim 1, characterized in that, Step S3 includes: Step S3.1: Predict the target state corresponding to the trajectory library at the previous time step based on the motion model, and determine the estimate of whether the target falls within the prediction gate of each trajectory at the current time step based on the tracking threshold, which is the threshold for determining whether there is a correlation; according to the following formula: ； in, express Always The state prediction vector at time t; express The state vector at any given time; express The target number at that moment; express Number of targets at any given time; express Time's up The covariance matrix of the estimation error at time; express The posterior estimation error covariance matrix at time t; This is the state transition matrix; This represents the transpose of the state transition matrix; The error covariance matrix corresponding to the process noise; This represents the predicted value of the target state; express The state vector at any given time; Represents the inverse of the error covariance matrix; To track and predict the gate; express The target number at that moment; express Number of targets at any given time; Step S3.2: Construct the allocation matrix, perform standard two-dimensional allocation association under the uniqueness constraint of the association between each trajectory and the target measurement within the predicted gate, and solve it using the auction algorithm; Step S3.3: Traverse the trajectory of the target candidate detection point associated with the current time, and use this state as the estimate of the trajectory at the current time to realize the continuation of the trajectory, based on the following formula: in, To link The trajectory This indicates that the trajectory is in Time estimation, This indicates the number of estimated trajectories associated with this two-dimensional assignment; Step S3.4: If all trajectories are associated with the estimated position, the association between the current trajectory and the measurement is completed; otherwise, the association matrix with respect to the estimated set is reconstructed, and the two-dimensional allocation is performed iteratively.

4. The spatial moving target trajectory association management method based on multi-frame optical images according to claim 1, characterized in that, In step S5, the detection and recording of target omissions in multi-target trajectory tracking includes the following three situations: 1) For the currently associated trajectory library, if the target point of a single trajectory appears in the previous frame but does not appear in the current frame, the missed detection information will be recorded. The target point of a single trajectory is the location information of each track that has been tracked and formed, which is the stored target point. 2) For the currently associated trajectory library, if a single trajectory storage target point did not appear in the previous frame but appears in the current frame, determine whether the conditions for notification fine detection algorithm intervention are met; 3) For the currently associated trajectory library, if a single trajectory storage target point does not appear in the previous frame and does not appear in the current frame, determine whether the conditions for notification fine detection algorithm intervention are met.

5. The spatial moving target trajectory association management method based on multi-frame optical images according to claim 1, characterized in that, In step S6, the condition for notifying the fine detection algorithm to intervene is that there are 5 consecutive frames of missed detections of the target point in a single trajectory storage; the single-frame optical image provided by the fine detection algorithm is cropped with the missed detection location information as the center point coordinates and the width and height of the cropping dynamic block set with a threshold.

6. The spatial moving target trajectory association management method based on multi-frame optical images according to claim 1, characterized in that, In step S8, the coordinates of the target position information output by the fine detection algorithm in the corresponding single-frame optical image are calculated according to the following formula: in, This indicates the x-coordinate of the provided missed detection location. This indicates the vertical coordinate of the provided missed detection location. This represents the x-coordinate of the initial target position output by the precision detection algorithm. This represents the initial target position ordinate output by the precision detection algorithm. This represents the width of the initial target detection box output by the fine detection algorithm. This represents the height of the initial target detection box output by the fine detection algorithm.

7. The spatial moving target trajectory association management method based on multi-frame optical images according to claim 1, characterized in that, Step S9 requires returning to the missed detection frame and re-associating the trajectory based on the fused target location information of the coarse and fine detection.

8. A spatial moving target trajectory association and management system based on multi-frame optical images, characterized in that, include: Module M1: Performs target pre-detection on a single frame of optical image and obtains the pre-detection results; Module M2: Determines whether there is a fine detection result for the current frame. If so, it reads the fusion result of the pre-detection and fine detection. Otherwise, it uses the pre-detection result of the current frame to perform trajectory association and form a trajectory library. Module M3: Performs multi-target trajectory tracking, which involves data association between the detection results of the current frame and the already associated trajectory library based on two-dimensional allocation; Module M4: Deletes the trajectory of the fine detection result used as the initial target point; Module M5: Identifies and records instances of missed target detection during multi-target trajectory tracking; Module M6: Based on the missed detection information under the set conditions of fine detection algorithm intervention, crop the single-frame optical image slice sequence; Module M7: Inputs the cropped slice sequence into the fine detection algorithm to obtain the initial target position information; Module M8: Converts the initial target position information output by the fine detection algorithm into coordinates in the corresponding single-frame optical image; Module M9: Removes redundancy from the initial target position information of pre-detection and fine detection in the same frame, and returns the frame number where the missed detection started to re-associate it; Module M1 needs to perform pre-detection processing based on the current frame, including: Module M1.1: Uses three concentric windows of different sizes to scan the entire single-frame optical image in a step-by-step manner, extracts image blocks, and obtains a three-layer model for each image block; Module M1.2: Enhances the potential target region by comparing the intensity of small targets in a single frame optical image and the three-layer model of each image patch; Module M1.3: Using the mean difference of variance between the inner and outer layers in a single frame optical image as the standard to distinguish between the target region and the clutter edge region, the enhanced image of the target is calculated; Module M1.4: Performs erosion morphological operations on the enhanced image to further filter out false alarm targets; Module M1.5: Utilizes connected component processing methods to obtain the target connected components of the enhanced image and outputs the bounding box positions of candidate targets; The fine detection algorithm module in module M7 includes the following steps: Module M7.1: Performs numerical normalization on single-frame optical slices; Module M7.2: Inputs the normalized single-frame optical image sequence into the ConvLSTM network for processing; Module M7.3: Performs threshold segmentation to obtain a binarized image; Module M7.4: Extracts suspected target regions and generates connected components, outputting the sub-pixel positions of the target and the original local data of the target; Module M7.5: Outputs the location information of the detected target.

Images