Deep learning-based automatic target detection and tracking method and system

Abstract

The application is suitable for the field of computer vision and artificial intelligence, and provides an automatic target detection and tracking method and system based on deep learning, which comprises the following steps: acquiring a video stream and a scene structure graph, extracting a target position bounding box, a category label and an appearance feature vector frame by frame, calculating a behavior parameter and a priority score; generating a detection frame interval and a tracking algorithm complexity parameter according to the priority score, and initializing a special tracker to realize automatic tracking; when the target is lost, generating a space-time probability graph based on historical data and the scene structure graph; determining a search parameter according to the priority score, and performing directional detection on the screened space-time region; after the target is found back, associating the identity and reassigning the parameter to restore the tracking. Through differentiated resource allocation, accurate loss prediction and directional recovery, the application takes into account the tracking accuracy, efficiency and robustness, and effectively solves the problems of target tracking interruption and resource waste in a complex scene.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and artificial intelligence, and particularly relates to a method and system for automatic target detection and tracking based on deep learning. Background Technology

[0002] With the rapid development of deep learning technology, target detection and tracking technologies in the field of computer vision have been widely applied in various scenarios such as security monitoring, intelligent transportation, and industrial inspection, becoming a core technological support for ensuring scenario security and improving management efficiency. Currently, deep learning-based target detection networks and tracking algorithms are constantly iterating, achieving significant progress in detection accuracy and tracking response speed. They can realize the simultaneous identification and continuous tracking of multiple targets, meeting the basic needs of dynamic target monitoring in various scenarios.

[0003] The core technical problem with existing technologies is that the allocation of tracking resources lacks specificity, resulting in insufficient tracking accuracy for high-value targets and excessive computational resources being consumed by low-value targets. At the same time, the efficiency of retrieving lost targets is low, and tracking interruptions and identity confusion are prone to occur. The robustness, continuity, and resource utilization efficiency of the overall tracking system are difficult to balance, and it cannot meet the needs of high-precision and high-efficiency target tracking in complex scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide an automatic target detection and tracking method and system based on deep learning, which aims to solve the technical problems existing in the prior art as identified in the background art.

[0005] This invention is implemented as follows: an automatic target detection and tracking method based on deep learning, the method comprising:

[0006] The video stream and scene structure map are acquired, and the video frame sequence is processed frame by frame. The location bounding box and category label of the detected target in the frame are extracted by the detection network. The target appearance feature vector is generated simultaneously, and the target behavior parameters and priority scores are calculated.

[0007] Based on the priority score, a detection frame interval parameter and a tracking algorithm complexity parameter are generated for each detection target. The detection frequency of the detection target is controlled based on the detection frame interval parameter, and a dedicated tracker is initialized for the detection target based on the tracking algorithm complexity parameter to achieve automatic tracking.

[0008] When the dedicated tracker determines that the target is lost, it generates a spatiotemporal probability map through a prediction network based on the target’s behavior parameters, appearance feature vector and scene structure diagram before the target was lost. The pixels in the map contain the probability value of the lost target appearing at the corresponding pixel in the future.

[0009] Based on the priority score and spatiotemporal probability map of the lost target, the probability threshold, the temporary detection confidence threshold, and the retrieval frame range are determined. Spatiotemporal regions with probability values ​​higher than the probability threshold are selected from the spatiotemporal probability map. In the subsequent video frames within the retrieval frame range, the image regions corresponding to the spatiotemporal regions are targeted for detection using the temporary detection confidence threshold.

[0010] If the lost target is rediscovered by directional detection, the lost target and the detected target are associated as the same target, and the detection frame interval and tracker are reallocated according to the priority score to resume tracking.

[0011] As a further aspect of the present invention, the step of generating the target appearance feature vector and calculating the target behavior parameters and priority score specifically includes:

[0012] Real-time video stream data is received from at least one image sensor, and the real-time video stream data is decoded into a sequence of video frames arranged in timestamp order. A predefined scene structure graph is loaded synchronously, and the scene structure graph stores the connection relationship and semantic attributes of each region node in the scene in a graph data structure.

[0013] The current frame in the video frame sequence is input into the target detection neural network, which outputs the bounding boxes of all detected targets in the current frame.

[0014] Based on the current frame and the location bounding box of each detected target, a fixed-dimensional floating-point vector is output for each detected target as the target appearance feature vector;

[0015] Based on the center coordinates of the bounding box of the same detected target in the current frame and the previous frame, the instantaneous displacement of the detected target on the two-dimensional image plane is calculated to obtain the target behavior parameters, which include instantaneous velocity and motion direction angle.

[0016] Based on the target's appearance feature vector and target behavior parameters, and combined with the semantic information of the target's current position in the scene structure diagram, a priority score representing the target's tracking value is calculated for each target.

[0017] As a further aspect of the present invention, the step of generating detection frame interval parameters and tracking algorithm complexity parameters for each target based on the priority score specifically includes:

[0018] Based on the comprehensive priority score of each detected target, the detection frame interval parameter and the tracking algorithm complexity parameter are calculated.

[0019] When processing each video frame sequence, the frame interval since the last detection of each target is checked. Only when the interval reaches the detection frame interval parameter stored in its state record is a new round of target detection performed on the image region containing the target to update the location bounding box.

[0020] Based on the tracking algorithm complexity parameter of each detected target, a tracking algorithm instance with the corresponding complexity is selected as the dedicated tracker for the detected target, and the target appearance feature vector and the updated position bounding box are initialized as the internal state of the dedicated tracker.

[0021] In each subsequent frame of the video frame sequence, a dedicated tracker corresponding to each detected target is invoked. The current frame image is input, and the dedicated tracker outputs the predicted tracking bounding box, thereby realizing automatic target tracking of the detected target between frames.

[0022] As a further aspect of the present invention, the step of generating a spatiotemporal probability graph through a prediction network specifically includes:

[0023] If the tracking bounding box continuously predicted by the dedicated tracker corresponding to a detected target is not confirmed by the new round of target detection within a preset number of frames, the detected target is determined to be lost and marked as a lost target.

[0024] Extract the historical data of the lost target in the last N frames before it was determined to be lost. The historical data includes N sets of target behavior parameters and N target appearance feature vectors, forming a sequence of the behavior and appearance of the lost target.

[0025] The behavior and appearance sequence and the scene structure map are input into the prediction neural network. The behavior and appearance sequence are encoded and combined with the passage constraints described by the scene structure map to decode the possible location distribution of the lost target on the image plane at the next T time points.

[0026] For each of the T future time points, output a probability map of the same size as the input image. The value of each pixel on the probability map represents the probability that the missing detected target appears at that pixel position at that time. Then, stack the T probability maps in chronological order to form a spatiotemporal probability map.

[0027] As a further aspect of the present invention, the step of performing directional detection on the image region corresponding to the spatiotemporal region using a temporary detection confidence threshold specifically includes:

[0028] Based on the final calculated comprehensive priority score of the lost target, calculate the probability threshold and planned retrieval frame range for the lost target;

[0029] Based on the probability threshold, the spatiotemporal probability map is segmented by threshold. Each pixel in the spatiotemporal probability map is traversed. If the probability value stored in any pixel is not less than the probability threshold, the pixel is marked as a valid point, and the connected region formed by all valid points is marked as the selected spatiotemporal region.

[0030] The current video frame is determined as the starting frame for retrieval, and the ending frame for retrieval is determined according to the planned range of the number of frames to be retrieved. Within this specified range of the number of frames from the starting frame to the ending frame, for each retrieval frame, the coordinates of all pixels falling within the selected spatiotemporal region are found based on the slice of the spatiotemporal probability map at the corresponding time.

[0031] The target detection neural network is invoked only within the local area enclosed by the pixel coordinates on the current retrieval frame image to perform directional detection and determine whether the missing target exists.

[0032] As a further aspect of the present invention, the reallocation of the detection frame interval and the tracker based on priority scores specifically includes:

[0033] If, in the detection results output by the directional detection within the selected spatiotemporal region, there is a target's location bounding box whose similarity to the historical target appearance feature vector of the lost target exceeds a preset re-identification threshold, then it is determined that the lost target has been detected again.

[0034] The identifier ID of the newly detected lost target is updated to the original identifier ID of the lost target. Based on the new location bounding box of the lost target, a new target appearance feature vector and new target behavior parameters are calculated.

[0035] Based on the comprehensive priority score of the original lost target, the new target appearance feature vector, and the new target behavior parameters, the detection frame interval parameters and tracking algorithm complexity parameters are recalculated, and a dedicated tracker is initialized to resume automatic tracking in subsequent frames.

[0036] Another object of the present invention is to provide an automatic target detection and tracking system based on deep learning, the system comprising:

[0037] The video stream and scene structure map acquisition module is used to acquire the video stream and scene structure map, process the video frame sequence frame by frame, extract the location bounding box and category label of the detected target in the frame through the detection network, generate the target appearance feature vector simultaneously, and calculate the target behavior parameters and priority score;

[0038] The target parameter generation module is used to generate a detection frame interval parameter and a tracking algorithm complexity parameter for each target based on the priority score, control the detection frequency of the target based on the detection frame interval parameter, and initialize a dedicated tracker for the target based on the tracking algorithm complexity parameter to achieve automatic tracking.

[0039] The target loss processing module is used to generate a spatiotemporal probability map by a prediction network when the dedicated tracker determines that the target is lost, based on the behavior parameters, appearance feature vector and scene structure diagram of the lost target before the loss. The pixels in the map contain the probability value of the lost target appearing at the corresponding pixel in the future.

[0040] The directional detection module is used to determine the probability threshold, temporary detection confidence threshold and retrieval frame range based on the priority score and spatiotemporal probability map of the lost target. It selects spatiotemporal regions with probability values ​​higher than the probability threshold from the spatiotemporal probability map, and performs directional detection on the image regions corresponding to the spatiotemporal regions in the subsequent video frames within the retrieval frame range using the temporary detection confidence threshold.

[0041] The tracking recovery module is used to associate the lost target with the detected target as the same target if the lost target is rediscovered by directional detection, and to reallocate the detection frame interval and tracker according to the priority score to resume tracking.

[0042] The beneficial effects of this invention are:

[0043] This invention constructs a full-process tracking mechanism, achieving precision, efficiency, and robustness in target detection and tracking. Priority scores are calculated based on target appearance features, behavioral parameters, and scene semantic information. The detection frame interval and tracking algorithm complexity are dynamically allocated, achieving optimal configuration of tracking resources. This ensures tracking accuracy for high-value targets while reducing overall computational overhead. For target loss scenarios, a spatiotemporal probability map is generated by fusing historical behavior sequences, appearance features, and scene constraints. Combined with a targeted detection mechanism, the search range is narrowed, significantly improving the efficiency and accuracy of finding lost targets. Simultaneously, identity feature matching and tracking parameter reinitialization ensure the continuity and consistency of target tracking, effectively solving problems such as unreasonable resource allocation, difficulty in finding lost targets, and insufficient tracking stability in existing technologies. This significantly improves the system's adaptability and practical value in complex scenarios, making it widely applicable to various application scenarios with high requirements for dynamic target monitoring. Attached Figure Description

[0044] Figure 1 A flowchart illustrating the deep learning-based automatic target detection and tracking method provided in this embodiment of the invention;

[0045] Figure 2 A flowchart for generating target appearance feature vectors and calculating target behavior parameters and priority scores provided in embodiments of the present invention;

[0046] Figure 3 A flowchart for generating detection frame interval parameters and tracking algorithm complexity parameters for each target based on the priority score, provided in an embodiment of the present invention;

[0047] Figure 4 A flowchart for generating a spatiotemporal probability graph through a prediction network, provided in an embodiment of the present invention;

[0048] Figure 5 This is a flowchart of a method for directional detection of image regions corresponding to spatiotemporal regions using a temporary detection confidence threshold, provided by an embodiment of the present invention.

[0049] Figure 6 A flowchart for reallocating detection frame intervals and trackers based on priority scores, provided in an embodiment of the present invention;

[0050] Figure 7 This is a structural block diagram of an automatic target detection and tracking system based on deep learning provided in an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0052] Figure 1 A flowchart of the deep learning-based automatic target detection and tracking method provided in the embodiments of the present invention is shown below. Figure 1 As shown, the method includes:

[0053] S100: Obtain the video stream and scene structure map, process the video frame sequence frame by frame, extract the location bounding box and category label of the detected target in the frame through the detection network, generate the target appearance feature vector simultaneously, and calculate the target behavior parameters and priority score;

[0054] The video frame sequence undergoes frame-by-frame refinement. Each frame image is input into the target detection neural network. Through the collaborative work of the network backbone layer and feature fusion layer, the location bounding boxes and category labels of all detected targets within the frame are accurately extracted, clarifying the spatial localization and attribute classification of the targets. Simultaneously, target appearance feature vectors are generated. Through feature extraction, dimensional standardization, and normalization of the target bounding box regions, a fixed-dimensional floating-point vector with identity recognition is formed.

[0055] Simultaneously, by combining the bounding box center coordinates of the same target in the current frame and the previous frame, the instantaneous displacement of the target in the two-dimensional image plane is calculated, thereby deriving behavioral parameters such as instantaneous velocity and motion direction angle, intuitively reflecting the target's motion state and trend. By comprehensively considering the discriminability of the target's appearance feature vector, the dynamic characteristics of the behavioral parameters, and the semantic priority of the target's location region in the scene structure map, a priority score representing the target's tracking value is calculated, completing the comprehensive value assessment of the target.

[0056] S200: Generate a detection frame interval parameter and a tracking algorithm complexity parameter for each detection target based on the priority score; control the detection frequency of the detection target based on the detection frame interval parameter; and initialize a dedicated tracker for the detection target based on the tracking algorithm complexity parameter to achieve automatic tracking.

[0057] Based on the comprehensive priority score of each detected target, the detection frame interval parameter and the tracking algorithm complexity parameter are accurately calculated, forming a resource allocation strategy that matches the value of the target. The detection frame interval parameter directly determines the time span between two target detections, while the tracking algorithm complexity parameter corresponds to tracking algorithm instances with different precision and computational overhead.

[0058] Based on these parameters, the system initializes a dedicated tracker for each detected target, imports the target's appearance feature vector and the updated location bounding box into the tracker, and builds a dedicated tracking model to ensure that the tracker is deeply adapted to the target's appearance and location characteristics.

[0059] During subsequent video frame sequence processing, the last detection time of each target is continuously recorded, and the frame interval is checked in real time. Only when the interval reaches the preset detection frame interval parameter is a new round of target detection started for the image region where the target is located, and the position bounding box is updated to correct possible tracking deviations. When the interval requirement is not met, there is no need to repeatedly call the detection network. Instead, the dedicated tracker corresponding to each target is directly called. The current frame image is input, and the tracker outputs an accurate tracking bounding box through inter-frame local feature matching and motion trend prediction, so as to achieve seamless tracking of the detected target between consecutive frames.

[0060] S300 When the dedicated tracker determines that the target is lost, it generates a spatiotemporal probability map through a prediction network based on the target’s behavior parameters, appearance feature vector and scene structure diagram before the target was lost. The pixels in the map contain the probability value of the lost target appearing at the corresponding pixel in the future.

[0061] The output of each dedicated tracker is continuously monitored. When the tracking bounding box predicted by the tracker fails to be confirmed by the new round of target detection within a preset number of frames, the target is determined to be lost and marked. Unconfirmed tracking bounding boxes may indicate tracking drift, target occlusion, drastic changes in pose, or temporary removal from the field of view. Continuing to rely on the tracker in this case will lead to error accumulation. Marking lost targets is a necessary measure to avoid invalid tracking.

[0062] Historical data from the last N frames before the target was determined to be lost were extracted. This data encompasses multiple sets of target behavior parameters and multiple appearance feature vectors, forming a complete behavior and appearance sequence. The behavior parameters carry information about the target's movement trend before it was lost, while the appearance feature vectors retain the target's identity-identifying features, providing core data support for subsequent predictions. This sequence data is then integrated with the scene structure. Figure 1 Using the same input prediction neural network, the behavior and appearance sequences are encoded to extract the target's motion patterns and identity features. Then, combined with the regional connectivity and passage constraints contained in the scene structure graph, the possible position distribution of the target on the image plane at the next T time points is decoded and generated.

[0063] Based on this, a probability map of the same size as the input image is output for each future moment. The value of each pixel intuitively reflects the probability that the target will appear at that location at that moment. Finally, the T probability maps are superimposed in chronological order to form a spatiotemporal probability map that combines temporal continuity and spatial orientation, fully presenting the possible spatiotemporal trajectory range after the target is lost.

[0064] S400 determines the probability threshold, temporary detection confidence threshold, and retrieval frame range based on the priority score and spatiotemporal probability map of the lost target. It then selects spatiotemporal regions with probability values ​​higher than the probability threshold from the spatiotemporal probability map and performs directional detection on the image regions corresponding to the spatiotemporal regions in the subsequent video frames within the retrieval frame range using the temporary detection confidence threshold.

[0065] Based on the comprehensive priority score of lost targets, the system dynamically calculates the appropriate probability threshold, temporary detection confidence threshold, and planned retrieval frame range. The priority score directly determines the level of retrieval resource investment, allowing high-value targets to receive stricter screening criteria and more sufficient retrieval time, while low-value targets are subject to relatively lenient thresholds and reasonable retrieval ranges, thus achieving personalized adaptation of the retrieval strategy.

[0066] Based on the calculated probability threshold, the spatiotemporal probability map is segmented using a thresholding process. All pixels in the map are traversed, and valid points with probability values ​​meeting the requirements are selected. These valid points converge to form a connected spatiotemporal region for filtering, focusing the potential range of the target from the entire image to a local high-probability area, significantly narrowing the detection range. Subsequently, the temporal boundary of the search is determined. Starting from the current video frame and considering the planned number of search frames, the ending frame is defined. Within this time interval, for each search frame, a slice of the spatiotemporal probability map corresponding to that moment is extracted, precisely locating the pixel coordinates falling within the filtered spatiotemporal region, thereby identifying the local image region to be detected in the current frame.

[0067] During the detection process, the target detection neural network is invoked only in the local area to perform targeted detection. At the same time, a temporary detection confidence threshold is used as the judgment criterion to determine whether the lost target exists. Through the coordination of priority scores and spatiotemporal probability maps, detection resources are concentrated on the spatiotemporal range where the target is most likely to appear.

[0068] In practical applications, when a high-priority target is lost, the system will set a higher probability threshold to filter out more reliable areas, extend the retrieval frame count, and perform targeted detection in key areas to improve the recovery probability.

[0069] S500: If the lost target is rediscovered by directional detection, the lost target and the detected target are associated as the same target, and the detection frame interval and tracker are reallocated according to the priority score to resume tracking.

[0070] After the targeted detection outputs detection results within the selected spatiotemporal region, the system initiates a target identity verification mechanism. This mechanism calculates the similarity between the appearance feature vector of the newly detected target and the historical appearance feature vector of the lost target, using this similarity as the core basis for identity matching. When the similarity reaches a preset re-identification threshold, the identity binding between the lost target and the newly detected target is completed, ensuring that the target being tracked is the original lost target and not other interfering targets. After identity verification, the identifier ID of the newly detected target is updated to the identifier ID of the original lost target, ensuring the continuity of the target tracking link and preventing identity breakage due to target loss.

[0071] Based on the newly detected target's bounding box, the target's appearance feature vector and behavioral parameters are recalculated to accommodate potential appearance changes or motion state adjustments during the period of target loss. The overall priority score of the original lost target is fused with the newly generated appearance feature vector and behavioral parameters to recalculate the detection frame interval parameters and tracking algorithm complexity parameters. This ensures that tracking resource allocation both maintains the original target's value weight and adapts to its current state. Finally, the dedicated tracker is initialized with the new parameters, and the updated bounding box and appearance feature vector are imported into the tracker's internal state. Automatic tracking of the target is resumed in subsequent video frames, bringing the entire detection and tracking process back to stable operation.

[0072] like Figure 2 As shown, the generation of the target appearance feature vector and the calculation of the target behavior parameters and priority scores specifically include:

[0073] S110, real-time video stream data is received from at least one image sensor, and the real-time video stream data is decoded into a sequence of video frames arranged in time stamp order, and a predefined scene structure graph is loaded synchronously. The scene structure graph stores the connection relationship and semantic attributes of each region node in the scene in a graph data structure.

[0074] The scene structure diagram is a scene information model organized using a graph data structure. It describes the spatial topology and semantic attributes of the monitoring scene. Nodes represent independent regions in the scene; edges represent the connection relationships between regions; and semantic attributes represent the semantic information attached to each node / edge.

[0075] S120, input the current frame in the video frame sequence into the target detection neural network, and output the bounding boxes of the positions of all detected targets in the current frame;

[0076] S130, based on the current frame and the position bounding box of each detected target, outputs a fixed-dimensional floating-point vector as the target appearance feature vector for each detected target;

[0077] 1. Perform preprocessing such as standardization and size adjustment on the decoded current frame.

[0078] 2. Input the preprocessed frame into the target detection neural network to extract image features and output the target's location bounding box (format: (i.e., the coordinates of the top left and bottom right corners), category labels, and detection confidence.

[0079] 3. The bounding box region for each detected target:

[0080] The feature map of the region is extracted from the Backbone / Neck layer of the detection network. The feature map is converted into a fixed-dimensional vector by RoI pooling / adaptive average pooling. The vector is then L2 normalized to obtain the final target appearance feature vector.

[0081] S140, based on the center coordinates of the bounding box of the same detected target in the current frame and the previous frame, calculate the instantaneous displacement of the detected target on the two-dimensional image plane to obtain the target behavior parameters, which include instantaneous velocity and motion direction angle;

[0082] 1. Instantaneous displacement

[0083] Let the current frame time be... The center coordinates of the target bounding box are ,in The previous frame was The center coordinates of the target bounding box are ;

[0084] The instantaneous displacement is:

[0085] ;

[0086] 2. Instantaneous velocity

[0087] Assuming the video frame rate is Then the frame interval time ;

[0088] Instantaneous velocity is:

[0089] ;

[0090] 3. Direction angle of motion

[0091] Motion direction angle (the angle between the motion direction angle and the positive horizontal direction of the image, range) )for:

[0092] ;

[0093] in:

[0094] : The pixel coordinates of the top-left / bottom-right corner of the target bounding box in the current frame;

[0095] : The center pixel coordinates of the target bounding box in the current frame;

[0096] : The center pixel coordinates of the target bounding box in the previous frame;

[0097] : Pixel displacement of the target along the x / y axis;

[0098] The time interval between two adjacent frames;

[0099] The instantaneous resultant velocity of the target;

[0100] Instantaneous velocity components along the x / y axes;

[0101] : The direction angle of the target's movement.

[0102] S150, based on the target appearance feature vector and target behavior parameters, and combined with the semantic information of the target's current position in the scene structure map, calculate a priority score representing the target tracking value for each target.

[0103] The priority score integrates the target's visual saliency, behavioral features, and scene semantics; the formula is as follows:

[0104] ;

[0105] in:

[0106] Priority score of the target, range The higher the score, the higher the tracking value;

[0107] : Appearance saliency, which is the ratio of the norm of the target feature vector to the norm of the background feature vector. For the target appearance feature vector, This represents the background feature vector of the region where the target is located.

[0108] Behavioral feature score, which is the normalized velocity value plus the degree of matching between the direction of movement and the direction of traffic in the scene. For direction weights, For the target instantaneous resultant velocity, The maximum speed of the target in the scene. The recommended travel direction angle for the area where the target is located in the scene structure diagram;

[0109] Scene semantic score, i.e., the priority of the area where the target is located;

[0110] Weighting coefficients, satisfying .

[0111] like Figure 3 As shown, the step of generating detection frame interval parameters and tracking algorithm complexity parameters for each target based on the priority score specifically includes:

[0112] S210, based on the comprehensive priority score of each detected target, calculate the detection frame interval parameter and the tracking algorithm complexity parameter of the detected target;

[0113] Detection frame interval parameter:

[0114] ;

[0115] Tracking algorithm complexity parameters:

[0116] ;

[0117] in:

[0118] : Detection frame interval, which is the interval between two detections of the target;

[0119] Minimum detection interval;

[0120] Interval coefficient;

[0121] Basic detection interval;

[0122] : Track algorithm complexity parameters; the larger the value, the more complex the algorithm.

[0123] Maximum complexity level;

[0124] Complexity coefficient;

[0125] : Basic complexity level.

[0126] The higher the priority ( (larger), detection interval The smaller the value (the more frequent the detection), the lower the complexity of the tracking algorithm. The higher (using a more precise algorithm).

[0127] S220, when processing each video frame sequence, check the frame interval since each detected target was last detected, and only when the interval reaches the detection frame interval parameter stored in its state record, perform a new round of target detection on the image region containing the target to update the location bounding box.

[0128] Performing detection at frame intervals balances tracking accuracy and computational efficiency. The tracker can predict the target position between frames using a motion model, eliminating the need for detection every frame. High-priority targets require more frequent detection to avoid tracking drift, hence the shorter detection intervals. Low-priority targets can have a lower detection frequency, saving GPU / CPU resources. Detection only when the frame interval reaches a threshold allows for periodic updates to the target position while reducing the number of calls to the detection network.

[0129] S230: Based on the tracking algorithm complexity parameter of each detected target, select a tracking algorithm instance with the corresponding complexity as a dedicated tracker for the detected target, and initialize the target appearance feature vector and the updated position bounding box as the internal state of the dedicated tracker.

[0130] A dedicated tracker is a tracking algorithm instance initialized separately for each detected target. It is built based on the target's appearance feature vector and initial bounding box, and belongs to instance-level trackers (as opposed to shared trackers).

[0131] S240: In each subsequent frame of the video frame sequence, the dedicated tracker corresponding to each detected target is called, the current frame image is input, and the dedicated tracker outputs the predicted tracking bounding box, so as to realize automatic target tracking of the detected target between frames.

[0132] Specifically:

[0133] 1. Initialize the target's bounding box. and appearance feature vector Input the selected tracking algorithm to initialize the tracker's internal state;

[0134] 2. Inter-frame prediction: For each subsequent frame, the tracker performs the following operations:

[0135] Extract local features around the tracking bounding box from the previous frame in the current frame;

[0136] Predicting and locating targets through feature matching or motion models;

[0137] Output the tracking bounding box of the current frame;

[0138] 3. State update: If detection is triggered, that is, the detection frame interval is reached, the tracker's prediction error is corrected with the new detection bounding box, and the template features are updated.

[0139] like Figure 4 As shown, the generation of the spatiotemporal probability map through the prediction network specifically includes:

[0140] S310: If the tracking bounding box continuously predicted by the dedicated tracker corresponding to a detected target is not confirmed by the new round of target detection within a preset number of frames, the detected target is determined to be lost and marked as a lost target.

[0141] The tracker's prediction results may drift due to occlusion, background interference, target deformation, etc., meaning the predicted bounding box deviates from the true target; if continuous If the tracking bounding box of a frame (preset number of frames, such as 5 frames) is not confirmed by the detection network, it indicates that there is no matching target in the detection result, meaning that the tracking is unreliable. After the loss is determined, spatiotemporal prediction and orientation detection are triggered to avoid continuous erroneous tracking, while attempting to find the real target.

[0142] S320, extract the historical data of the lost target in the last N frames before the loss was determined. The historical data includes N sets of target behavior parameters and N target appearance feature vectors, forming the behavior and appearance sequence of the lost target.

[0143] S330, The behavior and appearance sequence and the scene structure map are input into the prediction neural network. The behavior and appearance sequence are encoded and combined with the passage constraints described by the scene structure map to decode the possible location distribution of the lost target on the image plane at the next T time points.

[0144] S340 outputs a probability map of the same size as the input image for each of the T future time points. The value of each pixel on the probability map represents the probability that the lost detected target appears at that pixel position at that time. The T probability maps are superimposed in chronological order to form a spatiotemporal probability map.

[0145] The pixel probabilities predicted by the network output are generated using an encoder-decoder architecture, as shown in the formula:

[0146] ;

[0147] in:

[0148] Future Moments Pixels The probability and range of the target's appearance. ;

[0149] : Encoding features of behavioral appearance sequences Before it was lost Frame characteristics and behavioral parameters;

[0150] Scene structure diagram Graph convolutional coding features;

[0151] Future Moment Index ( ).

[0152] like Figure 5 As shown, the directional detection of image regions corresponding to spatiotemporal regions using a temporary detection confidence threshold specifically includes:

[0153] S410, based on the final calculated comprehensive priority score of the lost target, calculates the probability threshold and planned retrieval frame range for the lost target;

[0154] 1. Probability threshold

[0155] ;

[0156] 2. Planned Frame Count Range

[0157] ;

[0158] in:

[0159] : Probability threshold, which is the minimum probability of selecting valid pixels;

[0160] : Basic threshold;

[0161] Maximum threshold;

[0162] : Planned number of frames to retrieve, i.e., the number of video frames to retrieve;

[0163] Minimum number of search frames;

[0164] : Basic search frame count;

[0165] : Retrieve frame number offset.

[0166] High priority goals ( (Large) threshold Higher resolution (filters more reliable areas), more frames retrieved (extends retrieval time).

[0167] S420, according to the probability threshold, perform threshold segmentation on the spatiotemporal probability map, traverse each pixel in the spatiotemporal probability map, if the probability value stored in any pixel is not less than the probability threshold, then mark the pixel as a valid point, and mark the connected region formed by all valid points as the selected spatiotemporal region.

[0168] S430, determine the current video frame as the search start frame, and determine the search end frame according to the planned search frame range; within this specified frame range from the search start frame to the search end frame, for each search frame, find all pixel coordinates falling within the filtering spatiotemporal region according to the spatiotemporal probability map slice at the corresponding time.

[0169] S440 calls the target detection neural network to perform directional detection only within the local area enclosed by the pixel coordinates on the current retrieval frame image, to determine whether the lost target exists.

[0170] like Figure 6 As shown, the reallocation of detection frame intervals and trackers based on priority scores specifically includes:

[0171] S510, if in the detection results output by the directional detection within the selected spatiotemporal region, there is a target's location bounding box whose similarity to the historical target appearance feature vector of the lost target exceeds a preset re-identification threshold, then it is determined that the lost target has been detected again.

[0172] The appearance feature vector of a target has unique identity; the feature vectors of the same target have high similarity, while the similarity between different targets is low.

[0173] Calculate the cosine similarity between the feature vector of the newly detected target and the historical feature vector of the lost target:

[0174] ;

[0175] when When re-identifying the threshold, it is determined to be the same target, avoiding misidentification of other targets as lost targets;

[0176] By combining location (spatiotemporal probability map region) with appearance features, the accuracy of re-identification can be improved.

[0177] S520, update the ID of the newly detected lost target to the original ID of the lost target, and calculate the new target appearance feature vector and new target behavior parameters based on the new location bounding box of the lost target;

[0178] S530, based on the comprehensive priority score of the original lost target, the new target appearance feature vector, and the new target behavior parameters, recalculates the detection frame interval parameters and tracking algorithm complexity parameters, initializes the dedicated tracker, and resumes automatic tracking in subsequent frames.

[0179] Figure 7 The structural block diagram of the deep learning-based automatic target detection and tracking system provided in the embodiments of the present invention is as follows: Figure 7 As shown, the system includes:

[0180] The video stream and scene structure map acquisition module 100 is used to acquire the video stream and scene structure map, process the video frame sequence frame by frame, extract the location bounding box and category label of the detected target in the frame through the detection network, generate the target appearance feature vector simultaneously, and calculate the target behavior parameters and priority score.

[0181] The target parameter generation module 200 is used to generate a detection frame interval parameter and a tracking algorithm complexity parameter for each target based on the priority score, control the detection frequency of the target based on the detection frame interval parameter, and initialize a dedicated tracker for the target based on the tracking algorithm complexity parameter to achieve automatic tracking.

[0182] The target loss processing module 300 is used to generate a spatiotemporal probability map by a prediction network based on the behavior parameters, appearance feature vector and scene structure diagram of the lost target before it was lost, when the dedicated tracker determines that the target is lost. The pixels in the map contain the probability value of the lost target appearing at the corresponding pixel in the future.

[0183] The directional detection module 400 is used to determine the probability threshold, the temporary detection confidence threshold and the retrieval frame range based on the priority score and spatiotemporal probability map of the lost target. It selects spatiotemporal regions with probability values ​​higher than the probability threshold from the spatiotemporal probability map and performs directional detection on the image regions corresponding to the spatiotemporal regions in the subsequent video frames within the retrieval frame range using the temporary detection confidence threshold.

[0184] The tracking recovery module 500 is used to associate the lost target with the detected target as the same target if the lost target is rediscovered by directional detection, and to reallocate the detection frame interval and tracker according to the priority score to resume tracking.

[0185] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0186] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

[0187] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A deep learning-based automatic target detection and tracking method, characterized in that, The method includes: The video stream and scene structure map are acquired, and the video frame sequence is processed frame by frame. The location bounding box and category label of the detected target in the frame are extracted by the detection network. The target appearance feature vector is generated simultaneously, and the target behavior parameters and priority scores are calculated. Based on the priority score, a detection frame interval parameter and a tracking algorithm complexity parameter are generated for each detection target. The detection frequency of the detection target is controlled based on the detection frame interval parameter, and a dedicated tracker is initialized for the detection target based on the tracking algorithm complexity parameter to achieve automatic tracking. When the dedicated tracker determines that the target is lost, it generates a spatiotemporal probability map through a prediction network based on the target’s behavior parameters, appearance feature vector and scene structure diagram before the target was lost. The pixels in the map contain the probability value of the lost target appearing at the corresponding pixel in the future. Based on the priority score and spatiotemporal probability map of the lost target, the probability threshold, the temporary detection confidence threshold, and the retrieval frame range are determined. Spatiotemporal regions with probability values ​​higher than the probability threshold are selected from the spatiotemporal probability map. In the subsequent video frames within the retrieval frame range, the image regions corresponding to the spatiotemporal regions are targeted for detection using the temporary detection confidence threshold. If the lost target is rediscovered by directional detection, the lost target and the detected target are associated as the same target, and the detection frame interval and tracker are reallocated according to the priority score to resume tracking.

2. The method according to claim 1, characterized in that, The process of generating the target appearance feature vector and calculating the target behavior parameters and priority scores specifically includes: Real-time video stream data is received from at least one image sensor, and the real-time video stream data is decoded into a sequence of video frames arranged in timestamp order. A predefined scene structure graph is loaded synchronously, and the scene structure graph stores the connection relationship and semantic attributes of each region node in the scene in a graph data structure. The current frame in the video frame sequence is input into the target detection neural network, which outputs the bounding boxes of all detected targets in the current frame. Based on the current frame and the location bounding box of each detected target, a fixed-dimensional floating-point vector is output for each detected target as the target appearance feature vector; Based on the center coordinates of the bounding box of the same detected target in the current frame and the previous frame, the instantaneous displacement of the detected target on the two-dimensional image plane is calculated to obtain the target behavior parameters, which include instantaneous velocity and motion direction angle. Based on the target's appearance feature vector and target behavior parameters, and combined with the semantic information of the target's current position in the scene structure diagram, a priority score representing the target's tracking value is calculated for each target.

3. The method according to claim 2, characterized in that, The process of generating detection frame interval parameters and tracking algorithm complexity parameters for each target based on the priority score specifically includes: Based on the comprehensive priority score of each detected target, the detection frame interval parameter and the tracking algorithm complexity parameter are calculated. When processing each video frame sequence, the frame interval since the last detection of each target is checked. Only when the interval reaches the detection frame interval parameter stored in its state record is a new round of target detection performed on the image region containing the target to update the location bounding box. Based on the tracking algorithm complexity parameter of each detected target, a tracking algorithm instance with the corresponding complexity is selected as the dedicated tracker for the detected target, and the target appearance feature vector and the updated position bounding box are initialized as the internal state of the dedicated tracker. In each subsequent frame of the video frame sequence, a dedicated tracker corresponding to each detected target is invoked. The current frame image is input, and the dedicated tracker outputs the predicted tracking bounding box, thereby realizing automatic target tracking of the detected target between frames.

4. The method according to claim 3, characterized in that, The generation of the spatiotemporal probability map through the prediction network specifically includes: If the tracking bounding box continuously predicted by the dedicated tracker corresponding to a detected target is not confirmed by the new round of target detection within a preset number of frames, the detected target is determined to be lost and marked as a lost target. Extract the historical data of the lost target in the last N frames before it was determined to be lost. The historical data includes N sets of target behavior parameters and N target appearance feature vectors, forming a sequence of the behavior and appearance of the lost target. The behavior and appearance sequence and the scene structure map are input into the prediction neural network. The behavior and appearance sequence are encoded and combined with the passage constraints described by the scene structure map to decode the possible location distribution of the lost target on the image plane at the next T time points. For each of the T future time points, output a probability map of the same size as the input image. The value of each pixel on the probability map represents the probability that the missing detected target appears at that pixel position at that time. Then, stack the T probability maps in chronological order to form a spatiotemporal probability map.

5. The method according to claim 4, characterized in that, The method of performing directional detection of image regions corresponding to spatiotemporal regions using a temporary detection confidence threshold specifically includes: Based on the final calculated comprehensive priority score of the lost target, calculate the probability threshold and planned retrieval frame range for the lost target; Based on the probability threshold, the spatiotemporal probability map is segmented by threshold. Each pixel in the spatiotemporal probability map is traversed. If the probability value stored in any pixel is not less than the probability threshold, the pixel is marked as a valid point, and the connected region formed by all valid points is marked as the selected spatiotemporal region. The current video frame is determined as the starting frame for retrieval, and the ending frame for retrieval is determined according to the planned range of the number of frames to be retrieved. Within this specified range of the number of frames from the starting frame to the ending frame, for each retrieval frame, the coordinates of all pixels falling within the selected spatiotemporal region are found based on the slice of the spatiotemporal probability map at the corresponding time. The target detection neural network is invoked only within the local area enclosed by the pixel coordinates on the current retrieval frame image to perform directional detection and determine whether the missing target exists.

6. The method according to claim 5, characterized in that, The reallocation of detection frame intervals and trackers based on priority scores specifically includes: If, in the detection results output by the directional detection within the selected spatiotemporal region, there is a target's location bounding box whose similarity to the historical target appearance feature vector of the lost target exceeds a preset re-identification threshold, then it is determined that the lost target has been detected again. The identifier ID of the newly detected lost target is updated to the original identifier ID of the lost target. Based on the new location bounding box of the lost target, a new target appearance feature vector and new target behavior parameters are calculated. Based on the comprehensive priority score of the original lost target, the new target appearance feature vector, and the new target behavior parameters, the detection frame interval parameters and tracking algorithm complexity parameters are recalculated, and a dedicated tracker is initialized to resume automatic tracking in subsequent frames.

7. A deep learning-based automatic target detection and tracking system, characterized in that, The system includes: The video stream and scene structure map acquisition module is used to acquire the video stream and scene structure map, process the video frame sequence frame by frame, extract the location bounding box and category label of the detected target in the frame through the detection network, generate the target appearance feature vector simultaneously, and calculate the target behavior parameters and priority score; The target parameter generation module is used to generate a detection frame interval parameter and a tracking algorithm complexity parameter for each target based on the priority score, control the detection frequency of the target based on the detection frame interval parameter, and initialize a dedicated tracker for the target based on the tracking algorithm complexity parameter to achieve automatic tracking. The target loss processing module is used to generate a spatiotemporal probability map by a prediction network when the dedicated tracker determines that the target is lost, based on the behavior parameters, appearance feature vector and scene structure diagram of the lost target before the loss. The pixels in the map contain the probability value of the lost target appearing at the corresponding pixel in the future. The directional detection module is used to determine the probability threshold, temporary detection confidence threshold and retrieval frame range based on the priority score and spatiotemporal probability map of the lost target. It selects spatiotemporal regions with probability values ​​higher than the probability threshold from the spatiotemporal probability map, and performs directional detection on the image regions corresponding to the spatiotemporal regions in the subsequent video frames within the retrieval frame range using the temporary detection confidence threshold. The tracking recovery module is used to associate the lost target with the detected target as the same target if the lost target is rediscovered by directional detection, and to reallocate the detection frame interval and tracker according to the priority score to resume tracking.

Images