Automatic target detection and tracking method and system based on deep learning

Abstract

The invention is suitable for the field of computer vision and artificial intelligence, and provides an automatic target detection and tracking method and system based on deep learning, and the method comprises the steps: obtaining a video stream and a scene structure diagram, extracting a target position bounding box, a category label and an appearance feature vector frame by frame, and calculating a behavior parameter and a priority score; generating a detection frame interval and a tracking algorithm complexity parameter according to the priority score, and initializing a special tracker to realize automatic tracking; when the target is lost, generating a space-time probability graph based on historical data and the scene structure graph; determining retrieval parameters according to the priority scores, and performing directional detection on the screened space-time regions; and after the target is retrieved, the identity is associated and parameters are redistributed to recover tracking. According to the method, through differentiated resource allocation, accurate loss prediction and directional retrieval, tracking precision, efficiency and robustness are considered, and the problems of target tracking interruption, resource waste and the like in a complex scene are effectively solved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and artificial intelligence, and particularly relates to an automatic target detection and tracking method and system based on deep learning. BACKGROUND

[0002] With the rapid development of deep learning technology, target detection and tracking technology in the field of computer vision has been widely applied to security monitoring, intelligent transportation, industrial detection and other scenes, and has become a core technical support for ensuring scene safety and improving management efficiency. At present, target detection networks and tracking algorithms based on deep learning are constantly iterated, and significant progress has been made in detection accuracy, tracking response speed and other aspects, which can realize the simultaneous identification and continuous tracking of multiple targets and meet the basic needs of target dynamic monitoring in various scenes.

[0003] The core technical problem existing in the prior art is that the tracking resource allocation lacks pertinence, which leads to insufficient tracking accuracy of high-value targets, excessive occupation of computing resources by low-value targets, and low efficiency in finding back lost targets, which is prone to tracking interruption, identity confusion and other situations. The robustness, continuity and resource utilization efficiency of the overall tracking system are difficult to balance, and it is difficult to meet the high-precision and high-efficiency target tracking needs in complex scenes. SUMMARY

[0004] The purpose of the present application is to provide an automatic target detection and tracking method and system based on deep learning, which aims to solve the technical problems existing in the prior art identified in the background.

[0005] The present application is implemented as follows: an automatic target detection and tracking method based on deep learning, the method comprising: acquiring a video stream and a scene structure graph, and processing the video frame sequence frame by frame, extracting the position bounding box and class label of the detected target in the frame through a detection network, synchronously generating a target appearance feature vector, and calculating a target behavior parameter and a priority score; generating a detection frame interval parameter and a tracking algorithm complexity parameter for each detected target according to the priority score, controlling the detection frequency of the detected target according to the detection frame interval parameter, and initializing a dedicated tracker for the detected target according to the tracking algorithm complexity parameter to realize automatic tracking; when the dedicated tracker determines that the detected target is lost, generating a space-time probability map through a prediction network based on the behavior parameter, appearance feature vector and scene structure graph of the lost target before it is lost, wherein the pixel points in the map contain the probability value of the lost target appearing at the corresponding pixel point at a future time; According to the priority score of the lost target and the space-time probability map, a probability threshold, a temporary detection confidence threshold and a search frame number range are determined, a space-time region with a probability value higher than the probability threshold is screened from the space-time probability map, and an image region corresponding to the space-time region is detected in a subsequent search frame number range with the temporary detection confidence threshold. If the lost target is rediscovered through the directional detection, the lost target is associated with the detected target as the same target, and the detection frame interval and the tracker are re-allocated according to the priority score to resume tracking.

[0006] As a further scheme of the present application, the target appearance feature vector is generated, and the target behavior parameter and the priority score are calculated, specifically including: Real-time video stream data is accessed from at least one image sensor, and the real-time video stream data is decoded into a video frame sequence arranged in chronological order, and a predefined scene structure graph is synchronously loaded, the scene structure graph storing the connection relationship and semantic attributes of each region node in the scene in a graph data structure; A current frame in the video frame sequence is input into a target detection neural network, and a position bounding box of all detected targets in the current frame is output; Based on the current frame and the position bounding box of each detected target, a floating-point number vector of a fixed dimension is output for each detected target as a target appearance feature vector; According to the position bounding box center coordinates of the same detected target in the current frame and the previous frame, an instantaneous displacement of the detected target on a two-dimensional image plane is calculated to obtain a target behavior parameter, the target behavior parameter including an instantaneous speed and a motion direction angle; Based on the target appearance feature vector and the target behavior parameter, and in combination with the semantic information of the current position of the target in the scene structure graph, a priority score representing the tracking value of each target is calculated.

[0007] As a further scheme of the present application, the detection frame interval parameter and the tracking algorithm complexity parameter are generated for each target according to the priority score, specifically including: According to the comprehensive priority score of each detected target, a detection frame interval parameter and a tracking algorithm complexity parameter of the detected target are calculated; When each frame of the video frame sequence is processed, the frame interval of each detected target since the last detection is checked, and only when the interval reaches the detection frame interval parameter stored in the state record, a new round of target detection is performed on the image region containing the target to update the position bounding box; According to the tracking algorithm complexity parameter of each detected target, a tracking algorithm instance with a corresponding complexity is selected as a dedicated tracker of the detected target, and the target appearance feature vector and the updated position bounding box are initialized as the internal state of the dedicated tracker. In each subsequent frame of the video frame sequence, a dedicated tracker corresponding to each detected target is called, and a current frame image is input, and a predicted tracking bounding box is output by the dedicated tracker to realize automatic target tracking of the detected target between frames.

[0008] As a further scheme of the present application, the spatiotemporal probability map generated by the prediction network specifically comprises: When the tracking bounding box continuously predicted by the dedicated tracker corresponding to a detected target is not confirmed by a new round of target detection performed within more than a preset number of frames, it is determined that the detected target is lost, and is marked as a lost target; The historical data of the lost target in the last N frames before the loss is determined is extracted, the historical data including N sets of target behavior parameters and N target appearance feature vectors, forming a behavior and appearance sequence of the lost target; The behavior and appearance sequence and the scene structure graph are input into a prediction neural network, the behavior and appearance sequence are encoded, and the possible position distribution of the lost target on the image plane at T future time points is decoded in combination with the traffic constraints described by the scene structure graph; For each of the T future time points, a probability map with the same size as the input image is output, and the value of each pixel point on the probability map represents the probability of the lost detected target appearing at this pixel position at this time point, and the T probability maps are superimposed in chronological order to form a spatiotemporal probability map.

[0009] As a further scheme of the present application, the image region corresponding to the spatiotemporal region is detected in a targeted manner using a temporary detection confidence threshold, specifically comprising: Based on the last calculated comprehensive priority score of the lost target, a probability threshold and a planned search frame number range for the lost target are calculated; According to the probability threshold, the spatiotemporal probability map is threshold segmented, and each pixel point in the spatiotemporal probability map is traversed, and if the probability value stored in any pixel point is not less than the probability threshold, the pixel point is marked as an effective point, and the connected region formed by all effective points is marked as a screening spatiotemporal region; The current video frame is determined as a search starting frame, and the search ending frame is determined according to the planned search frame number range; in the specified frame number range from the search starting frame to the search ending frame, for each search frame, all pixel coordinates falling within the screening spatiotemporal region are found according to the slice of the spatiotemporal probability map at the corresponding time point; Only in the local region surrounded by the pixel coordinates on the current search frame image, the target detection neural network is called for targeted detection to determine whether the lost target exists.

[0010] As a further scheme of the present application, the detection frame interval and the tracker are reassigned according to the corresponding dynamic priority score, specifically comprising: If the orientation detection exists a target position bounding box in the detection result output in the screening spatio-temporal region, and the similarity between the target position bounding box and the historical target appearance feature vector of the lost target exceeds the preset re-identification threshold, it is determined that the lost target is re-detected. The identification ID of the re-detected lost target is updated as the identification ID of the original lost target, a new target appearance feature vector is calculated based on the new position bounding box of the lost target, and a new target behavior parameter is calculated. Based on the comprehensive priority score of the original lost target, the new target appearance feature vector and the new target behavior parameter, the detection frame interval parameter and the tracking algorithm complexity parameter are re-calculated, and the dedicated tracker is initialized to resume automatic tracking in the subsequent frames.

[0011] Another object of the present application is to provide an automatic target detection and tracking system based on deep learning, which comprises: A video stream and scene structure graph acquisition module is configured to acquire a video stream and a scene structure graph, and process the video frame sequence frame by frame, extract the position bounding box of the detected target in the frame, the category label, the target appearance feature vector, the target behavior parameter and the priority score through a detection network, and synchronously generate the target appearance feature vector. A detection target parameter generation module is configured to generate the detection frame interval parameter and the tracking algorithm complexity parameter for each detected target according to the priority score, control the detection frequency of the detected target according to the detection frame interval parameter, and initialize the dedicated tracker for the detected target according to the tracking algorithm complexity parameter to realize automatic tracking. A target loss processing module is configured to, when the dedicated tracker determines that the detected target is lost, generate a spatio-temporal probability map through a prediction network based on the behavior parameter, the appearance feature vector and the scene structure graph of the lost target before the loss, wherein the pixel points in the map contain the probability value of the lost target appearing at the corresponding pixel point at a future time. An orientation detection module is configured to determine the probability threshold, the temporary detection confidence threshold and the search frame number range according to the priority score of the lost target and the spatio-temporal probability map, screen out the spatio-temporal region with a probability value higher than the probability threshold from the spatio-temporal probability map, and perform orientation detection on the image region corresponding to the spatio-temporal region in the video frame within the subsequent search frame number range with the temporary detection confidence threshold. A tracking recovery module is configured to associate the lost target and the detected target as the same target if the orientation detection re-discovers the lost target, and re-allocate the detection frame interval and the tracker according to the priority score to recover the tracking.

[0012] The present application has the following advantages: This invention constructs a full-process tracking mechanism, achieving precision, efficiency, and robustness in target detection and tracking. Priority scores are calculated based on target appearance features, behavioral parameters, and scene semantic information. The detection frame interval and tracking algorithm complexity are dynamically allocated, achieving optimal configuration of tracking resources. This ensures tracking accuracy for high-value targets while reducing overall computational overhead. For target loss scenarios, a spatiotemporal probability map is generated by fusing historical behavior sequences, appearance features, and scene constraints. Combined with a targeted detection mechanism, the search range is narrowed, significantly improving the efficiency and accuracy of finding lost targets. Simultaneously, identity feature matching and tracking parameter reinitialization ensure the continuity and consistency of target tracking, effectively solving problems such as unreasonable resource allocation, difficulty in finding lost targets, and insufficient tracking stability in existing technologies. This significantly improves the system's adaptability and practical value in complex scenarios, making it widely applicable to various application scenarios with high requirements for dynamic target monitoring. Attached Figure Description

[0013] Figure 1 A flowchart illustrating the deep learning-based automatic target detection and tracking method provided in this embodiment of the invention; Figure 2 A flowchart for generating target appearance feature vectors and calculating target behavior parameters and priority scores provided in embodiments of the present invention; Figure 3 A flowchart for generating detection frame interval parameters and tracking algorithm complexity parameters for each target based on the priority score, provided in an embodiment of the present invention; Figure 4 A flowchart for generating a spatiotemporal probability graph through a prediction network, provided in an embodiment of the present invention; Figure 5 This is a flowchart of a method for directional detection of image regions corresponding to spatiotemporal regions using a temporary detection confidence threshold, provided by an embodiment of the present invention. Figure 6 A flowchart for reallocating the detection frame interval and tracker based on the corresponding dynamic priority score provided in an embodiment of the present invention; Figure 7 This is a structural block diagram of an automatic target detection and tracking system based on deep learning provided in an embodiment of the present invention. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0015] Figure 1 A flowchart of the deep learning-based automatic target detection and tracking method provided in the embodiments of the present invention is shown below.Figure 1 As shown, the method comprises: S100, acquiring a video stream and a scene structure diagram, and processing the video frame sequence frame by frame, extracting the position bounding box and the category label of the detection target in the frame through the detection network, synchronously generating the target appearance feature vector, and calculating the target behavior parameter and the priority score; The video frame sequence is processed frame by frame, each frame of image is input into the target detection neural network, the position bounding box and the category label of all detection targets in the frame are accurately extracted through the collaborative work of the network backbone layer and the feature fusion layer, and the spatial positioning and attribute classification of the target are determined. The synchronous target appearance feature vector generation is performed through feature extraction, dimension standardization and normalization processing on the target bounding box region, forming a fixed dimension floating point vector with identity recognition.

[0016] At the same time, the instantaneous displacement of the target in the two-dimensional image plane is calculated by combining the bounding box center coordinates of the same target in the current frame and the previous frame, and the behavior parameters such as instantaneous speed and motion direction angle are derived, which intuitively reflect the motion state and trend of the target. The recognition degree of the target appearance feature vector, the dynamic characteristics of the behavior parameter and the semantic priority of the region where the target is located in the scene structure diagram are comprehensively considered to calculate the priority score representing the tracking value of the target, and the comprehensive value evaluation of the target is completed.

[0017] S200, generating a detection frame interval parameter and a tracking algorithm complexity parameter for each detection target according to the priority score, controlling the detection frequency of the detection target according to the detection frame interval parameter, and initializing a special tracker for the detection target according to the tracking algorithm complexity parameter to realize automatic tracking; According to the comprehensive priority score of each detection target, the detection frame interval parameter and the tracking algorithm complexity parameter are accurately calculated, forming a resource allocation strategy matched with the value of the target. The detection frame interval parameter directly determines the time span of two detections of the target, and the tracking algorithm complexity parameter corresponds to tracking algorithm instances with different accuracies and computational overheads.

[0018] Based on these parameters, the system initializes a special tracker for each detection target, imports the target appearance feature vector and the updated position bounding box into the tracker, constructs a dedicated tracking model, and ensures that the tracker is deeply adapted to the appearance and position characteristics of the target.

[0019] In the subsequent video frame sequence processing process, the last detection time of each target is continuously recorded, the frame interval is checked in real time, and only when the interval reaches the preset detection frame interval parameter, a new round of target detection is started for the image area where the target is located, and the position bounding box is updated to correct the possible tracking deviation; when the interval requirement is not met, the detection network does not need to be repeatedly called, and the special tracker corresponding to each target is directly called to input the current frame image, and the tracker outputs accurate tracking bounding box through inter-frame local feature matching and motion trend prediction, realizing seamless tracking of the detected target between continuous frames.

[0020] S300, when the special tracker determines that the detected target is lost, a spatio-temporal probability map is generated based on the behavior parameters, appearance feature vectors and scene structure graph of the lost target before the loss, and the pixel points in the map contain the probability value of the lost target appearing at the corresponding pixel point at a future time; The output results of each special tracker are continuously monitored, and when the tracking bounding box continuously predicted by the tracker is not confirmed by a new round of target detection within more than a preset number of frames, it is determined that the target is in a lost state and is marked: the tracking bounding box that is not confirmed by detection means that there may be tracking drift, target occlusion, dramatic changes in posture or temporary moving out of the field of view, and at this time, continuing to rely on the tracker will lead to error accumulation, and marking the loss is a necessary measure to avoid invalid tracking.

[0021] The historical data of the target in the last N frames before the loss is determined, which covers multiple sets of target behavior parameters and multiple appearance feature vectors, forming a complete behavior and appearance sequence, the behavior parameters carry the motion trend information before the target is lost, and the appearance feature vectors retain the identity recognition features of the target, providing core data support for subsequent prediction. These sequence data and the scene structure Figure 1 The behavior and appearance sequence is encoded into the input prediction neural network, the motion law and identity features of the target are extracted, and the region connection relationship and traffic constraints contained in the scene structure graph are combined to decode and generate the possible position distribution of the target on the image plane at T future times.

[0022] On this basis, a probability map with the same size as the input image is output for each future time, and the value of each pixel point directly reflects the possibility of the target appearing at that position at that time, and finally T probability maps are stacked in chronological order to form a spatio-temporal probability map with time continuity and spatial directionality, which completely presents the possible spatio-temporal trajectory range of the lost target.

[0023] S400, according to the priority score of the missing target and the spatiotemporal probability map, determining a probability threshold, a temporary detection confidence threshold and a search frame number range, screening out a spatiotemporal region with a probability value higher than the probability threshold from the spatiotemporal probability map, and performing directional detection on an image region corresponding to the spatiotemporal region in a video frame within the subsequent search frame number range with the temporary detection confidence threshold; With the comprehensive priority score of the missing target as the core basis, the probability threshold, the temporary detection confidence threshold and the planned search frame number range are dynamically calculated and adapted. The priority score directly determines the degree of investment of search resources, so that high-value targets obtain stricter screening standards and more sufficient search time, and low-value targets use relatively loose thresholds and reasonable search ranges, thereby realizing personalized adaptation of the search strategy.

[0024] Based on the calculated probability threshold, threshold segmentation processing is performed on the spatiotemporal probability map, all pixel points in the map are traversed, and valid points with probability values meeting the requirements are screened out. These valid points converge to form a connected screening spatiotemporal region, focusing the range where the target may appear from the entire image to a local high-probability region, and greatly reducing the detection range. Then, the time boundary of the search is determined, the current video frame is taken as the starting point, the end frame is determined in combination with the planned search frame number range, and in the time interval, for each search frame, a slice corresponding to the time of the spatiotemporal probability map is extracted, pixel coordinates falling within the screening spatiotemporal region are accurately located, and then the local image region to be detected in the current frame is locked.

[0025] During the detection process, the target detection neural network is called only in the local region to perform directional detection, and the temporary detection confidence threshold is used as the judgment standard to determine whether the missing target exists. Through the cooperation of the priority score and the spatiotemporal probability map, the detection resources are concentrated in the spatiotemporal range where the target is most likely to appear.

[0026] In actual application, after a high-priority target is lost, the system sets a higher probability threshold to screen out more reliable regions, extends the search frame number, and performs directional detection in the key region to improve the recovery probability.

[0027] S500, if the missing target is found again through directional detection, the missing target and the detected target are associated as the same target, and the detection frame interval and the tracker are re-allocated according to the priority score to resume tracking.

[0028] After the directional detection outputs the detection result in the spatiotemporal region, the system starts the target identity verification mechanism, calculates the similarity between the appearance feature vector of the new detection target and the historical appearance feature vector of the lost target, and uses the similarity as the core basis for identity matching. When the similarity reaches the preset re-identification threshold, the identity binding of the lost target and the new detection target is completed, ensuring that the target recovered for tracking is the original lost target rather than other interference targets. After identity confirmation, the identification ID of the new detection target is updated to the identification ID of the original lost target, ensuring the continuity of the target tracking link and avoiding identity breakage caused by target loss.

[0029] Based on the position bounding box of the new detection target, the appearance feature vector and the behavior parameter of the target are recalculated to adapt to the appearance change or motion state adjustment that may occur during the loss of the target. The comprehensive priority score of the original lost target, the newly generated appearance feature vector, and the behavior parameter are fused to recalculate the detection frame interval parameter and the tracking algorithm complexity parameter, so that the tracking resource allocation not only continues the value weight of the original target but also adapts to its current state. Finally, the dedicated tracker is initialized with new parameters, and the updated position bounding box and appearance feature vector are imported into the tracker internal state to resume automatic tracking of the target in subsequent video frames, so that the entire detection and tracking process returns to the stable operation track.

[0030] As shown in Figure 2 , the target appearance feature vector is generated, and the target behavior parameter and priority score are calculated, specifically including: S110, accessing real-time video stream data from at least one image sensor, and decoding the real-time video stream data into a video frame sequence arranged in chronological order, and synchronously loading a predefined scene structure graph, the scene structure graph storing the connection relationship and semantic attributes of each region node in the scene in a graph data structure; The scene structure graph organizes the scene information model in a graph data structure, describes the spatial topology and semantic attributes of the monitoring scene, and the node represents an independent region in the scene; the edge represents the connection relationship between the regions; and the semantic attributes represent the semantic information attached to each node / edge.

[0031] S120, inputting a current frame in the video frame sequence into a target detection neural network to output the position bounding box of all detection targets in the current frame; S130, based on the current frame and the position bounding box of each detection target, outputting a fixed-dimension floating-point number vector as the target appearance feature vector for each detection target; 1. The decoded current frame is preprocessed, such as standardization and size adjustment.

[0032] 2. The preprocessed frame is input into the target detection neural network to extract image features and output the position bounding box of the target (format: , the top-left and bottom-right coordinates), a class label, and a detection confidence.

[0033] 3. The bounding box region of each detection target: The feature map of the region is extracted from the Backbone / Neck layer of the detection network, the feature map is converted into a fixed-dimensional vector through RoI pooling / adaptive average pooling, the vector is subjected to L2 normalization, and the final target appearance feature vector is obtained.

[0034] In S140, the instantaneous displacement of the detection target on the two-dimensional image plane is calculated according to the position bounding box center coordinates of the same detection target in the current frame and the previous frame, and the target behavior parameter is obtained, the target behavior parameter includes instantaneous speed and motion direction angle; 1. Instantaneous displacement Let the current frame time be , and the target bounding box center coordinates be , where ; the previous frame time be , and the target bounding box center coordinates be ; The instantaneous displacement is: ; 2. Instantaneous speed Let the video frame rate be , then the frame interval time ; The instantaneous speed is: ; 3. Motion direction angle The motion direction angle (the angle with the horizontal positive direction of the image, the range ) is: ; Wherein: : the top-left / right-bottom pixel coordinates of the current frame target bounding box; : the center pixel coordinates of the current frame target bounding box; : the center pixel coordinates of the previous frame target bounding box; : the pixel displacement of the target in the x / y axis; : the time interval of the adjacent two frames; : the instantaneous total speed of the target; : the instantaneous component speed in the x / y axis; : the moving direction angle of the target.

[0035] S150, based on the target appearance feature vector and the target behavior parameter, combining the semantic information of the target current position in the scene structure graph, calculating the priority score of each target representing the target tracking value.

[0036] The priority score integrates the target appearance saliency, behavior feature and scene semantics, and the formula is: ; Wherein: : the priority score of the target, the range , the higher the score, the higher the tracking value; : appearance saliency, that is, the ratio of the norm of the target feature vector to the norm of the background feature vector, is the target appearance feature vector, is the background feature vector of the target region; : behavior feature score, that is, speed normalization value + matching degree of moving direction and scene passing direction, is the direction weight, is the target instantaneous combined speed, is the maximum speed of the target in the scene, is the recommended passing direction angle of the target region in the scene structure graph; : scene semantic score, that is, the priority of the target region; : weight coefficient, satisfying .

[0037] As Figure 3 shown, the priority score is used to generate detection frame interval parameters and tracking algorithm complexity parameters for each target, specifically including: S210, according to the comprehensive priority score of each detection target, calculating the detection frame interval parameter and the tracking algorithm complexity parameter of the detection target; Detection frame interval parameter: ; Tracking algorithm complexity parameter: ; Wherein: : detection frame interval, that is, the frame interval of two detections of the target; : minimum detection interval; : interval coefficient; : base detection interval; : tracking algorithm complexity parameter, the larger the value, the more complex the algorithm; : maximum complexity level; : complexity coefficient; : base complexity level.

[0038] The higher the priority (the larger the value), the smaller the detection interval (the more frequent the detection), the higher the tracking algorithm complexity (the more accurate the algorithm).

[0039] S220, when processing each frame of the sequence of video frames, check the frame interval since the last detection of each detection target, and only when the interval reaches the detection frame interval parameter stored in its state record, perform a new round of target detection on the image region containing the target to update the position bounding box; Performing detection according to the detection frame interval can balance tracking accuracy and computational efficiency. The tracker can predict the target position through the motion model between frames without detection every frame. High-priority targets require more frequent detection to avoid tracking drift, so the detection interval is small. Low-priority targets can reduce the detection frequency to save GPU / CPU resources. Only when the frame interval reaches the threshold, detection is performed, which can periodically update the target position and reduce the number of calls to the detection network.

[0040] S230, according to the tracking algorithm complexity parameter of each detection target, select a tracking algorithm instance of corresponding complexity as the dedicated tracker of the detection target, and initialize the target appearance feature vector and the updated position bounding box as the internal state of the dedicated tracker; The dedicated tracker is a tracking algorithm instance initialized separately for each detection target, which is constructed based on the appearance feature vector and the initial bounding box of the target, and belongs to an instance-level tracker (different from a shared tracker).

[0041] S240, in each subsequent frame of the sequence of video frames, call the dedicated tracker corresponding to each detection target, input the current frame image, and output the predicted tracking bounding box by the dedicated tracker to realize automatic target tracking of the detection target between frames.

[0042] Specifically: 1. input the initial bounding box and the appearance feature vector of the target into the selected tracking algorithm to initialize the internal state of the tracker; ​​​2. Inter-frame prediction: for each subsequent frame, the tracker performs the following operations: extract local features around the previous frame's tracking bounding box in the current frame; localize the target through feature matching or motion model prediction; output the tracking bounding box of the current frame; 3. State update: if detection trigger is reached, i.e. the detection interval is met, correct the tracker's prediction error with the new detection bounding box and update the template features.

[0043] As shown in Figure 4 , the generation of the spatio-temporal probability map by the prediction network specifically comprises: S310, when a tracking bounding box continuously predicted by a dedicated tracker corresponding to a detection target is not confirmed by a new round of target detection performed within a preset number of frames, it is determined that the detection target is lost, and the detection target is marked as a lost target; The prediction result of the tracker may drift due to occlusion, background interference, target deformation, etc., i.e. the prediction box deviates from the real target; if the tracking bounding box of the target is not confirmed by the detection network for a continuous number of frames (the preset number of frames, such as 5 frames), it means that there is no matching target in the detection result, indicating that the tracking is unreliable. After determining that the target is lost, spatio-temporal prediction and directional detection are triggered to avoid continuous false tracking and attempt to find the real target.

[0044] S320, extract historical data of the lost target in the last N frames before the determination of loss, the historical data including N sets of target behavior parameters and N target appearance feature vectors, forming a behavior and appearance sequence of the lost target; S330, input the behavior and appearance sequence and the scene structure graph into the prediction neural network, encode the behavior and appearance sequence, and decode the possible position distribution of the lost target on the image plane at T future time points in combination with the traffic constraints described by the scene structure graph; S340, for each of the T future time points, output a probability map of the same size as the input image, the value of each pixel point on the probability map representing the probability of the lost detection target appearing at this pixel position at the time point, and stack the T probability maps in chronological order to form a spatio-temporal probability map.

[0045] The pixel probability output by the prediction network is generated through an encoding-decoding architecture, and the formula is: ; wherein: : the probability of the target appearing at pixel at future time ; ; ​: encoding features of behavior appearance sequence, , i.e. before loss frame features and behavior parameters; : scene structure graph graph convolutional encoding features; : future time index .

[0046] As shown in Figure 5 , the temporary detection confidence threshold is used to perform directional detection on the image region corresponding to the spatiotemporal region, specifically including: S410, based on the last calculated comprehensive priority score of the missing target, calculate the probability threshold and the planned retrieval frame number range for the missing target; 1. Probability threshold ; 2. Planned retrieval frame number range ; Wherein: : probability threshold, i.e. the minimum probability of screening valid pixel points; : basic threshold; : maximum threshold; : planned retrieval frame number, i.e. the number of video frames retrieved; : minimum retrieval frame number; : basic retrieval frame number; : retrieval frame number offset.

[0047] High priority targets The threshold is higher (screening more reliable regions), and the retrieval frame number is more (extending the recovery time).

[0048] S420, according to the probability threshold, threshold segmentation is performed on the spatiotemporal probability graph, each pixel point in the spatiotemporal probability graph is traversed, if the probability value stored in any pixel point is not less than the probability threshold, the pixel point is marked as a valid point, and the connected region formed by all valid points is marked as a screening spatiotemporal region; S430, determine the current video frame as the retrieval starting frame, according to the planned retrieval frame number range, determine the retrieval ending frame; in the specified frame number range from the retrieval starting frame to the retrieval ending frame, for each retrieval frame, according to the slice of the spatiotemporal probability graph at the corresponding time, find out all pixel coordinates falling within the screening spatiotemporal region; S440, only in the local region surrounded by the pixel coordinates on the current search frame image, the target detection neural network is called for directional detection to determine whether the missing target exists.

[0049] As shown in Figure 6 , the re-distribution of the detection frame interval and the tracker according to the corresponding dynamic priority score specifically includes: S510, if the directional detection exists a target position bounding box in the detection result output in the screening spatiotemporal region, and the similarity of the historical target appearance feature vector of the missing target exceeds the preset re-identification threshold, it is determined that the missing target is re-detected; The appearance feature vector of the target has identity uniqueness, the feature vector of the same target has high similarity, and the similarity of different targets is low; The cosine similarity of the feature vector of the newly detected target and the historical feature vector of the missing target is calculated: ; When the re-identification threshold, it is determined that the same target, avoiding misjudging other targets as missing targets; Combining the position (spatiotemporal probability graph region) and the appearance feature, the accuracy of re-identification is improved.

[0050] S520, the identification ID of the re-detected missing target is updated to the identification ID of the original missing target, and a new target appearance feature vector and a new target behavior parameter are calculated based on the new position bounding box of the missing target; S530, based on the comprehensive priority score of the original missing target, the new target appearance feature vector and the new target behavior parameter, the detection frame interval parameter and the tracking algorithm complexity parameter are re-calculated, and the special tracker is initialized to restore automatic tracking in the subsequent frames.

[0051] Figure 7 The structural block diagram of the automatic target detection and tracking system based on deep learning provided by the embodiment of the application is shown in Figure 7 , the system includes: A video stream and scene structure graph acquisition module 100 is used to acquire a video stream and a scene structure graph, and process video frames in sequence frame by frame, extract the position bounding box of the detection target in the frame through the detection network, generate the target appearance feature vector synchronously, and calculate the target behavior parameter and the priority score; A detection target parameter generation module 200 is used to generate the detection frame interval parameter and the tracking algorithm complexity parameter for each detection target according to the priority score, control the detection frequency of the detection target according to the detection frame interval parameter, and initialize the special tracker for the detection target according to the tracking algorithm complexity parameter to realize automatic tracking; The target loss processing module 300 is configured to, when the dedicated tracker determines to detect the target loss, generate a spatiotemporal probability map by a prediction network based on the behavior parameter, the appearance feature vector and the scene structure graph of the lost target before the target loss, wherein a pixel point in the map contains a probability value of the lost target appearing at a corresponding pixel point at a future time; The directional detection module 400 is configured to determine a probability threshold, a temporary detection confidence threshold and a search frame number range according to the priority score of the lost target and the spatiotemporal probability map, filter out a spatiotemporal region with a probability value higher than the probability threshold from the spatiotemporal probability map, and perform directional detection on an image region corresponding to the spatiotemporal region in a video frame within the search frame number range with the temporary detection confidence threshold. The tracking recovery module 500 is configured to associate the lost target and the detected target as the same target if the directional detection rediscover the lost target, and reallocate a detection frame interval and a tracker according to the priority score to recover the tracking.

[0052] The technical features of the above-described embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described, but it should be considered that any combination of the technical features that does not cause contradiction is within the scope of the present disclosure.

[0053] The above-described embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as limiting the scope of the present patent. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of protection of the present patent should be subject to the appended claims.

[0054] The above-described embodiments are only the preferred embodiments of the present application, and are not used to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A method for automatic target detection and tracking based on deep learning, characterized in that, The method comprises: acquiring a video stream and a scene structure diagram, and processing the video frame sequence frame by frame, extracting the position bounding box and category label of the detection target in the frame through a detection network, synchronously generating a target appearance feature vector, and calculating a target behavior parameter and a priority score; generating a detection frame interval parameter and a tracking algorithm complexity parameter for each detection target according to the priority score, controlling the detection frequency of the detection target according to the detection frame interval parameter, and initializing a special tracker for the detection target according to the tracking algorithm complexity parameter to realize automatic tracking; when the special tracker determines that the detection target is lost, generating a space-time probability map through a prediction network based on the behavior parameter, the appearance feature vector of the lost target before the target is lost, and the scene structure diagram, wherein a pixel point in the map contains a probability value of the lost target appearing at a corresponding pixel point at a future time; determining a probability threshold, a temporary detection confidence threshold and a search frame number range according to the priority score of the lost target and the space-time probability map, screening a space-time region with a probability value higher than the probability threshold from the space-time probability map, and performing directional detection on an image region corresponding to the space-time region in the video frames within the search frame number range with the temporary detection confidence threshold; if the lost target is rediscovered through the directional detection, associating the lost target and the detection target as the same target, and reallocating the detection frame interval and the tracker according to the priority score to restore tracking.

2. The method of claim 1, wherein, The method further comprises: accessing real-time video stream data from at least one image sensor, decoding the real-time video stream data into a video frame sequence arranged in chronological order, and synchronously loading a predefined scene structure diagram, wherein the scene structure diagram stores the connection relationship and semantic attributes of each region node in the scene in a graph data structure; inputting a current frame in the video frame sequence into a target detection neural network to output the position bounding box of all detection targets in the current frame; outputting a fixed-dimension floating-point number vector as a target appearance feature vector for each detection target based on the position bounding box of each detection target in the current frame; calculating the instantaneous displacement of the same detection target in the two-dimensional image plane based on the position bounding box center coordinates of the same detection target in the current frame and the previous frame to obtain a target behavior parameter, wherein the target behavior parameter comprises an instantaneous speed and a motion direction angle; calculating a priority score representing the tracking value of each target based on the target appearance feature vector and the target behavior parameter, and combining the semantic information of the current position of the target in the scene structure diagram.

3. The method of claim 2, wherein, The method further comprises: calculating the detection frame interval parameter and the tracking algorithm complexity parameter of each detection target according to the comprehensive priority score of each detection target; checking the frame interval of each detection target since the last detection when processing each frame of the video frame sequence, and only when the interval reaches the detection frame interval parameter stored in the state record, a new round of target detection is performed on the image region containing the target to update the position bounding box. According to the tracking algorithm complexity parameter of each detection target, a tracking algorithm instance corresponding to the complexity is selected as a dedicated tracker of the detection target, and the target appearance feature vector and the updated position bounding box are initialized as the internal state of the dedicated tracker; In each subsequent frame of the video frame sequence, the dedicated tracker corresponding to each detection target is called, and the current frame image is input, and the predicted tracking bounding box is output by the dedicated tracker to realize automatic target tracking of the detection target between frames.

4. The method of claim 3, wherein, The spatiotemporal probability map generated by the prediction network specifically includes: When the tracking bounding box continuously predicted by the dedicated tracker of a detection target is not confirmed by a new round of target detection performed within more than a preset number of frames, it is determined that the detection target is lost, and is marked as a lost target; Extract the historical data of the lost target in the last N frames before the loss is determined, and the historical data includes N groups of target behavior parameters and N target appearance feature vectors to form a behavior and appearance sequence of the lost target; The behavior and appearance sequence and the scene structure graph are input into the prediction neural network, the behavior and appearance sequence are encoded, and the possible position distribution of the lost target on the image plane at T future time points is decoded by combining the traffic constraints described by the scene structure graph; For each of the T future time points, a probability map with the same size as the input image is output, and the value of each pixel point on the probability map represents the probability of the lost detection target appearing at this pixel position at this time point, and the T probability maps are superimposed in time sequence to form a spatiotemporal probability map.

5. The method of claim 4, wherein, The image region corresponding to the spatiotemporal region is detected in a targeted manner with a temporary detection confidence threshold, specifically including: Based on the last calculated comprehensive priority score of the lost target, the probability threshold and the planned search frame number range for the lost target are calculated; According to the probability threshold, the spatiotemporal probability map is threshold segmented, and each pixel point in the spatiotemporal probability map is traversed, if the probability value stored in any pixel point is not less than the probability threshold, the pixel point is marked as an effective point, and the connected region formed by all effective points is marked as a screening spatiotemporal region; The current video frame is determined as the search starting frame, and the search ending frame is determined according to the planned search frame number range; in the specified frame number range from the search starting frame to the search ending frame, for each search frame, all pixel coordinates falling within the screening spatiotemporal region are found according to the slice of the spatiotemporal probability map at the corresponding time; Only in the local region surrounded by the pixel coordinates on the current search frame image, the target detection neural network is called for targeted detection to determine whether the lost target exists.

6. The method of claim 5, wherein, The detection frame interval and the tracker are re-allocated according to the corresponding dynamic priority score, specifically including: If there is a position bounding box of a target in the detection result output by the targeted detection in the screening spatiotemporal region, and the similarity between the historical target appearance feature vector of the lost target and the position bounding box exceeds a preset re-identification threshold, it is determined that the lost target is detected again; The identification ID of the re-detected lost target is updated to the identification ID of the original lost target, and a new target appearance feature vector and a new target behavior parameter are calculated based on the new position bounding box of the lost target; Based on the original missing target comprehensive priority score, the new target appearance feature vector and the new target behavior parameter, the detection frame interval parameter and the tracking algorithm complexity parameter are recalculated, and the dedicated tracker is initialized to resume automatic tracking in the subsequent frames.

7. An automatic target detection and tracking system based on deep learning, characterized in that, The system comprises: A video stream and scene structure graph acquisition module is configured to acquire a video stream and a scene structure graph, and to process video frame sequences frame by frame, to extract the position bounding box and the category label of a detection target in a frame through a detection network, to synchronously generate a target appearance feature vector, and to calculate a target behavior parameter and a priority score. A detection target parameter generation module is configured to generate a detection frame interval parameter and a tracking algorithm complexity parameter for each detection target according to the priority score, to control the detection frequency of the detection target according to the detection frame interval parameter, and to initialize a dedicated tracker for the detection target according to the tracking algorithm complexity parameter to realize automatic tracking. A target loss processing module is configured to, when the dedicated tracker determines that a detection target is missing, generate a space-time probability map through a prediction network based on the behavior parameter, the appearance feature vector and the scene structure graph of the missing target before the target is lost. A directional detection module is configured to determine a probability threshold, a temporary detection confidence threshold and a search frame number range according to the priority score of the missing target and the space-time probability map, to select a space-time region with a probability value higher than the probability threshold from the space-time probability map, and to perform directional detection on an image region corresponding to the space-time region in the video frames within the search frame number range with the temporary detection confidence threshold. A tracking recovery module is configured to, if the directional detection rediscovers the missing target, associate the missing target and the detection target as the same target, and to reallocate the detection frame interval and the tracker according to the priority score to recover the tracking.

Images