A dynamic target tracking and behavior prediction method based on deep learning

Abstract

The application discloses a kind of dynamic target tracking and behavior prediction method based on deep learning, it is related to AI technical field, including steps: S1, target feature extraction: using deep learning model, the appearance feature and motion feature of target are extracted from video frame;S2, dynamic target tracking;S3, target historical trajectory and action sequence construction;S4, behavior prediction.The dynamic target tracking and behavior prediction method based on deep learning, the appearance and motion feature of target are automatically extracted using deep learning model, can effectively deal with target occlusion, light change, deformation and background complex etc., realizes stable and accurate dynamic target tracking.Meanwhile, by combining time series deep learning model, the historical trajectory and action sequence of target are deeply learned, the motion law and behavior mode of target can be captured, and then the behavior of target in future short time is accurately judged.

Description

Technical Field

[0001] This invention relates to the field of AI technology, specifically to a method for dynamic target tracking and behavior prediction based on deep learning. Background Technology

[0002] In the fields of computer vision and artificial intelligence, dynamic target tracking and behavior prediction are two highly challenging tasks, widely used in security monitoring, autonomous driving, drone inspection, and robot interaction. Traditionally, dynamic target tracking mainly relies on algorithms based on hand-designed features, such as background subtraction, inter-frame differencing, and optical flow. These methods perform reasonably well in simple or controlled environments, but their performance deteriorates significantly when faced with complex and changing real-world scenarios, such as target occlusion, drastic lighting changes, target deformation, or highly complex backgrounds, leading to tracking loss or errors.

[0003] In the field of behavior prediction, early research focused primarily on rule-based methods, which predetermine target behavior by establishing a set of behavioral patterns or rules. However, this approach lacks flexibility and adaptability, making it difficult to handle complex and ever-changing real-world scenarios. With the development of machine learning technology, behavior prediction methods based on statistical models have gradually emerged, such as Hidden Markov Models (HMMs) and Conditional Random Fields (CRFs). Although these methods have improved prediction accuracy to some extent, they are still limited by model complexity and data representation capabilities, making it difficult to capture the long-term dependencies and complex patterns of target behavior.

[0004] In summary, traditional methods have significant limitations when dealing with dynamic target tracking and behavior prediction in complex scenarios. Manually designed features are insufficient to fully cover all possible scene changes, leading to unstable tracking; while behavior prediction methods based on rules or simple statistical models lack the ability to understand and capture the deep-seated patterns of target behavior. Summary of the Invention

[0005] The purpose of this invention is to provide a deep learning-based method for dynamic target tracking and behavior prediction to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for dynamic target tracking and behavior prediction based on deep learning, comprising the following steps: S1. Target Feature Extraction: Using a deep learning model, extract the appearance and motion features of the target from video frames to provide basic data for subsequent dynamic target tracking and behavior prediction. S2, Dynamic Target Tracking: Locks onto a specified target in consecutive video frames and outputs the target's (such as a person, vehicle, or drone) location information in real time; S3. Construction of target historical trajectory and action sequence: Record and organize the target's motion trajectory and action sequence over a period of time to provide data support for behavior prediction; S4. Behavior Prediction: By combining the target's historical trajectory and action sequence, predict the target's next movement direction and behavioral intention.

[0007] Furthermore, in step S1, the video frames are processed by a pre-trained deep learning model. The model outputs feature vectors of the target, which can represent the uniqueness of the target, facilitating matching and tracking in subsequent frames. Specifically, a combination of CNN and RNN is used, as follows: CNN extracts appearance features: It uses pre-trained convolutional neural network (CNN) models (such as VGG, ResNet, etc.) to extract appearance features such as color, texture, and shape of the target. The CNN model automatically learns and extracts high-level feature representations in the image through structures such as convolutional layers and pooling layers. RNN captures motion features: The appearance features extracted by CNN are used as input to RNN, and combined with time series information to capture the motion features of the target. RNN models (especially their variants such as LSTM and GRU) can process sequence data and capture the dynamic changes of the target over time.

[0008] Furthermore, the feature extraction can also employ a two-stream network approach, using two independent neural network branches (i.e., "two-stream") to process inputs from different modalities respectively: Spatial stream: processes static video frames, i.e., single-frame images, capturing the appearance features of the target; Temporal flow: Processes optical flow information between consecutive frames (optical flow describes the changes of pixels in an image over time and can reflect the motion state of the target) to capture the motion features of the target.

[0009] Furthermore, in step S1, the appearance features include color, texture, shape, etc., and the motion features include speed, direction, acceleration, etc. The extracted appearance features and motion features are fused to obtain a comprehensive feature representation of the target.

[0010] Furthermore, in step S2, based on the extracted target features, a deep learning-based feature matching and tracking algorithm is implemented in consecutive video frames to lock and obtain the location information of the specified target in real time. Even if the target is briefly occluded or deformed, it can be accurately re-locked through feature matching.

[0011] Furthermore, the implementation process of the feature matching and tracking algorithm is as follows: Initialization: In the first frame of the video, the target features extracted in step S1 (such as appearance features extracted by CNN and motion features captured by RNN, or comprehensive features extracted by a two-stream network) are used as the initial template, and a tracker is initialized. The tracker will use these features as references to match and track the target in subsequent frames. Feature matching and tracking: For each subsequent video frame, the same deep learning model is used to extract features of possible targets in the current frame, calculate the similarity between the features of each candidate target in the current frame and the features of the initial template (such as using cosine similarity, Euclidean distance, etc.), and select the candidate target that is most similar to the features of the initial template as the tracking result in the current frame. Handling occlusion and deformation: When the target is briefly occluded or deformed, the tracker may temporarily lose the target. In this case, the target's motion history information is used to predict the possible location of the target. At the same time, the tracker continues to search for targets with similar features to the initial template near the predicted location to achieve re-locking. Output results: For each frame of video, output the location information of the tracked target (such as the coordinates of the bounding box). This location information will be used in subsequent steps S3 and S4.

[0012] Furthermore, the tracking template is updated periodically based on time (e.g., every N frames) or tracking quality (e.g., when the tracking confidence is below a certain threshold) to adapt to changes in the target's appearance during the tracking process. When updating, the target features tracked in the current frame are used as new template features.

[0013] Furthermore, the specific operation of step S3 is as follows: the target position information obtained during the dynamic target tracking process is arranged in chronological order to form the target's historical trajectory; at the same time, the target's action characteristics (such as walking, running, standing still, etc.) at each time point are recorded to construct an action sequence; in this step, the target's position information and action characteristics are stored through data structures (such as lists, arrays) to ensure the continuity and integrity of the data.

[0014] Furthermore, in step S3, the historical trajectory and action sequence are normalized and smoothed. The processing operations are as follows: 1) Normalization process: Convert data of different dimensions or ranges to a unified dimension or range so that the model can better process and compare the data: Historical trajectory normalization: For the location information of the target (such as the coordinates of the bounding box), each coordinate value is normalized to the [0,1] interval or other fixed interval by dividing each coordinate value by the width or height of the video frame, thereby eliminating the influence of different video frame sizes on the trajectory data; Action sequence normalization: For action features represented in numerical form (such as velocity, acceleration, etc.), the maximum-min normalization method is used. The minimum value is subtracted from each action feature value and then divided by the range (maximum value - minimum value) to obtain the normalized value. 2) Smoothing process: Eliminating noise and outliers in historical trajectories and action sequences makes the data smoother and more stable, thereby improving the reliability of behavior prediction; Historical trajectory smoothing: The trajectory data is smoothed using methods such as sliding window averaging, median filtering, or Gaussian filtering; for example, the sliding window averaging method replaces the value of the center point with the average value of all trajectory points within the window, thereby eliminating local noise; Action sequence smoothing: The same smoothing method is used for action sequence data; for example, a low-pass filter can be used to filter out high-frequency noise and retain the low-frequency action change trend.

[0015] Furthermore, step S4 includes the following sub-steps: S41. Use temporal deep learning models (such as LSTM and Transformer) to learn the historical trajectory and action sequence of the target, and capture the target's motion patterns and behavior patterns. S42. Based on the learned model, predict the target's behavior in the near future, such as whether a vehicle will change lanes or whether a pedestrian will cross the road. In practical applications, the historical trajectory and action sequence of the target obtained through real-time tracking are input into the trained model, and the model outputs the prediction results of the target's future behavior. Based on the prediction results, early warnings can be issued or corresponding measures can be taken, such as warning of dangerous behavior of suspicious persons in security monitoring and avoiding collisions in autonomous driving.

[0016] This invention provides a deep learning-based method for dynamic target tracking and behavior prediction, which has the following advantages: This invention utilizes a deep learning model to automatically extract the appearance and motion features of a target, effectively addressing situations such as target occlusion, lighting changes, deformation, and complex backgrounds, achieving stable and accurate dynamic target tracking. Simultaneously, by combining a temporal deep learning model to perform deep learning on the target's historical trajectory and action sequences, it can capture the target's motion patterns and behavioral patterns, thereby accurately predicting the target's behavior in the near future. This upgrade from passive identification to proactive prediction not only enhances the level of intelligence but also provides strong technical support for fields such as security monitoring, autonomous driving, drone inspection, and robot interaction, helping to mitigate risks and make informed decisions. Attached Figure Description

[0017] Figure 1This is a schematic diagram illustrating the operational steps of a deep learning-based dynamic target tracking and behavior prediction method according to the present invention. Detailed Implementation

[0018] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0019] Example 1 A deep learning-based dynamic target tracking and behavior prediction method (dynamic target tracking and behavior prediction based on CNN and LSTM) includes the following steps: Step S1, Target Feature Extraction: Select a surveillance video containing pedestrians and vehicles, with a video resolution of 1920x1080 and a frame rate of 30fps.

[0020] A pre-trained VGG16 model (CNN part) is used to extract appearance features of targets and vehicles from video frames, including color, texture, and shape. VGG16 automatically learns and extracts high-level feature representations from images through its convolutional and pooling layer structures. The appearance features extracted by the CNN are then used as input to an LSTM (a variant of RNN), which, combined with time-series information, captures motion features of targets and vehicles, such as speed and direction.

[0021] Step S2, Dynamic Target Tracking: In the first frame of the video, the target features extracted in step S1 are used as the initial template to initialize a deep learning-based feature matching tracker.

[0022] For each subsequent video frame, the same VGG16+LSTM model is used to extract features of possible targets in the current frame. The cosine similarity between the features of each candidate target in the current frame and the features of the initial template is calculated, and the candidate target with the highest similarity is selected as the tracking result of the current frame.

[0023] When the target is briefly occluded, the target's motion history information stored in the LSTM is used to predict the possible location of the target, and similar features are searched near the predicted location to achieve re-locking.

[0024] In this embodiment, the tracking template is updated every 10 frames, using the target features tracked in the current frame as the new template features.

[0025] Step S3: Construction of target historical trajectory and action sequence: Record the position information and motion characteristics of pedestrians and vehicles in the video over a continuous 30 seconds. Arrange the target position information (boundary box coordinates) obtained during dynamic target tracking in chronological order to form the target's historical trajectory. Simultaneously record the target's motion characteristics at each time point, such as walking, running, standing still, turning, etc., to construct a motion sequence.

[0026] The historical trajectory and action sequence are normalized by normalizing the position coordinates to the [0,1] interval, and the action features are normalized using the max-min normalization method. The historical trajectory is smoothed using a sliding window averaging method with a window size of 5 frames.

[0027] Step S4, Behavior Prediction: The LSTM model is used to learn the target's historical trajectory and action sequence to capture the target's movement patterns and behavior. Based on the learned model, the target's behavior within the next 5 seconds is predicted, such as whether a pedestrian will cross the road or whether a vehicle will change lanes.

[0028] Implementation Results The system successfully tracked the movement of pedestrians and vehicles in the surveillance video, maintaining stable tracking even when the targets were briefly obscured or deformed. Behavior prediction accuracy exceeded 85%, effectively alerting pedestrians to dangerous behaviors and vehicles to abnormal lane changes, providing strong support for security monitoring and autonomous driving.

[0029] Example 2 A deep learning-based dynamic target tracking and behavior prediction method (dynamic target tracking and behavior prediction based on a two-stream network) includes the following steps: Step S1, Target Feature Extraction: Select a segment of aerial video from a drone inspection, with a resolution of 1280x720 and a frame rate of 25fps. Use a dual-stream network to process the spatial and temporal information of the video frames separately.

[0030] Spatial Stream: Processes static video frames, using the ResNet50 model to capture the appearance features of the drone, such as color and shape.

[0031] Temporal Flow: Processes optical flow information between consecutive frames and uses the FlowNet model to capture the motion characteristics of the drone, such as speed and acceleration.

[0032] By fusing the features extracted from the spatial and temporal streams, a comprehensive feature representation of the UAV is obtained.

[0033] Step S2, Dynamic Target Tracking: In the first frame of the video, the integrated drone features extracted from S1 are used as the initial template to initialize a feature matching tracker based on a two-stream network.

[0034] For each subsequent video frame, ResNet50 and FlowNet are used to extract the spatial and temporal features of possible drones in the current frame. The Euclidean distance between each candidate drone feature in the current frame and the initial template feature is calculated, and the candidate drone with the smallest distance is selected as the tracking result of the current frame.

[0035] When a drone deforms due to changes in flight attitude, the system uses time-stream information to predict the drone's possible location and searches for similar features near the predicted location to achieve relocking.

[0036] When the tracking confidence is below 0.8, the tracking template is updated, using the drone features tracked in the current frame as the new template features.

[0037] Step S3: Construction of target historical trajectory and action sequence: Record the drone's position information and motion characteristics over a continuous 1-minute period. Arrange the drone's position information obtained during dynamic target tracking in chronological order to form the drone's historical trajectory. Simultaneously record the drone's motion characteristics at each time point, such as ascent, descent, hovering, and turning, to construct a motion sequence.

[0038] Historical trajectories and action sequences are normalized by normalizing position coordinates to the [0,1] interval, and action features are normalized using the Z-score method. Median filtering is used to smooth historical trajectories within a 3-frame window.

[0039] Step S4, Behavior Prediction: The Transformer model is used to learn the historical trajectory and action sequence of the drone, capturing its motion patterns and behavior. Based on the learned model, the drone's behavior in the next 10 seconds is predicted, such as whether it will enter a no-fly zone or collide with other drones.

[0040] Implementation Results In the drone inspection video, the drone's flight trajectory was successfully tracked, maintaining stable tracking even when the drone underwent deformation or rapid changes in flight attitude. The behavior prediction accuracy reached over 90%, effectively providing early warning of abnormal drone flight behavior and offering strong assurance for the safe conduct of drone inspections.

[0041] The embodiments of the present invention are given for illustrative and descriptive purposes only, and are not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of the invention, and to enable those skilled in the art to understand the invention and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A method for dynamic target tracking and behavior prediction based on deep learning, characterized in that, Includes the following steps: S1. Target Feature Extraction: Using a deep learning model, extract the appearance and motion features of the target from video frames; S2, Dynamic Target Tracking: Locks onto a specified target in consecutive video frames and outputs the target's position information in real time; S3. Construction of target historical trajectory and action sequence: Record and organize the target's movement trajectory and action sequence over a period of time; S4. Behavior Prediction: By combining the target's historical trajectory and action sequence, predict the target's next movement direction and behavioral intention.

2. The method for dynamic target tracking and behavior prediction based on deep learning according to claim 1, characterized in that, In step S1, the video frames are processed by a pre-trained deep learning model, and the model outputs the feature vector of the target. Specifically, a combination of CNN and RNN is used, and the operation is as follows: CNN is used to extract the appearance features of the target, and the appearance features extracted by CNN are used as the input of RNN, combined with time series information to capture the motion features of the target.

3. The method for dynamic target tracking and behavior prediction based on deep learning according to claim 2, characterized in that, The feature extraction can also employ a two-stream network approach, using two independent neural network branches to process inputs from different modalities: Spatial stream: processes static video frames, i.e., single-frame images, capturing the appearance features of the target; Temporal flow: Processes optical flow information between consecutive frames to capture the motion characteristics of the target.

4. The method for dynamic target tracking and behavior prediction based on deep learning according to claim 3, characterized in that, In step S1, the appearance features include color, texture, and shape, and the motion features include speed, direction, and acceleration. The extracted appearance features and motion features are fused to obtain a comprehensive feature representation of the target.

5. The method for dynamic target tracking and behavior prediction based on deep learning according to claim 1, characterized in that, In step S2, based on the extracted target features, a deep learning-based feature matching and tracking algorithm is implemented in consecutive video frames to lock and obtain the location information of the specified target in real time.

6. The method for dynamic target tracking and behavior prediction based on deep learning according to claim 5, characterized in that, The implementation process of the feature matching and tracking algorithm is as follows: Initialization: In the first frame of the video, the extracted target features are used as the initial template, and a tracker is initialized; Feature matching and tracking: For each subsequent video frame, calculate the similarity between each candidate target feature in the current frame and the initial template feature, and select the candidate target that is most similar to the initial template feature as the tracking result in the current frame; Handling occlusion and deformation: When the target is temporarily occluded or deformed, the target's motion history information is used to predict the target's position. At the same time, the search continues to find targets with similar features to the initial template near the predicted position to achieve re-locking. Output: For each frame of video, output the location information of the tracked target.

7. The method for dynamic target tracking and behavior prediction based on deep learning according to claim 6, characterized in that: The tracking template is updated periodically, either based on time or tracking quality, and the target features tracked in the current frame are used as the new template features during the update.

8. The method for dynamic target tracking and behavior prediction based on deep learning according to claim 1, characterized in that, The specific operation of step S3 is as follows: the target position information obtained during the dynamic target tracking process is arranged in chronological order to form the target's historical trajectory; at the same time, the action characteristics of the target at each time point are recorded to construct an action sequence.

9. The method for dynamic target tracking and behavior prediction based on deep learning according to claim 1, characterized in that, In step S3, the historical trajectory and action sequence are normalized and smoothed. The processing operations are as follows: 1) Normalization process: Convert data of different dimensions or ranges into a unified dimension or range: Historical trajectory normalization: For the target's location information, each coordinate value is normalized to the [0,1] interval by dividing it by the width or height of the video frame; Action sequence normalization: For action features represented in numerical form, the max-min normalization method is used to subtract the minimum value from each action feature value and then divide by the range to obtain the normalized value; 2) Smoothing process: Eliminate noise and outliers in historical trajectories and action sequences; Historical trajectory smoothing: The trajectory data is smoothed using methods such as sliding window averaging, median filtering, or Gaussian filtering; Action sequence smoothing: The same smoothing method is used for action sequence data.

10. The method for dynamic target tracking and behavior prediction based on deep learning according to claim 1, characterized in that, Step S4 includes the following sub-steps: S41. Use a time-series deep learning model to learn the target's historical trajectory and action sequence to capture the target's motion patterns and behavior patterns. S42. Based on the learned model, predict the target's behavior in the near future.

Images