Target tracking method and device, electronic equipment and storage medium
Through the collaborative work of the object detection network, prediction network and trajectory generation network, the problem of high difficulty in tracking small targets in complex environments is solved, and accurate and continuous tracking of small targets is achieved, which is suitable for multi-objective scenarios in complex environments.
Patent Information
- Application Number
- CN202411177562.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-17
AI Technical Summary
Small-target tracking technology faces challenges such as high detection difficulty, occlusion, rapid movement and lighting changes in complex environments, making it difficult to meet tracking accuracy and real-time requirements.
The coordinated work of the target detection network, the target prediction network and the trajectory generation network is adopted to detect the target center point through a sequence of continuous multi-frame images, predict future locations, and build a trajectory tree to achieve accurate tracking of small goals.
It improves the detection accuracy and tracking stability of small targets, reduces tracking loss caused by occlusion and fast motion, and is suitable for multi-objective scenarios in complex environments.
Smart Images
Figure CN119151989A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of target tracking, and in particular to a target tracking method, device, electronic device and storage medium. Background Art
[0002] In the field of computer vision, target tracking technology aims to achieve accurate recognition and continuous tracking of target objects in continuous image sequences. With the evolution of technology and the expansion of application scenarios, the necessity of small target tracking has become more and more significant, especially in the fields of unmanned driving, video surveillance, motion analysis and sports competitions, which plays a decisive role in ensuring safety, obtaining accurate data and making effective decisions.
[0003] Despite this, small target tracking technology still faces a series of technical challenges: first, the small size of the target leads to a limited pixel share in the image, which is easily affected by background noise, thus increasing the difficulty of detection; second, frequent occlusions in complex environments can easily lead to tracking interruptions; third, small targets often present fast and irregular motion trajectories, which places higher demands on the stability and accuracy of the tracking algorithm; in addition, the rapid changes in lighting conditions also pose a challenge to small target detection; finally, real-time requirements set strict standards for the algorithm's computational efficiency, and the processing speed must be improved while maintaining tracking accuracy. Summary of the invention
[0004] The present application provides a target tracking method, device, electronic device and storage medium, aiming to solve the problem of how to improve the detection accuracy of small target tracking to achieve accurate tracking.
[0005] In a first aspect, the present application provides a target tracking method, the method comprising:
[0006] Obtain a continuous multi-frame image sequence of the target, and input it into a preset target detection network, and output a target position map, which marks the coordinate position and confidence of the target center point in each frame;
[0007] Obtaining target state information and inputting it into a preset target prediction network to output a position prediction distribution; the target state information includes the target's historical motion trajectory and the initial position of the current trajectory, and the position prediction distribution includes multiple possible future position prediction points and their corresponding probability values;
[0008] The target position map and the position prediction distribution are input into a preset trajectory generation network to construct a trajectory tree according to the matching results of the target center point in the target position map and the future position prediction point in the position prediction distribution, and output at least one trajectory tree, wherein the trajectory tree represents the motion trajectory of the tracked target.
[0009] In a second aspect, the present application further provides a target tracking device, the device comprising:
[0010] The target detection module is used to obtain a continuous multi-frame image sequence of the target, input it into a preset target detection network, and output a target position map, which marks the coordinate position and confidence of the target center point in each frame;
[0011] A target prediction module is used to obtain target state information and input it into a preset target prediction network to output a position prediction distribution; the target state information includes the historical motion trajectory of the target and the initial position of the current trajectory, and the position prediction distribution includes multiple possible future position prediction points and their corresponding probability values;
[0012] A trajectory generation module is used to input the target position map and the position prediction distribution into a preset trajectory generation network to construct a trajectory tree according to the matching result between the target center point in the target position map and the future position prediction point in the position prediction distribution, and output at least one trajectory tree, wherein the trajectory tree represents the motion trajectory of the tracked target.
[0013] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the steps of the target tracking method as described in any one of the first aspects are implemented.
[0014] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the target tracking method as described in any one of the first aspects.
[0015] The present application provides a target tracking method, device, electronic device and storage medium, which aims to solve the problem of how to improve the detection accuracy of small target tracking to achieve accurate tracking. The method can accurately detect the center point of the target and its confidence from a continuous multi-frame image sequence by using a preset target detection network. This is especially important for small targets, because small targets occupy fewer pixels in the image and are easily disturbed by background noise. The target detection network can effectively identify and locate small targets through the learned feature information, thereby improving the detection accuracy. The target prediction network uses the target state information to generate a position prediction distribution. The position prediction distribution contains multiple possible future position prediction points and their corresponding probability values, providing prediction support for the dynamic tracking of small targets. By predicting the future position of the small target, it is possible to keep tracking during the target movement and reduce the tracking loss caused by the target moving too fast or being blocked. The trajectory generation network constructs a trajectory tree representing the target trajectory based on the matching results of the target center point in the target position map and the prediction point in the position prediction distribution. This trajectory tree representation method helps the system to maintain continuous tracking of small targets in complex environments (such as dense targets or complex backgrounds). The output of the trajectory tree is multiple trajectories, each trajectory corresponds to a target, which helps the system distinguish and track small targets in multi-target scenes.
[0016] Therefore, the present application can effectively improve the detection accuracy of small targets through the collaborative work of three networks: target detection network, target prediction network and trajectory generation network, and realize accurate and continuous tracking of small targets through position prediction and trajectory generation. It is suitable for scenarios where small targets are not easy to be detected in images and are easily affected by environmental interference. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0018] Figure 1 is a flow chart of the target tracking method provided by the present application;
[0019] Figure 2 is a schematic diagram of a network architecture provided in an embodiment of the present application;
[0020] Figure 3 is a schematic diagram of the structure of the target detection network provided in the embodiment of the present application;
[0021] Figure 4 is a schematic diagram of a trajectory tree provided in an embodiment of the present application;
[0022] Figure 5 It is a schematic diagram of the structure of the target tracking device provided by the present application;
[0023] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0025] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.
[0026] In order to solve the problem of how to improve the detection accuracy of small target tracking to achieve accurate tracking, the present application provides a target tracking method, device, electronic device and storage medium. The method can accurately detect the center point of the target and its confidence from a continuous multi-frame image sequence by using a preset target detection network. This is especially important for small targets, because small targets occupy fewer pixels in the image and are easily disturbed by background noise. The target detection network can effectively identify and locate small targets through the learned feature information, thereby improving the detection accuracy. The target prediction network uses the target state information to generate a position prediction distribution. The position prediction distribution contains multiple possible future position prediction points and their corresponding probability values, providing prediction support for the dynamic tracking of small targets. By predicting the future position of the small target, it is possible to keep tracking during the target movement and reduce the tracking loss caused by the target moving too fast or being blocked. The trajectory generation network constructs a trajectory tree representing the target trajectory based on the matching results of the target center point in the target position map and the prediction point in the position prediction distribution. This trajectory tree representation method helps the system to maintain continuous tracking of small targets in complex environments (such as dense targets or complex backgrounds). The output of the trajectory tree is multiple trajectories, each trajectory corresponds to a target, which helps the system distinguish and track small targets in multi-target scenarios.
[0027] Combine the following Figure 1-Figure 6 The present application describes a target tracking method, device, electronic device and storage medium.
[0028] Please refer to Figure 1 , Figure 1 : is a flow chart of a target tracking method provided by the present application. A target tracking method comprises the following steps:
[0029] S110, obtaining a continuous multi-frame image sequence of the target, and inputting it into a preset target detection network, and outputting a target position map, which marks the coordinate position and confidence level of the target center point in each frame.
[0030] Specifically, a series of continuous image frames are captured from the video source, which form a time series, and each frame records the visual information of the target at different time points. These continuous image frames are input into a pre-trained target detection network. The target detection network is a deep learning model used to identify and locate targets in images. After processing each frame of the image, the target detection network outputs a target location map. The target location map is annotated with the coordinate position of the target center point detected in each frame. In the target location map, each detection result is also accompanied by a confidence value, which indicates the confidence level of the target detection network in the detection result, that is, the probability that the position is indeed the center point of the target.
[0031] S120, obtaining target state information and inputting it into a preset target prediction network to output a position prediction distribution; the target state information includes the historical motion trajectory of the target and the initial position of the current trajectory, and the position prediction distribution includes multiple possible future position prediction points and their corresponding probability values.
[0032] Specifically, the system collects information about the current state of the target, which includes the historical motion trajectory of the target and the initial position of the current trajectory. In addition, the input of the target prediction network also uses the hidden state information saved by itself, which is the motion information and pattern of the target in the past period of time. The target state information is input into a pre-trained target prediction network. The target prediction network is a deep learning model used to predict the future position of the target. After the target prediction network processes the input target state information, it outputs a position prediction distribution. The position prediction distribution contains multiple possible future position prediction points. Each prediction point in the position prediction distribution represents the position where the target may appear at a certain point in the future. Each prediction point is also accompanied by a probability value, which represents the probability that the target prediction network believes that the position is the actual future position of the target.
[0033] S130, input the target position map and the position prediction distribution into a preset trajectory generation network to construct a trajectory tree according to the matching results of the target center point in the target position map and the future position prediction point in the position prediction distribution, and output at least one trajectory tree, wherein the trajectory tree represents the motion trajectory of the tracked target.
[0034] Specifically, the target position map (including the coordinate position of the target center point and its confidence) and the position prediction distribution (including the future position prediction point and its corresponding probability value) are input into a preset trajectory generation network. The trajectory generation network constructs a trajectory tree based on the matching results of the target center point in the target position map and the future position prediction point in the position prediction distribution. This trajectory tree is a data structure that represents the motion trajectory of the target through tree-like branches. The system will output at least one trajectory tree, each corresponding to a target. This means that if there are multiple targets in the scene, the system will generate an independent trajectory tree for each target. Each trajectory tree is used to track the motion trajectory of the corresponding target. Each node in the trajectory tree represents the position of the target at a certain point in time, and the connection between the nodes represents the motion path of the target. The trajectory tree can represent and track the motion trajectory of the target, which helps to maintain continuous and accurate tracking of the target in complex environments (such as multiple targets, occlusion, fast movement, etc.).
[0035] In summary, the present application can effectively improve the detection accuracy of small targets through the collaborative work of three networks: target detection network, target prediction network and trajectory generation network, and realize accurate and continuous tracking of small targets through position prediction and trajectory generation.
[0036] The above steps S110 to S130 are described in detail below in conjunction with an embodiment.
[0037] Please refer to Figure 2 , Figure 2 Schematic diagram of the network architecture provided by the embodiment of the present application. The network architecture includes a target detection network, a target prediction network and a trajectory generation network.
[0038] Exemplarily, the target detection network includes an encoder and a decoder. The encoder adopts the residual connection structure in the deep convolutional neural network; through the residual connection structure, the encoder performs multi-level feature extraction operations on the input continuous multi-frame image sequence and generates a series of hierarchical feature maps, which capture visual features from low-level to high-level. The decoder adopts the skip connection structure in the convolutional neural network, which is used to directly transfer the feature map of each level in the encoder to the corresponding level of the decoder; the decoder upsamples and performs feature fusion processing on the received feature map layer by layer to generate a target position map.
[0039] For more information, see Figure 3 , Figure 3 : is a schematic diagram of the structure of the target detection network provided by the embodiment of the present application. Among them, the encoder steps include S301 to S318, and the decoder steps include S319 to S329. The details are as follows:
[0040] S301, input. That is, the target detection network receives a series of continuous image frames as input, which are extracted from the video stream to capture the movement and change of the target.
[0041] For example, the target detection network can receive 5 consecutive pictures as input, each of which has a size of (3,512,288), that is, each picture contains three RGB color channels and a resolution of 512x288 pixels.
[0042] S302, convolution. That is, the input image sequence first passes through a convolution layer. The convolution layer uses a set of learnable filters (convolution kernels) to slide on the image and calculate the dot product of the local area to extract the initial features.
[0043] S303, nonlinearity. That is, the output of the convolutional layer is processed by the ReLU (Rectified Linear Unit) activation function. The ReLU function sets all negative values to zero and keeps positive values unchanged, introducing nonlinearity and enhancing the expression ability of the target detection network.
[0044] S304, convolution. That is, the feature map after ReLU processing is further passed through a convolution layer for deeper feature extraction.
[0045] S305, non-linearity. That is, the output of the convolutional layer is processed again by the ReLU activation function to introduce non-linearity.
[0046] S306, convolution. That is, the feature map after ReLU processing continues to pass through a convolution layer to refine the features.
[0047] S304 to S306 above represent multi-level convolution operations. The feature map after ReLU processing is further passed through multiple convolution layers. These convolution layers gradually extract deeper features, and each convolution layer increases the depth of the feature map while reducing its spatial dimension.
[0048] S307, convolution. That is, the feature map passes through another convolution layer to further extract features.
[0049] S308, merging. That is, the output of S307 is merged with the output of S306. The merging is achieved through a residual connection structure, which helps to retain the feature information of the previous level and enhance the feature representation.
[0050] S309, convolution. That is, the merged feature map continues to pass through a convolution layer for further feature extraction.
[0051] S310, nonlinearity. That is, the output of the convolutional layer is processed by the ReLU activation function, introducing nonlinearity.
[0052] S311, convolution. That is, the feature map after ReLU processing continues to pass through a convolution layer to refine the features.
[0053] S312, convolution. That is, the feature map passes through another convolution layer to further extract features.
[0054] S313, merge. That is, the output of S312 is merged with the output of S311. This merging operation helps the network learn deeper feature representations.
[0055] S314, convolution. That is, the merged feature map continues to pass through a convolution layer for further feature extraction.
[0056] S315, nonlinearity. That is, the output of the convolutional layer is processed by the ReLU activation function, introducing nonlinearity.
[0057] S316, convolution. That is, the feature map after ReLU processing continues to pass through a convolution layer to refine the features.
[0058] S317, convolution. That is, the feature map passes through another convolution layer to further extract features.
[0059] S318, merge. That is, the output of S317 is merged with the output of S316. This merging operation helps the network capture more complex feature relationships.
[0060] S319, upsampling. That is, upsampling is performed on the merged feature map. Upsampling increases the spatial resolution of the feature map to make it closer to the size of the original image.
[0061] It should be noted that in this step, the decoder uses the skip connection structure in the convolutional neural network to directly pass the feature map output by step S318 in the encoder to the layer corresponding to step S319 in the decoder. The design of the skip connection structure allows the decoder to fuse the low-level features from the encoder while upsampling, thereby retaining more spatial detail information.
[0062] S320, convolution + nonlinearity + normalization. That is, the upsampled feature map and the output after S313 are combined through the convolution layer, ReLU activation function and batch normalization layer to further process and normalize the feature map to make it more suitable for subsequent processing.
[0063] It should be noted that in this step, the decoder adopts the skip connection structure in the convolutional neural network to directly pass the feature map output by step S313 in the encoder to the layer corresponding to step S320 of the decoder.
[0064] S321, convolution + nonlinearity + normalization. That is, the feature map continues to pass through the convolution layer, ReLU activation function and batch normalization layer for further feature processing and normalization.
[0065] S322, upsampling. That is, the feature map passes through an upsampling layer in the decoder and is gradually restored to the resolution of the original image.
[0066] S323, convolution + nonlinearity + normalization. That is, the upsampled feature map and the output after merging in S308 continue to pass through the convolution layer, ReLU activation function and batch normalization layer for feature processing.
[0067] It should be noted that in this step, the decoder adopts the skip connection structure in the convolutional neural network to directly pass the feature map output by step S308 in the encoder to the layer corresponding to step S303 of the decoder.
[0068] S324, convolution + nonlinearity + normalization. That is, the feature map continues to pass through the convolution layer, ReLU activation function and batch normalization layer for further feature processing and normalization.
[0069] S325, upsampling. That is, the feature map passes through an upsampling layer in the decoder and is gradually restored to the resolution of the original image.
[0070] S326, convolution + nonlinearity + normalization. That is, the upsampled feature map and the output after nonlinear processing in S303 continue to pass through the convolution layer, ReLU activation function and batch normalization layer for feature processing.
[0071] S327, convolution + nonlinearity + normalization. That is, the feature map continues to pass through the convolution layer, ReLU activation function and batch normalization layer for further feature processing and normalization.
[0072] S328, the final feature map passes through a convolution layer with a Sigmoid activation function. The Sigmoid function compresses the output between 0 and 1 to generate a target position map, which marks the position of the target (i.e., the coordinate position of the target center point) and its confidence.
[0073] S329, output. That is, the target detection network outputs the target position map.
[0074] In some embodiments, in S110, inputting it into a preset target detection network to output a target position map further includes post-processing the target position map output by the decoder, including:
[0075] S330, performing threshold filtering processing on the target position map to exclude pixels whose confidence is lower than a preset threshold, and retaining pixels in the target position map with higher confidence and more likely to be target center points, to obtain a target position map with low confidence filtered out.
[0076] Specifically, the target detection network outputs a target position map, in which each pixel represents the probability (i.e., confidence) that the position is the target center point. The target position map is generated by the last layer (i.e., step S328) of the target detection network using the Sigmoid activation function, so the value of each pixel is between 0 and 1. Threshold filtering refers to setting a preset threshold (e.g., 0.5) and then judging each pixel in the target position map. If the confidence (i.e., probability value) of a certain pixel is lower than the preset threshold, it is considered that the pixel is unlikely to be the target center point, so it is excluded. If the confidence of a certain pixel is higher than or equal to the preset threshold, it is considered that the pixel is more likely to be the target center point, so its value is retained. After threshold filtering, the target position map obtained will only contain those pixels with higher confidence and more likely to be the target center point. Such processing can effectively exclude those pixels that are unlikely to be the target center point, thereby reducing false detection and improving the accuracy of target detection.
[0077] S331, using a preset contour recognition function, contour extraction is performed on high-confidence pixels in the target position map with low confidence filtered out to determine a bounding box of each detected target.
[0078] Specifically, for example, the findContours function of OpenCV (a tool for detecting and extracting contours from binary images) is used to identify and extract the contours of objects in the image. Contour extraction refers to processing high-confidence pixels in the target position map using the contour recognition function to identify the contours of the targets represented by these pixels. The contour is composed of a series of continuous boundary points that define the shape of the target. After the contour is extracted, the bounding box of each target can be determined by calculating the circumscribed rectangle of the contour. The bounding box is a rectangle that surrounds the contour of the target and provides information about the position and size of the target in the image. The bounding box can be defined by the coordinates of its upper left corner and lower right corner.
[0079] S332, counting the pixel points in each bounding box, and calculating the detection score of the target as the confidence of the target; wherein each pixel value in the target position map represents the probability that the position of the pixel is the center point of the target.
[0080] Specifically, a statistical analysis is performed on the pixels in each bounding box, specifically, the values of these pixels in the target location map are averaged. Each pixel value in the target location map represents the probability that the location of the pixel is the center point of the target, so these values can reflect the confidence of the target. The detection score refers to the value obtained by averaging the values of all pixels in the bounding box. The mean can reflect the overall confidence of the target in the bounding box. The higher the mean, the higher the confidence of the target, and vice versa. The calculated detection score is used as the confidence of the target. Confidence is an indicator to measure the reliability of the target detection result, which is used to evaluate the performance of the target detection algorithm.
[0081] For example, for each of the five input images, a corresponding target location map is generated, and the detection score (i.e., confidence) of each target is calculated through the above steps. In other words, for each input image, a target location map is generated, and the detection score of each detected target is calculated. The above steps are repeated for each input image to calculate its detection score until all input images are processed, and finally the detection score of each target in the five input images is obtained.
[0082] In some embodiments, the target tracking method described in the present application, before executing S110, further includes:
[0083] S100, simulating different moving speeds by modifying the distance change of the target relative to its initial position in each frame. For example, different moving speeds of the target in the video sequence can be simulated by cutting out, blocking, random speed calculation and pasting operations.
[0084] Specifically, each frame in the video sequence is processed based on the Ground Truth (GT, which refers to the bounding box and category label of the object in each frame) information. First, the image area of the target is extracted from each frame, that is, the target is cut out. Then, the image areas in other frames where the target does not exist are selected, and these areas are used to cover the image areas of the target extracted in the current frame, thereby achieving occlusion of the target. Next, based on the randomly generated speed change, the new position to which the target should move in the current frame is calculated. The speed change can be random or set according to the possible moving speed range of the target in the actual scene. Paste the previously cut-out object image to the calculated new position. Make sure that the pasted target image blends naturally with the background image to avoid obvious splicing marks.
[0085] Through the above steps, new training samples are generated. The training samples contain the movement of the target at different speeds, thereby increasing the diversity of the data. The generated enhanced samples are added to the training data set for subsequent model training.
[0086] This data augmentation method not only expands the diversity of target moving speeds, but also improves the detection ability of the target detection network for targets of different speeds. It enhances the robustness of the model, enabling it to maintain stable performance under various speed changes, and also improves the generalization ability of the model, enabling it to better adapt to unseen data.
[0087] It can be seen that the target detection network can make full use of the information in the time dimension by receiving preset (for example, 5) continuous pictures, which helps to capture the motion trajectory of dynamic targets, thereby improving the accuracy and robustness of detection. And receive 5 consecutive pictures as input, and output the processing results of the corresponding number (ie, 5). This usage method helps the model to analyze and predict in time series. In addition, the efficient feature extraction capability of the encoder of the target detection network and the multi-scale feature fusion structure of the decoder make the target detection network have significant advantages in detection accuracy and real-time performance, especially when dealing with small targets in fast motion and complex backgrounds.
[0088] For example, Figure 2 As shown in the figure, the target prediction network includes a long short-term memory network (LSTM) and a Gaussian mixture density network (GMM). LSTM is responsible for encoding the target state information and capturing dynamic change information through recursive processing of time series. These dynamic change information contains the characteristics of the target moving over time. In other words, LSTM analyzes and learns the historical trajectory data of the target movement. These historical trajectory data may include information such as the position, velocity, acceleration, etc. of the target at continuous time points to capture the dynamic changes of the object's movement. The historical trajectory data can be output by the trajectory generation network and input to the LSTM. After processing the historical trajectory data, LSTM outputs a series of features. These features are encodings of the historical trajectory data, which contain dynamic change information in the time series and can be used to describe the movement pattern and trend of the target. This helps GMM understand the movement pattern and trend of the target, thereby making more accurate predictions.
[0089] GMM generates multiple Gaussian distributions based on the features output by LSTM, and samples the mixture of these multiple Gaussian distributions to output different prediction points and their corresponding probability values, which represent the credibility of the prediction point as the future location of the target. Since each Gaussian distribution has its own probability density function, sampling from the mixed distribution can obtain different prediction points, and each point has its corresponding probability value.
[0090] It can be seen that through the structure of LSTM and GMM, the target prediction network can not only provide multiple potential prediction positions, but also provide confidence information for each position, thereby improving the robustness and accuracy of target tracking.
[0091] Please refer to Figure 4 , Figure 4 It is a schematic diagram of a trajectory tree provided in an embodiment of the present application. Figure 4 The hierarchical and branching structures of two trajectory trees (T1 and T2) are shown. T1 is a trajectory tree with multiple layers and branches. is the root node, yes The direct child node of There are two branches below, pointing to P mdn and And P mdn and They each have child nodes, namely and P mdn . T2 is also a trajectory tree, where is the root node, The direct child nodes are The child node is P mdn .
[0092] In the target tracking system, the trajectory generation network can use multiple trajectory trees to represent and maintain the motion trajectories of multiple targets. Each node represents the target position at a specific time point, and the branch represents the possible motion direction of the target. Through this tree structure, the system can effectively manage and update the trajectory information of the target. Among them, each trajectory tree includes multiple nodes, including node structure and node information, as follows:
[0093] Node structure: Each node contains a parent node and a set of child nodes. The parent node indicates the parent position of the node, thus helping to build the hierarchical structure of the trajectory. The child nodes in the child node set represent multiple possible trajectory directions starting from the node. The child node set is used to extend the trajectory and capture the future movement possibilities of the target.
[0094] Node information, including the location information of the point, whether it is a detected point, confidence, hidden state, and one or more combinations of the predicted point set and corresponding confidence, as follows:
[0095] The location information of a point indicates the specific location of the node in the image, that is, the specific coordinates of the target in the current frame. This helps track the movement trajectory of the target.
[0096] Whether it is a detected point indicates whether the node is a point directly detected by the target detection network, that is, whether it is the actual position detected by the target detection network. This mark is used to help distinguish between actual detection points and predicted points.
[0097] Confidence, each node has a confidence score, which indicates the credibility of the node's location information. The higher the confidence, the greater the possibility that the node is the target location.
[0098] Hidden state, the node stores the hidden state information passed down by the LSTM network. The hidden state information contains the historical motion information and pattern of the target, which is used for further prediction and trajectory generation.
[0099] Prediction point set and corresponding confidence,Based on the trajectory represented by the node, the GMM network will generate multiple prediction points, each of which contains its location information and corresponding confidence.,These prediction point sets are used to guide the trajectory extension of the next frame.
[0100] Therefore, the trajectory tree includes the hierarchical relationship, location information, detection status, confidence, hidden state and prediction point set of nodes, which are used to generate, extend and predict trajectories in the target tracking system.
[0101] In some embodiments, in step S130, the step of matching the target center point in the target position map with the future position prediction point in the position prediction distribution includes:
[0102] S131, each target center point is regarded as an independent detection point and assigned to the corresponding trajectory tree. After receiving the detection point, each trajectory tree will match it according to the prediction subset contained in the terminal node of its activation state.
[0103] S132, for each combination of the detection point and the prediction point, calculate its matching score, which is a weighted summation result based on the confidence of the detection target and the confidence of the prediction point, and is used to quantify the matching degree between the detection point and the prediction point.
[0104] For example, for each pairing of a detection point (DT) and a prediction point (PD), a matching score (F(S_dt, S_pd)) is calculated. The matching score is the result of a weighted summation based on the confidence of the detection target (S_dt) and the confidence of the prediction point (S_pd), which is used to measure the degree of matching between the detection point and the prediction point.
[0105] S133, determining the node to which each prediction point belongs in the trajectory tree, where the node reflects the position of the prediction point in the time series, that is, the time level of the prediction point in the trajectory tree.
[0106] For example, determine the node (Node) to which each prediction point (PD) belongs in the trajectory tree.
[0107] S134, combining the calculated matching score, the node to which the prediction point belongs, the detection point, and the prediction point into a four-tuple, recorded as [matching score, node, detection point, prediction point].
[0108] For example, the matching score, the node to which the prediction point belongs, the detection point and the prediction point calculated above are combined into a four-tuple [F(S_dt, S_pd), Node, DT, PD]. This four-tuple represents the matching of a specific detection point and a prediction point. Among them, F is the result of weighted summation based on the detection target confidence and the prediction point confidence, Node refers to the node to which the prediction point belongs, DT refers to the detection point, and PD refers to the prediction point.
[0109] Finally, all combinations of detection points and prediction points are iterated, the matching scores of each combination are calculated one by one, and all quadruplets are generated. After the calculation is completed, they are sorted according to the F score.
[0110] In some embodiments, in step S130, the step of matching the target center point in the target position map with the future position prediction point in the position prediction distribution further includes:
[0111] S135, if the matching score is lower than the first threshold, it is determined that the prediction point coincides with the detection point, then a new node is created under the terminal node of the current trajectory tree based on the detection point, and a prediction subset of the next frame is predicted, and the new node is extended to the terminal branch node of the trajectory tree, and the trajectory is extended based on this.
[0112] For example, if the matching score is 0.1, which is lower than the first threshold (e.g., 0.2), the system considers that the predicted point and the detected point are almost coincident. At this time, the system takes the detected point DT as the reference, creates a new node N2 under the terminal node N1 of the current trajectory tree, representing the detected point DT, and predicts the predicted subset of the next frame based on N2, and extends the new node N2 to the terminal branch node of the trajectory tree, and extends the trajectory based on this.
[0113] S136, if the matching score is between the first threshold and the second threshold, it is determined that both the prediction point and the detection point are likely to belong to the trajectory of the current trajectory tree, and new trajectory branches are expanded based on the prediction point and the detection point respectively, and these two possible trajectory branches are retained.
[0114] For example, if the matching score is 0.3, which is between the first threshold (e.g., 0.2) and the second threshold (e.g., 0.5), the system considers that both the detection point and the prediction point are possible. Therefore, the system creates new nodes N2 and N3 under the terminal node N1, representing the detection point DT and the prediction point PD, respectively, and continues to predict the child nodes of the next frame under N2 and N3, retaining two possible trajectory branches.
[0115] S137, if the matching score is higher than the second threshold, it is determined that the current detection point does not belong to the trajectory of the current trajectory tree, then a new trajectory branch is expanded based on the prediction point, and the detection point is retained; if in subsequent frames, the detection point fails to match any trajectory, a new trajectory tree is created with the detection point as the starting point.
[0116] For example, if the matching score is 0.7, which is higher than the second threshold (e.g., 0.5), the system determines that the detection point DT does not belong to the trajectory of the current trajectory tree. In this case, the system only creates a new node N2 under the end node N1, representing the predicted point PD, and continues to predict the child nodes of the next frame under N2. At the same time, the system records the detection point DT and evaluates whether there are other trajectories that match DT in subsequent frames. If there is no matching trajectory, the system will create a new trajectory tree starting from DT to independently track the potential new target.
[0117] It should be noted that during the target tracking process, in order to ensure the uniqueness and accuracy of the trajectory tree, the system implements the following two key rules:
[0118] First, the uniqueness of the prediction point.
[0119] The system will not expand the same prediction point (PD) repeatedly. This means that once a prediction point has been used to create or expand a node in the trajectory tree, the system will not use the prediction point again to create or expand another node. This can avoid duplicate or redundant nodes in the trajectory tree and ensure the uniqueness and clarity of the trajectory.
[0120] Second, the exclusivity of the detection point.
[0121] The system will use the same detection point (DT) exclusively, that is, a detection point will not be used twice by the system. If a duplicate detection point is encountered during processing, the system will recognize this situation and skip the duplicate detection point without processing it. This prevents the same detection point from being incorrectly assigned to multiple tracks, thereby ensuring the accuracy and consistency of the track.
[0122] Therefore, through the above strategy, the system can flexibly handle the relationship between detection points and prediction points according to different intervals of matching scores, ensuring that the trajectory tree can be efficiently updated and maintained in complex or dynamically changing scenarios, thereby achieving accurate tracking of multiple targets.
[0123] In some embodiments, the present application designs a pruning mechanism to improve the efficiency and accuracy of the trajectory tree. The pruning mechanism optimizes the structure of the trajectory tree by removing unnecessary or unreliable trajectory branches. Specifically, the pruning mechanism includes the following two conditions:
[0124] First pruning condition: If the number of consecutive predictions of a node reaches a preset threshold, the node will enter the inactivation state and pass the inactivation state upward.
[0125] That is, if the number of consecutive predictions for a node exceeds the set threshold, the system will consider that the trajectory represented by the node has deviated or terminated, and will therefore mark it as inactive. The node and all its child nodes will no longer be expanded further. This pruning condition is used to deal with predicted nodes that have no detection point matches for a long time. By removing these nodes, the computational burden can be reduced while avoiding tracking useless trajectories.
[0126] Among them, the inactive state means that a node and its child nodes in the trajectory tree are marked as no longer participating in further expansion or updates. When a node enters the inactive state, it means that the system believes that the trajectory represented by the node is no longer valid or no longer worth tracking. When a node enters the inactive state, the system will stop further expansion and updates of the node and its child nodes. In addition, the inactive state will be passed upward, which means that the parent node of the node and its upper nodes may also be affected, which may eventually cause the entire branch to be removed or no longer updated.
[0127] Second pruning condition: If the detection score of a node does not reach a certain threshold in multiple consecutive frames, the node will enter the inactivated state and pass the inactivated state upward.
[0128] That is, if the average detection result of a node in multiple consecutive frames is lower than the set threshold, the system will consider the trajectory represented by the node to be unreliable or inaccurate, and thus mark it as inactive. This helps to remove those trajectory branches with unreliable or unstable detection results. By removing these unreliable nodes, the trajectory tree can be kept streamlined and effective, thereby improving the accuracy and reliability of tracking.
[0129] Therefore, by implementing these pruning conditions, it is possible to automatically identify and remove unnecessary or unreliable trajectory branches, thereby optimizing the structure of the trajectory tree and improving the overall performance of the system. This mechanism is particularly important for target tracking problems in complex scenes, because it helps the system focus resources on more likely target trajectories and improves tracking efficiency and accuracy.
[0130] Through the above steps, the system can track multiple targets at the same time and maintain an independent track for each target. These track trees can accurately track each target, ensuring that the position and motion trajectory of each target are accurately recorded and updated. In addition, it can adapt to the complex movement of the target, such as sudden acceleration, deceleration, turning, etc., as well as factors such as occlusion and lighting changes in the environment. Through dynamic adjustment and optimization, the efficiency and accuracy of target tracking can be ensured. For example, efficiency is reflected in the system's ability to quickly process and update trajectory information, while accuracy is reflected in the system's ability to provide accurate target positions and motion trajectories.
[0131] The target tracking device provided in the present application is described below. The target tracking device and the target tracking method described below can be referenced to each other.
[0132] Please refer to Figure 5 , Figure 5 Schematic diagram of the structure of the target tracking device provided by the present application. A target tracking device 500 includes a target detection module 510 , a target prediction module 520 and a trajectory generation module 530 .
[0133] Exemplarily, the target detection module 510 is used to obtain a continuous multi-frame image sequence of the target, and input it into a preset target detection network, and output a target position map, which marks the coordinate position and confidence level of the target center point in each frame.
[0134] Exemplarily, the target prediction module 520 is used to obtain target state information and input it into a preset target prediction network to output a position prediction distribution; the target state information includes the historical motion trajectory of the target and the initial position of the current trajectory, and the position prediction distribution includes multiple possible future position prediction points and their corresponding probability values.
[0135] Exemplarily, the trajectory generation module 530 is used to input the target position map and the position prediction distribution into a preset trajectory generation network to construct a trajectory tree based on the matching result between the target center point in the target position map and the future position prediction point in the position prediction distribution, and output at least one trajectory tree, wherein the trajectory tree represents the motion trajectory of the tracked target.
[0136] Exemplarily, the target detection network includes an encoder that adopts a residual connection structure in a deep convolutional neural network; through the residual connection structure, the encoder performs multi-level feature extraction operations on an input continuous multi-frame image sequence and generates a series of hierarchical feature maps, which capture visual features from low-level to high-level.
[0137] Exemplarily, the target detection network also includes a decoder, which adopts a skip connection structure in a convolutional neural network, and the skip connection structure is used to directly pass the feature map of each level in the encoder to the corresponding level of the decoder; the decoder upsamples and performs feature fusion processing on the received feature map layer by layer to generate the target position map.
[0138] Exemplarily, the target detection module 510 is further used for:
[0139] Performing threshold filtering on the target position map to exclude pixels whose confidence is lower than a preset threshold, and retaining pixels in the target position map with higher confidence and more likely to be target center points, thereby obtaining a target position map with low confidence filtered out;
[0140] Using a preset contour recognition function, contour extraction is performed on high-confidence pixels in the target position map that has been filtered out with low confidence, so as to determine the bounding box of each detected target;
[0141] The pixel points in each bounding box are counted and the detection score of the target is calculated as the confidence of the target; among them, each pixel value in the target position map represents the probability that the position of the pixel is the center point of the target.
[0142] Exemplarily, the target prediction network includes a long short-term memory network and a Gaussian mixture density network. The long short-term memory network is responsible for encoding target state information and capturing dynamic change information through recursive processing of time series. The dynamic change information includes the characteristics of the target moving over time; the Gaussian mixture density network generates multiple Gaussian distributions based on the dynamic change information output by the long short-term memory network, and samples the multiple Gaussian distribution mixtures to output different prediction points and their corresponding probability values.
[0143] Exemplarily, the trajectory generation network uses multiple trajectory trees to represent and maintain the motion trajectories of multiple targets, each trajectory tree includes multiple nodes, each node includes a parent node and a set of child nodes; the node information corresponding to each node includes the location information of the point, whether it is a detected point, confidence, hidden state, and a combination of one or more of the predicted point set and the corresponding confidence.
[0144] Exemplarily, the trajectory generation module 530 is further used for:
[0145] Each target center point is regarded as an independent detection point and assigned to the corresponding trajectory tree. After receiving the detection point, each trajectory tree will match it according to the prediction subset contained in its activated terminal node;
[0146] For each combination of a detection point and a prediction point, calculate its matching score, which is the result of weighted summation based on the confidence of the detection target and the confidence of the prediction point, and is used to quantify the degree of matching between the detection point and the prediction point;
[0147] Determine the node to which each prediction point belongs in the trajectory tree. This node reflects the position of the prediction point in the time series, that is, the time level of the prediction point in the trajectory tree;
[0148] The calculated matching score, the node to which the prediction point belongs, the detection point, and the prediction point are combined into a four-tuple, recorded as [matching score, node, detection point, prediction point].
[0149] Exemplarily, the trajectory generation module 530 is further used for:
[0150] If the matching score is lower than the first threshold, it is determined that the prediction point coincides with the detection point. Then, based on the detection point, a new node is created under the terminal node of the current trajectory tree, and the prediction subset of the next frame is predicted. The new node is extended to the terminal branch node of the trajectory tree, and the trajectory is extended based on this.
[0151] If the matching score is between the first threshold and the second threshold, it is determined that both the prediction point and the detection point are likely to belong to the trajectory of the current trajectory tree, and new trajectory branches are expanded based on the prediction point and the detection point respectively, and these two possible trajectory branches are retained;
[0152] If the matching score is higher than the second threshold, it is determined that the current detection point does not belong to the trajectory of the current trajectory tree, then a new trajectory branch is expanded based on the prediction point, and the detection point is retained; if in subsequent frames, the detection point fails to match any trajectory, a new trajectory tree is created with the detection point as the starting point.
[0153] Exemplarily, the trajectory generation module 530 is further used for:
[0154] If the number of consecutive predictions of a node reaches a preset threshold, the node will enter the inactive state and pass the inactive state upward;
[0155] If the detection score of a node does not reach the preset threshold in multiple consecutive frames, the node will enter the inactive state and pass the inactive state upward.
[0156] It should be noted here that the above-mentioned target tracking device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.
[0157] Figure 6is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the electronic device may include: a processor (Processor) 610, a communication interface (Communications Interface) 620, a memory (Memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the target tracking method.
[0158] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to ensure that a computer device (which can be a personal computer, a server, or a network device, etc.) executes all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.
[0159] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the target tracking method provided by the above methods.
[0160] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the target tracking method provided above.
[0161] The embodiments of the present application provide an electronic device, a computer program product, and a processor-readable storage medium, on which the computer program stored enables the processor to implement all the method steps implemented in the above-mentioned method embodiments and to achieve the same technical effects. The parts and beneficial effects of this embodiment that are the same as those in the method embodiments will not be described in detail herein.
[0162] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0163] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions to ensure that a computer device (which can be a personal computer, a server, or a network device, etc.) executes the methods described in each embodiment or some parts of the embodiments.
[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A target tracking method, characterized in that: The target tracking method comprises: Obtain a continuous multi-frame image sequence of the target, and input it into a preset target detection network, and output a target position map, which marks the coordinate position and confidence of the target center point in each frame; Obtaining target state information and inputting it into a preset target prediction network to output a position prediction distribution; the target state information includes the target's historical motion trajectory and the initial position of the current trajectory, and the position prediction distribution includes multiple possible future position prediction points and their corresponding probability values; The target position map and the position prediction distribution are input into a preset trajectory generation network to construct a trajectory tree according to the matching result between the target center point in the target position map and the future position prediction point in the position prediction distribution, and output at least one trajectory tree, wherein the trajectory tree represents the motion trajectory of the tracked target.
2. The target tracking method according to claim 1, characterized in that: The target detection network includes an encoder, which adopts the residual connection structure in the deep convolutional neural network; through the residual connection structure, the encoder performs multi-level feature extraction operations on the input continuous multi-frame image sequence and generates a series of hierarchical feature maps, which capture visual features from low-level to high-level.
3. The target tracking method according to claim 2, characterized in that: The target detection network also includes a decoder, which adopts a skip connection structure in a convolutional neural network, and the skip connection structure is used to directly transfer the feature map of each level in the encoder to the corresponding level of the decoder; the decoder upsamples and performs feature fusion processing on the received feature map layer by layer to generate the target position map.
4. The target tracking method according to claim 1, characterized in that: The step of inputting it into a preset target detection network to output a target position map comprises: Performing threshold filtering on the target position map to exclude pixels whose confidence is lower than a preset threshold, and retaining pixels in the target position map with higher confidence and more likely to be target center points, thereby obtaining a target position map with low confidence filtered out; Using a preset contour recognition function, contour extraction is performed on high-confidence pixels in the target position map that has been filtered out with low confidence, so as to determine the bounding box of each detected target; The pixel points in each bounding box are counted and the detection score of the target is calculated as the confidence of the target; among them, each pixel value in the target position map represents the probability that the position of the pixel is the center point of the target.
5. The target tracking method according to claim 1, characterized in that: The target prediction network includes a long short-term memory network and a Gaussian mixture density network. The long short-term memory network is responsible for encoding target state information and capturing dynamic change information through recursive processing of time series. The dynamic change information includes characteristics of target movement over time. The Gaussian mixture density network generates multiple Gaussian distributions according to the dynamic change information output by the long short-term memory network, and samples the multiple Gaussian distribution mixtures to output different prediction points and their corresponding probability values.
6. The target tracking method according to claim 1, characterized in that: The trajectory generation network uses multiple trajectory trees to represent and maintain the motion trajectories of multiple targets. Each trajectory tree includes multiple nodes, each node includes a parent node and a child node set; the node information corresponding to each node includes the location information of the point, whether it is a detected point, confidence, hidden state, and a combination of one or more of the predicted point set and the corresponding confidence.
7. The target tracking method according to claim 1, characterized in that: The step of matching the target center point in the target position map with the future position prediction point in the position prediction distribution comprises: Each target center point is regarded as an independent detection point and assigned to the corresponding trajectory tree. After receiving the detection point, each trajectory tree will match it according to the prediction subset contained in its activated terminal node; For each combination of a detection point and a prediction point, calculate its matching score, which is the result of weighted summation based on the confidence of the detection target and the confidence of the prediction point, and is used to quantify the degree of matching between the detection point and the prediction point; Determine the node to which each prediction point belongs in the trajectory tree. This node reflects the position of the prediction point in the time series, that is, the time level of the prediction point in the trajectory tree; The calculated matching score, the node to which the prediction point belongs, the detection point, and the prediction point are combined into a four-tuple, recorded as [matching score, node, detection point, prediction point].
8. The target tracking method according to claim 7, characterized in that: The step of matching the target center point in the target position map with the future position prediction point in the position prediction distribution also includes: If the matching score is lower than the first threshold, it is determined that the prediction point coincides with the detection point. Then, based on the detection point, a new node is created under the terminal node of the current trajectory tree, and the prediction subset of the next frame is predicted. The new node is extended to the terminal branch node of the trajectory tree, and the trajectory is extended based on this. If the matching score is between the first threshold and the second threshold, it is determined that both the prediction point and the detection point are likely to belong to the trajectory of the current trajectory tree, and new trajectory branches are expanded based on the prediction point and the detection point respectively, and these two possible trajectory branches are retained; If the matching score is higher than the second threshold, it is determined that the current detection point does not belong to the trajectory of the current trajectory tree, then a new trajectory branch is expanded based on the prediction point, and the detection point is retained; if in subsequent frames, the detection point fails to match any trajectory, a new trajectory tree is created with the detection point as the starting point.
9. The target tracking method according to claim 1, characterized in that: The step of matching the target center point in the target position map with the future position prediction point in the position prediction distribution also includes: If the number of consecutive predictions of a node reaches a preset threshold, the node will enter the inactive state and pass the inactive state upward; If the detection score of a node does not reach the preset threshold in multiple consecutive frames, the node will enter the inactive state and pass the inactive state upward.
10. A target tracking device, characterized in that: The target tracking device comprises: The target detection module is used to obtain a continuous multi-frame image sequence of the target, input it into a preset target detection network, and output a target position map, which marks the coordinate position and confidence of the target center point in each frame; A target prediction module is used to obtain target state information and input it into a preset target prediction network to output a position prediction distribution; the target state information includes the historical motion trajectory of the target and the initial position of the current trajectory, and the position prediction distribution includes multiple possible future position prediction points and their corresponding probability values; A trajectory generation module is used to input the target position map and the position prediction distribution into a preset trajectory generation network to construct a trajectory tree according to the matching result between the target center point in the target position map and the future position prediction point in the position prediction distribution, and output at least one trajectory tree, wherein the trajectory tree represents the motion trajectory of the tracked target.
11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the target tracking method according to any one of claims 1 to 10 are implemented.
12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the target tracking method according to any one of claims 1 to 10 are implemented.