A personnel tracking method and device based on state space model

Abstract

The present invention discloses a person tracking method and device based on a state-space model. The method comprises the following steps: acquiring surveillance video stream data; constructing a target detection model in conjunction with the state-space model and training the model; passing each frame of the data into the trained target detection model, detecting people in the image, and obtaining target detection results, wherein the class probability is the confidence level; predicting the next frame tracking result of the tracked target using a Kalman filter; performing Hungarian matching on high-confidence detection results and all tracking results based on appearance loss and EIoU loss; performing Hungarian matching on low-confidence detection results and unmatched tracking results based on EIoU loss; removing tracking target trajectories that have not been successfully matched within a certain period of time; and creating tracking trajectories for consecutive unmatched high-confidence detection results and incorporating them into a set of tracked targets. The present invention can improve the robustness of multi-target tracking.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-target tracking, and in particular to a personnel tracking method and device based on a state space model. Background Art

[0002] Multi-target tracking technology plays a crucial role in the field of computer vision, widely used in scenarios such as intelligent surveillance, autonomous driving, intelligent robotics, and sports analysis. With the increasing urbanization and widespread deployment of intelligent systems, the demand for this technology is rapidly increasing, especially in the field of people tracking. Therefore, the rational application of efficient and reliable deep learning algorithm models to accurately track people and targets has become a key issue.

[0003] In recent years, target detection and tracking technology, as a research hotspot in the field of computer vision, has made significant progress. Currently, common multi-target detection schemes include the following steps: First, high-performance target detection models, such as Faster R-CNN and YOLOX, are used to detect the target bounding box and confidence in each frame of the video in real time; second, motion model-based tracking algorithms, such as SORT and DeepSORT, are used. These algorithms predict the target's motion trajectory through the Kalman filter and use techniques such as the Hungarian algorithm to associate targets in consecutive frames; finally, the DeepSORT algorithm further integrates the re-identification model to capture the target's appearance features, thereby reducing the frequency of ID switching during the tracking process. In terms of matching mechanism, it not only relies on the traditional IOU (region of interest)-based cost matrix matching, but also adopts a cascade matching strategy combined with IOU matching to improve the accuracy and robustness of matching.

[0004] In summary, multi-target tracking is a key application of computer vision, with widespread demand across many fields. However, current solutions for tracking people in high-density environments still have shortcomings. With technological advancements, developing a more efficient and reliable multi-target tracking method is becoming increasingly important.

[0005] The performance of multi-object tracking relies heavily on the accuracy of the object detection model. While high-performance detectors such as Faster R-CNN and YOLOX excel in providing object bounding boxes and confidence scores, their performance can degrade significantly in complex environments, such as those with occlusion and illumination variations. Furthermore, many tracking algorithms, such as SORT and DeepSORT, are based on linear motion models, which assume that the object's motion is uniform and linear in time. However, real-world object motion is often nonlinear, often leading to inaccurate tracking results.

[0006] Occlusion is a common and challenging problem in multi-object tracking. When objects are highly similar, occlude each other, or blend into the background, re-identification models may struggle to distinguish between different objects. This can easily lead to tracking loss or misassignment of IDs, compromising tracking continuity and accuracy. Furthermore, even with techniques like the Hungarian algorithm, correctly matching detection boxes to existing tracks remains challenging when there are numerous or close objects in a scene. Data association errors can lead to tracking confusion, especially in scenes with frequent object interactions and dynamic changes.

[0007] While the re-identification model used in the DeepSORT algorithm may perform well on specific datasets, its generalization capabilities are often limited. The model may not be able to effectively handle objects with similar or changing appearances, resulting in degraded tracking performance. Furthermore, while DeepSORT employs a cascade matching strategy and IOU matching to improve matching accuracy, this approach may not be flexible enough when dealing with objects with irregular shapes or varying sizes. The cascade matching strategy may not be able to adapt to complex changes in the object, such as deformation or scale changes, which can lead to matching errors and unstable tracking during the tracking process.

[0008] Therefore, in order to improve the robustness and accuracy of tracking, a personnel tracking method based on the state space model is urgently needed for further research and innovation. Summary of the Invention

[0009] The purpose of the present invention is to address the deficiencies of the prior art and to propose a personnel tracking method and device based on a state space model.

[0010] The object of the present invention is achieved through the following technical solutions: In a first aspect, an embodiment of the present invention provides a personnel tracking method based on a state space model, comprising the following steps:

[0011] (1) Capturing the surveillance video according to the set frame rate and adjusting it to the specified size to obtain the surveillance video stream data;

[0012] (2) Based on YOLOv5, the feature extraction modules in the backbone network and feature pyramid network are optimized and improved by combining the state space model to build a target detection model; the target detection model is iteratively trained using the collected training data, and the parameters of the target detection model are optimized using the GIoU loss during the training process to obtain a trained target detection model;

[0013] (3) The surveillance video stream data is fed into the trained target detection model frame by frame, and the people in the picture are detected to obtain the target detection results, including the location, size and category of the target, and the probability of the category is used as its confidence;

[0014] (4) All target detection results are divided into high confidence group and low confidence group according to the confidence level; the target detection results of each frame are processed by Kalman filter to predict the tracking results of the next frame of the tracked target; the target detection results of the high confidence group and all tracking results are matched by Hungarian algorithm based on appearance loss and EIoU loss; the target detection results of the low confidence group and the remaining unmatched tracking results are matched by Hungarian algorithm based on EIoU loss; the tracking target trajectories that have not been successfully matched within a certain period of time are removed; the target detection results of the high confidence group that appear continuously and have not been successfully matched are used to create tracking trajectories and included in the set of tracked targets;

[0015] (5) Repeat steps (3) to (4) until the monitoring video stream ends.

[0016] Furthermore, the target detection model includes a backbone network, a feature pyramid network and a detection head. The backbone network includes a downsampling module, a feature extraction module, a feature fusion module and a pyramid pooling layer; the feature pyramid network includes a feature joint layer, an upsampling layer, a downsampling layer, a convolution layer and a feature extraction module; the detection head includes three types: small-size detection head, medium-size detection head and large-size detection head. Each type of detection head includes three branches, which are respectively used to determine the position, size and category of the target; wherein the feature extraction module includes two dimensional convolutional layer, batch normalization layer, Mish activation function, local space module, layer normalization, 2D selection scanning module, layer normalization and residual gate unit, the local space module sequentially includes a depth-separable convolutional layer, a batch normalization layer, a two-dimensional convolutional layer, a Mish activation function and a two-dimensional convolutional layer, the 2D selection scanning module sequentially includes a linear layer, a depth-separable convolutional layer, a Mish activation function, a local feature extraction layer, layer normalization and a linear layer, the residual gate unit includes a two-dimensional convolutional layer, a depth-separable convolutional layer and a Mish activation function;

[0017] Each frame of video image in the monitoring video stream data is input into the target detection model, first entering the backbone network, and preliminarily extracting image features through the downsampling module to obtain the initial feature map; the initial feature map extracts deep features through the first feature extraction module to obtain the first feature map; the first feature map further extracts deep features through the first feature fusion module and the second feature extraction module in turn to obtain the second feature map; the second feature map further extracts deep features through the second feature fusion module and the third feature extraction module in turn to obtain the third feature map; the third feature map further extracts deep features through the third feature fusion module and the fourth feature extraction module in turn to obtain the fourth feature map; the fourth feature map obtains the fourth feature map after pooling through the pyramid pooling layer; then enters the feature pyramid network, and the fourth feature map after pooling is fused with the third feature map through the feature union layer after the upsampling layer to obtain the first fused feature map, and the fourth feature map is obtained. After passing through the feature extraction module and the upsampling layer in sequence, a fused feature map is fused with the second feature map through a feature union layer to obtain a second fused feature map. After passing through the feature extraction module, the second fused feature map obtains a small-scale feature map. After passing through the convolution layer, the small-scale feature map is fused with the first fused feature map after passing through the feature extraction module through a feature union layer to obtain a third fused feature map. After passing through the feature extraction module, the third fused feature map obtains a medium-scale feature map. After passing through the convolution layer, the medium-scale feature map is fused with the fourth feature map after pooling through a feature union layer to obtain a fourth fused feature map. After passing through the feature extraction module, the fourth fused feature map obtains a large-scale feature map. Finally, it enters the detection head, and the small-scale feature map is sent to the small-size detection head to obtain a small-size detection result; the medium-scale feature map is sent to the medium-size detection head to obtain a medium-size detection result; the large-scale feature map is sent to the large-size detection head to obtain a large-size detection result.

[0018] Furthermore, the step (4) includes the following sub-steps:

[0019] (4.1) Target detection results with a confidence level greater than a preset high confidence threshold are classified into a high confidence group, target detection results with a confidence level less than or equal to the preset high confidence threshold and greater than or equal to the preset low confidence threshold are classified into a low confidence group, and target detection results with a confidence level less than the preset low confidence threshold are discarded;

[0020] (4.2) The target detection result of the current frame and the tracking result of the previous frame of the tracked target are passed to the Kalman filter to obtain the predicted tracking result of the tracked target in the current frame; wherein the tracking result is the coordinates of the tracked target and the corresponding unique ID;

[0021] (4.3) Perform the first round of high-confidence detection and matching, and use the Hungarian algorithm to complete the matching of the target detection results of the high-confidence group and all tracking results. Specifically, based on the target detection results of the high-confidence group and all tracking results, calculate the appearance loss and EIoU loss between each target detection result and each tracking result, filter out unmatched target associations by setting the appearance loss threshold and EIoU loss threshold, and then use the Hungarian algorithm to complete the matching. During this matching process, retain the tracking results that are not successfully matched, that is, retain the tracking targets that are not successfully matched;

[0022] (4.4) Perform a second round of low-confidence detection and matching, using the Hungarian algorithm to complete the matching of the target detection results of the low-confidence group and the remaining unsuccessfully matched tracking results from the previous round of matching. Specifically, based on the target detection results of the low-confidence group and the remaining unsuccessfully matched tracking results from the previous round of matching, calculate the EIoU loss between each target detection result and each tracking result, filter out unmatched target associations based on the EIoU loss threshold, and then use the Hungarian algorithm to complete the matching;

[0023] (4.5) After the matching process is completed, the tracking target that has not been successfully matched within a certain period of time will be removed;

[0024] (4.6) Track newly appeared targets, which come from unmatched high-confidence detection results. For each target detection result, if its confidence is greater than the preset tracking threshold and appears in two consecutive frames, a new tracking target trajectory is created for it and included in the set of tracking targets.

[0025] Furthermore, the calculation formula of the appearance loss is:

[0026]

[0027] Among them, C A Represents the appearance loss between the target detection result and the tracking result, Cosine Similarity represents the cosine similarity, a and b represent the appearance features of the tracked target in the tracking result and the appearance features of the target in the target detection result, respectively;

[0028] The calculation formula of the EIoU loss is:

[0029]

[0030] Among them, C EIoU represents the EIoU loss between the target detection result and the tracking result, IoU represents the intersection over union ratio, A and B represent the expanded target box in the tracking result and the expanded target box in the target detection result, respectively.

[0031] Furthermore, in the EIoU loss, the expanded target box is obtained by the following method:

[0032] First, based on the height and width of the original target frame and the initial expansion scale, the height and width of the expanded target frame are obtained according to the following formula:

[0033] h * =(2E t +1)h

[0034] w * =(2E t +1)w

[0035] E t =E0+λt

[0036] Among them, h * and w * They represent the height and width of the expanded target frame, h and w represent the height and width of the original target frame, E t Represents the expansion scale after iterative expansion, E0 represents the initial expansion scale, λ represents the step size of the iterative expansion process, t represents the number of iterations, in the high confidence detection and matching stage, t≥0; in the low confidence detection and matching stage, t=0;

[0037] Then, the coordinates of the expanded target frame are obtained based on the coordinates of the original target frame and the height and width of the expanded target frame;

[0038] Finally, the expanded target frame is obtained according to its coordinates, height, and width.

[0039] Furthermore, filtering out unmatched target associations by setting an appearance loss threshold and an EIoU loss threshold specifically includes:

[0040] If the calculated appearance loss is greater than the appearance loss threshold or the EIoU loss is greater than the EIoU loss threshold, the adjusted appearance loss is set to 1; otherwise, the adjusted appearance loss is set to half of the original appearance loss; the smaller value of the adjusted appearance loss and the EIoU loss is selected as the final loss of the first round of matching; the target association corresponding to the minimum value of the final loss is retained.

[0041] A second aspect of an embodiment of the present invention provides a personnel tracking device based on a state-space model, comprising one or more processors and a memory, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned personnel tracking method based on the state-space model.

[0042] A third aspect of an embodiment of the present invention provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, it is used to implement the above-mentioned personnel tracking method based on the state space model.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] (1) The present invention innovatively combines the state space model into a high-performance target detection model, processes each frame in the video frame to extract the position and size information of the human target; further optimizes the processing of low-confidence detection frames and the recognition ability of human appearance features, and performs multi-target tracking on the motion and appearance feature information of the detected people; at the same time, integrates the person re-identification technology and improves the person's motion model to adapt to the target dynamics in complex environments, thereby realizing a high-performance person target tracking method; the present invention provides an innovative solution to the problem of personnel tracking in high-density environments. Applying it to actual scenarios can provide better detection performance, reduce dependence on manual verification, thereby saving costs, and directly improve the efficiency and quality of business operations.

[0045] (2) The target detection model in the present invention adopts depthwise separable convolution, which reduces the amount of calculation and improves the calculation efficiency; the present invention combines the state-space model to improve the backbone network and feature extraction network of the target detection model, optimizes the processing of long-distance dependencies, reduces the computational complexity of the self-attention mechanism, and improves the processing performance of the target detection model on large-scale data; at the same time, a more efficient activation function is used to improve the nonlinear expression ability of the target detection model.

[0046] (3) The present invention adopts a multi-target tracking method based on Tracking-by-Detection, which divides all target detection results into high-confidence and low-confidence detection frames according to the confidence level. By evaluating the similarity between the detection frame and the existing tracking trajectory, it not only retains the high-confidence detection results, but also filters out the real objects from the low-confidence detection results, which may include targets that are difficult to identify due to challenging conditions such as occlusion or blur; it effectively reduces missed detections and enhances the continuity and integrity of the tracking trajectory.

[0047] (4) The present invention uses a target association algorithm to match target detection results with tracking targets. The target association algorithm considers both appearance loss and EIoU loss. Through iterative expansion of EIoU, it is used in scenes with dense crowds and complex motions. It can effectively solve the problem of short-term multi-target tracking with nonlinear, irregular motion and similar appearance, and improve the robustness of tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1A flow chart of an implementation of the personnel tracking method based on the state space model of the present invention;

[0049] Figure 2 Another implementation flow chart of the personnel tracking method based on the state space model of the present invention;

[0050] Figure 3 A flowchart of the network architecture of the target detection model constructed in combination with the state-space model of the present invention;

[0051] Figure 4 This is a flow chart of the network architecture of the feature extraction module in the target detection model of the present invention;

[0052] Figure 5 The figure is a structural diagram of a personnel tracking device based on a state space model of the present invention. DETAILED DESCRIPTION

[0053] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numbers in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims. It should be understood that the foregoing general description and the detailed description that follows are exemplary and illustrative only and do not limit the present application.

[0054] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0055] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of..." or "when..." or "in response to determination." Moreover, the term "comprises," "comprising," or any other variant thereof is intended to cover non-exclusive inclusion, so that the process or method comprising a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process or method. In the absence of further restrictions, the elements defined by the statement "comprising a..." do not exclude the presence of other identical elements in the process, method, article, or device comprising the elements.

[0056] The present invention will be described in detail below with reference to the accompanying drawings. Unless there is any conflict, the features of the following embodiments and implementations may be combined with each other.

[0057] The personnel tracking method based on the state space model of the present invention is applicable to common multi-target tracking applications and has good performance in environments with similar clothing and high density, such as Figure 1 and Figure 2 As shown, the specific steps include:

[0058] (1) Capture the surveillance video according to the set frame rate and adjust it to the specified size to obtain the surveillance video stream data.

[0059] It should be noted that the specified size is the size required by the target detection model input.

[0060] (2) Based on YOLOv5, the feature extraction modules in the backbone network and feature pyramid network are optimized and improved by combining the state space model to construct a target detection model. The target detection model is iteratively trained using the collected training data. During the training process, the GIoU (Generalized Intersection over Union) loss is used to optimize the parameters of the target detection model to obtain a trained target detection model.

[0061] In this embodiment, the target detection model includes a backbone network, a feature pyramid network and a detection head. The backbone network includes a downsampling (Simple Stem) module, a feature extraction module, a feature fusion module and a pyramid pooling layer; the feature pyramid network includes a feature joint layer, an upsampling layer, a downsampling layer, a convolution layer and a feature extraction module; the detection head includes three types: small-size detection head, medium-size detection head and large-size detection head. Each type of detection head includes three branches, which are used to determine the position, size and category of the target, such as Figure 3 As shown in Figure 1, the feature extraction module includes a two-dimensional convolutional layer, a batch normalization layer, a Mish activation function, a local space module, a layer normalization, a 2D selection scanning module, a layer normalization, and a residual gate unit in sequence. The local space module includes a depth-separable convolutional layer, a batch normalization layer, a two-dimensional convolutional layer, a Mish activation function, and a two-dimensional convolutional layer in sequence. The 2D selection scanning module includes a linear layer, a depth-separable convolutional layer, a Mish activation function, a local feature extraction layer, a layer normalization, and a linear layer in sequence. The residual gate unit includes a two-dimensional convolutional layer, a depth-separable convolutional layer, and a Mish activation function. Figure 4 shown.

[0062] It should be understood that replacing all the original activation functions with the Mish activation function can show better smoothness and nonlinear characteristics compared to the traditional ReLU activation function, which helps to improve the expression ability of the target detection model when processing complex data and the stability during training.

[0063] In this embodiment, the backbone network processes each input video frame through a downsampling module, including using convolutional layers with large strides to reduce the spatial size of the image, reducing the computational burden of subsequent processing, and simultaneously extracting preliminary image features. The feature extraction module then performs deep feature extraction on the pre-processed feature map. This feature extraction module, combined with state-space model optimization, effectively captures features at different levels. The combination of the feature fusion module and the feature extraction module extracts features at multiple levels, while also processing temporal dynamic information in the image, which is particularly important for moving pedestrians.

[0064] In this embodiment, a feature pyramid network is used to perform further multi-scale processing on the image features extracted at different levels by the backbone network. At this stage, the feature map is adjusted and refined in spatial resolution through a combination of upsampling and downsampling operations and additional convolution. This configuration enables the object detection model to simultaneously process both high-resolution, large-scale features and low-resolution, small-scale features, enabling more accurate identification and classification of pedestrians of varying scales appearing simultaneously in the image.

[0065] In this embodiment, the detection head receives feature maps processed by the feature pyramid network. Three different types of detection heads are configured based on different sizes: small, medium, and large. These heads are responsible for detecting small, medium, and large objects, respectively. Each type of detection head includes three branches, each used to determine the location, size, and category of an object. Specifically, these branches are configured using a specifically configured convolutional layer to determine the precise location, size, and category of the object.

[0066] It should be understood that the so-called large size, medium size and small size can be defined according to actual needs.

[0067] Specifically, if Figure 3 As shown, each frame of video image in the monitoring video stream data is input into the target detection model, first entering the backbone network, and the image features are preliminarily extracted through the downsampling module to obtain the initial feature map; the initial feature map extracts deep features through the first feature extraction module to obtain the first feature map; the first feature map further extracts deep features through the first feature fusion module and the second feature extraction module in sequence to obtain the second feature map; the second feature map further extracts deep features through the second feature fusion module and the third feature extraction module in sequence to obtain the third feature map; the third feature map further extracts deep features through the third feature fusion module and the fourth feature extraction module in sequence to obtain the fourth feature map; the fourth feature map obtains the pooled fourth feature map through the pyramid pooling layer. Then, each feature map output by the backbone network enters the feature pyramid network. The fourth feature map after pooling passes through the upsampling layer and is fused with the third feature map through the feature union layer to obtain the first fused feature map. The first fused feature map passes through the feature extraction module and the upsampling layer in turn and is fused with the second feature map through the feature union layer to obtain the second fused feature map. The second fused feature map passes through the feature extraction module to obtain a small-scale feature map; the small-scale feature map passes through the convolution layer and is fused with the first fused feature map after passing through the feature extraction module through the feature union layer to obtain the third fused feature map. The third fused feature map passes through the feature extraction module to obtain a medium-scale feature map; the medium-scale feature map passes through the convolution layer and is fused with the fourth feature map after pooling through the feature union layer to obtain the fourth fused feature map. The fourth fused feature map passes through the feature extraction module to obtain a large-scale feature map. Finally, the feature maps of each scale output by the feature pyramid network enter the detection head, and the small-scale feature map is sent to the small-size detection head to obtain the small-size detection result (i.e., the position, size and category of the small-size target); the medium-scale feature map is sent to the medium-size detection head to obtain the medium-size detection result (i.e., the position, size and category of the medium-size target); the large-scale feature map is sent to the large-size detection head to obtain the large-size detection result (i.e., the position, size and category of the large-size target).

[0068] like Figure 4 As shown, when any feature map passes through the feature extraction module, the feature map Z l-3 After passing through the two-dimensional convolution layer, batch normalization layer and Mish activation function, the feature map Z is obtained. l-2 , feature map Z l-2 After passing through the local space module, layer normalization, 2D selection scanning module, and then combined with the feature map Z l-2 Add and fuse the features to get the feature map Z l-1 , feature map Z l-1 After layer normalization and residual gate unit, it is combined with feature map Z l-1 Add and fuse the features to get the feature map Z l , which is the output of the feature extraction module. Among them, the input X of the local space module l-2 After passing through the depth separable convolution layer, batch normalization layer, two-dimensional convolution layer, Mish activation function, two-dimensional convolution layer and X l-2 Add and fuse the features to get the output X of the local space module l The input of the 2D selection scanning module passes through the linear layer, depth-separable convolution layer, Mish activation function, local feature extraction layer, layer normalization and linear layer in sequence to obtain the output of the 2D selection scanning module. After passing through the two-dimensional convolution layer of the first branch and the second branch, we get and After the depth-wise separable convolution layer, After adding the features for fusion, the output is processed by Mish activation function. After the multiplication operation, the output of the third two-dimensional convolution layer is Add and fuse the features to get the output F of the residual gate unit l .

[0069] In this embodiment, when using multiple optimization strategies to iteratively train the target detection model, it is necessary to first collect some training data, and then use data enhancement methods such as Mixup and Mosaic to process it. For example, Mixup is used to randomly select two images from each batch and mix them in a certain proportion to generate a new image, and Mosaic is used to combine multiple images into one image in a certain proportion to increase the diversity of training data and improve the generalization ability of the target detection model. Using the mixed precision training (FP16) method to train the target detection model can reduce video memory usage and increase training speed. Using the AdamW optimizer can reduce overfitting while accelerating model convergence. Using GIoU loss during training, and adjusting the parameters of the target detection model with minimizing GIoU loss as the optimization goal can improve bounding box regression and improve detection accuracy. After training is completed, a trained target detection model can be obtained.

[0070] It should be understood that the target detection model can be trained using commonly used training methods. These are all commonly used methods and will not be described in detail here.

[0071] (3) The surveillance video stream data is passed frame by frame into the trained target detection model to detect the people in the picture and obtain the target detection results, including the location, size and category of the target, and the probability of the category is used as its confidence.

[0072] (4) All target detection results are divided into high confidence group and low confidence group according to the confidence level; the target detection results of each frame are processed by Kalman filter to predict the tracking results of the next frame of the tracked target; the target detection results of the high confidence group and all tracking results are matched by Hungarian algorithm based on appearance loss and EIoU loss; the target detection results of the low confidence group and the remaining unmatched tracking results are matched by Hungarian algorithm based on EIoU loss; the tracking target trajectories that have not been successfully matched within a certain period of time are removed; the target detection results of the high confidence group that appear continuously and have not been successfully matched are used to create tracking trajectories and included in the set of tracked targets.

[0073] It should be understood that the Hungarian algorithm is a common partial graph matching algorithm. Its core is to find augmenting paths. It is an algorithm that uses augmenting paths to find the maximum matching of bipartite graphs. Therefore, the Hungarian algorithm can be used to match target detection results and tracking results.

[0074] In this embodiment, after obtaining the target detection results of all surveillance video stream data through the trained target detection model, all target detection results are retained, and all target detection results are divided into two groups of high confidence and low confidence according to the confidence level. First, the target detection results of the high confidence group are matched with the tracking targets. Since some tracking targets may fail due to factors such as blur and image jitter, the target detection results of the low confidence group are matched with the tracking targets that have not yet been successfully matched, so as to retain the target detection results of the low confidence group and eliminate the interference of the background. Among them, the target detection result is the output of the target detection model. It is the result of each frame and does not have a correlation with the previous and next frames; the tracking target is the trajectory of a target person in the entire video, that is, the corresponding position of the tracking target in each frame has a correlation with the previous and next frames; therefore, it is necessary to jointly determine the trajectory of the tracking target based on the target detection result and the matching algorithm. The specific implementation process is as follows:

[0075] (4.1) Target detection results with a confidence greater than a preset high confidence threshold T_high are classified into a high confidence group, target detection results with a confidence less than or equal to the preset high confidence threshold T_high and greater than or equal to the preset low confidence threshold T_low are classified into a low confidence group, and target detection results with a confidence less than the preset low confidence threshold T_low are discarded.

[0076] (4.2) The target detection result of the current frame and the tracking result of the previous frame of the tracked target are fed into the Kalman filter to obtain the predicted tracking result of the tracked target in the current frame. The tracking result is the coordinates of the tracked target and its corresponding unique ID.

[0077] (4.3) Perform the first round of high-confidence detection and matching, and use the Hungarian algorithm to complete the matching of the target detection results of the high-confidence group and all tracking results. Specifically, based on the target detection results of the high-confidence group and all tracking results, calculate the appearance loss and EIoU (ExpansionIoU, intersection-over-union) loss between each target detection result and each tracking result, filter out unmatched target associations by setting the appearance loss threshold and the EIoU loss threshold, and then use the Hungarian algorithm to complete the matching. During this matching process, the tracking results that were not successfully matched are retained, that is, the tracking targets that were not successfully matched are retained.

[0078] Furthermore, the calculation formula of appearance loss is:

[0079]

[0080] Among them, C AC represents the appearance loss between the target detection result and the tracking result, Cosine Similarity represents the cosine similarity, a and b represent the appearance features of the tracked target in the tracking result and the appearance features of the target in the target detection result, respectively. A The smaller it is, the greater the cosine similarity is, which means the greater the appearance similarity is, that is, the more the target detection results and the tracking results match.

[0081] Furthermore, the calculation formula of EIoU loss is:

[0082]

[0083] Among them, C EIoU Represents the EIoU loss between the target detection result and the tracking result, IoU represents the intersection over union ratio, A and B represent the expanded target box in the tracking result and the expanded target box in the target detection result, respectively. EIoU The smaller it is, the closer the target detection result and the target in the tracking result are.

[0084] Furthermore, the expanded target frame is obtained by the following method: First, the height and width of the expanded target frame are obtained according to the following formula based on the height and width of the original target frame and the initial expansion scale:

[0085] h * =(2E t +1)h

[0086] w * =(2E t +1)w

[0087] E t =E0+λt

[0088] Among them, h * and w * They represent the height and width of the expanded target frame, h and w represent the height and width of the original target frame, E t The expanded scale after iterative expansion, E0, the initial expansion scale, λ, and t represent the step size of the iterative expansion process. During the high-confidence detection and matching phase, t ≥ 0; during the low-confidence detection and matching phase, t = 0. Next, the coordinates of the expanded target box are obtained based on the coordinates of the original target box and the height and width of the expanded target box. Finally, the expanded target box is obtained based on the coordinates of the expanded target box and its height and width.

[0089] It should be noted that further expanding the target box during the target association process can achieve better tracking performance for moving people. Traditional IoU often lacks flexibility. When the target boxes in the tracking results and target detection results share a small IoU or no IoU between adjacent frames, they will not be able to take into account the large amount of movement of the target. EIoU solves this limitation by modifying the size of the target box, expanding the width and height, and considering a wider range of associations. The expansion of the target box is controlled by setting the expansion scale. The height and width of the expanded target box are h and h respectively. * and w * If the coordinates of the upper left corner and lower right corner of the original target box are (t, l) and (b, r) respectively, the coordinates of the upper left corner of the expanded target box can be obtained and the coordinates of the lower right corner The expansion of the target box needs to be applied to both the last frame tracking result and the new target detection result of the tracking trajectory. The expanded target box is further used to calculate the EIoU loss between the target detection result and the tracking result. The calculated EIoU loss is used for the Hungarian algorithm association of adjacent frames.

[0090] It should be understood that the present invention can first associate tracking trajectories and target detection results with a higher expansion scale, and then gradually search for tracking trajectories and detection results with a smaller overlapping area to enhance the robustness of the association. Iterative expansion of EIoU only applies to the high-confidence detection and matching stage. When the number of iterations reaches the preset total number of iterations, the high-confidence detection and matching stage will be terminated and the low-confidence detection and matching stage will be entered.

[0091] In this embodiment, the appearance loss threshold and the EIoU loss threshold are set to filter out unmatched target associations. Specifically, if the calculated appearance loss C A Greater than the appearance loss threshold τ A Or EIoU loss C EIoU Greater than the EIoU loss threshold τ EIoU , then the adjusted appearance loss Set to 1; otherwise, the adjusted appearance loss Set as the original appearance loss C A Half of the adjusted appearance loss The smaller value of the EIoU loss is used as the final loss C of the first round of matching stage1 ;Retain the final loss C stage1 The target association corresponding to the minimum value of .

[0092] It should be understood that since the tracking target 1 is associated with multiple detection targets, the tracking target 1 will have multiple target associations, that is, there will be multiple final losses C stage1 , when the final loss C stage1When it is the smallest, it means that the tracking target 1 is the best match for the detection target.

[0093] Furthermore, the adjusted appearance loss Expressed as:

[0094]

[0095] Furthermore, the final loss C of the first round of matching stage1 Expressed as:

[0096]

[0097] (4.4) Perform the second round of low-confidence detection and matching, and use the Hungarian algorithm to complete the matching of the target detection results of the low-confidence group and the remaining unsuccessful matching tracking results of the previous round. Specifically, based on the target detection results of the low-confidence group and the remaining unsuccessful matching tracking results of the previous round, calculate the EIoU loss between each target detection result and each tracking result, filter out unmatched target associations according to the EIoU loss threshold, and then use the Hungarian algorithm to complete the matching.

[0098] It should be understood that since the appearance of the target detection results of the low confidence group is mostly degraded, only the EIoU loss is used as the matching basis in this round of matching.

[0099] (4.5) After the matching process is completed, the tracking targets that have not been successfully matched within a certain period of time will be removed. In the actual process, the tracking targets that have not been successfully matched in the second round will be temporarily retained and will not be removed until a certain period of time has passed.

[0100] (4.6) Track newly appeared targets, which come from unmatched high-confidence detection results. For each target detection result, if its confidence is greater than the preset tracking threshold T_E and it appears in two consecutive frames, a new tracking target trajectory is created for it and included in the set of tracking targets.

[0101] In this embodiment, in the process of matching the target detection results and the tracking results, it is necessary to associate the target detection results and the tracking results based on appearance and EIoU. Appearance similarity and EIoU (intersection-over-union) similarity are important bases for target association. Among them, appearance similarity is obtained by calculating the cosine similarity between the appearance features of the target in the target detection result and the appearance features of the tracked target in the tracking result; EIoU (intersection-over-union) similarity is obtained by calculating the intersection-over-union ratio between the target box in the target detection result and the tracked target box in the tracking result. The smaller the appearance loss and the EIoU loss, the greater the appearance similarity and the EIoU similarity. The greater the appearance similarity and the EIoU similarity, the more matched the two are. Therefore, some impossible associations can be filtered out using appearance similarity and EIoU similarity.

[0102] Specifically, appearance loss and EIoU loss are used to match target detection results with tracking results through confidence score perception. High-confidence target detection results typically indicate less occlusion and are more likely to retain reliable appearance features. Therefore, the first round of matching is the matching of target detection results from the high-confidence group with all tracking results. This is based on appearance loss and EIoU loss, denoted as Cstage1. The first round of matching is based on several rounds of iterative association and will be gradually expanded. The second round of matching is the matching of target detection results from the low-confidence group with the remaining unmatched tracking results. Only EIoU loss is used in this matching process, denoted as Cstage2.

[0103] In the first round of matching, unmatched target associations are filtered out by setting the appearance loss threshold and EIoU loss threshold. If the calculated appearance loss C A Greater than the appearance loss threshold τ A Or EIoU loss C EIoU Greater than the EIoU loss threshold τ EIoU , then the adjusted appearance loss Set to 1; otherwise, the adjusted appearance loss Set as the original appearance loss C A The final loss Cstage1 of the first round of matching is the smaller value of the adjusted appearance loss and EIoU loss. For the target detection results of the high confidence group and the matching process of all tracking results, unmatched target associations are filtered out based on the final loss Cstage1.

[0104] In the second round of matching, only the EIoU loss C is used EIoU As the final loss Cstage2 of the second round of matching. For the matching process of the target detection results of the low confidence group and the remaining unsuccessfully matched tracking results of the previous round, the unmatched target associations are filtered out based on the final loss Cstage2.

[0105] In summary, the present invention integrates large-scale public datasets such as CrowdHuman and CityPersons for training to improve the detection performance of target detection models in high-density environments. The integration of the spatial state model improves the target detection model, resulting in improvements in both training speed and model accuracy, and reduces missed detections of densely populated areas. When performing multi-target tracking, the target detection results of the high-confidence group and the low-confidence group are comprehensively considered, resulting in a significant improvement in tracking accuracy when processing videos of scenes with dense crowds and complex movements, reducing the ID switch problem that is prone to occur when people are obstructed.

[0106] This invention combines appearance loss and EIoU loss for target association, achieving state-of-the-art performance on the SportsMOT and SoccerNetTracking datasets. This demonstrates the effectiveness of the tracking method described in this invention for tracking people in sports scenes with nonlinear, irregular motion, and similar appearances. The tracking method described in this invention effectively addresses issues such as tracking ID switching and track breakage that can occur in crowded and complex scenes, improving its applicability and providing an innovative approach for multi-target tracking tasks, demonstrating its high practical value.

[0107] (5) Repeat steps (3) to (4) until the monitoring video stream ends.

[0108] Corresponding to the aforementioned embodiment of the personnel tracking method based on the state-space model, the present invention also provides an embodiment of a personnel tracking device based on the state-space model.

[0109] See also Figure 5 An embodiment of the present invention provides a personnel tracking device based on a state-space model, comprising one or more processors and a memory coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the personnel tracking method based on the state-space model in the above embodiment.

[0110] The embodiment of the personnel tracking device based on the state space model of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 5As shown in the figure, it is a hardware structure diagram of any device with data processing capability where the personnel tracking device based on the state space model of the present invention is located. Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0111] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0112] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0113] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the personnel tracking method based on the state space model in the above embodiment is implemented.

[0114] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0115] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A personnel tracking method based on a state space model, characterized in that: The following steps are involved: (1) Capturing the surveillance video according to the set frame rate and adjusting it to the specified size to obtain the surveillance video stream data; (2) Based on YOLOv5, the feature extraction modules in the backbone network and feature pyramid network are optimized and improved by combining the state space model to build a target detection model; the target detection model is iteratively trained using the collected training data, and the parameters of the target detection model are optimized using the GIoU loss during the training process to obtain a trained target detection model; (3) The surveillance video stream data is fed into the trained target detection model frame by frame, and the people in the picture are detected to obtain the target detection results, including the location, size and category of the target, and the probability of the category is used as its confidence; (4) Divide all target detection results into high confidence group and low confidence group according to confidence; Use the Kalman filter to process the target detection results of each frame and predict the tracking results of the next frame of the tracked target; perform Hungarian algorithm matching on the target detection results of the high-confidence group and all tracking results based on appearance loss and EIoU loss; perform Hungarian algorithm matching on the target detection results of the low-confidence group and the remaining unmatched tracking results based on EIoU loss; remove the tracking target tracks that have not been successfully matched within a certain period of time; For the target detection results of the high-confidence group that appear continuously and are not successfully matched, a tracking track is created for them and included in the set of tracked targets; (5) Repeat steps (3) to (4) until the monitoring video stream ends.

2. The personnel tracking method based on the state space model according to claim 1, characterized in that: The target detection model includes a backbone network, a feature pyramid network and a detection head, wherein the backbone network includes a downsampling module, a feature extraction module, a feature fusion module and a pyramid pooling layer; the feature pyramid network includes a feature union layer, an upsampling layer, a downsampling layer, a convolution layer and a feature extraction module; the detection head includes three types: a small-size detection head, a medium-size detection head and a large-size detection head, and each type of detection head includes three branches, which are respectively used to determine the position, size and category of the target; wherein the feature extraction module includes a two-dimensional convolution layer, a batch normalization layer, a Mish activation function, a local space module, layer normalization, a 2D selection scanning module, layer normalization and a residual gate unit in sequence, the local space module includes a depth-separable convolution layer, a batch normalization layer, a two-dimensional convolution layer, a Mish activation function and a two-dimensional convolution layer in sequence, the 2D selection scanning module includes a linear layer, a depth-separable convolution layer, a Mish activation function, a local feature extraction layer, layer normalization and a linear layer in sequence, and the residual gate unit includes a two-dimensional convolution layer, a depth-separable convolution layer and a Mish activation function; Each frame of video image in the monitoring video stream data is input into the target detection model, first entering the backbone network, and preliminarily extracting image features through the downsampling module to obtain the initial feature map; the initial feature map extracts deep features through the first feature extraction module to obtain the first feature map; the first feature map further extracts deep features through the first feature fusion module and the second feature extraction module in turn to obtain the second feature map; the second feature map further extracts deep features through the second feature fusion module and the third feature extraction module in turn to obtain the third feature map; the third feature map further extracts deep features through the third feature fusion module and the fourth feature extraction module in turn to obtain the fourth feature map; the fourth feature map obtains the fourth feature map after pooling through the pyramid pooling layer; then enters the feature pyramid network, and the fourth feature map after pooling is fused with the third feature map through the feature union layer after the upsampling layer to obtain the first fused feature map, and the fourth feature map is obtained. After passing through the feature extraction module and the upsampling layer in sequence, a fused feature map is fused with the second feature map through a feature union layer to obtain a second fused feature map. After passing through the feature extraction module, the second fused feature map obtains a small-scale feature map. After passing through the convolution layer, the small-scale feature map is fused with the first fused feature map after passing through the feature extraction module through a feature union layer to obtain a third fused feature map. After passing through the feature extraction module, the third fused feature map obtains a medium-scale feature map. After passing through the convolution layer, the medium-scale feature map is fused with the fourth feature map after pooling through a feature union layer to obtain a fourth fused feature map. After passing through the feature extraction module, the fourth fused feature map obtains a large-scale feature map. Finally, it enters the detection head, and the small-scale feature map is sent to the small-size detection head to obtain a small-size detection result; the medium-scale feature map is sent to the medium-size detection head to obtain a medium-size detection result; the large-scale feature map is sent to the large-size detection head to obtain a large-size detection result.

3. The personnel tracking method based on the state space model according to claim 1, characterized in that: The step (4) includes the following sub-steps: (4.1) Target detection results with a confidence level greater than a preset high confidence threshold are classified into a high confidence group, target detection results with a confidence level less than or equal to the preset high confidence threshold and greater than or equal to the preset low confidence threshold are classified into a low confidence group, and target detection results with a confidence level less than the preset low confidence threshold are discarded; (4.2) The target detection result of the current frame and the tracking result of the previous frame of the tracked target are passed to the Kalman filter to obtain the predicted tracking result of the tracked target in the current frame; wherein the tracking result is the coordinates of the tracked target and the corresponding unique ID; (4.3) Perform the first round of high-confidence detection and matching, and use the Hungarian algorithm to complete the matching of the target detection results of the high-confidence group and all tracking results. Specifically, based on the target detection results of the high-confidence group and all tracking results, calculate the appearance loss and EIoU loss between each target detection result and each tracking result, filter out unmatched target associations by setting the appearance loss threshold and EIoU loss threshold, and then use the Hungarian algorithm to complete the matching. During this matching process, retain the tracking results that are not successfully matched, that is, retain the tracking targets that are not successfully matched; (4.4) Perform a second round of low-confidence detection and matching, using the Hungarian algorithm to complete the matching of the target detection results of the low-confidence group and the remaining unsuccessfully matched tracking results from the previous round of matching. Specifically, based on the target detection results of the low-confidence group and the remaining unsuccessfully matched tracking results from the previous round of matching, calculate the EIoU loss between each target detection result and each tracking result, filter out unmatched target associations based on the EIoU loss threshold, and then use the Hungarian algorithm to complete the matching; (4.5) After the matching process is completed, the tracking target that has not been successfully matched within a certain period of time will be removed; (4.6) Track newly appeared targets, which come from unmatched high-confidence detection results. For each target detection result, if its confidence is greater than the preset tracking threshold and appears in two consecutive frames, a new tracking target trajectory is created for it and included in the set of tracking targets.

4. The personnel tracking method based on the state space model according to claim 3, characterized in that: The calculation formula of the appearance loss is: Among them, C A Represents the appearance loss between the target detection result and the tracking result, Cosine Similarity represents the cosine similarity, a and b represent the appearance features of the tracked target in the tracking result and the appearance features of the target in the target detection result, respectively; The calculation formula of the EIoU loss is: Among them, C EIoU represents the EIoU loss between the target detection result and the tracking result, IoU represents the intersection over union ratio, A and B represent the expanded target box in the tracking result and the expanded target box in the target detection result, respectively.

5. The personnel tracking method based on the state space model according to claim 4, characterized in that: In the EIoU loss, the expanded target box is obtained by the following method: First, based on the height and width of the original target frame and the initial expansion scale, the height and width of the expanded target frame are obtained according to the following formula: h * =(2E t +1)h w * =(2E t +1)w BY t =E0+λt Among them, h * and w * They represent the height and width of the expanded target frame, h and w represent the height and width of the original target frame, E t Represents the expansion scale after iterative expansion, E0 represents the initial expansion scale, λ represents the step size of the iterative expansion process, t represents the number of iterations, in the high confidence detection and matching stage, t≥0; in the low confidence detection and matching stage, t=0; Then, the coordinates of the expanded target frame are obtained based on the coordinates of the original target frame and the height and width of the expanded target frame; Finally, the expanded target frame is obtained according to its coordinates, height, and width.

6. The personnel tracking method based on the state space model according to claim 3, characterized in that: The method of filtering out unmatched target associations by setting the appearance loss threshold and the EIoU loss threshold specifically includes: If the calculated appearance loss is greater than the appearance loss threshold or the EIoU loss is greater than the EIoU loss threshold, the adjusted appearance loss is set to 1; otherwise, the adjusted appearance loss is set to half of the original appearance loss; the smaller value of the adjusted appearance loss and the EIoU loss is selected as the final loss of the first round of matching; the target association corresponding to the minimum value of the final loss is retained.

7. A personnel tracking device based on a state space model, comprising one or more processors and a memory, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the personnel tracking method based on the state space model according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, it is used to implement the personnel tracking method based on the state space model according to any one of claims 1 to 6.

Images