A method, device, intelligent terminal and storage medium for tracking multiple pedestrians
Through the integration and separation structure of the target neural network model, the sharing of features is solved to combine detection and appearance feature extraction sub-tasks, and the problem of slow tracking of pedestrians in the prior art is solved, achieving efficient real-time tracking effect.
Patent Information
- Application Number
- CN202011641353.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-31
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-12-31
Smart Images

Figure CN112750147B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of multi-target tracking, and in particular to a method, device, intelligent terminal and storage medium for multi-target tracking of pedestrians. Background Art
[0002] With the continuous development of deep learning technology, multi-target tracking technology has received more and more attention. The main task of multi-target tracking is to find the moving objects in a given image sequence, match the moving objects in different frames of the image sequence one by one, and then give the motion trajectories of different objects. These objects can be arbitrary, such as pedestrians, vehicles, athletes, various animals, etc., and pedestrian tracking is the most studied, most valuable, and most widely used.
[0003] In recent years, with the development of driverless technology, pedestrian multi-target tracking technology has gradually become a research hotspot. Multi-target tracking technology can be used to predict the corresponding target running trajectories around the vehicle, avoid obstacles in advance, and ensure driving safety. In this scenario, the tracking speed of pedestrian multi-target tracking is required to be fast and meet real-time requirements. However, in the prior art, pedestrian multi-target tracking is usually divided into two separate subtasks: detection and appearance feature prediction. The problem with the prior art is that dividing pedestrian multi-target tracking into two separate subtasks, detection and appearance feature prediction, affects the tracking speed and cannot meet real-time requirements.
[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the invention
[0005] The purpose of the present invention is to provide a method, device, intelligent terminal and storage medium for multi-target pedestrian tracking, aiming to solve the problem that the multi-target pedestrian tracking in the prior art has a slow tracking speed and cannot meet the real-time requirements.
[0006] In order to achieve the above object, the present invention provides a method for multi-target pedestrian tracking in a first aspect, wherein the method comprises:
[0007] Acquire a target neural network model, wherein the target neural network model comprises: a unified model for extracting the total features of the input image to obtain a feature map, a separation structure for separating the feature map, and an appearance feature extraction sub-model and a detection sub-model for respectively predicting the appearance feature vector and target information of the acquired target based on the separated feature map;
[0008] Obtain target video sequence;
[0009] Based on the target neural network model, the target video sequence is subjected to multi-target pedestrian tracking to obtain tracking results;
[0010] Output the above tracking results.
[0011] Optionally, the above-mentioned obtaining the target neural network model includes:
[0012] Build a neural network model to be trained;
[0013] Get the training set;
[0014] The above-mentioned neural network model to be trained is trained based on the above-mentioned training set to obtain the above-mentioned target neural network model.
[0015] Optionally, the above-mentioned obtaining of the training set includes:
[0016] Obtain detection datasets and person re-identification datasets;
[0017] The training set is obtained by automatically annotating the data in the above detection dataset and the above person re-identification dataset.
[0018] Optionally, the automatic labeling based on the data in the detection dataset and the person re-identification dataset to obtain a training set includes:
[0019] Acquire a detection image and a person re-identification image based on the detection dataset and the person re-identification dataset respectively;
[0020] Segment the above person re-identification image into black and white binary images;
[0021] Extracting the corresponding part of the segmented mask in the above person re-identification image as a segmented image;
[0022] Predicting depth information of the detection image, and dividing the detection image into regions based on the depth information;
[0023] Based on the size of the segmented image, the segmented image is pasted on a corresponding area of the detection image to obtain a training image;
[0024] The training set is obtained based on the set of training images.
[0025] Optionally, the training of the neural network model to be trained based on the training set to obtain the target neural network model includes:
[0026] Training the neural network model to be trained based on the training set to obtain a trained neural network model;
[0027] Performing sparse training on the above trained neural network model;
[0028] Based on the model weights after sparse training, the preset target sparse number of convolution channels is deleted to obtain the above target neural network model.
[0029] Optionally, performing pedestrian multi-target tracking on the target video sequence based on the target neural network model to obtain tracking results includes:
[0030] For an input image corresponding to a frame in the target video sequence, predicting the position of a pedestrian target in the input image by using a detector in the target neural network model;
[0031] Use the Kalman filter to predict the position of the next frame based on the existing trajectory;
[0032] Based on pedestrian features and intersection-over-union ratio, the predicted target of the above detector is associated with the existing trajectory, the identity ID of the same pedestrian target in the previous and next frames is kept consistent, and the above tracking result is obtained.
[0033] Optionally, the outputting of the tracking result includes: based on the tracking result, visually outputting the trajectory of each of the pedestrian targets, and when outputting, the bounding boxes of pedestrian targets with different identity IDs have different colors.
[0034] A second aspect of the present invention provides a pedestrian multi-target tracking device, wherein the device comprises:
[0035] A target neural network model acquisition module, used to acquire a target neural network model, wherein the target neural network model includes: a unified model for extracting the total features of an input image to acquire a feature map, a separation structure for separating the feature map, and an appearance feature extraction sub-model and a detection sub-model for respectively predicting and acquiring an appearance feature vector and target information of a target based on the separated feature map;
[0036] A target video sequence acquisition module is used to acquire a target video sequence;
[0037] A tracking module, used to perform pedestrian multi-target tracking on the target video sequence based on the target neural network model to obtain tracking results;
[0038] The output module is used to output the above tracking results.
[0039] A third aspect of the present invention provides an intelligent terminal, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of any one of the above-mentioned pedestrian multi-target tracking methods.
[0040] A fourth aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the above-mentioned pedestrian multi-target tracking methods are implemented.
[0041] As can be seen from the above, the target neural network model is obtained by the scheme of the present invention, wherein the target neural network model includes: a unified model for extracting the total features of the input image to obtain a feature map, a separation structure for separating the feature map, and an appearance feature extraction sub-model and a detection sub-model for predicting the appearance feature vector and target information of the target based on the separated feature map; obtaining a target video sequence; performing multi-target pedestrian tracking on the target video sequence based on the target neural network model to obtain tracking results; and outputting the tracking results. Since the target neural network model obtained in the scheme of the present invention can extract the total features of the input image and obtain the feature map through the unified model, and separate the feature map through the separation structure, first unify and then separate, so that the appearance feature extraction sub-model and the detection sub-model can share features, and the two different sub-tasks are combined for joint prediction, which is conducive to improving the tracking speed and meeting the real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0043] Figure 1 It is a schematic diagram of a two-stage detection and tracking framework in the prior art provided by an embodiment of the present invention;
[0044] Figure 2 It is a schematic diagram of a one-stage detection and tracking framework in the prior art provided by an embodiment of the present invention;
[0045] Figure 3 It is a flowchart of a method for multi-target pedestrian tracking provided by an embodiment of the present invention;
[0046] Figure 4 Embodiment of the present invention Figure 3 Specific flow diagram of step S10;
[0047] Figure 5 It is a schematic diagram of a framework of a neural network model to be trained provided by an embodiment of the present invention;
[0048] Figure 6 Embodiment of the present invention Figure 5 Schematic diagram of the structure of the Zhongheyi model;
[0049] Figure 7 Embodiment of the present invention Figure 6 Schematic diagram of the specific structure of the basic model;
[0050] Figure 8Embodiment of the present invention Figure 6 Schematic diagram of the specific structure of the fusion block;
[0051] Fig. 9 Embodiment of the present invention Figure 4 Specific flow diagram of step S12;
[0052] Fig.10 Embodiment of the present invention Fig. 9 Specific flow diagram of step S122;
[0053] Fig.11 is a pedestrian re-identification image provided by an embodiment of the present invention;
[0054] Fig.12 Embodiment of the present invention Fig.11 The black and white binary image obtained by segmentation;
[0055] Fig.13 Embodiment of the present invention Fig.12 Schematic diagram of the mask extracted from ;
[0056] Fig.14 is a schematic diagram of depth information provided by an embodiment of the present invention;
[0057] Fig.15 The embodiment of the present invention provides a method based on Fig.13 and Fig.14 The obtained integrated graph;
[0058] Fig.16 Embodiment of the present invention Figure 4 Specific flow diagram of step S13;
[0059] Fig.17 Embodiment of the present invention Figure 3 Specific flow diagram of step S30;
[0060] Fig.18 It is a structural schematic diagram of a pedestrian multi-target tracking device provided by an embodiment of the present invention;
[0061] Fig.19 It is a block diagram of the internal structure principle of an intelligent terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0062] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.
[0063] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.
[0064] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0065] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
[0066] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.
[0067] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0068] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0069] 70% of the information we obtain every day comes from our eyes. If we give machines and equipment the characteristics of human eyes, we can save a lot of labor and improve people's lifestyles. For example, shopping malls and streets need to be monitored by special security personnel. After a long time, visual fatigue makes it difficult to find and track the offender in time. By using multi-target tracking technology, we can automatically track the corresponding person based on the monitoring screen and grasp his or her trajectory. In the field of unmanned driving, multi-target tracking technology can be used to predict the target trajectory around the vehicle, avoid obstacles in advance, and ensure driving safety.
[0070] Multi-target tracking refers to the technology of tracking multiple target objects of interest in a target video, identifying the position of the target objects from each image frame of the video, and then determining the motion trajectory of the target objects in the video. As an intermediate-level task in computer vision, multi-target tracking technology needs to rely on other underlying tasks to complete, such as target detection and pedestrian re-identification. It is also the basis for other high-level tasks (such as action recognition and behavior analysis). The main application scenarios of multi-target tracking include: video surveillance, which is used to detect abnormal behavior and save a lot of manpower and material resources; human-computer interaction, which realizes the recognition and processing of target interactions in complex scenes; virtual reality and augmented reality, such as the setting of character movement in games; medical diagnosis, etc.
[0071] The most popular multi-target tracking framework in the industry is the tracking-by-detection mode, that is, the detection-first-then-tracking mode. There are also some studies on the detection-and-tracking method. One method in the prior art uses YOLOv3 as a detector. The original detection end only predicts the target category, position, and confidence of the target's existence. But at the same time, a new appearance feature vector prediction is added to determine whether the targets of the previous and next frames are consistent during the subsequent tracking process. This implementation method can reduce memory consumption. However, the backbone network (such as ResNet50) of this method model to extract the total features usually uses a deep convolutional neural network. The model size is too large, which is not conducive to the transplantation of embedded devices; and the detection task and the task of predicting appearance features interfere with each other during the training process, and the tracking performance cannot reach the optimal level.
[0072] Multi-target tracking methods are mainly divided into two categories: two-stage and one-stage. The two-stage method separates the detection and pedestrian re-identification (ReID) tasks, and uses separate networks for prediction. First, the location of the target is selected through the detection algorithm, and then the corresponding area is cropped according to the predicted location and sent to the ReID sub-network to extract the embedding features. Because the target detection task and the embedding feature extraction task do not reuse features, the computational cost is high and the speed is slow, making it difficult to support real-time applications. Figure 1 A schematic diagram of a corresponding two-stage detection and tracking framework is shown in Figure 1 As shown in the figure, it includes a detection sub-model and an appearance feature extraction sub-model connected to the detection sub-model. The detection sub-model is first used to predict the pedestrian position in the current frame, and then the detection result is sent to the appearance feature extraction sub-model to predict the appearance feature vector of each target. The detection result is associated with the current trajectory based on the appearance feature vector to achieve multi-target tracking. This tracking method will increase additional time overhead and the tracking real-time performance is low.
[0073] The one-stage method uses a single network to complete detection and embedding prediction simultaneously under the premise of shared features, reducing some repeated calculations. For example, Wang et al. published "Towards Real-Time Multi-Object Tracking" at the ECCV conference in 2020. The paper added an embedding prediction branch to the detection end of YOLOv3 and performed end-to-end learning; for example, at the CPPR 2020 conference, Zhang et al. added an embedding prediction branch based on the anchor-free detection method CenterNet in the paper "A Simple Baseline for Multi-Object Tracking". Compared with traditional anchor-based methods, this method is conducive to reducing the number of exchanges of the same target identity. Figure 2 It is a schematic diagram of a one-stage detection and tracking framework, such as Figure 2 As shown in the figure, two corresponding outputs are directly predicted on the feature map extracted by the unified model. However, the detection task and the appearance feature extraction task are two completely unrelated tasks. Therefore, directly using one feature map to represent the results of the two types of tasks will cause confusion in the direction of parameter optimization during training. Ultimately, the two types of tasks cannot learn the optimal parameters, which affects the tracking performance.
[0074] At present, the multi-target tracking technology for pedestrians has attracted much attention. What needs to be achieved in multi-target tracking of pedestrians is: for the continuous video sequence captured by the monitoring, the designed algorithm or model is used to locate the position of the pedestrian target in the image and continue to track it. Based on the needs of actual use, the designed algorithm or model is required to be able to be ported to some embedded platforms or devices without GPU acceleration, with real-time speed and excellent performance. In the prior art, multi-target tracking of pedestrians is usually divided into two separate sub-tasks: detection and appearance feature prediction. The problem with the prior art is that multi-target tracking of pedestrians is divided into two separate sub-tasks: detection and appearance feature prediction, which affects the tracking speed and cannot meet the real-time requirements.
[0075] In order to solve the problems in the prior art, the present invention provides a method, device, intelligent terminal and storage medium for multi-target pedestrian tracking. The multi-target pedestrian tracking method obtains a target neural network model, wherein the target neural network model includes: a unified model for extracting the total features of an input image to obtain a feature map, a separation structure for separating the feature map, and an appearance feature extraction sub-model and a detection sub-model for respectively predicting the appearance feature vector and target information of the acquired target based on the separated feature map; obtaining a target video sequence; performing multi-target pedestrian tracking on the target video sequence based on the target neural network model to obtain a tracking result; and outputting the tracking result. Since the target neural network model obtained in the scheme of the present invention can extract the total features of the input image and obtain a feature map through the unified model, and separate the feature map through the separation structure, the appearance feature extraction sub-model and the detection sub-model can share features and alleviate the mutual interference of the two sub-tasks based on the separation structure, and combine the two different sub-tasks for joint prediction, it is beneficial to improve the tracking speed and meet the real-time requirements.
[0076] Exemplary Methods
[0077] like Figure 3 As shown, an embodiment of the present invention provides a method for multi-target pedestrian tracking, the method comprising the following steps:
[0078] Step S10, obtaining a target neural network model, wherein the target neural network model includes: a unified model for extracting the total features of an input image to obtain a feature map, a separation structure for separating the feature map, and an appearance feature extraction sub-model and a detection sub-model for respectively predicting the appearance feature vector and target information of the target based on the separated feature map.
[0079] The target neural network model is a neural network model for multi-target pedestrian tracking, and the input image is an image input into the target neural network model for tracking and detection. The target neural network model is a structure that is first integrated and then separated. The integrated model can be used to extract the total features of the input image to obtain a feature map. The total features extracted can be shared by the detection sub-model and the appearance feature extraction sub-model, thereby improving the real-time tracking. Then, the features in the feature map are separated by the separation structure to alleviate the mutual interference between the two sub-models, so that the detection sub-model and the appearance feature extraction sub-model can learn the optimal parameters, further optimizing the tracking performance.
[0080] Step S20, obtaining a target video sequence.
[0081] Among them, the above-mentioned target video sequence is a video sequence that requires multi-target pedestrian tracking. It can be a pre-recorded video or a video acquired in real time through a camera or other device. No specific restrictions are made here. The above-mentioned target image can be an image of a frame in the above-mentioned target video sequence.
[0082] Step S30, performing pedestrian multi-target tracking on the target video sequence based on the target neural network model to obtain tracking results.
[0083] Step S40: output the above tracking result.
[0084] As can be seen from the above, the pedestrian multi-target tracking method provided by the embodiment of the present invention obtains a target neural network model, wherein the target neural network model includes: a unified model for extracting the total features of the input image to obtain a feature map, a separation structure for separating the feature map, and an appearance feature extraction sub-model and a detection sub-model for respectively predicting the appearance feature vector and target information of the acquired target based on the separated feature map; obtaining a target video sequence; performing multi-target pedestrian tracking on the target video sequence based on the target neural network model to obtain tracking results; and outputting the tracking results. Since the target neural network model obtained in this scheme can extract the total features of the input image and obtain the feature map through the unified model, and separate the feature map through the separation structure, first unify and then separate, so that the appearance feature extraction sub-model and the detection sub-model can share features, and combine the two different sub-tasks for joint prediction, it is beneficial to improve the tracking speed and meet the real-time requirements.
[0085] Specifically, Figure 4 As shown, in this embodiment, the above step S10 includes:
[0086] Step S11, constructing a neural network model to be trained.
[0087] Step S12, obtaining a training set.
[0088] Step S13, training the neural network model to be trained based on the training set to obtain the target neural network model.
[0089] Specifically, Figure 5 is a schematic diagram of a framework of a neural network model to be trained provided in this embodiment, such as Figure 5As shown, in the above neural network model to be trained, the detection sub-model and the appearance feature extraction sub-model are combined in one network, and the method of first merging and then separating is adopted to save computing resources and improve the speed and performance of tracking to meet the real-time requirements. Specifically, the first half of the unified model extracts the total features of the input image and stores them in the form of feature maps. The features of this part are shared by the detection sub-model and the appearance extraction sub-model, thereby saving computing resources. Then, the features flow to two branches, the upper branch is responsible for predicting the appearance feature vector of the target, and the lower branch is responsible for predicting the target information (including the location, category, etc. of the target). In this embodiment, the above separation structure includes the above upper branch and the lower branch, wherein the upper branch is composed of 3 convolutions, and the convolution kernel sizes are 1, 3, and 1 respectively; the lower branch is composed of 2 convolutions with a convolution kernel size of 3. In actual use, the specific structure of the above separation structure and the specific convolution kernel size can also be adjusted according to actual needs, and no specific limitation is made here.
[0090] The above separation structure can alleviate the mutual interference between the two sub-models (detection sub-model and appearance extraction sub-model) during the training process without affecting the tracking speed, so that both sub-models can learn the optimal parameters and further optimize the tracking performance.
[0091] Furthermore, Figure 6 is a schematic diagram of the structure of the unified model provided in the embodiment of the present invention. The unified model provided in the embodiment of the present invention is a lightweight unified model, such as Figure 6 As shown, it includes the basic model, fusion block and top-down feature pyramid structure. Figure 6 In , the input image is fed into the above lightweight unified model for analysis. Figure 7 and Figure 8 The structures of the above basic model and fusion block are shown respectively. Figure 7 As shown, the design of the basic model starts from reducing the amount of convolution calculation. If the input size is H x W x Ci (H, W, Ci represent the height, width and number of channels of the input respectively), the size of the convolution kernel is K x K, and the number of output channels is Co, then the total amount of calculation is H x W x K x K x Co x Ci. The size of the convolution kernel is generally set to 1 or 3, and the number of channels can be set manually, so the size of the input determines the amount of calculation to a large extent. The above basic model mainly consists of two repeated structures, with a convolution kernel of 3 and a stride of 2 combined with a depth-separable convolution to quickly downsample the original image to 4 times the size. In the initial stage of the network, that is, the stage when the input image resolution is relatively large, this fast downsampling operation can reduce the amount of calculation to the greatest extent, but lacks the extraction of image semantic information. Therefore, the present invention cooperates with the fusion block in the subsequent stage to strengthen the missing image features.
[0092] Specifically, Figure 8 As shown in the figure, the above fusion block is divided into two branches. The left branch is the residual structure, which is added to the output of the right branch. The right branch contains two repeated structures, each of which also contains a depth-separable convolution. First, the input is deeply convolved, and the result is continuously sent to the two branches. The left branch is a depth-separable convolution, and the right branch is a 1x1, 3x3, and 1x1 convolution small structure. The results of the two branches are superimposed according to the channel. Because the depth-separable convolution is embedded, the fusion block increases the learning ability of the convolutional neural network while maintaining lightweight, and refines the loss features of the front-end basic model in a fusion manner. The fusion block is divided into 3 stages, and each stage will perform a downsampling, and the number of repetitions in the network is 3, 4, and 4 respectively. The output of the three-stage fusion block is finally sent to the feature pyramid network to supplement the semantic information of the deep network to the shallow network.
[0093] Among them, the size and number of repetitions of each of the above convolution kernels can be adjusted according to actual needs and are not specifically limited here. In this way, the lightweight unified model provided by the embodiment of the present invention is smaller in size than the backbone network used to extract the overall features in the prior art, which is conducive to the transplantation of embedded devices and can maintain good performance.
[0094] Specifically, Fig. 9 As shown, in this embodiment, the above step S12 includes:
[0095] Step S121, obtaining a detection data set and a person re-identification data set.
[0096] Step S122: Automatically annotate the data in the detection dataset and the person re-identification dataset to obtain a training set.
[0097] Specifically, the model training of the multi-target tracking method based on deep learning relies on a large number of multi-target tracking datasets. These datasets are cumbersome in the manual annotation process, as it is necessary not only to annotate the coordinate position and category information of the target in each image, but also to annotate its identity ID information. The amount of existing public multi-target tracking data is relatively small. In public pedestrian datasets (such as MOT17, MOT20, etc.), approximately 40,000 annotated images can be obtained. The amount of 40,000 data is difficult to support the training of a strong robust model. Manual shooting, video capture and annotation will consume a huge amount of time and manpower.
[0098] Therefore, in the embodiment of the present invention, the data in the public pedestrian detection data set (such as Wider Pedestrian, etc.) and the pedestrian re-identification image (such as Market-1501, etc.) can be integrated and automatically annotated to obtain a training set, thereby fundamentally alleviating the problem of small data volume and high manual annotation cost, and enriching the detection data. Of course, if a sufficient amount of multi-target tracking data can be obtained, the training set can also be directly obtained based on the multi-target tracking data, which is not specifically limited here.
[0099] Specifically, Fig.10 As shown, in this embodiment, the above step S122 includes:
[0100] Step S1221 , obtaining a detection image and a person re-identification image based on the detection dataset and the person re-identification dataset, respectively.
[0101] Step S1222: segment the above-mentioned pedestrian re-identification image into a black and white binary image.
[0102] Step S1223 , extracting a portion of the segmented mask corresponding to the person re-identification image as a segmented image.
[0103] Step S1224: predict the depth information of the detection image, and divide the detection image into regions based on the depth information.
[0104] Step S1225 , based on the size of the segmented image, pasting the segmented image onto a corresponding area of the detection image to obtain a training image.
[0105] Step S1226, obtaining the above training set based on the above set of training images.
[0106] The segmented mask is a mask in the segmented binary black-and-white image, specifically a mask corresponding to the detected pedestrian.
[0107] Specifically, Fig.11 This embodiment provides a pedestrian re-identification image. The pedestrian re-identification image is segmented into a black and white binary image using an image saliency method. The result is as follows: Fig.12 As shown. Extract the corresponding part of the segmented mask in the original image of pedestrian re-identification, as shown Fig.13 As shown. Predict the depth information of the above detection image, such as Fig.14As shown, the detection image is divided into regions based on the depth information. In this embodiment, the image is divided into three regions (small, medium, and large) according to the depth value. Among them, the pixel value of the normalized depth map is 0-0.3 for the large area, 0.3-0.6 for the medium area, and 0.6-1 for the small area. According to the size of the segmented pedestrian re-identification image, it is pasted in the corresponding area according to the principle of near big and far small. Specifically, if the area of the pedestrian re-identification image is S and the area of the detection image is Q, then S / Q<0.3 is pasted in the small area, 0.3≤S / Q<0.6 is pasted in the medium area, and S / Q≥0.6 is pasted in the large area. The obtained integrated image is as follows Fig.15 The number of the above-mentioned divided areas and the corresponding area division ranges can be adjusted according to actual needs and are not specifically limited here. Fig.15 In the figure, the gray part corresponding to each person's face is a mosaic added to protect the privacy of the relevant persons and has nothing to do with the content of the present invention.
[0108] Specifically, Fig.16 As shown, in this embodiment, the above step S13 includes:
[0109] Step S131, training the neural network model to be trained based on the training set to obtain a trained neural network model.
[0110] Step S132, performing sparse training on the trained neural network model.
[0111] Step S133, based on the model weights after sparse training, delete the preset target sparse number of convolution channels to obtain the above-mentioned target neural network model.
[0112] Among them, the preset target sparsity number is the preset number of convolution channels that need to be deleted. Specifically, after training the constructed neural network model, in order to achieve the purpose of "network slimming", reduce the network size, and facilitate transplantation, further sparse training can be performed. Specifically, the scaling variables of the BN layer in the neural network are added to the loss function in the form of L1 regularization, and a preset number of iterative training is performed. According to the model weights after sparse training, convolution channels of an appropriate number of layers are deleted without losing too much performance. Among them, the above-mentioned preset number of times is a preset number of iterations. In this embodiment, 30 times can be taken. It can be adjusted in actual use and is not specifically limited here.
[0113] Specifically, Fig.17 As shown, in this embodiment, the above step S30 includes:
[0114] Step S31, for an input image corresponding to a frame in the target video sequence, predict the position of the pedestrian target in the input image through the detector in the target neural network model.
[0115] Step S32, predicting the position of the next frame based on the existing trajectory through a Kalman filter.
[0116] Step S33, based on pedestrian features and intersection-over-union ratio, the predicted target of the above detector is associated with the existing trajectory, the identity ID of the same pedestrian target in the previous and next frames is kept consistent, and the above tracking result is obtained.
[0117] In this embodiment, the Kalman filter is used for trajectory detection, and the detector will consume more time than the tracker. Therefore, the target can be detected by skipping frames, and the Kalman filter is used for tracking during the non-detection period. Optionally, the number of frame intervals can be pre-set, and detection is performed every number of frames corresponding to the number of frame intervals, for example, 3 to 5 frames are set for detection. In this embodiment, detection is preferably performed every 5 frames. Specifically, the video sequence acquired in real time is input into the trained target neural network model to obtain the position and appearance feature information of the pedestrian target; the detection results are correlated with the existing trajectory according to the predicted appearance features; the matched trajectory is associated according to the maximum intersection-combination ratio principle, and for the unmatched detection results, a new trajectory is generated. During the tracking process, detection is performed once every 5 seconds (in this embodiment, one frame corresponds to one second), and the trajectory is predicted by the Kalman filter in the remaining frames, and the Kalman filter state of the corresponding trajectory is updated with the result during detection. In this way, the tracking speed can be improved to meet the real-time tracking.
[0118] Optionally, the step S40 includes: based on the tracking result, visually outputting the trajectory of each pedestrian target, wherein the bounding boxes of pedestrian targets with different identity IDs have different colors, while the bounding boxes of pedestrian targets with the same identity ID have the same color, so as to facilitate observation by staff.
[0119] Optionally, the above method can be used not only for multi-target tracking of pedestrians, but also for multi-target tracking of other moving objects after appropriate adjustment, which is not specifically limited here.
[0120] Specifically, the following table shows the experimental performance and model size comparison results of each tracking method provided in this embodiment. This experiment was conducted on an Intel Core i7-10750H CPU, using a public pedestrian dataset and a street monitoring scene dataset that was photographed, collected and annotated for training and testing. This model first performs a classification pre-training on ImageNet and loads the subsequent tasks of the weight training, with a total of 140 iterations. Finally, it was tested on MOT16, and the specific results are shown in the table below. When evaluating tracking performance, there are two main reference indicators, MOTA and IDS. The higher the MOTA, the better the performance, and the lower the IDS, the fewer identity ID exchanges during the tracking process. As can be seen from the table, the model in this solution maintains a leading real-time speed of 24s, and is small in size and easy to port.
[0121] Tracking Methods FPS MOTA IDs IDF1 MOTP Model Size Traditional two-stage approach 2s 64.1 1496 66.8 0.221 298M Lightweight all-in-one model 24s 60.4 1252 63.9 0.195 8M Lightweight one-piece model + separation structure 24s 62.6 1084 65.5 0.203 8.5M
[0122] Exemplary Devices
[0123] like Fig.18 As shown in , corresponding to the above-mentioned pedestrian multi-target tracking method, an embodiment of the present invention further provides a pedestrian multi-target tracking device, and the above-mentioned pedestrian multi-target tracking device includes:
[0124] The target neural network model acquisition module 810 is used to acquire the target neural network model, wherein the target neural network model includes: a unified model for extracting the total features of the input image to obtain a feature map, a separation structure for separating the feature map, and an appearance feature extraction sub-model and a detection sub-model for predicting the appearance feature vector and target information of the target based on the separated feature map.
[0125] The target neural network model is a neural network model for multi-target pedestrian tracking, and the input image is an image input into the target neural network model for tracking and detection. The target neural network model is a structure that is first integrated and then separated. The integrated model can be used to extract the total features of the input image to obtain a feature map. The total features extracted can be shared by the detection sub-model and the appearance feature extraction sub-model, thereby improving the real-time tracking. Then, the features in the feature map are separated by the separation structure to alleviate the mutual interference between the two sub-models, so that the detection sub-model and the appearance feature extraction sub-model can learn the optimal parameters, further optimizing the tracking performance.
[0126] The target video sequence acquisition module 820 is used to acquire the target video sequence.
[0127] Among them, the above-mentioned target video sequence is a video sequence that requires multi-target pedestrian tracking. It can be a pre-recorded video or a video acquired in real time through a camera or other device. No specific restrictions are made here. The above-mentioned target image can be an image of a frame in the above-mentioned target video sequence.
[0128] The tracking module 830 is used to perform multi-target pedestrian tracking on the target video sequence based on the target neural network model to obtain tracking results.
[0129] The output module 840 is used to output the above tracking results.
[0130] As can be seen from the above, the pedestrian multi-target tracking device provided by the embodiment of the present invention obtains the target neural network model through the target neural network model acquisition module 810, wherein the above target neural network model includes: a unified model for extracting the total features of the input image to obtain a feature map, a separation structure for separating the above feature map, and an appearance feature extraction sub-model and a detection sub-model for respectively predicting the appearance feature vector and target information of the target based on the separated feature map; obtaining the target video sequence through the target video sequence acquisition module 820; performing pedestrian multi-target tracking on the above target video sequence based on the above target neural network model through the tracking module 830 to obtain the tracking result; and outputting the above tracking result through the output module 840. Since the target neural network model obtained in this scheme can extract the total features of the input image and obtain the feature map through the unified model, and separate the feature map through the separation structure, the first unification and then separation are performed, so that the appearance feature extraction sub-model and the detection sub-model can share features, and the two different sub-tasks are combined for joint prediction, which is conducive to improving the tracking speed and meeting the real-time requirements.
[0131] Optionally, the target neural network model acquisition module 810 is specifically used to: construct a neural network model to be trained; obtain a training set; and train the neural network model to be trained based on the training set to obtain the target neural network model.
[0132] Among them, the above-mentioned construction of the neural network model to be trained includes constructing Figures 5 to 8 The neural network model shown. The obtaining of the training set includes: obtaining a detection data set and a pedestrian re-identification data set; and automatically annotating the data in the detection data set and the pedestrian re-identification data set to obtain the training set.
[0133] Specifically, the model training of the multi-target tracking method based on deep learning relies on a large number of multi-target tracking datasets. These datasets are cumbersome in the manual annotation process, as it is necessary not only to annotate the coordinate position and category information of the target in each image, but also to annotate its identity ID information. The amount of existing public multi-target tracking data is relatively small. In public pedestrian datasets (such as MOT17, MOT20, etc.), approximately 40,000 annotated images can be obtained. The amount of 40,000 data is difficult to support the training of a strong robust model. Manual shooting, video capture and annotation will consume a huge amount of time and manpower.
[0134] Therefore, in the embodiment of the present invention, the data in the public pedestrian detection data set (such as Wider Pedestrian, etc.) and the pedestrian re-identification image (such as Market-1501, etc.) can be integrated and automatically annotated to obtain a training set, thereby fundamentally alleviating the problem of small data volume and high manual annotation cost, and enriching the detection data. Of course, if a sufficient amount of multi-target tracking data can be obtained, the training set can also be directly obtained based on the multi-target tracking data, which is not specifically limited here.
[0135] Specifically, in this embodiment, the target neural network model acquisition module 810 obtains the training set through the following process: obtaining a detection image and a pedestrian re-identification image based on the detection data set and the pedestrian re-identification data set respectively; segmenting the pedestrian re-identification image into a black and white binary image; extracting the corresponding part of the segmented mask in the pedestrian re-identification image as a segmented image; predicting the depth information of the detection image, and dividing the detection image into regions based on the depth information; pasting the segmented image on the corresponding region of the detection image based on the size of the segmented image to obtain a training image; and obtaining the training set based on the set of the training images.
[0136] The segmented mask is a mask in the segmented binary black-and-white image, specifically a mask corresponding to the detected pedestrian.
[0137] Optionally, the above-mentioned training of the neural network model to be trained based on the above-mentioned training set to obtain the above-mentioned target neural network model includes: training the above-mentioned neural network model to be trained based on the above-mentioned training set to obtain the trained neural network model; performing sparse training on the above-mentioned trained neural network model; based on the model weights after sparse training, deleting the preset target sparse number of convolution channels to obtain the above-mentioned target neural network model. Thereby achieving the purpose of "network slimming" and reducing the network size for transplantation.
[0138] Optionally, the tracking module 830 is specifically used for: for an input image corresponding to a frame in the target video sequence, predicting the position of the pedestrian target in the input image through the detector in the target neural network model; predicting the position of the next frame based on the existing trajectory through the Kalman filter; and associating the predicted target of the detector with the existing trajectory based on pedestrian features and intersection-union ratio, keeping the identity ID of the same pedestrian target in the previous and next frames consistent, and obtaining the tracking result. Furthermore, the target can be detected by skipping frames, and the Kalman filter is used for tracking during non-detection periods. Optionally, detection can be set every 3 to 5 frames, and in this embodiment, detection is preferably performed every 5 frames.
[0139] Optionally, the output module 840 is specifically used to: based on the tracking results, visualize the trajectory of each pedestrian target, and when outputting, the bounding boxes of pedestrian targets with different identity IDs have different colors, while the bounding boxes of pedestrian targets with the same identity ID have the same color, so that it is convenient for staff to observe.
[0140] Based on the above embodiments, the present invention further provides an intelligent terminal, whose principle block diagram can be shown as follows: Fig.19 As shown. The above-mentioned intelligent terminal includes a processor, a memory, a network interface and a display screen connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps of any one of the above-mentioned pedestrian multi-target tracking methods are implemented. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen.
[0141] Those skilled in the art will understand that Fig.19 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the smart terminal to which the scheme of the present invention is applied. The specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0142] In one embodiment, a smart terminal is provided, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the program performs the following operation instructions when executed by the processor:
[0143] Acquire a target neural network model, wherein the target neural network model comprises: a unified model for extracting the total features of the input image to obtain a feature map, a separation structure for separating the feature map, and an appearance feature extraction sub-model and a detection sub-model for respectively predicting the appearance feature vector and target information of the acquired target based on the separated feature map;
[0144] Obtain target video sequence;
[0145] Based on the target neural network model, the target video sequence is subjected to multi-target pedestrian tracking to obtain tracking results;
[0146] Output the above tracking results.
[0147] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of any one of the pedestrian multi-target tracking methods provided by the embodiments of the present invention are implemented.
[0148] It should be understood that the serial numbers of the steps in the above embodiments do not imply a sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0149] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0150] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments. For quantities with the same expression form in various formulas, unless otherwise specified, they represent the same quantity, and various formulas can refer to each other.
[0151] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0152] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / terminal equipment and method can be implemented in other ways. For example, the apparatus / terminal equipment embodiments described above are only illustrative, for example, the division of the above modules or units is only a logical function division, and in actual implementation, other division methods can be used, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0153] If the above-mentioned integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.
[0154] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features therein may be replaced by equivalents. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.
Claims
1. A pedestrian multi-target tracking method, characterized in that: The method comprises: Acquire a target neural network model, wherein the target neural network model comprises: a unified model for extracting the total features of an input image to obtain a feature map, a separation structure for separating the feature map, and an appearance feature extraction sub-model and a detection sub-model for respectively predicting an appearance feature vector of a target and target information based on the separated feature map, wherein the target information comprises a position and a category of the target, the separation structure comprises an upper branch and a lower branch, the upper branch comprises three convolutional layers, and the lower branch comprises two convolutional layers; Obtain target video sequence; Performing pedestrian multi-target tracking on the target video sequence based on the target neural network model to obtain tracking results; The tracking result is outputted.
2. The pedestrian multi-target tracking method according to claim 1, characterized in that: The step of obtaining a target neural network model comprises: Build a neural network model to be trained; Get the training set; The neural network model to be trained is trained based on the training set to obtain the target neural network model.
3. The pedestrian multi-target tracking method according to claim 2, characterized in that: The obtaining of the training set comprises: Obtain detection datasets and person re-identification datasets; A training set is obtained by automatically annotating data in the detection data set and the pedestrian re-identification data set.
4. The pedestrian multi-target tracking method according to claim 3, characterized in that: The step of automatically labeling the data in the detection dataset and the person re-identification dataset to obtain a training set comprises: Acquire a detection image and a person re-identification image based on the detection dataset and the person re-identification dataset respectively; Segmenting the pedestrian re-identification image into a black and white binary image; Extracting a corresponding portion of the segmented mask in the person re-identification image as a segmented image; Predicting depth information of the detection image, and dividing the detection image into regions based on the depth information; Pasting the segmented image on a corresponding area of the detection image based on the size of the segmented image to obtain a training image; The training set is obtained based on the set of training images.
5. The pedestrian multi-target tracking method according to claim 2, characterized in that: The step of training the neural network model to be trained based on the training set to obtain the target neural network model includes: Training the neural network model to be trained based on the training set to obtain a trained neural network model; Performing sparse training on the trained neural network model; Based on the model weights after sparse training, a preset target sparse number of convolution channels is deleted to obtain the target neural network model.
6. The method for multi-target pedestrian tracking according to claim 1, characterized in that: The performing multi-target pedestrian tracking on the target video sequence based on the target neural network model to obtain tracking results includes: For an input image corresponding to a frame in the target video sequence, predicting the position of a pedestrian target in the input image by a detector in the target neural network model; Use the Kalman filter to predict the position of the next frame based on the existing trajectory; Based on pedestrian features and intersection-over-union ratio, the predicted target of the detector is associated with the existing trajectory, the identity ID of the same pedestrian target in the previous and next frames is kept consistent, and the tracking result is obtained.
7. The method for multi-target pedestrian tracking according to claim 6, characterized in that: Outputting the tracking result includes: based on the tracking result, visually outputting the trajectory of each pedestrian target, and when outputting, the bounding boxes of pedestrian targets with different identity IDs have different colors.
8. A pedestrian multi-target tracking device, characterized in that: The device comprises: A target neural network model acquisition module, used to acquire a target neural network model, wherein the target neural network model includes: a unified model for extracting the total features of an input image to acquire a feature map, a separation structure for separating the feature map, and an appearance feature extraction sub-model and a detection sub-model for respectively predicting and acquiring an appearance feature vector of a target and target information based on the separated feature map, wherein the target information includes a position and a category of the target, and the separation structure includes an upper branch and a lower branch, wherein the upper branch is composed of three convolutional layers, and the lower branch is composed of two convolutional layers; A target video sequence acquisition module is used to acquire a target video sequence; A tracking module, used for performing pedestrian multi-target tracking on the target video sequence based on the target neural network model to obtain tracking results; An output module is used to output the tracking result.
9. An intelligent terminal, characterized in that: The method comprises a memory, a processor and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
