Pedestrian tracking method and system, storage medium and electronic equipment

Abstract

The invention provides a pedestrian tracking method and system, a storage medium and electronic equipment, and relates to the technical field of target detection, and the method comprises the steps: obtaining a video image; performing pedestrian target detection on the target video frame of the video image to obtain a pedestrian detection frame of each pedestrian target; according to area coordinate information fed back by the pedestrian detection frame, pedestrian images are extracted and input to a feature recognition network, a passenger flow feature library formed by first high-dimensional features corresponding to the pedestrian targets is obtained, and number information is distributed to the pedestrian targets; if the number of pedestrians changes, updating a first high-dimensional feature in a passenger flow feature library to obtain a second high-dimensional feature; and according to the first high-dimensional feature and the second high-dimensional feature, determining a motion track of the pedestrian targets with the same number information in the video image. According to the invention, based on the high-dimensional features of the pedestrians, the high-dimensional features are bound with the pedestrian targets of the corresponding number information, so that the problem of failure when the intersection-to-parallel distance is adopted when the pedestrians are shielded is avoided, and the pedestrian tracking precision is improved.

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a pedestrian tracking method, system, storage medium and electronic device. Background Art

[0002] Currently, pedestrian tracking mainly involves frame-by-frame object detection and associating the objects in the previous and next frames to form a trajectory. Existing methods usually use a Kalman filter for each tracked object, match the estimated state of the target with the detection, and use the matched detection to update the state of the trajectory.

[0003] The earliest pedestrian tracking method used Kalman filtering to predict the position of the object in the next frame, and used the Hungarian algorithm to associate the detection with the trajectory. The cost matrix of the Hungarian algorithm is calculated based on the intersection-over-union distance, which has low computational complexity and strong real-time performance. However, when pedestrians are blocked (cross-blocking between pedestrians or building blockage), the intersection-over-union distance cost matrix will fail, which can easily cause the pedestrian target to be lost and reduce the accuracy of pedestrian tracking. Summary of the invention

[0004] The purpose of this application is to provide a pedestrian tracking method, a pedestrian tracking system, a computer-readable storage medium and an electronic device, which track a target pedestrian by determining the high-dimensional features of the target in a video image, avoid tracking loss due to occlusion of the target, and improve tracking accuracy.

[0005] In order to solve the above technical problems, this application provides a pedestrian tracking method, and the specific technical solution is as follows:

[0006] Acquire video images;

[0007] Performing pedestrian target detection on the target video frame of the video image to obtain a pedestrian detection frame for each pedestrian target;

[0008] According to the regional coordinate information fed back by the pedestrian detection frame, the pedestrian image is extracted and input into the feature recognition network to obtain a passenger flow feature library composed of the first high-dimensional features corresponding to each pedestrian target, and number information is assigned to each pedestrian target;

[0009] If there is a change in the number of pedestrians between the target video frame and its previous video frame, the first high-dimensional feature in the passenger flow feature library is updated according to all pedestrian targets in the target video frame to obtain a second high-dimensional feature;

[0010] The motion trajectory of the pedestrian target with the same number information in the video image is determined according to the first high-dimensional feature and the second high-dimensional feature.

[0011] Optionally, performing pedestrian target detection on a target video frame of the video image to obtain a pedestrian detection frame of each pedestrian target includes:

[0012] Obtain a pedestrian detection data set containing annotation information; the annotation information includes pedestrian detection frame coordinate position information and detection confidence;

[0013] Training a pre-trained target detection network based on the pedestrian detection data set to obtain a pedestrian detection model;

[0014] The pedestrian detection model is applied to perform pedestrian target detection on the target video frame of the video image to obtain a pedestrian detection frame for each pedestrian target.

[0015] Optionally, the pedestrian image is extracted and input into the feature recognition network to obtain a passenger flow feature library consisting of the first high-dimensional features corresponding to each pedestrian target, including:

[0016] Using a feature recognition network to identify color features, angle features, and light features in the pedestrian image, to obtain a first high-dimensional feature of the pedestrian target;

[0017] A passenger flow feature library is constructed based on the first high-dimensional features of all pedestrian targets.

[0018] Optionally, determining the motion trajectory of a pedestrian target with the same serial number information in the video image according to the first high-dimensional feature and the second high-dimensional feature includes:

[0019] Calculate a two-dimensional cost matrix of the same serial number information in the previous and next images according to the first high-dimensional feature and the second high-dimensional feature; the number of rows of the two-dimensional cost matrix is ​​the number of pedestrians in the target video frame, and the number of columns is the number of high-dimensional features in the passenger flow feature library;

[0020] The two-dimensional cost matrix is ​​used as the input of the Hungarian algorithm, and the motion trajectory of the pedestrian target corresponding to the same numbering information is determined according to the optimal row and column indexes output by the Hungarian algorithm.

[0021] Optionally, before using the two-dimensional cost matrix as an input of the Hungarian algorithm and determining the motion trajectory of the pedestrian target corresponding to the same numbering information according to the optimal row and column index output by the Hungarian algorithm, the method further includes:

[0022] Calculating the moving speed of the pedestrian target according to the pedestrian detection frame of the same pedestrian target in the target video frame and the previous video frame;

[0023] Determine a speed matching threshold of the number information corresponding to the pedestrian target based on the movement speed;

[0024] Determine a two-dimensional velocity representation matrix based on a pedestrian detection frame of the pedestrian target in the target video frame and a pedestrian detection frame of the same pedestrian target in the passenger flow feature library in the previous video frame;

[0025] The elements in the two-dimensional velocity representation matrix that are greater than the velocity matching threshold are set to 1, and the elements that do not exceed the velocity matching threshold are set to 0;

[0026] The two-dimensional velocity representation matrix is ​​added to the two-dimensional cost matrix to obtain a cost matrix; the cost matrix is ​​used to replace the two-dimensional cost matrix as an input of the Hungarian algorithm.

[0027] Optionally, when the first high-dimensional feature in the passenger flow feature library is updated according to all pedestrian targets in the target video frame to obtain the second high-dimensional feature, the method further includes:

[0028] Calculating the intersection-over-union distance between pedestrian detection frames corresponding to each pedestrian target in the target video frame;

[0029] If the intersection-over-union ratio distance is less than 1, stop updating the high-dimensional features of the pedestrian targets corresponding to the two corresponding pedestrian detection frames;

[0030] If the intersection-over-union ratio distance is greater than or equal to 1, the step of updating the first high-dimensional feature in the passenger flow feature library according to all pedestrian targets in the target video frame to obtain a second high-dimensional feature is performed.

[0031] Optionally, updating the first high-dimensional feature in the passenger flow feature library according to all pedestrian targets in the target video frame to obtain the second high-dimensional feature includes:

[0032] Determine a weight vector according to the time difference between the high-dimensional feature and the last update; wherein the longer the time difference from the last update, the smaller the weight vector;

[0033] updating the first high-dimensional feature in the passenger flow feature library according to an updating formula including the weight vector to obtain a second high-dimensional feature;

[0034] The update formula is:

[0035] ;

[0036] in, is the first high-dimensional feature of the pedestrian target, is the second highest dimensional feature of the pedestrian target, The feature recognition network identifies the features of the pedestrian target. is the weight vector.

[0037] The present application also provides a pedestrian tracking system, comprising:

[0038] A video acquisition module, used for acquiring video images;

[0039] A pedestrian detection module, used to perform pedestrian target detection on the target video frame of the video image to obtain a pedestrian detection frame for each pedestrian target;

[0040] A feature recognition module is used to extract pedestrian images according to the regional coordinate information fed back by the pedestrian detection frame and input them into a feature recognition network to obtain a passenger flow feature library consisting of the first high-dimensional features corresponding to each pedestrian target, and assign number information to each pedestrian target;

[0041] A feature updating module, configured to update the first high-dimensional feature in the passenger flow feature library according to all pedestrian targets in the target video frame to obtain a second high-dimensional feature if there is a change in the number of pedestrians between the target video frame and its previous video frame;

[0042] A pedestrian tracking module is used to determine the movement trajectory of the pedestrian target with the same number information in the video image according to the first high-dimensional feature and the second high-dimensional feature.

[0043] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the pedestrian tracking method described above are implemented.

[0044] The present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the pedestrian tracking method described above when calling the computer program in the memory.

[0045] The present application provides a pedestrian tracking method, comprising: acquiring a video image; performing pedestrian target detection on a target video frame of the video image to obtain a pedestrian detection frame for each pedestrian target; extracting the pedestrian image based on the regional coordinate information fed back by the pedestrian detection frame and inputting it into a feature recognition network to obtain a passenger flow feature library composed of first high-dimensional features corresponding to each pedestrian target, and assigning number information to each pedestrian target; if there is a change in the number of pedestrians between the target video frame and its previous video frame, updating the first high-dimensional feature in the passenger flow feature library based on all pedestrian targets in the target video frame to obtain a second high-dimensional feature; determining the movement trajectory of pedestrian targets with the same number information in the video image based on the first high-dimensional feature and the second high-dimensional feature.

[0046] The present application performs pedestrian target detection on the target video frame in the video image to determine the pedestrian detection frame of each pedestrian target, locates the pedestrian position based on the regional coordinate information of the pedestrian detection frame, and extracts the pedestrian image and inputs it into the feature recognition network to obtain the first high-dimensional feature of the pedestrian. Thereafter, once the number of pedestrians in adjacent video frames changes, the passenger flow feature library containing the first high-dimensional features of all pedestrians is updated, and the first high-dimensional feature and the second high-dimensional feature are compared to determine the motion trajectory of the pedestrian target corresponding to the same numbering information. The present application is based on the high-dimensional features of pedestrians, and binds the high-dimensional features to the pedestrian targets with the corresponding numbering information to avoid the failure problem of using the intersection-over-union distance when pedestrians are occluded, thereby improving the pedestrian tracking accuracy.

[0047] The present application also provides a pedestrian tracking system, a computer-readable storage medium and an electronic device, which have the above-mentioned beneficial effects and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0049] Figure 1 A flowchart of a pedestrian tracking method provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of a pedestrian tracking process provided by an embodiment of the present application;

[0051] Figure 3 A schematic diagram of a pedestrian tracking process for an oblique passenger flow provided by an embodiment of the present application;

[0052] Figure 4 A schematic diagram of the structure of a pedestrian tracking system provided in an embodiment of the present application;

[0053] Figure 5 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] The object information involved in this application, including but not limited to the object device information, object personal information, etc., and data, including but not limited to data used for analysis, stored data, displayed data, etc., are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the laws, regulations and standards of relevant countries and regions.

[0056] See also Figure 1 , Figure 1 A flowchart of a pedestrian tracking method provided in an embodiment of the present application, the method comprising:

[0057] S101: Acquire video images;

[0058] S102: performing pedestrian target detection on the target video frame of the video image to obtain a pedestrian detection frame of each pedestrian target;

[0059] S103: extracting pedestrian images according to the regional coordinate information fed back by the pedestrian detection frame and inputting them into a feature recognition network to obtain a passenger flow feature library consisting of first high-dimensional features corresponding to each pedestrian target, and assigning number information to each pedestrian target;

[0060] S104: if there is a change in the number of pedestrians between the target video frame and its previous video frame, update the first high-dimensional feature in the passenger flow feature library according to all pedestrian targets in the target video frame to obtain a second high-dimensional feature;

[0061] S105: Determine the movement trajectory of the pedestrian target with the same number information in the video image according to the first high-dimensional feature and the second high-dimensional feature.

[0062] This application can be applied to scenes such as oblique passenger flow tracking, and can track and identify pedestrian targets in video images transmitted by oblique cameras. The pedestrian tracking method provided by this application can be applied when the camera installation conditions are not ideal to optimize the accuracy of passenger flow statistics.

[0063] There is no limitation on how to obtain the video image. The real-time monitoring screen of the camera can be used as the video image, or historical monitoring data can be obtained as the video image.

[0064] When detecting pedestrian targets in step S102, a pre-trained target detection model may be used for detection. For example, based on a general target detection model, a labeled pedestrian data set is used for training to obtain a passenger flow detection model.

[0065] In a feasible implementation manner, step S102 may be performed in the following manner:

[0066] The first step is to obtain a pedestrian detection data set containing annotation information; the annotation information includes the coordinate position information of the pedestrian detection box and the detection confidence;

[0067] Step 2: training a pre-trained target detection network based on the pedestrian detection dataset to obtain a pedestrian detection model;

[0068] The third step is to apply the pedestrian detection model to perform pedestrian target detection on the target video frame of the video image to obtain a pedestrian detection frame for each pedestrian target.

[0069] In the actual application process, the one-stage target detection model YOLOv5 / v7 / v6 / v8 with strong real-time performance can be flexibly selected according to the specific use requirements, and the public pre-trained network of the target detection model can be selected as the basic network of the oblique passenger flow detection model. Generally, these general target detection models have been fully trained on large-scale public data sets such as ImageNet and MSCOCO, and the trained network models often have a certain pedestrian target detection effect. It should be noted that since the data category of the public data set may not fully conform to the scenario of oblique camera passenger flow detection, after adjusting the output category of the general target detection pre-trained network model accordingly, the pre-annotated oblique camera pedestrian detection data set can be used for targeted training. In this embodiment, the pedestrian detection data set is mainly the image / detection frame information of the selected pedestrians, and the output result of the trained pedestrian detection model includes the detection frame coordinate position information of the pedestrians in the image and the confidence level of the detection.

[0070] In step 103, a labeled pedestrian re-identification dataset can be used for training in advance. The pedestrian re-identification dataset includes a pedestrian re-identification dataset publicly available on the Internet and a labeled data set for oblique camera scenes to obtain the feature recognition network, and the feature recognition network outputs a feature vector of the pedestrian.

[0071] The feature recognition network needs to extract effective discriminative features from the image, and these features should be robust to environmental changes (such as lighting, angle, etc.). You can choose a public feature extraction network such as resnet\densenet\swin-transformer\mobilenet.

[0072] In a feasible implementation, a feature recognition network can be used to identify color features, angle features and light features in the pedestrian image to obtain the first high-dimensional features of the pedestrian target, and a passenger flow feature library can be constructed based on the first high-dimensional features of all pedestrian targets.

[0073] In order to better adapt to the oblique passenger flow tracking scenario, a pedestrian re-identification dataset under an oblique camera can be generated in advance. Each pedestrian in the dataset includes the state under different conditions such as angle and lighting. The feature extraction network is used for training to obtain a feature recognition network suitable for the oblique passenger flow tracking scenario.

[0074] When constructing a passenger flow feature library based on the first high-dimensional features of all pedestrian targets, the pedestrian image corresponding to the regional coordinate information can be extracted according to the regional coordinate information fed back by the pedestrian detection frame and input into the feature recognition network to form an initial passenger flow feature library, and at the same time, number information is assigned to the pedestrian corresponding to each pedestrian image.

[0075] In step S104, if the number of pedestrians in the target video frame and its previous video frame changes, the first high-dimensional feature in the passenger flow feature library is updated according to all pedestrian targets in the target video frame to obtain the second high-dimensional feature. The present application updates the passenger flow feature library only when the number of pedestrians in adjacent video frames changes, that is, new pedestrians appear or pedestrians disappear (pedestrians are blocked by buildings or pedestrians are blocked by intersections). Pedestrian target detection can be performed on the target video frame to obtain a pedestrian detection frame for each pedestrian target, and then the pedestrian image is extracted according to the regional coordinate information fed back by the pedestrian detection frame and input into the feature recognition network to obtain the second high-dimensional feature. That is, the direct feature acquisition method of the first high-dimensional feature and the second high-dimensional feature can be the same.

[0076] Thereafter, the motion trajectory of the pedestrian target with the same number information in the video image can be determined according to the first high-dimensional feature and the second high-dimensional feature. In order to achieve efficient pedestrian tracking, different matching cost matrix calculation methods can be selected by comparing the number of targets in adjacent frame images. The matching cost matrix is ​​constructed using pedestrian features, and the pedestrian tracking result is obtained by matching according to the feature distance value.

[0077] Specifically, after obtaining the second high-dimensional feature of each pedestrian, the two-dimensional cost matrix of the same number information in the previous and next images can be calculated based on the first high-dimensional feature and the second high-dimensional feature. At this time, the number of rows of the two-dimensional cost matrix is ​​the number of pedestrians in the target video frame, and the number of columns is the number of high-dimensional features in the passenger flow feature library;

[0078] The two-dimensional cost matrix is ​​used as the input of the Hungarian algorithm, and the motion trajectory of the pedestrian target corresponding to the same number information is determined according to the optimal row and column index output by the Hungarian algorithm.

[0079] The feature matching between the first high-dimensional feature and the second high-dimensional feature is based on global matching of the entire frame image. In the real-time tracking process, the travel distance of the same pedestrian target between adjacent frames is limited. Therefore, in order to ensure the robustness of feature matching, the pedestrian running speed can be added as a priori constraint. Specifically, the following steps can be included:

[0080] The first step is to calculate the moving speed of the pedestrian target according to the pedestrian detection frame of the same pedestrian target in the target video frame and the previous video frame;

[0081] Step 2: determining a speed matching threshold of the pedestrian target corresponding number information based on the movement speed;

[0082] Step 3: Determine a two-dimensional velocity representation matrix based on the pedestrian detection frame of the pedestrian target in the target video frame and the pedestrian detection frame of the same pedestrian target in the passenger flow feature library in the previous video frame;

[0083] Step 4: Set the elements in the two-dimensional velocity representation matrix that are greater than the velocity matching threshold to 1, and set the elements that do not exceed the velocity matching threshold to 0;

[0084] Step 5: Add the two-dimensional velocity representation matrix to the two-dimensional cost matrix to obtain a cost matrix; the cost matrix is ​​used to replace the two-dimensional cost matrix as the input of the Hungarian algorithm.

[0085] The pedestrian's running speed is calculated based on the pedestrian coordinate information obtained from the pedestrian detection of adjacent frame images, and is used as the threshold for matching each numbered information (the threshold can be set to α times the speed, which can be set according to the on-site situation). The pedestrian coordinate information obtained in the current frame is calculated with the coordinate information of the numbered information in the feature library (for the numbered information that was not matched in the previous frame, the last position of the numbered information is used for calculation, and the duration is the frame length of the last match to the current frame). A two-dimensional speed representation matrix representing the speed can be obtained, and its size is the same as the two-dimensional cost matrix of the feature distance. The matrix is ​​processed according to the threshold for matching each numbered information. The elements in the two-dimensional matrix that exceed the threshold are set to 1 (then the feature will not be used for matching, and some mismatched trajectories with similar features can be filtered out), and those that do not exceed the threshold are set to 0. Finally, the two-dimensional speed representation matrix representing the speed is added to the two-dimensional cost matrix representing the feature distance. The matching result is obtained using the Hungarian algorithm based on the generated cost matrix, and the trajectory of the pedestrian target is obtained.

[0086] When the number information is matched, the real-time features of the current number information in the passenger flow feature library are updated. The validity of the updated feature library is ensured by filtering out occluded features and introducing time distance.

[0087] If the number of pedestrians in the target video frame and its previous video frame has not changed, a two-dimensional cost matrix can be obtained based on the intersection and union distance of the target position information fed back by the pedestrian detection frame and the pedestrian detection frame with the same number information in the passenger flow feature library corresponding to the target video frame. Based on the generated cost matrix, the Hungarian algorithm is used to obtain the matching pedestrian trajectory.

[0088] Considering that when there is a pedestrian crossing, the target in the detection frame is an occluded target. At this time, the features generated by the feature recognition network are different from the original target, that is, the real-time features of the numbering information are contaminated. In the high-dimensional feature update stage of the passenger flow feature library, this numbering information does not update the corresponding high-dimensional features.

[0089] The specific implementation can be based on the intersection-and-union distance between pedestrian detection frames, that is, the intersection-and-union distance between pedestrian detection frames in the target video frame is calculated. When the distance is less than 1, it means that there is an intersection between pedestrian detection frames, and the feature in the current pedestrian detection frame is judged to be a contaminated feature, and the feature is not updated. When the distance is 1, it means that there is no intersection between pedestrian detection frames, and the feature of the pedestrian is updated.

[0090] The present application performs pedestrian target detection on the target video frame in the video image to determine the pedestrian detection frame of each pedestrian target, locates the pedestrian position based on the regional coordinate information of the pedestrian detection frame, and extracts the pedestrian image and inputs it into the feature recognition network to obtain the first high-dimensional feature of the pedestrian. Thereafter, once the number of pedestrians in adjacent video frames changes, the passenger flow feature library containing the first high-dimensional features of all pedestrians is updated, and the first high-dimensional feature and the second high-dimensional feature are compared to determine the motion trajectory of the pedestrian target corresponding to the same numbering information. The present application is based on the high-dimensional features of pedestrians, and binds the high-dimensional features to the pedestrian targets with the corresponding numbering information to avoid the failure problem of using the intersection-over-union distance when pedestrians are occluded, thereby improving the pedestrian tracking accuracy.

[0091] On the basis of the above embodiments, in order to avoid the long update interval of the first high-dimensional feature in the passenger flow feature library, which causes the first high-dimensional feature to gradually become invalid, when updating the first high-dimensional feature and obtaining the second high-dimensional feature, a weighted feature update method can be used to weight the first high-dimensional feature in the passenger flow feature library and the feature output by the feature recognition network as the updated second high-dimensional feature. At the same time, a time coefficient is introduced in the feature weighting stage to weigh the historical first high-dimensional feature and the current second high-dimensional feature in the updated feature. The specific process includes:

[0092] Determine a weight vector according to the time difference between the high-dimensional feature and the last update; wherein the longer the time difference between the high-dimensional feature and the last update, the smaller the weight vector;

[0093] updating the first high-dimensional feature in the passenger flow feature library according to an updating formula including the weight vector to obtain a second high-dimensional feature;

[0094] The update formula is:

[0095] ;

[0096] in, is the first high-dimensional feature of the pedestrian target, is the second highest dimensional feature of the pedestrian target, The feature recognition network identifies the features of the pedestrian target. is the weight vector.

[0097] However, in the above formula, when the number of pedestrians in adjacent video frames does not change for a long time, resulting in the long-term non-update of features, the duration factor is ignored during the update, so that the first high-dimensional feature will not decay due to duration. To prevent this from happening, duration decay can be introduced during the feature update process. Multiply by the update interval frame number t l To simulate feature attenuation, the second update formula can be as follows:

[0098] ;

[0099] in, is the second weight vector.

[0100] See also Figure 2 , Figure 2 A schematic diagram of a pedestrian tracking process provided in an embodiment of the present application, Figure 2 In the model, the passenger flow detection model is used to identify pedestrian detection frames in video images. Then, the number of pedestrian targets in adjacent image frames is compared. If they are the same, the intersection-over-union distance between the pedestrian detection frames corresponding to the same number information in the previous video frame and the target video frame is calculated, and the trajectory of the pedestrian is matched according to the intersection-over-union distance. Otherwise, the target video frame is input into the feature recognition network for high-dimensional feature recognition of pedestrian targets, and the target features are output. , and then the passenger flow feature database Update the passenger flow feature library It should be noted that the passenger flow feature database By the first high-dimensional feature Composition, passenger flow feature library By the second high dimensional feature constitute.

[0101] See also Figure 3 , Figure 3 A schematic diagram of a pedestrian tracking process of an oblique passenger flow provided in an embodiment of the present application is provided. The following uses a stable tracking process of an oblique passenger flow as a specific application process of the present application, including the following steps:

[0102] The first step is to install an oblique camera device in a supermarket or store to collect video images of customer flow in a set area.

[0103] The second step is to call the edge computing unit to preload the trained pedestrian detection model and feature recognition network.

[0104] The third step is to process the received video images frame by frame. First, run the pedestrian detection model to detect pedestrians in the image, and obtain the pedestrian detection frame of each pedestrian target in the frame image. The number of pedestrian targets and the corresponding coordinate information can be fed back.

[0105] Step 4: Compare the number of pedestrians in adjacent frames and select different matching cost calculation methods according to the number changes to achieve efficient tracking, including:

[0106] (1) When the number of pedestrians in adjacent frame images changes (increases or decreases), the feature matching method is used to calculate the cosine distance between the first high-dimensional feature and the second high-dimensional feature corresponding to the same number information, that is, the two-dimensional cost matrix of the same number information in the previous and next images is calculated based on the first high-dimensional feature and the second high-dimensional feature. At the same time, a speed prior constraint is added in the process of feature distance matching, and some mismatched trajectories with similar features are filtered out through the speed prior constraint.

[0107] (2) When the number of pedestrians in adjacent frame images has not changed, the intersection-over-union (IoU) distance between the pedestrian detection frames corresponding to the same number information in the previous video frame and the target video frame is calculated, and the pedestrian trajectories are matched based on the IoU distance.

[0108] Step 5: Use the obtained two-dimensional cost matrix as input and use the Hungarian algorithm to obtain the matching result of the pedestrian trajectory.

[0109] Step 6: Update the high-dimensional features in the passenger flow feature library based on the matching results;

[0110] Specifically, when executing the sixth step, it can be determined whether to update the high-dimensional features according to different matching results.

[0111] 1) By calculating the positional relationship of all targets in the current frame image, determine whether there is occlusion between targets;

[0112] 2) For occluded targets, features are not updated; for matching targets without occlusion, features are updated.

[0113] 3) The feature update adopts the weighted update method, and the historical features and current matching features in the passenger flow feature library are weighted to serve as the updated features. At the same time, the duration coefficient is introduced in the feature weighting stage to weigh the historical features and current features in the updated features.

[0114] In view of the special visual requirements of obliquely mounted cameras, this embodiment builds a pedestrian detection model and feature recognition network based on a general model and a self-built data set for stable tracking of passenger flow in obliquely mounted camera scenarios. In view of the characteristics of feature distance matching, which has high accuracy but low efficiency, and intersection-over-union distance matching, which has high efficiency but low robustness to occluded scenes, this embodiment organically combines feature distance matching and intersection-over-union distance matching, selects a specific matching cost matrix by changing the number of targets in adjacent frames, so that the tracking algorithm can effectively take into account both real-time and accuracy, and proposes a speed prior to convert feature matching from global to local, further improving tracking accuracy.

[0115] In the feature library update stage, the embodiment of the present application analyzes the position of the pedestrian detection frame, filters out the update of the occlusion feature, maintains the stability of the feature library features, and adopts a weighted feature update method. The updated feature is the weighted result of the historical features in the passenger flow feature library and the current matching features. In the feature weighting stage, the duration coefficient can be introduced to weigh the historical features and current features in the updated features.

[0116] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of a pedestrian tracking system provided in an embodiment of the present application. The present application also provides a pedestrian tracking system, including:

[0117] A video acquisition module, used for acquiring video images;

[0118] A pedestrian detection module, used to perform pedestrian target detection on the target video frame of the video image to obtain a pedestrian detection frame for each pedestrian target;

[0119] A feature recognition module is used to extract pedestrian images according to the regional coordinate information fed back by the pedestrian detection frame and input them into a feature recognition network to obtain a passenger flow feature library consisting of the first high-dimensional features corresponding to each pedestrian target, and assign number information to each pedestrian target;

[0120] A feature updating module, configured to update the first high-dimensional feature in the passenger flow feature library according to all pedestrian targets in the target video frame to obtain a second high-dimensional feature if there is a change in the number of pedestrians between the target video frame and its previous video frame;

[0121] A pedestrian tracking module is used to determine the movement trajectory of the pedestrian target with the same number information in the video image according to the first high-dimensional feature and the second high-dimensional feature.

[0122] Based on the above embodiments, as a preferred embodiment, the pedestrian detection module is a module for performing the following steps:

[0123] Obtain a pedestrian detection data set containing annotation information; the annotation information includes pedestrian detection frame coordinate position information and detection confidence;

[0124] Training a pre-trained target detection network based on the pedestrian detection data set to obtain a pedestrian detection model;

[0125] The pedestrian detection model is applied to perform pedestrian target detection on the target video frame of the video image to obtain a pedestrian detection frame for each pedestrian target.

[0126] Based on the above embodiment, as a preferred embodiment, the feature recognition module includes:

[0127] The feature extraction unit is used to use a feature recognition network to identify the color features, angle features and light features in the pedestrian image to obtain the first high-dimensional features of the pedestrian target; and to build a passenger flow feature library based on the first high-dimensional features of all pedestrian targets.

[0128] Based on the above embodiments, as a preferred embodiment, the pedestrian tracking module is a module for performing the following steps:

[0129] Calculate a two-dimensional cost matrix of the same serial number information in the previous and next images according to the first high-dimensional feature and the second high-dimensional feature; the number of rows of the two-dimensional cost matrix is ​​the number of pedestrians in the target video frame, and the number of columns is the number of high-dimensional features in the passenger flow feature library;

[0130] The two-dimensional cost matrix is ​​used as the input of the Hungarian algorithm, and the motion trajectory of the pedestrian target corresponding to the same numbering information is determined according to the optimal row and column indexes output by the Hungarian algorithm.

[0131] Based on the above embodiment, as a preferred embodiment, the pedestrian tracking module further includes:

[0132] The speed optimization unit is used to perform the following steps:

[0133] Calculating the moving speed of the pedestrian target according to the pedestrian detection frame of the same pedestrian target in the target video frame and the previous video frame;

[0134] Determine a speed matching threshold of the number information corresponding to the pedestrian target based on the movement speed;

[0135] Determine a two-dimensional velocity representation matrix based on a pedestrian detection frame of the pedestrian target in the target video frame and a pedestrian detection frame of the same pedestrian target in the passenger flow feature library in the previous video frame;

[0136] The elements in the two-dimensional velocity representation matrix that are greater than the velocity matching threshold are set to 1, and the elements that do not exceed the velocity matching threshold are set to 0;

[0137] The two-dimensional velocity representation matrix is ​​added to the two-dimensional cost matrix to obtain a cost matrix; the cost matrix is ​​used to replace the two-dimensional cost matrix as an input of the Hungarian algorithm.

[0138] Based on the above embodiments, as a preferred embodiment, it also includes:

[0139] The distance calculation module is used to calculate the intersection-and-union distance between the pedestrian detection frames corresponding to each pedestrian target in the target video frame; if the intersection-and-union distance is less than 1, stop updating the high-dimensional features of the pedestrian targets corresponding to the two corresponding pedestrian detection frames; if the intersection-and-union distance is greater than or equal to 1, enter the feature update module.

[0140] Based on the above embodiment, as a preferred embodiment, the feature update module includes:

[0141] A weight calculation unit, used to determine a weight vector according to a time difference between the high-dimensional feature and the last update; wherein the longer the time difference from the last update, the smaller the weight vector;

[0142] An updating unit, configured to update the first high-dimensional feature in the passenger flow feature library according to an updating formula including the weight vector to obtain a second high-dimensional feature;

[0143] The update formula is:

[0144] ;

[0145] in, is the first high-dimensional feature of the pedestrian target, is the second highest dimensional feature of the pedestrian target, The feature recognition network identifies the features of the pedestrian target. is the weight vector.

[0146] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0147] The present application also provides an electronic device, see Figure 5 , a structural diagram of an electronic device provided in an embodiment of the present application, such as Figure 5 As shown, a processor 1410 and a memory 1420 may be included.

[0148] Among them, the processor 1410 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1410 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1410 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1410 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1410 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0149] The memory 1420 may include one or more computer-readable storage media, which may be non-transitory. The memory 1420 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 1420 is at least used to store the following computer program 1421, wherein, after the computer program is loaded and executed by the processor 1410, it can implement the relevant steps in the method performed by the electronic device side disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 1420 may also include an operating system 1422 and data 1423, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 1422 may include Windows, Linux, Andro numbering information, etc.

[0150] In some embodiments, the electronic device may further include a display screen 1430 , an input / output interface 1440 , a communication interface 1450 , a sensor 1460 , a power source 1470 , and a communication bus 1480 .

[0151] certainly, Figure 5 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiments of the present application. In actual applications, the electronic device may include Figure 5 More or fewer components than shown, or combinations of certain components.

[0152] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the system provided in the embodiment, since it corresponds to the method provided in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0153] Specific examples are used herein to illustrate the principles and implementation methods of the present application, and the description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

[0154] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A pedestrian tracking method, characterized in that: include: Acquire video images; Performing pedestrian target detection on the target video frame of the video image to obtain a pedestrian detection frame for each pedestrian target; According to the regional coordinate information fed back by the pedestrian detection frame, the pedestrian image is extracted and input into the feature recognition network to obtain a passenger flow feature library composed of the first high-dimensional features corresponding to each pedestrian target, and number information is assigned to each pedestrian target; If there is a change in the number of pedestrians between the target video frame and its previous video frame, the first high-dimensional feature in the passenger flow feature library is updated according to all pedestrian targets in the target video frame to obtain a second high-dimensional feature; The motion trajectory of the pedestrian target with the same number information in the video image is determined according to the first high-dimensional feature and the second high-dimensional feature.

2. The pedestrian tracking method according to claim 1, characterized in that: Performing pedestrian target detection on the target video frame of the video image to obtain a pedestrian detection frame for each pedestrian target includes: Obtain a pedestrian detection data set containing annotation information; the annotation information includes pedestrian detection frame coordinate position information and detection confidence; Training a pre-trained target detection network based on the pedestrian detection data set to obtain a pedestrian detection model; The pedestrian detection model is applied to perform pedestrian target detection on the target video frame of the video image to obtain a pedestrian detection frame for each pedestrian target.

3. The pedestrian tracking method according to claim 1, characterized in that: Pedestrian images are extracted and input into the feature recognition network to obtain the passenger flow feature library consisting of the first high-dimensional features corresponding to each pedestrian target, including: Using a feature recognition network to identify color features, angle features, and light features in the pedestrian image, to obtain a first high-dimensional feature of the pedestrian target; A passenger flow feature library is constructed based on the first high-dimensional features of all pedestrian targets.

4. The pedestrian tracking method according to claim 1, characterized in that: Determining the motion trajectory of the pedestrian target with the same number information in the video image according to the first high-dimensional feature and the second high-dimensional feature includes: Calculate a two-dimensional cost matrix of the same serial number information in the previous and next images according to the first high-dimensional feature and the second high-dimensional feature; the number of rows of the two-dimensional cost matrix is ​​the number of pedestrians in the target video frame, and the number of columns is the number of high-dimensional features in the passenger flow feature library; The two-dimensional cost matrix is ​​used as the input of the Hungarian algorithm, and the motion trajectory of the pedestrian target corresponding to the same numbering information is determined according to the optimal row and column indexes output by the Hungarian algorithm.

5. The pedestrian tracking method according to claim 4, characterized in that: Before using the two-dimensional cost matrix as the input of the Hungarian algorithm and determining the motion trajectory of the pedestrian target corresponding to the same number information according to the optimal row and column index output by the Hungarian algorithm, the method further includes: Calculating the moving speed of the pedestrian target according to the pedestrian detection frame of the same pedestrian target in the target video frame and the previous video frame; Determine a speed matching threshold of the number information corresponding to the pedestrian target based on the movement speed; Determine a two-dimensional velocity representation matrix based on a pedestrian detection frame of the pedestrian target in the target video frame and a pedestrian detection frame of the same pedestrian target in the passenger flow feature library in the previous video frame; The elements in the two-dimensional velocity representation matrix that are greater than the velocity matching threshold are set to 1, and the elements that do not exceed the velocity matching threshold are set to 0; The two-dimensional velocity representation matrix is ​​added to the two-dimensional cost matrix to obtain a cost matrix; the cost matrix is ​​used to replace the two-dimensional cost matrix as an input of the Hungarian algorithm.

6. The pedestrian tracking method according to claim 1, characterized in that: When the first high-dimensional feature in the passenger flow feature library is updated according to all pedestrian targets in the target video frame to obtain the second high-dimensional feature, the method further includes: Calculating the intersection-over-union distance between pedestrian detection frames corresponding to each pedestrian target in the target video frame; If the intersection-over-union ratio distance is less than 1, stop updating the high-dimensional features of the pedestrian targets corresponding to the two corresponding pedestrian detection frames; If the intersection-over-union ratio distance is greater than or equal to 1, the step of updating the first high-dimensional feature in the passenger flow feature library according to all pedestrian targets in the target video frame to obtain a second high-dimensional feature is performed.

7. The pedestrian tracking method according to claim 1 or 6, characterized in that: The first high-dimensional feature in the passenger flow feature library is updated according to all pedestrian targets in the target video frame, and the second high-dimensional feature is obtained, including: Determine a weight vector according to the time difference between the high-dimensional feature and the last update; wherein the longer the time difference between the high-dimensional feature and the last update, the smaller the weight vector; updating the first high-dimensional feature in the passenger flow feature library according to an updating formula including the weight vector to obtain a second high-dimensional feature; The update formula is: ； in, is the first high-dimensional feature of the pedestrian target, is the second highest dimensional feature of the pedestrian target, The feature recognition network identifies the features of the pedestrian target. is the weight vector.

8. A pedestrian tracking system, characterized in that: include: A video acquisition module, used for acquiring video images; A pedestrian detection module, used to perform pedestrian target detection on the target video frame of the video image to obtain a pedestrian detection frame for each pedestrian target; A feature recognition module is used to extract pedestrian images according to the regional coordinate information fed back by the pedestrian detection frame and input them into a feature recognition network to obtain a passenger flow feature library consisting of the first high-dimensional features corresponding to each pedestrian target, and assign number information to each pedestrian target; A feature updating module, configured to update the first high-dimensional feature in the passenger flow feature library according to all pedestrian targets in the target video frame to obtain a second high-dimensional feature if there is a change in the number of pedestrians between the target video frame and its previous video frame; A pedestrian tracking module is used to determine the movement trajectory of the pedestrian target with the same number information in the video image according to the first high-dimensional feature and the second high-dimensional feature.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which implements the steps of the method according to any one of claims 1 to 7 when executed.

Images