Cross-lens trajectory tracking method, system and device and storage medium

By obtaining the pedestrian location and converting it into global map coordinates, combining ReID features and Kalman filtering algorithm, the deviation problem of downward pedestrian trajectory analysis is solved in multiple lenses, and the accuracy and robustness of the cross-lens tracking system are improved. sex.

CN118887261APending Publication Date: 2024-11-01CHINA UNITED NETWORK COMM GRP CO LTD

Patent Information

Application Number
CN202410903026.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-05
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

In scenes where multiple lenses and no overlapping areas between the lenses, pedestrian position information cannot be used for pedestrian trajectory analysis, and the pedestrian detection box represents the problem that the current position of the pedestrian is prone to cause obvious deviations in the dynamic tracking of pedestrians.

Method used

By obtaining the pedestrian locations in the surveillance video stream, using the projection matrix to convert the pedestrian locations into coordinate positions on the global map, and combining ReID features to match the historical trajectory library, Kalman filtering algorithm is used for dynamic analysis to achieve cross-multi-camera cross-multi-camera pedestrian track tracking and identification.

Benefits of technology

It improves the accuracy and robustness of the cross-lens tracking system, and can maintain good performance when pedestrian posture changes or images are blocked, realizing pedestrian trajectory tracking and recognition across multiple cameras.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118887261A_ABST
    Figure CN118887261A_ABST
Patent Text Reader

Abstract

The invention provides a cross-lens trajectory tracking method and system, electronic equipment and a storage medium, and aims to solve the problems that pedestrian position information cannot be used for pedestrian trajectory analysis and a pedestrian detection frame represents the current position of a pedestrian to easily generate obvious deviation on dynamic tracking of the pedestrian in a scene with multiple lenses and no overlapping area among the lenses. The method comprises the following steps: acquiring a monitoring video stream, detecting pedestrians in the video stream, and acquiring pedestrian attachment points; using the projection matrix to convert the pixel position of the pedestrian grounding point in the lens into the coordinate position of the pedestrian on the global map; and matching the pedestrian with all historical tracks in a historical track library according to the coordinate position of the pedestrian on the global map and the ReID feature of the pedestrian so as to determine the track of the pedestrian and realize real-time tracking and track analysis of the pedestrian. On the basis of improving the accuracy of the tracking system, the robustness of the algorithm is improved, and the cross-lens tracking capability of pedestrians is comprehensively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence visual recognition technology, and in particular to a cross-shot trajectory tracking method, a cross-shot trajectory tracking system, an electronic device, and a computer-readable storage medium. Background Art

[0002] Cross-lens multi-pedestrian trajectory tracking is an important and challenging problem in the field of intelligent visual surveillance. With the increasing number of cameras, a large number of video streams are connected to the monitoring management center. The traditional cross-lens multi-pedestrian tracking method based on manual methods can no longer meet the actual scene requirements in terms of timeliness, accuracy, labor costs, etc. Therefore, it is urgent to use an intelligent processing algorithm to automatically perform real-time target tracking and trajectory analysis of cross-lens multi-pedestrians to improve the work efficiency of the video surveillance process. Therefore, an efficient cross-lens multi-pedestrian trajectory tracking system has become a key research direction in the field of intelligent monitoring.

[0003] At present, there are a lot of academic research works on single-lens multi-pedestrian trajectory tracking, which greatly promoted the development of intelligent video surveillance, but multi-target tracking in multi-camera scenarios still needs to be explored. In fact, due to the limited field of view of a single camera, it is difficult to observe the complete trajectory of the target in a large range with only one camera. In areas with dense crowds and high security levels such as factories, schools, and military areas, multiple cameras are often required for all-round monitoring, so cross-lens multi-target tracking is more in line with the needs of real-world scenarios. The cross-lens multi-pedestrian trajectory tracking system focuses on associating the trajectories of the same person between different cameras and generating a complete global trajectory of the pedestrian. The position distribution relationship between multiple cameras can be divided into two categories: with overlapping fields of view and without overlapping fields of view. The cross-camera pedestrian trajectory association with overlapping fields of view is relatively simple and has a high recognition accuracy. However, due to the open blind spots of the viewing angle, there are many uncertainties when pedestrians disappear, so the inter-camera trajectory association between non-overlapping cameras is more complicated.

[0004] The current cross-lens multi-pedestrian tracking relies on a single-lens multi-target tracking algorithm, most of which need to use Kalman filtering to establish a motion model of the target in the lens image, and add the appearance features of the pedestrians for joint tracking (such as patent CN115760921A, patent CN 116645402A). These patents first detect pedestrians and then perform Kalman filtering analysis on the pixel positions of the detection frame. Since there is no clear correlation between the pixel positions of different lenses, such methods cannot be extended to scenes with multiple lenses.

[0005] In multi-camera scenarios, since the temporal and spatial displacement information of pedestrians in different lens images is difficult to use, many cross-lens multi-pedestrian tracking algorithms no longer use cross-lens displacement information for tracking. Taking patent CN 112200841A as an example, based on the single-lens multi-target tracking algorithm, it extracts the posture sequence fusion features of each pedestrian trajectory under each lens, and compares and calibrates it with the pedestrian features under other cameras to achieve multi-pedestrian tracking in cross-camera scenarios. However, affected by lighting conditions, picture clarity, appearance and other conditions, the appearance features of the same pedestrian under different lenses may vary greatly, which in turn affects the performance of these cross-lens tracking systems.

[0006] In addition, some algorithms also use pedestrian location information, such as patent CN

[0007] 114972445A analyzes facial information, appearance features (which are encoded into feature sequences by deep neural networks such as osnet and resnet50, also known as ReID (Person re-identification) features), and position mapping information, and combines the three to achieve cross-lens trajectory tracking. Since this type of method uses the similarity of static map coordinates for pedestrian re-identification tasks, it is only applicable to scenes where multiple lenses can capture the same pedestrian at the same time, and cannot be extended to multi-lens scenes without overlapping fields of view. In addition, these methods use the pixel position of the pedestrian detection box to estimate the global map coordinates. When the pedestrian's posture changes (such as squatting, bending over, sitting) or when part of the pedestrian's image is blocked, the pedestrian detection box changes significantly, which will cause a large number of errors in the estimation process of the global map coordinates. Summary of the invention

[0008] In order to at least solve the problem in the prior art that in a scenario with multiple lenses and no overlapping areas between lenses, pedestrian position information cannot be used for pedestrian trajectory analysis, and the pedestrian detection frame represents the current position of the pedestrian, which easily causes obvious deviation in the dynamic tracking of the pedestrian, the present disclosure provides a cross-lens trajectory tracking method, a cross-lens trajectory tracking system, an electronic device and a computer-readable storage medium, which can improve the robustness of the algorithm on the basis of improving the accuracy of the tracking system, and comprehensively enhance the cross-lens tracking capability of pedestrians.

[0009] In a first aspect, the present disclosure provides a cross-shot trajectory tracking method, the method comprising:

[0010] Obtain surveillance video stream, detect pedestrians in the video stream, and obtain the location of pedestrians;

[0011] Use the projection matrix to convert the pixel position of the pedestrian's contact point in the camera into the coordinate position of the pedestrian on the global map;

[0012] According to the coordinate position of the pedestrian on the global map and the ReID features of the pedestrian, the pedestrian is matched with all historical trajectories in the historical trajectory library to determine the pedestrian's trajectory and realize real-time tracking and trajectory analysis of the pedestrian.

[0013] Furthermore, the step of acquiring a surveillance video stream, detecting pedestrians in the video stream, and acquiring the location of the pedestrians includes:

[0014] Detect the key points of the pedestrians in the camera and output the pixel positions x, y and corresponding confidence of the key points of the pedestrians’ postures.

[0015] If the confidence of a key point is less than the threshold, the key point is considered to be occluded;

[0016] If one or more of the left and right foot key points are blocked, the pedestrian posture key points are restored using the pre-trained pedestrian posture point restorer;

[0017] Calculate the center point of the pedestrian's left and right foot key points as the location representing the pedestrian.

[0018] Furthermore, the method further comprises:

[0019] Train a pedestrian pose point restorer based on an encoder-decoder framework to restore the pose key points of occluded pedestrians, including:

[0020] Prepare a dataset of pedestrian posture key points;

[0021] Randomly perform mask operations on the pose key points in the dataset;

[0022] Construct an encoder to reduce the dimension of the pedestrian's posture key points after the mask operation to obtain the pedestrian's action state;

[0023] Build a decoder to upgrade the pedestrian's motion state and obtain all the pedestrian's posture key points;

[0024] The masked pedestrian posture key points are used as the model input, the unmasked pedestrian posture points are used as the model target, and MSE (Mean Squared Error Loss) is used as the model loss function to train the encoder-decoder.

[0025] Furthermore, the use of the projection matrix to convert the pixel position of the pedestrian's contact point in the lens into the coordinate position of the pedestrian on the global map includes:

[0026] For each camera, calculate the corresponding lens-global map coordinate transformation matrix;

[0027] Using the lens-global map coordinate conversion matrix corresponding to the camera that captured the pedestrian, the following equation (1) is used to convert the pixel position of the pedestrian in the lens into the global map coordinate;

[0028]

[0029] Where (x, y) is the pixel position in the lens, (x', y') is its corresponding global map coordinate, and M is the lens-global map coordinate conversion matrix of the camera;

[0030] Among them, for each camera, the corresponding lens-global map coordinate transformation matrix is ​​calculated separately, including:

[0031] For a certain camera, obtain n locations with obvious identification in the lens, obtain the lens pixel positions of these locations, and simultaneously obtain the global map positions corresponding to these locations;

[0032] Substitute the lens pixel positions of these locations and the corresponding global map positions into equation (1) respectively, and use the least squares method to estimate the optimal transformation matrix as the lens-global map coordinate transformation matrix M of the camera;

[0033] Repeat this operation for all cameras to obtain the lens-global map coordinate transformation matrix corresponding to each camera.

[0034] Furthermore, matching the pedestrian with all historical trajectories in the historical trajectory library according to the coordinate position of the pedestrian on the global map and the ReID feature of the pedestrian to determine the trajectory of the pedestrian includes:

[0035] The detected pedestrian image is subjected to feature extraction. A general ReID feature extraction network is used to transcode the pedestrian image into a 512-dimensional feature vector. At the same time, the coordinate position of the pedestrian in the global map is recorded, which is recorded as

[0036] According to the ReID distance and motion distance between the pedestrian and each historical trajectory, the pedestrian is matched with all the historical trajectories in the historical trajectory library to determine whether the pedestrian belongs to the pedestrian trajectory that already exists in the trajectory library;

[0037] If yes, the pedestrian trajectory is updated, including adding the ReID features of the newly observed pedestrian to the pedestrian trajectory and updating the corresponding Kalman parameters;

[0038] If not, a new trajectory is created for the pedestrian and stored in the historical trajectory library.

[0039] Furthermore, matching the pedestrian with all historical trajectories in the historical trajectory library according to the ReID distance and the movement distance between the pedestrian and each historical trajectory, and determining whether the pedestrian belongs to a pedestrian trajectory already existing in the trajectory library, includes:

[0040] Calculate the ReID distance between the pedestrian and each historical trajectory stored in the historical trajectory library, that is, 1 minus the cosine similarity between the 512-dimensional feature vectors of the two after transcoding, as the feature distance between the pedestrian and the trajectory;

[0041] Perform the following operations on each historical track: calculate the prior state of the historical track at the current moment based on the global map coordinates, the velocity vector, and the time length from the current moment, wherein the prior state includes the global map prior coordinates and the velocity vector of the track;

[0042] Use Kalman filtering to calculate the covariance matrix of the predicted state of the trajectory The coordinate error covariance matrix S of the trajectory k , according to the prior coordinates of the trajectory Coordinate error covariance matrix S k , and the observation coordinates Calculate the square of the Mahalanobis distance between the prior coordinates and the observed coordinates as the motion distance between the detected pedestrian and the historical trajectory;

[0043] The total distance between the observed pedestrian and the historical trajectory is calculated by the following formula (2):

[0044] Total distance = r * characteristic distance + (1-r) * movement distance (2)

[0045] r is a pre-set weight coefficient. If the total distance is less than the threshold, it is considered that the observed pedestrian and the historical trajectory in the historical trajectory library belong to the same person. Otherwise, it is considered that the newly observed pedestrian and the current trajectory do not belong to the same person.

[0046] Furthermore, the prior state of the historical trajectory at the current moment is calculated based on the global map coordinates, the velocity vector, and the time length from the current moment, including

[0047] For each historical trajectory, the prior state of the historical trajectory at the current moment is calculated in the following way:

[0048]

[0049] Among them, (x k-1 ,y k-1 ) represents the global map coordinates of the pedestrian trajectory when it was last updated, Δ(x k-1 , Δy k-1 ) represents the velocity vector of the pedestrian trajectory last updated, is the current prior estimation state of the trajectory, that is, the current position and speed estimated based on the coordinates and speed of the trajectory at the last update, F(t) is the dynamic transfer matrix, and the specific parameters are:

[0050]

[0051] Among them, t k Represents the time of the kth update.

[0052] In a second aspect, the present disclosure provides a cross-shot trajectory tracking system, the system comprising:

[0053] An acquisition module is configured to acquire a surveillance video stream, detect pedestrians in the video stream, and acquire the location of the pedestrians;

[0054] A conversion module, which is configured to use a projection matrix to convert a pixel position of a pedestrian's contact point in the camera lens into a coordinate position of the pedestrian on a global map;

[0055] The matching module is configured to match the pedestrian with all historical trajectories in the historical trajectory library according to the coordinate position of the pedestrian on the global map and the ReID features of the pedestrian, so as to determine the trajectory of the pedestrian and realize real-time tracking and trajectory analysis of the pedestrian.

[0056] In a third aspect, the present disclosure provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the cross-shot trajectory tracking method as described in any one of the first aspects.

[0057] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the cross-shot trajectory tracking method described in any one of the first aspects above is implemented.

[0058] Beneficial effects:

[0059] The cross-lens trajectory tracking method, cross-lens trajectory tracking system, electronic device and storage medium provided by the present disclosure obtain the location of the pedestrian. Using the location to replace the traditional detection frame center point or the upper left and lower right points of the detection frame can better capture the dynamic trajectory of the pedestrian, and the pedestrian trajectory obtained by this method still has good robustness even when the pedestrian posture changes significantly or the pedestrian image is partially blocked. After obtaining the pedestrian's location, the projection matrix is ​​used to convert the pixel position of the pedestrian's location in the lens into the coordinate position of the pedestrian on the global map, and then the Kalman filter algorithm is used to dynamically analyze the coordinate position of the pedestrian; when the pedestrian's trajectory spans different cameras, the moving vector (i.e., movement direction and movement speed) of the pedestrian's trajectory and the ReID feature of the pedestrian can be combined to jointly complete the pedestrian trajectory tracking and identification across multiple cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic diagram of a flow chart of a cross-shot trajectory tracking method provided in the first embodiment of the present disclosure;

[0061] Figure 2 A schematic diagram of a process of implementing a cross-shot trajectory tracking method provided in the second embodiment of the present disclosure;

[0062] Figure 3 This is an architecture diagram of a cross-lens trajectory tracking system provided in Embodiment 3 of the present disclosure;

[0063] Figure 4 This is an architecture diagram of an electronic device provided in Embodiment 4 of the present disclosure. DETAILED DESCRIPTION

[0064] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the present disclosure is further described in detail below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention, rather than to limit the present invention.

[0065] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence; and, in the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be arbitrarily combined with each other.

[0066] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. The singular forms of "a", "said" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings.

[0067] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present disclosure, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0068] The Kalman filter algorithm combines the laws of object (pedestrian) motion and the measurement information of sensors (surveillance cameras), providing an efficient, accurate and adaptive estimation method for pedestrian trajectories. Therefore, it is widely used in pedestrian trajectory analysis algorithms under a single lens. However, when there are multiple lenses, the Kalman filter algorithm is difficult to apply due to the lack of obvious correlation between the lenses. Therefore, the current cross-lens pedestrian trajectory analysis methods either no longer use pedestrian position information, or only calculate the position similarity of each pedestrian in the overlapping area of ​​the lenses and perform pedestrian ID fusion. In scenarios with multiple lenses and no overlapping areas between lenses, pedestrian position information cannot be used for pedestrian trajectory analysis.

[0069] In addition, the Kalman filter algorithm needs to identify the position coordinates of pedestrians. The current mainstream algorithm uses the center point of the pedestrian detection frame or the upper left and lower right points of the detection frame to represent the current position of the pedestrian. When the pedestrian's posture changes (or bends, squats, sits) or when part of the pedestrian's image is blocked, the center point and the upper left and lower right points of the detection frame will change significantly, resulting in a significant deviation in the dynamic tracking of the pedestrian.

[0070] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems existing in the prior art are described in detail below with specific embodiments. It is understandable that in the embodiments of the present application, the execution subject can perform some or all of the steps in the embodiments of the present application, and these steps or operations are only examples. The embodiments of the present application can also perform other operations or deformations of various operations. In addition, each step can be performed in a different order presented in the embodiments of the present application, and it is possible not to perform all the operations in the embodiments of the present application. And, the following several specific embodiments can be combined with each other, and may not be repeated in some embodiments for the same or similar concepts or processes.

[0071] Figure 1 A schematic diagram of a cross-shot trajectory tracking method provided in the first embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the method includes:

[0072] Step S101: Obtain a surveillance video stream, detect pedestrians in the video stream, and obtain the location of the pedestrians;

[0073] Step S102: using a projection matrix to convert the pixel position of the pedestrian's contact point in the camera into the coordinate position of the pedestrian on the global map;

[0074] Step S103: Match the pedestrian with all historical trajectories in the historical trajectory library according to the coordinate position of the pedestrian on the global map and the ReID feature of the pedestrian to determine the trajectory of the pedestrian and realize real-time tracking and trajectory analysis of the pedestrian.

[0075] The disclosed embodiment is a cross-lens trajectory tracking method based on pedestrian position analysis and Kalman filtering. First, a historical trajectory library is prepared for storing pedestrian trajectories, which stores displacement motion model data such as ReID features and trajectory coordinates of detected pedestrians. By acquiring a monitoring video stream, a pedestrian detector is used to detect pedestrian frames, and a posture detector (such as YoloPose, OpenPose, etc.) is used to detect posture key points of pedestrians in the lens. The center points of the key points of the left and right feet of the pedestrian are used as the locations representing the pedestrian. Then, the lens-global map coordinate conversion matrix corresponding to the camera that captures the pedestrian is used to convert the pixel position of the pedestrian's location into global map coordinates, and feature extraction is performed on the detected pedestrian image. A general ReID feature extraction network (such as resnet50, etc.) is used. Then, according to the ReID features and the coordinate positions on the global map, the pedestrian is matched with all historical trajectories in the historical trajectory library to determine whether the pedestrian belongs to the pedestrian trajectory that already exists in the trajectory library, thereby determining the pedestrian trajectory that matches it, and realizing real-time tracking and trajectory analysis of the pedestrian.

[0076] The disclosed embodiment uses a contact point to replace the traditional center point of the detection frame or the upper left and lower right points of the detection frame, which can better capture the dynamic trajectory of the pedestrian. The pedestrian trajectory obtained by this method still has good robustness even when the pedestrian's posture changes significantly or the pedestrian image is partially blocked. After obtaining the pedestrian's contact point, the projection matrix is ​​used to convert the pixel position of the pedestrian's contact point in the lens into the coordinate position of the pedestrian on the global map, and then the Kalman filter algorithm is used to dynamically analyze the coordinate position of the pedestrian. When the pedestrian's trajectory spans different cameras, the moving vector (i.e., the direction and speed of movement) of the pedestrian's trajectory and the pedestrian's ReID feature can be combined to jointly complete the pedestrian trajectory tracking and identification across multiple cameras.

[0077] Furthermore, the step of acquiring a surveillance video stream, detecting pedestrians in the video stream, and acquiring the location of the pedestrians includes:

[0078] Detect the key points of the pedestrians in the camera and output the pixel positions x, y and corresponding confidence of the key points of the pedestrians’ postures.

[0079] If the confidence of a key point is less than the threshold, the key point is considered to be occluded;

[0080] If one or more of the left and right foot key points are blocked, the pedestrian posture key points are restored using the pre-trained pedestrian posture point restorer;

[0081] Calculate the center point of the pedestrian's left and right foot key points as the location representing the pedestrian.

[0082] Before performing trajectory tracking, the disclosed embodiment needs to train a pedestrian posture point restorer based on an encoder-decoder framework to restore the key points of the obscured pedestrian, and the posture key points include left and right feet, and multiple positions in the following, such as left and right eyes, nose, left and right ears, neck, left and right shoulders, left and right elbows, left and right hands, left and right hips, left and right knees, etc. The main function of this component is to receive the pixel coordinates of all unobstructed posture key points of the pedestrian, analyze and complete the coordinates of the obscured key points, read multiple surveillance video streams, use the YoloPose model to detect pedestrians in the video, and obtain the detection frame of each pedestrian, each posture key point and the confidence corresponding to the key point. If the confidence is less than the threshold, it is considered that the key point is obscured. The threshold can be set according to the situation of the trainer and the actual situation, such as 0.4. If the pedestrian's left and right foot key points are not obscured, the center point of the two points is calculated as the pedestrian's contact point. If the pedestrian's left and right foot key points are obscured, the pedestrian's posture points are restored using the pre-trained pedestrian posture point restorer.

[0083] In addition to obtaining all posture key points first, it is also possible to determine whether the left and right foot key points are blocked. The method of obtaining the pedestrian's location also includes:

[0084] Detect the left and right foot key points of pedestrians in the camera.

[0085] If the left and right foot key points of the pedestrian are not blocked, the center point of the two points is calculated as the pedestrian's contact point;

[0086] If one or more of the left and right foot key points are blocked, the pedestrian in the camera is detected for posture key points, and the pedestrian posture points are restored using a pre-trained pedestrian posture point restorer; and,

[0087] Use the center point of the restored left and right foot key points as the pedestrian attachment point.

[0088] The dynamic trajectory of pedestrians can be better captured by sticking to the location, and the pedestrian trajectory obtained by this method still has good robustness even when the pedestrian's posture changes significantly or the pedestrian image is partially occluded.

[0089] Furthermore, the method further comprises:

[0090] Train a pedestrian pose point restorer based on an encoder-decoder framework to restore the pose key points of occluded pedestrians, including:

[0091] Prepare a dataset of pedestrian posture key points;

[0092] Randomly perform mask operations on the pose key points in the dataset;

[0093] Construct an encoder to reduce the dimension of the pedestrian's posture key points after the mask operation to obtain the pedestrian's action state;

[0094] Build a decoder to upgrade the pedestrian's motion state and obtain all the pedestrian's posture key points;

[0095] The masked pedestrian posture key points are used as the model input, the unmasked pedestrian posture points are used as the model target, and MSE is used as the model loss function to train the encoder-decoder.

[0096] In preparation for trajectory tracking, it is necessary to train a pedestrian posture point restorer based on the encoder-decoder framework. The specific training process is as follows:

[0097] a) Prepare a dataset of pedestrian pose key points. You can use a public dataset or a pose detection model (such as YoloPose, OpenPose, etc.) to detect unobstructed pedestrian images and obtain a dataset of pose key points.

[0098] b) Randomly perform mask operations on the posture key points in the dataset, that is, randomly replace some posture detection key points with (0, 0). This operation is used to simulate the situation where pedestrians are partially occluded.

[0099] c) Construct an encoder to reduce the dimension of the pedestrian's posture key points after the mask operation to obtain the pedestrian's action state. According to the needs of the actual scene, the structure of the encoder can select a classic neural network architecture such as a multi-layer perceptron, a one-dimensional convolutional neural network, or LSTM.

[0100] d) Construct a decoder to upgrade the pedestrian's motion state and obtain the full set of pedestrian posture key points. The decoder generally adopts a symmetrical structure with the encoder.

[0101] e) The masked pedestrian posture key points are used as the model input, the unmasked pedestrian posture points are used as the model target, and the MSE is used as the model loss function to train the encoder-decoder.

[0102] By restoring the key points of the pedestrian's posture through the posture point restorer, when the pedestrian's posture changes or part of the pedestrian's image is blocked, or when the pedestrian detection frame changes significantly, the obtained location can be guaranteed to be more accurate, so that the estimation process of the global map coordinates will not produce large errors.

[0103] Furthermore, the use of the projection matrix to convert the pixel position of the pedestrian's contact point in the lens into the coordinate position of the pedestrian on the global map includes:

[0104] For each camera, calculate the corresponding lens-global map coordinate transformation matrix;

[0105] Using the lens-global map coordinate conversion matrix corresponding to the camera that captured the pedestrian, the following equation (1) is used to convert the pixel position of the pedestrian in the lens into the global map coordinate;

[0106]

[0107] Where (x, y) is the pixel position in the lens, (x', y') is its corresponding global map coordinate, and M is the lens-global map coordinate conversion matrix of the camera;

[0108] Among them, for each camera, the corresponding lens-global map coordinate transformation matrix is ​​calculated separately, including:

[0109] For a certain camera, obtain n locations with obvious identification in the lens, obtain the lens pixel positions of these locations, and simultaneously obtain the global map positions corresponding to these locations;

[0110] Substitute the lens pixel positions of these locations and the corresponding global map positions into equation (1) respectively, and use the least squares method to estimate the optimal transformation matrix as the lens-global map coordinate transformation matrix M of the camera;

[0111] Repeat this operation for all cameras to obtain the lens-global map coordinate transformation matrix corresponding to each camera.

[0112] When preparing for trajectory tracking, you also need to calculate the lens-global map coordinate transformation matrix for each camera.

[0113] The function of the lens-global map coordinate conversion matrix is ​​to convert the pixel position in the camera into the global map coordinate. This method requires that the vertical coordinate of the pixel position is a fixed value, so the pedestrian's contact point is selected as the pedestrian's pixel position in the camera. If (x, y) is the lens pixel position, (x', y') is its corresponding global map coordinate, and M is the lens-global map coordinate conversion matrix of the camera, then there is a linear relationship in equation (1).

[0114] For each lens, obtain n clearly identified points in the lens, obtain the lens pixel positions of these points, and obtain the corresponding global map positions of these points. Substitute these points into equation (1) and use the least squares method to estimate the optimal transformation matrix M. After obtaining the pedestrian's point position (x, y), the global map coordinate position of the point can be obtained through equation (1). Repeat this operation for all cameras to obtain the lens-global map coordinate transformation matrix corresponding to each camera.

[0115] Furthermore, matching the pedestrian with all historical trajectories in the historical trajectory library according to the coordinate position of the pedestrian on the global map and the ReID feature of the pedestrian to determine the trajectory of the pedestrian includes:

[0116] The detected pedestrian image is subjected to feature extraction. A general ReID feature extraction network is used to transcode the pedestrian image into a 512-dimensional feature vector. At the same time, the coordinate position of the pedestrian in the global map is recorded, which is recorded as

[0117] According to the ReID distance and motion distance between the pedestrian and each historical trajectory, the pedestrian is matched with all the historical trajectories in the historical trajectory library to determine whether the pedestrian belongs to the pedestrian trajectory that already exists in the trajectory library;

[0118] If yes, the pedestrian trajectory is updated, including adding the ReID features of the newly observed pedestrian to the pedestrian trajectory and updating the corresponding Kalman parameters;

[0119] If not, a new trajectory is created for the pedestrian and stored in the historical trajectory library.

[0120] After obtaining the coordinate position of the pedestrian on the global map and the ReID features of the pedestrian, it is necessary to match them with the pedestrian trajectories in the historical trajectory library to determine the trajectory corresponding to the pedestrian. If the observed pedestrian belongs to the same person as the historical trajectory in the trajectory library. The trajectory is updated, that is, the ReID features of the newly observed pedestrian are added to the trajectory, and the corresponding Kalman parameters (including the posterior state, Kalman gain, and the covariance matrix of the posterior state) are updated. If the newly observed pedestrian does not belong to the same person as all historical trajectories, a new trajectory is created for the pedestrian and stored in the historical trajectory library. Repeating the above steps, the embodiment of the present disclosure can perform real-time tracking and trajectory analysis on pedestrians appearing in multiple shots.

[0121] Furthermore, matching the pedestrian with all historical trajectories in the historical trajectory library according to the ReID distance and the movement distance between the pedestrian and each historical trajectory, and determining whether the pedestrian belongs to a pedestrian trajectory already existing in the trajectory library, includes:

[0122] Calculate the ReID distance between the pedestrian and each historical trajectory stored in the historical trajectory library, that is, 1 minus the cosine similarity between the 512-dimensional feature vectors of the two after transcoding, as the feature distance between the pedestrian and the trajectory;

[0123] Perform the following operations on each historical track: calculate the prior state of the historical track at the current moment based on the global map coordinates, the velocity vector, and the time length from the current moment, wherein the prior state includes the global map prior coordinates and the velocity vector of the track;

[0124] Use Kalman filtering to calculate the covariance matrix of the predicted state of the trajectory The coordinate error covariance matrix S of the trajectory k , according to the prior coordinates of the trajectory Coordinate error covariance matrix S k , and the observation coordinates Calculate the square of the Mahalanobis distance between the prior coordinates and the observed coordinates as the motion distance between the detected pedestrian and the historical trajectory;

[0125] The total distance between the observed pedestrian and the historical trajectory is calculated by the following formula (2):

[0126] Total distance = r*feature distance + (1-r)*movement distance (2)

[0127] r is a pre-set weight coefficient. If the total distance is less than the threshold, it is considered that the observed pedestrian and the historical trajectory in the historical trajectory library belong to the same person. Otherwise, it is considered that the newly observed pedestrian and the current trajectory do not belong to the same person.

[0128] The calculation method of the ReID distance and the Mahalanobis distance may adopt the existing calculation method, and the embodiments of the present disclosure do not limit this.

[0129] This paper analyzes the motion information of pedestrians on the global map through Kalman filtering, extracts the appearance feature information of pedestrians through deep neural network, and combines the two to realize cross-lens pedestrian trajectory analysis. A cross-lens trajectory analysis algorithm for analyzing pedestrian motion information in a cross-lens (non-overlapping field of view) scenario is proposed. By combining the pedestrian's movement vector (i.e., movement direction and movement speed) and the pedestrian's ReID feature, the pedestrian's trajectory can be tracked with high precision in a cross-lens scenario.

[0130] Furthermore, the prior state of the historical trajectory at the current moment is calculated based on the global map coordinates, the velocity vector, and the time length from the current moment, including

[0131] For each historical trajectory, the prior state of the historical trajectory at the current moment is calculated in the following way:

[0132]

[0133] Among them, (x k-1 ,y k-1 ) represents the global map coordinates of the pedestrian trajectory when it was last updated, Δ(x k-1 , Δy k-1 ) represents the velocity vector of the pedestrian trajectory last updated, is the current prior estimation state of the trajectory, that is, the current position and speed estimated based on the coordinates and speed of the trajectory at the last update, F(t) is the dynamic transfer matrix, and the specific parameters are:

[0134]

[0135] Among them, t k Represents the time of the kth update.

[0136] For each historical trajectory, the current position and speed are estimated through the coordinates and speed of the trajectory at the last update, so as to obtain the prior state at the current moment and compare it with the coordinates of the pedestrian, etc., without the need for overlapping areas between cameras, so as to better realize pedestrian trajectory recognition under cross-lens.

[0137] The embodiment of the disclosure is based on the cross-lens multi-pedestrian trajectory tracking of pedestrians based on the planar motion and appearance features of pedestrians. It is not limited by the camera arrangement scene. The coordinates of the pedestrian's location are calculated based on the pedestrian's posture points, and the displacement motion model of the pedestrian is established on the global map. The multi-pedestrian trajectory tracking in the cross-lens scene is carried out in combination with the pedestrian's appearance features. The movement route map of each pedestrian can be drawn in real time on the plane map. On the basis of improving the accuracy of the tracking system, the disclosure improves the robustness of the algorithm and comprehensively improves the cross-lens tracking capability of pedestrians.

[0138] The present invention uses a pedestrian posture detector and an encoder-decoder framework to analyze the key points of the foot of the pedestrian posture to obtain the pedestrian's contact point. Using the contact point instead of the traditional detection frame center point or the upper left and lower right points of the detection frame can better capture the dynamic trajectory of the pedestrian, and the pedestrian trajectory obtained by this method still has good robustness even when the pedestrian posture changes significantly or the pedestrian image is partially blocked.

[0139] After obtaining the location of the pedestrian, the present invention uses a projection matrix to convert the pixel position of the pedestrian's location in the lens into the coordinate position of the pedestrian on the global map. Then, the Kalman filter algorithm is used to dynamically analyze the coordinate position of the pedestrian. When the pedestrian's trajectory crosses different cameras, the pedestrian's movement vector (i.e., movement direction and movement speed) and the pedestrian's ReID features can be combined to complete the pedestrian trajectory tracking and identification across multiple cameras.

[0140] In order to make the design scheme and workflow of the cross-shot multi-pedestrian trajectory tracking method and system proposed in the present disclosure clearer, the second embodiment of the present disclosure also provides a cross-shot trajectory tracking method, which is described in detail, including:

[0141] 1. Train a pedestrian posture point restorer based on the encoder-decoder framework. This example uses 17 human body key points (including left and right eyes, nose, left and right ears, neck, left and right shoulders, left and right elbows, left and right hands, left and right hips, left and right knees, and left and right feet). The function of this component is to receive some key points and infer the positions of other key points.

[0142] 1.1 Data preparation: Obtain training data of 3,000 unobstructed pedestrian images and the annotations of corresponding posture key points. Mask the key points to simulate the scene where pedestrians are obscured. The mask operation is to replace the coordinates of the key points with (0, 0). In this experiment, 10% of the images are masked with 2 key points, 20% of the images are masked with 4 key points, 20% of the images are masked with 6 key points, 20% of the images are masked with 8 key points, 20% of the images are masked with 10 key points, and 10% of the images are masked with 12 key points.

[0143] 1.2 Constructing the encoder, this experiment uses two linear layers to construct, the first layer has 64 neurons, the second layer has 128 neurons, and each layer uses the relu function as the activation function. The encoder reduces the 17 key points (each key contains two input data x and y) to a 32-dimensional vector.

[0144] 1.3 Construct the decoder. This experiment uses two linear layers to construct the decoder. The first layer has 128 neurons and the second layer has 64 neurons. Each layer uses the relu function as the activation function. The decoder upgrades the 32-dimensional vector obtained by the encoder to 17 key points (each key point contains two output data, x and y).

[0145] 1.4 Use 3,000 unobstructed pedestrian images without masking and the corresponding posture key point annotations for model training. Obtain the pedestrian posture key point restorer.

[0146] 2. Construct a lens-global map coordinate transformation matrix for each camera.

[0147] 2.1 Find multiple fixed reference objects, such as electric poles and pillars, in the image captured by a single camera, and obtain the coordinates of the reference objects in the lens image;

[0148] 2.2 Establish a global plane map, find the plane coordinates of these reference objects on the map in the plane map, and use the least squares method to calculate the coordinate transformation matrix from the camera to the plane map;

[0149] 2.3 Repeat the above operation for each camera and record the lens-global map coordinate transformation matrix of each camera respectively.

[0150] 3. Prepare a historical trajectory library to store pedestrian trajectories.

[0151] 4. Read multiple surveillance video streams, use the YoloPose model to detect pedestrians in the video, and obtain the detection frame, 17 posture key points and the corresponding confidence of each pedestrian. If the confidence is less than 0.4, the key point is considered to be blocked.

[0152] 5. If the left and right foot key points of the pedestrian are not blocked, calculate the center point of the two points as the pedestrian's contact point. If the left and right foot key points of the pedestrian are blocked, use the posture key point restorer trained in 1. to restore the pedestrian's posture key points. And use the center point of the restored left and right foot key points as the pedestrian's contact point.

[0153] 6. Retrieve the lens-global map coordinate conversion matrix of the camera that detected the pedestrian, and multiply the pedestrian's location by the matrix to obtain the pedestrian's coordinate point on the global map.

[0154] 7. Match the pedestrian with all historical trajectories in the trajectory library as follows:

[0155] 7.1. Use Kalman filtering to calculate the prior coordinate position of each historical trajectory at the current moment.

[0156] 7.2. Use Kalman filtering to calculate the coordinate error covariance matrix of each historical trajectory at the current moment.

[0157] 7.3. Based on the coordinate error covariance matrix, the square of the Mahalanobis distance between the prior coordinate position of the historical trajectory and the global map coordinate position of the detected pedestrian is calculated as the movement distance between the detected pedestrian and the historical trajectory.

[0158] 7.4. Use resnet50 to extract features of pedestrians, obtain 512-dimensional ReID feature vectors, match pedestrians with all historical feature vectors stored in historical trajectories, and calculate the minimum cosine distance as the feature distance between pedestrians and the trajectory.

[0159] 7.5. Take the weighted sum of the movement distance and feature distance between the pedestrian and the trajectory to obtain the total distance between the pedestrian and the trajectory. If the distance is less than the threshold, the pedestrian is judged to match the trajectory. The latest features of the pedestrian are updated to the trajectory. At the same time, the Kalman-related parameters of the trajectory are updated. If the pedestrian does not match any trajectory, a new trajectory ID is assigned to the pedestrian and stored in the trajectory library.

[0160] The specific implementation process of an implementation method of the present disclosure is as follows: Figure 2 As shown, including:

[0161] Obtain multiple surveillance video stream data, and use pedestrian detectors to detect whether pedestrians are detected;

[0162] If not, the next frame is detected;

[0163] If yes, two steps are performed. On the one hand, the pedestrian posture detector is used to detect whether the pedestrian's foot cloth is blocked. If not, the pedestrian posture foot key points are directly obtained. If yes, the posture key point restorer is used to restore and obtain the pedestrian posture foot key points, which are converted by the corresponding lens map coordinates to obtain the pedestrian map coordinates; the Kalman filter is used to calculate the Mahalanobis distance between the pedestrian's global map coordinates and the historical trajectory coordinates; on the other hand, the ReID feature of the pedestrian is obtained through the ReID feature extraction component, and the cosine similarity between the pedestrian's ReID feature and the historical trajectory ReID feature is calculated;

[0164] The calculation results of the above two aspects are used to determine whether the detected pedestrian matches the historical trajectory;

[0165] If yes, then update the trajectory information and Kalman parameters. If not, create a new trajectory for the pedestrian and save it in the historical trajectory library.

[0166] Make a judgment on the next frame.

[0167] The technical solution disclosed in the present invention has been applied to the 1.0 version of the real-time monitoring system for key populations that the company is about to release. Through pedestrian position analysis, the position information of pedestrians under different lenses can be accurately captured. Combined with the Kalman filter algorithm, the movement trajectory of pedestrians can be effectively predicted and corrected, thereby improving the accuracy of cross-lens trajectory tracking.

[0168] The third embodiment of the present disclosure also provides a cross-lens trajectory tracking system, such as Figure 3 As shown, the system comprises:

[0169] An acquisition module 11 is configured to acquire a surveillance video stream, detect pedestrians in the video stream, and acquire the location of the pedestrians;

[0170] A conversion module 12, which is configured to use a projection matrix to convert the pixel position of the pedestrian's contact point in the camera lens into the coordinate position of the pedestrian on the global map;

[0171] The matching module 13 is configured to match the pedestrian with all historical trajectories in the historical trajectory library according to the coordinate position of the pedestrian on the global map and the ReID feature of the pedestrian, so as to determine the trajectory of the pedestrian and realize real-time tracking and trajectory analysis of the pedestrian.

[0172] Furthermore, the acquisition module 11 is specifically configured as follows:

[0173] Detect the key points of the pedestrians in the camera and output the pixel positions x, y and corresponding confidence of the key points of the pedestrians’ postures.

[0174] If the confidence of a key point is less than the threshold, the key point is considered to be occluded;

[0175] If one or more of the left and right foot key points are blocked, the pedestrian posture key points are restored using the pre-trained pedestrian posture point restorer;

[0176] Calculate the center point of the pedestrian's left and right foot key points as the location representing the pedestrian.

[0177] Furthermore, the system also includes a training module 14;

[0178] The training module 14 is configured to train a pedestrian posture point restorer based on an encoder-decoder framework, for restoring posture key points of obscured pedestrians, including:

[0179] Prepare a dataset of pedestrian posture key points;

[0180] Randomly perform mask operations on the pose key points in the dataset;

[0181] Construct an encoder to reduce the dimension of the pedestrian's posture key points after the mask operation to obtain the pedestrian's action state;

[0182] Build a decoder to upgrade the pedestrian's motion state and obtain all the pedestrian's posture key points;

[0183] The masked pedestrian posture key points are used as the model input, the unmasked pedestrian posture points are used as the model target, and MSE is used as the model loss function to train the encoder-decoder.

[0184] Furthermore, the conversion module 12 is specifically configured as follows:

[0185] For each camera, calculate the corresponding lens-global map coordinate transformation matrix;

[0186] Using the lens-global map coordinate conversion matrix corresponding to the camera that captured the pedestrian, the following equation (1) is used to convert the pixel position of the pedestrian in the lens into the global map coordinate;

[0187]

[0188] Where (x, y) is the pixel position in the lens, (x', y') is its corresponding global map coordinate, and M is the lens-global map coordinate conversion matrix of the camera;

[0189] Among them, for each camera, the corresponding lens-global map coordinate transformation matrix is ​​calculated separately, including:

[0190] For a certain camera, obtain n locations with obvious identification in the lens, obtain the lens pixel positions of these locations, and simultaneously obtain the global map positions corresponding to these locations;

[0191] Substitute the lens pixel positions of these locations and the corresponding global map positions into equation (1) respectively, and use the least squares method to estimate the optimal transformation matrix as the lens-global map coordinate transformation matrix M of the camera;

[0192] Repeat this operation for all cameras to obtain the lens-global map coordinate transformation matrix corresponding to each camera.

[0193] Furthermore, the matching module 13 includes:

[0194] The extraction unit is configured to extract features from the detected pedestrian image, and uses a general ReID feature extraction network to transcode the pedestrian image into a 512-dimensional feature vector, and at the same time records the coordinate position of the pedestrian in the global map, which is recorded as

[0195] A matching unit is configured to match the pedestrian with all historical trajectories in the historical trajectory library according to the ReID distance and motion distance between the pedestrian and each historical trajectory, and determine whether the pedestrian belongs to a pedestrian trajectory that already exists in the trajectory library;

[0196] an updating unit, which is configured to update the pedestrian trajectory if the matching unit determines that the pedestrian trajectory is positive, including adding the ReID feature of the newly observed pedestrian to the pedestrian trajectory and updating the corresponding Kalman parameters; and

[0197] If the matching unit determines that the pedestrian is not the pedestrian, a new trajectory is created for the pedestrian and stored in a historical trajectory library.

[0198] Furthermore, the matching unit is specifically configured as follows:

[0199] Calculate the ReID distance between the pedestrian and each historical trajectory stored in the historical trajectory library, that is, 1 minus the cosine similarity between the 512-dimensional feature vectors of the two after transcoding, as the feature distance between the pedestrian and the trajectory;

[0200] Perform the following operations on each historical track: calculate the prior state of the historical track at the current moment based on the global map coordinates, the velocity vector, and the time length from the current moment, wherein the prior state includes the global map prior coordinates and the velocity vector of the track;

[0201] Use Kalman filtering to calculate the covariance matrix of the predicted state of the trajectory The coordinate error covariance matrix S of the trajectory k , according to the prior coordinates of the trajectory Coordinate error covariance matrix S k , and the observation coordinates Calculate the square of the Mahalanobis distance between the prior coordinates and the observed coordinates as the motion distance between the detected pedestrian and the historical trajectory;

[0202] The total distance between the observed pedestrian and the historical trajectory is calculated by the following formula (2):

[0203] Total distance = r*feature distance + (1-r)*movement distance (2)

[0204] r is a pre-set weight coefficient. If the total distance is less than the threshold, it is considered that the observed pedestrian and the historical trajectory in the historical trajectory library belong to the same person. Otherwise, it is considered that the newly observed pedestrian and the current trajectory do not belong to the same person.

[0205] Furthermore, the matching unit is further configured to:

[0206] For each historical trajectory, the prior state of the historical trajectory at the current moment is calculated in the following way:

[0207]

[0208] Among them, (x k-1 ,y k-1 ) represents the global map coordinates of the pedestrian trajectory when it was last updated, Δ(x k-1 , Δy k-1 ) represents the velocity vector of the pedestrian trajectory last updated, is the current prior estimation state of the trajectory, that is, the current position and speed estimated based on the coordinates and speed of the trajectory at the last update, F(t) is the dynamic transfer matrix, and the specific parameters are:

[0209]

[0210] Among them, t k Represents the time of the kth update.

[0211] The cross-shot trajectory tracking system of the disclosed embodiment is used to implement the cross-shot trajectory tracking method in method embodiment 1 and embodiment 2, so the description is relatively simple. For details, please refer to the relevant description in the previous method embodiment, which will not be repeated here.

[0212] In addition, if Figure 4 As shown, the fourth embodiment of the present disclosure further provides an electronic device, including a memory 100 and a processor 200, wherein the memory 100 stores a computer program, and when the processor 200 runs the computer program stored in the memory 100, the processor 200 executes the above-mentioned various possible methods.

[0213] The memory 100 is connected to the processor 200. The memory 100 may be a flash memory, a read-only memory or other memory, and the processor 200 may be a central processing unit or a single-chip microcomputer.

[0214] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to perform the above-mentioned various possible methods.

[0215] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), Digital Video Disc (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0216] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A cross-shot trajectory tracking method, characterized in that: The method comprises: Obtain surveillance video stream, detect pedestrians in the video stream, and obtain the location of pedestrians; Use the projection matrix to convert the pixel position of the pedestrian's contact point in the camera into the coordinate position of the pedestrian on the global map; According to the coordinate position of the pedestrian on the global map and the pedestrian re-identification (ReID) features, the pedestrian is matched with all historical trajectories in the historical trajectory library to determine the pedestrian's trajectory, thereby achieving real-time tracking and trajectory analysis of the pedestrian.

2. The method according to claim 1, characterized in that The method of obtaining a surveillance video stream, detecting pedestrians in the video stream, and obtaining the location of the pedestrians includes: Detect the key points of the pedestrians in the camera and output the pixel positions x, y and corresponding confidence of the key points of the pedestrians’ postures. If the confidence of a key point is less than the threshold, the key point is considered to be occluded; If one or more of the left and right foot key points are blocked, the pedestrian posture key points are restored using the pre-trained pedestrian posture point restorer; Calculate the center point of the pedestrian's left and right foot key points as the location representing the pedestrian.

3. The method according to claim 2, characterized in that The method further comprises: Train a pedestrian pose point restorer based on an encoder-decoder framework to restore the pose key points of occluded pedestrians, including: Prepare a dataset of pedestrian posture key points; Randomly perform mask operations on the pose key points in the dataset; Construct an encoder to reduce the dimension of the pedestrian's posture key points after the mask operation to obtain the pedestrian's action state; Build a decoder to upgrade the pedestrian's motion state and obtain all the pedestrian's posture key points; The masked pedestrian posture key points are used as the model input, the unmasked pedestrian posture points are used as the model target, and the mean square error loss MSE is used as the model loss function to train the encoder-decoder.

4. The method according to claim 1, characterized in that The method of using the projection matrix to convert the pixel position of the pedestrian's contact point in the lens into the coordinate position of the pedestrian on the global map includes: For each camera, calculate the corresponding lens-global map coordinate transformation matrix; Using the lens-global map coordinate conversion matrix corresponding to the camera that captured the pedestrian, the following equation (1) is used to convert the pixel position of the pedestrian in the lens into the global map coordinate; Where (x, y) is the pixel position in the lens, (x', y') is its corresponding global map coordinate, and M is the lens-global map coordinate conversion matrix of the camera; Among them, for each camera, the corresponding lens-global map coordinate transformation matrix is ​​calculated separately, including: For a certain camera, obtain n locations with obvious identification in the lens, obtain the lens pixel positions of these locations, and simultaneously obtain the global map positions corresponding to these locations; Substitute the lens pixel positions of these locations and the corresponding global map positions into equation (1) respectively, and use the least squares method to estimate the optimal transformation matrix as the lens-global map coordinate transformation matrix M of the camera; Repeat this operation for all cameras to obtain the lens-global map coordinate transformation matrix corresponding to each camera.

5. The method according to claim 1, characterized in that The method of matching the pedestrian with all historical trajectories in the historical trajectory library according to the coordinate position of the pedestrian on the global map and the ReID feature of the pedestrian to determine the trajectory of the pedestrian includes: The detected pedestrian image is subjected to feature extraction. A general ReID feature extraction network is used to transcode the pedestrian image into a 512-dimensional feature vector. At the same time, the coordinate position of the pedestrian in the global map is recorded, which is recorded as According to the ReID distance and motion distance between the pedestrian and each historical trajectory, the pedestrian is matched with all the historical trajectories in the historical trajectory library to determine whether the pedestrian belongs to the pedestrian trajectory that already exists in the trajectory library; If yes, the pedestrian trajectory is updated, including adding the ReID features of the newly observed pedestrian to the pedestrian trajectory and updating the corresponding Kalman parameters; If not, a new trajectory is created for the pedestrian and stored in the historical trajectory library.

6. The method according to claim 1 or 5, characterized in that: The method of matching the pedestrian with all historical trajectories in the historical trajectory library according to the ReID distance and movement distance between the pedestrian and each historical trajectory, and determining whether the pedestrian belongs to a pedestrian trajectory already existing in the trajectory library, includes: Calculate the ReID distance between the pedestrian and each historical trajectory stored in the historical trajectory library, that is, 1 minus the cosine similarity between the 512-dimensional feature vectors of the two after transcoding, as the feature distance between the pedestrian and the trajectory; Perform the following operations on each historical track: calculate the prior state of the historical track at the current moment based on the global map coordinates, the velocity vector, and the time length from the current moment, wherein the prior state includes the global map prior coordinates and the velocity vector of the track; Use Kalman filtering to calculate the covariance matrix of the predicted state of the trajectory The coordinate error covariance matrix S of the trajectory k , according to the prior coordinates of the trajectory Coordinate error covariance matrix S k , and the observation coordinates Calculate the square of the Mahalanobis distance between the prior coordinates and the observed coordinates as the motion distance between the detected pedestrian and the historical trajectory; The total distance between the observed pedestrian and the historical trajectory is calculated by the following formula (2): Total distance = r*feature distance + (1-r)*movement distance (2) r is a pre-set weight coefficient. If the total distance is less than the threshold, it is considered that the observed pedestrian and the historical trajectory in the historical trajectory library belong to the same person. Otherwise, it is considered that the newly observed pedestrian and the current trajectory do not belong to the same person.

7. The method according to claim 6, characterized in that The prior state of the historical trajectory at the current moment is calculated based on the global map coordinates, velocity vector, and time length from the current moment, including For each historical trajectory, the prior state of the historical trajectory at the current moment is calculated in the following way: Among them, (x k-1 , g k-1 ) represents the global map coordinates of the pedestrian trajectory when it was last updated, Δ(x k-1 , Δy k-1 ) represents the velocity vector of the pedestrian trajectory last updated, is the current prior estimation state of the trajectory, that is, the current position and speed estimated based on the coordinates and speed of the trajectory at the last update, F(t) is the dynamic transfer matrix, and the specific parameters are: Among them, t k Represents the time of the kth update.

8. A cross-shot trajectory tracking system, characterized in that: The system comprises: An acquisition module is configured to acquire a surveillance video stream, detect pedestrians in the video stream, and acquire the location of the pedestrians; A conversion module, which is configured to use a projection matrix to convert a pixel position of a pedestrian's contact point in the camera lens into a coordinate position of the pedestrian on a global map; The matching module is configured to match the pedestrian with all historical trajectories in the historical trajectory library according to the coordinate position of the pedestrian on the global map and the ReID features of the pedestrian, so as to determine the trajectory of the pedestrian and realize real-time tracking and trajectory analysis of the pedestrian.

9. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor runs the computer program stored in the memory, the processor executes the cross-shot trajectory tracking method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the cross-shot trajectory tracking method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Cross-domain multi-camera tracking method and device based on pedestrian posture

    CN112200841A

  • Cross-lens personnel tracking and re-identification method and system

    CN114972445A

  • Pedestrian trajectory prediction method and system based on multi-target tracking

    CN115760921A

  • Online pedestrian tracking method based on improved target detection network

    CN116645402A

  • Quick pedestrian positioning method based on multi-camera geometrical constraint

    CN105894505A

Cited By

  • Target tracking method and apparatus, and medium

    WO2026107770A1