Trajectory restoration method based on person re-identification
Through the trajectory reduction method based on pedestrian recognition, FasterRCNN, ResNet and SENet are used for feature extraction, combined with K-means clustering and optimized path generation algorithm, the problem of low efficiency in searching specific target trajectories under multi-camera network is solved, and efficient and accurate trajectory analysis is achieved.
Patent Information
- Application Number
- CN202210844696.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-18
- Publication Date
- 2025-08-19
- Estimated Expiration
- 2042-07-18
Smart Images
Figure CN115331138B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the service field, and in particular to a trajectory restoration method based on pedestrian re-identification. Background Art
[0002] With the development of the times, public safety faces increasing challenges and challenges. Surveillance cameras, as a technological guarantee in public environments, rely heavily on identifying specific targets within the vast video feeds captured by numerous cameras and analyzing their movements. The development of this technology can safeguard the safety of public property and public life, while also providing effective technical support to relevant departments. For example, in the process of tracing the movements of infected individuals, manual phone screening and inquiries to surveillance cameras are currently the primary means of obtaining these movements, effectively laying the foundation for subsequent epidemic prevention and control efforts. Furthermore, in the process of tracking criminal suspects, the coordinated operation of different cameras in different locations can assist in tracking the suspect's movements and analyzing their next steps. Currently, trajectory analysis of specific targets relies primarily on manual camera tracking and tagging. Relying solely on manual effort to obtain the trajectory of a specific target across a multi-camera network is laborious and inefficient, potentially leading to missed or false positives. Summary of the Invention
[0003] The technical problem to be solved by the embodiments of the present invention is to provide a trajectory restoration method based on pedestrian re-identification to improve the efficiency of searching for specific pedestrians under different cameras and obtain the movement trajectory of specific pedestrians through the results of multiple cameras to provide assistance for subsequent tasks.
[0004] In order to solve the above technical problems, an embodiment of the present invention proposes a trajectory restoration method based on pedestrian re-identification, comprising the following steps:
[0005] S100: Acquire a target pedestrian image and store it in a target pedestrian database;
[0006] S200: Establishing a video data acquisition environment to collect video data from multiple cameras;
[0007] S300: Reading the video data collected by the camera, dividing the video data into frames, extracting images containing pedestrians, and storing them in a pedestrian image database;
[0008] S400: Comparing the image containing the pedestrian detected with the target pedestrian image in the target pedestrian database;
[0009] S500: Return the matching result. If the current camera finds the target pedestrian, save the camera's latitude and longitude information on the map and the target's appearance time.
[0010] S600: Generate a possible path to the target using the optimized path generation algorithm and the stored latitude and longitude information and time information.
[0011] Furthermore, step S300 includes:
[0012] Read the frame format of the video data collected from multiple cameras and determine the frame number;
[0013] By performing pedestrian detection on consecutive frames of images, features are extracted for analysis and it is determined whether the image contains pedestrians;
[0014] For images determined to contain pedestrians, the images containing pedestrians are stored in the pedestrian image database for comparison with the target pedestrians, and images determined not to contain pedestrians are not processed.
[0015] Furthermore, in step S400 and step S500,
[0016] First, extract the features of the saved image containing pedestrians and the features of the target pedestrian;
[0017] Then calculate the Euclidean distance between the pedestrian features of different pedestrians in the image containing pedestrians and the pedestrian features in the target pedestrian database. If the calculated result is greater than the preset threshold, it is considered that the current pedestrian does not match. If the calculated result is less than the set threshold, it is considered that the current image contains the target pedestrian, and the location information and time information of the camera corresponding to the image are saved.
[0018] Furthermore, in step S400, FasterRCNN is combined with ResNet to extract features from each frame of the image, and the ResASPP structure based on dilated convolution is used to obtain multi-receptive field features. The feature map after cov1 is convolved by dilated convolution with different expansion rates, and different weights are assigned by SENet to obtain a feature map of multiple receptive fields. Bilinear interpolation is performed on the features finally obtained by FasterRCNN and fused with the previous features. A total of three layers of feature layers at different levels are added as the final feature output. Finally, by comparing the Euclidean distance between the features of different pedestrians, it is determined whether they are the same pedestrian, and the middle-level features are passed through DANet to model rich contextual dependencies on local features. The network uses the following formula to calculate the training error:
[0019]
[0020] where v i is the pedestrian feature vector marked in the input image, y i is the true label of the target pedestrian, x is the pedestrian feature vector detected by the network, u k is the unlabeled pedestrian feature vector in the input image, Indicates the similarity between the pedestrians detected by the network and the saved marked pedestrians, It represents the similarity between the pedestrians detected by the network and the unlabeled pedestrians that have been saved, and τ is the softened probability distribution parameter.
[0021] Furthermore, in step S400, considering the detection of pedestrians, a K-means clustering algorithm is used to obtain a suitable anchor frame size and shape based on the characteristics of pedestrians. The existing network model is fine-tuned using the personnel data to accurately regress the anchor frame to the pedestrian to be detected. The following steps are used to obtain the anchor frame:
[0022] (1) Traverse the real box of each pedestrian in the given image in the dataset and save the width, height and area of the box;
[0023] (2) Set the number of cluster centers and use the K-means algorithm for clustering;
[0024] (3) The aspect ratio of the cluster center is used as the ratio of the anchor box, and the area is used as the scale of the anchor box.
[0025] Furthermore, in step S600, the location information is first mapped on a map, different cameras are regarded as vertices in a graph data structure, and different priorities are assigned to the vertices according to time information;
[0026] Then, possible paths are obtained by using a path generation algorithm in the constructed graph.
[0027] Furthermore, in step S600, the map path and the cameras distributed on the map are modeled as an undirected graph structure, wherein each path on the map is an edge in the graph structure, each camera is a node in the graph structure, and each node contains information such as whether the target pedestrian appears at the node and the time priority of the target pedestrian appearing. The path is generated using the following steps:
[0028] (1) First, find the starting node by comparing the time priority information of all nodes;
[0029] (2) Compare the nodes adjacent to the starting node to see whether the target pedestrian appears and their appearance priority, compare the priorities, and select the node with higher priority (i.e., earlier appearance time) as the next node;
[0030] (3) In the loop (2) phase, if the target pedestrian does not appear in any of the adjacent nodes of the current node, the loop ends and the saved path node is compared with the node where the target pedestrian appears. If the two are different, an error message is output; if the two are the same, the saved path is output.
[0031] The beneficial effects of the present invention are as follows: compared with the existing personnel trajectory restoration process, the present invention replaces the traditional restoration process with computer equipment instead of manual search and path planning, which can speed up the efficiency of finding pedestrians and analyzing target trajectories. The present invention has high promotion and use value. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 4 is a flowchart of a trajectory restoration method based on pedestrian re-identification according to an embodiment of the present invention.
[0033] Figure 2 This is a schematic diagram of a pedestrian re-identification principle for preparing for path restoration according to an embodiment of the present invention.
[0034] Figure 3 It is a pseudo code flow chart of step S600 in an embodiment of the present invention. DETAILED DESCRIPTION
[0035] It should be noted that, unless there is a conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The present invention is further described in detail below with reference to the drawings and specific embodiments.
[0036] Please refer to Figure 1 , a trajectory restoration method based on pedestrian re-identification according to an embodiment of the present invention, the method comprising:
[0037] S100: Acquire a target pedestrian image and store it in a target pedestrian database;
[0038] S200: Establishing a video data acquisition environment to collect video data from multiple cameras;
[0039] S300: Reading the video data collected by the camera, dividing the video data into frames to extract pedestrian images, and storing them in a pedestrian image database;
[0040] S400: comparing the detected pedestrian image with a target image in a target pedestrian database;
[0041] S500: Return the matching result. If the current camera finds the target pedestrian, save the latitude and longitude information of the camera on the map and the time when the target appeared.
[0042] S600: Generate a possible path to the target using a preset optimized path generation algorithm and the stored latitude, longitude, and time information. Traditional path generation algorithms primarily find all paths between points A and B and cannot be integrated with re-identification results. Therefore, this embodiment of the present invention adds pedestrian re-identification results as additional parameters to the algorithm and incorporates them into the algorithm's logic, making the algorithm more suitable for the proposed task.
[0043] The present invention provides a trajectory restoration method based on pedestrian re-identification, which adopts an algorithm that integrates detection and re-identification to perform pedestrian detection and feature extraction on multiple pedestrian images, and match them with the target object features in turn. It then finds the camera where the target appears among multiple cameras, and uses the camera position information and time information of the target to map the target's movement trajectory within a certain range.
[0044] The following is a detailed description of a trajectory restoration method based on pedestrian re-identification provided by the present invention.
[0045] See also Figure 1 , which is a flow chart of a trajectory restoration method based on pedestrian re-identification in one embodiment of the present invention.
[0046] In step S100, an image of a target pedestrian to be searched is obtained and stored in a target pedestrian database.
[0047] In this embodiment, the target image is obtained by searching the camera where the target has been determined to appear. The acquisition method can use a detection algorithm or manual screenshot. In order to improve the subsequent matching effect, the target image acquisition principle is to be unobstructed and not blurred.
[0048] In step S200, a video data acquisition environment is established to acquire video data from multiple cameras.
[0049] In this embodiment, the position information of different cameras should be bound to the camera video content for subsequent trajectory restoration task processing.
[0050] In step S300, the video data collected by the camera is read, and the video data is frame-divided to extract pedestrian images, which are stored in a pedestrian image database.
[0051] In step S400, the collected pedestrian image data and the target pedestrian image data are read and matched.
[0052] In this embodiment, the specific implementation algorithm of step S400 is as follows: Figure 2 shown.
[0053] See also Figure 2 , is an example of the present invention Figure 1 Detailed structural diagram of comparing the image of the detected pedestrian with the target image in the target pedestrian database in step S400.
[0054] In this embodiment, the computer reads the same camera image in the pedestrian image database.
[0055] exist Figure 2In the CNN, each pedestrian image in the database is subjected to feature extraction for each pedestrian in the image through a convolutional neural network.
[0056] In this embodiment, FasterRCNN is combined with ResNet to extract features from each frame of the image. ResNet relies on residual connections to solve the gradient vanishing problem caused by the network being too deep, making it possible to build a deeper network. However, the deeper the network, the more information will be lost during the convolution process. Therefore, the ResASPP structure based on dilated convolution is adopted to obtain multi-receptive field features. The feature map after cov1 is convolved by dilated convolution with different expansion rates, and different weights are assigned by SENet to obtain feature maps with multiple receptive fields. At the same time, from the bottom to the top, the neural network feature map often corresponds to more abstract and higher semantic level visual concepts. Therefore, in the pedestrian re-identification task, the top-level output vector is generally used as the expression feature. However, the visual features at the intermediate semantic level can also effectively express the differences between different pedestrians. Therefore, the features obtained by FasterRCNN are bilinearly interpolated and fused with the previous features. A total of three layers of features are added together as the final feature output. Finally, the Euclidean distance between different pedestrian features is compared to determine whether they are the same pedestrian. The middle-level features are passed through DANet to model rich contextual dependencies on local features. The network uses the following formula to calculate the training error:
[0057]
[0058] where v i y is the pedestrian feature vector marked in the input image. i is the true label of the target pedestrian, and x is the pedestrian feature vector detected by the network. k is the unlabeled pedestrian feature vector in the input image, Indicates the similarity between the pedestrians detected by the network and the saved marked pedestrians, It represents the similarity between the pedestrians detected by the network and the unlabeled pedestrians that have been saved, and τ is the softened probability distribution parameter.
[0059] In this implementation, considering pedestrian detection, the K-means clustering algorithm is used to obtain the appropriate anchor box size and shape based on pedestrian characteristics. Fine-tuning is performed using human data within the existing network model to more accurately regress the anchor box to the pedestrian to be detected. The main process of obtaining the anchor box using K-means is as follows:
[0060] (1) Traverse the real frame of each pedestrian in the given image in the dataset and save the width, height and area of the frame.
[0061] (2) Set the number of cluster centers and use the K-means algorithm for clustering
[0062] (3) The aspect ratio of the cluster center is used as the ratio of the anchor box, and the area is used as the scale of the anchor box.
[0063] Please continue reading Figure 1 In step S500, the matching result is returned. If the current camera finds the target pedestrian, the target appearance time on the camera video is saved.
[0064] In this example, the latitude and longitude information of different cameras on the map obtained in step S200 is obtained. In addition, the result of whether the target pedestrian appears in each camera obtained in step S400 is also included.
[0065] In step S600, a possible path to the target is generated using the optimized path generation algorithm and the stored latitude and longitude information and time information.
[0066] In this embodiment, the map paths and cameras distributed on the map are modeled as an undirected graph structure, where each path on the map is an edge in the graph structure and each camera is a node in the graph structure. Each node contains information such as whether the target pedestrian appears at the node and the time priority of the target pedestrian. Figure 3 The proposed path generation algorithm obtains the path.
[0067] See also Figure 3 , which is a pseudo code flow chart of step S600 in an embodiment of the present invention.
[0068] exist Figure 3 In , the path generation algorithm process is as follows:
[0069] (1) First, find the starting node by comparing the time priority information of all nodes.
[0070] (2) Compare the nodes adjacent to the starting node to see whether the target pedestrian appears and their appearance priority, compare the priorities, and select the node with higher priority (i.e., earlier appearance time) as the next node.
[0071] (3) In the loop (2) phase, if the target pedestrian does not appear in any of the adjacent nodes of the current node, the loop ends and the saved path node is compared with the node where the target pedestrian appears. If the two are different, an error message is output; if the two are the same, the saved path is output.
[0072] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.
Claims
1. A trajectory restoration method based on pedestrian re-identification, characterized in that: The following steps are involved: S100: Acquire a target pedestrian image and store it in a target pedestrian database; S200: Establishing a video data acquisition environment to collect video data from multiple cameras; S3 00: Read the collected video data from the camera, extract the images containing pedestrians from the video data by frames, and store them in the pedestrian image database; S400: Comparing the image containing the pedestrian detected with the target pedestrian image in the target pedestrian database; In step S400, FasterRCNN is combined with ResNet to extract features from each frame of the image, and the ResASPP structure based on void convolution is used to obtain multi-receptive field features. The feature map after cov1 is convolved by void convolution with different expansion rates, and different weights are assigned by SENet to obtain a feature map of multiple receptive fields; the features finally obtained by FasterRCNN are bilinearly interpolated and fused with the previous features. A total of three layers of feature layers at different levels are added as the final feature output. Finally, the Euclidean distance between different pedestrian features is compared to determine whether they are the same pedestrian, and the middle-level features are passed through DANet to model rich contextual dependencies on local features; the network uses the following formula to calculate the training error: ； in is the pedestrian feature vector marked in the input image, is the true label of the target pedestrian, is the pedestrian feature vector detected by the network, is the unlabeled pedestrian feature vector in the input image, Indicates the similarity between the pedestrians detected by the network and the saved marked pedestrians, Indicates the similarity between the pedestrians detected by the network and the unlabeled pedestrians that have been saved, is the softening probability distribution parameter; S500: Return the matching result. If the current camera finds the target pedestrian, save the camera's latitude and longitude information on the map and the target's appearance time. S600: Generate a possible path to the target using the optimized path generation algorithm and the stored latitude and longitude information and time information; In step S600, the location information is first mapped on a map, and different cameras are regarded as vertices in a graph data structure, and different priorities are assigned to the vertices according to time information; Then, possible paths are obtained by using a path generation algorithm in the constructed graph; In step S600, the map path and the cameras distributed on the map are modeled as an undirected graph structure, where each path on the map is an edge in the graph structure, each camera is a node in the graph structure, and each node contains information such as whether the target pedestrian appears at the node and the time priority of the target pedestrian appearing. The path is generated using the following steps: (1) First, find the starting node by comparing the time priority information of all nodes; (2) Compare the nodes adjacent to the starting node to see whether the target pedestrian appears and their appearance priority, compare the priorities, and select the node with higher priority, i.e., the node with earlier appearance time, as the next node; (3) In the loop (2) phase, if the target pedestrian does not appear in any of the adjacent nodes of the current node, the loop ends and the saved path node is compared with the node where the target pedestrian appears. If the two are different, an error message is output; if the two are the same, the saved path is output.
2. The trajectory restoration method based on pedestrian re-identification according to claim 1, characterized in that Step S300 includes: Read the frame format of the video data collected from multiple cameras and determine the frame number; By performing pedestrian detection on consecutive frames of images, features are extracted for analysis and it is determined whether the image contains pedestrians; For images determined to contain pedestrians, the images containing pedestrians are stored in the pedestrian image database for comparison with the target pedestrians, and images determined not to contain pedestrians are not processed.
3. The trajectory restoration method based on pedestrian re-identification according to claim 1, characterized in that In step S400 and step S500, First, extract the features of the saved image containing pedestrians and the features of the target pedestrian; Then calculate the Euclidean distance between the pedestrian features of different pedestrians in the image containing pedestrians and the pedestrian features in the target pedestrian database. If the calculated result is greater than the preset threshold, it is considered that the current pedestrian does not match. If the calculated result is less than the set threshold, it is considered that the current image contains the target pedestrian, and the location information and time information of the camera corresponding to the image are saved.
4. The trajectory restoration method based on pedestrian re-identification according to claim 1, characterized in that In step S400, considering the detection of pedestrians, the K-means clustering algorithm is used to obtain the appropriate anchor box size and shape based on the characteristics of pedestrians. The existing network model is fine-tuned using the personnel data to accurately regress the anchor box to the pedestrian to be detected. The following steps are used to obtain the anchor box: (1) Traverse the real box of each pedestrian in the given image in the dataset and save the width, height and area of the box; (2) Set the number of cluster centers and use the K-means algorithm for clustering; (3) The aspect ratio of the cluster center is used as the ratio of the anchor box, and the area is used as the scale of the anchor box.