Cross-camera video pedestrian search method, terminal device and computer-readable storage medium based on representation learning

Abstract

The present invention discloses a method for searching pedestrians across camera videos based on representation learning, a terminal device, and a computer-readable storage medium, comprising the following steps: obtaining pedestrian data across camera videos, constructing a query set and a candidate video library corresponding to the query set; inputting the video into a target detection network to learn each pedestrian bounding box, the confidence score of the bounding box, and the features of each pedestrian; inputting the pedestrian bounding box and features into a multi-target tracking network based on temporal feature fusion, performing data association on the video frames, obtaining the trajectory of each pedestrian and the temporal features of each trajectory; calculating the similarity between the pedestrian trajectory feature vector in the query set and all the pedestrian trajectory feature vectors of the pedestrian in the candidate video library, and performing accuracy calculation. Aiming at real-world video surveillance scenarios, the present invention realizes the search for target pedestrians in cross-camera scenarios by extracting pedestrian features and data association using long-term temporal features in the video.

Description

Technical Field

[0001] The present invention relates to the fields of target detection, pedestrian re-identification and target tracking, and specifically to a cross-camera video pedestrian search method based on representation learning, a terminal device and a computer-readable storage medium. Background Art

[0002] Pedestrian search is a combination of pedestrian detection and pedestrian re-identification. That is, given a target pedestrian, it accurately detects and identifies the target pedestrian in the uncropped surveillance images across cameras, and can simultaneously answer the questions of "who is he and where is he", greatly enhancing and improving the functionality and efficiency of intelligent visual surveillance.

[0003] Although pedestrian search has achieved good results on the two mainstream datasets CUHK-SYSU and PRW, the contents of these two datasets are images, which still have certain deviations from the actual video surveillance scenes and are difficult to implement in actual surveillance video systems. Therefore, video pedestrian search is also proposed based on multi-camera multi-target tracking. Multi-camera multi-target tracking aims to determine the location of each person at any time from the video streams shot by multiple cameras, and video pedestrian search adds a query module on this basis to find the trajectory of the pedestrian to be queried in the video data across cameras in the entire scene. Therefore, video pedestrian search can also be considered as a combination of multi-target tracking and video pedestrian re-identification.

[0004] In terms of datasets, DukeMTMC is a key dataset for multi-camera multi-target tracking. This dataset, developed by Duke University in 2014 for campus surveillance videos, has been used in numerous video research fields. However, DukeMTMC was disabled in 2019 due to privacy concerns, impacting research on multi-camera video tasks, including video person search. Datasets for other video tasks, such as multi-target tracking, video person re-identification, and video understanding, do not meet the requirements of video person search. Therefore, video person search also faces the challenge of dataset shortage. Regarding features, compared to images, videos have an additional temporal dimension, providing richer spatiotemporal information, which is currently lacking in image-based person search. This feature avoids the model's reliance on a single frame and mitigates the recognition difficulties caused by occlusion in a single image. Improving the accuracy of video person re-identification requires utilizing and integrating these multi-frame features. Therefore, leveraging this temporal information is a core issue in video person search.

[0005] Therefore, based on the above technical problems, it is necessary to design a new cross-camera video pedestrian search method. Summary of the Invention

[0006] Purpose of the invention: In response to the above-mentioned existing technologies, a cross-camera video pedestrian search method, terminal device and computer-readable storage medium based on representation learning are proposed to realize cross-camera video pedestrian search.

[0007] Technical solution:

[0008] A pedestrian search method for cross-camera videos based on representation learning, including:

[0009] Obtain pedestrian data from cross-camera videos and divide them into training and test sets; construct a query set and a candidate video library corresponding to the query set;

[0010] Input the pedestrian data across camera videos into the object detection network to learn the bounding box of each pedestrian, the confidence score of the bounding box, and the features of each pedestrian;

[0011] Obtain the trajectory of each pedestrian and the temporal features of each trajectory;

[0012] The similarity between the pedestrian trajectory feature vector in the query set and all the pedestrian trajectory feature vectors of the pedestrian in the candidate video library is calculated, and the accuracy is calculated.

[0013] Preferably, obtaining pedestrian data from cross-camera videos and dividing them into a training set and a test set; and constructing a query set and a candidate video library corresponding to the query set specifically include:

[0014] A cross-camera video pedestrian search dataset MTA-PS containing cross-camera video pedestrian data is obtained, and image enhancement processing is performed on the cross-camera video pedestrian data. Then, the cross-camera video pedestrian data after image enhancement processing is divided into a training set and a test set; a query set is constructed, and a frame of image is randomly selected for each pedestrian that appears in at least two cameras and added to the query set, and the video in which the pedestrian appears in other cameras is added to the candidate video library of the pedestrian, and the interference video that does not contain the pedestrian is selected and added to the candidate video library of the pedestrian, ensuring that the candidate video library of each pedestrian in the query set contains at least five videos.

[0015] Preferably, the pedestrian data across the camera videos is fed into the object detection network to learn the bounding box of each pedestrian, the confidence score of the bounding box, and the features of each pedestrian, including:

[0016] The training set is subjected to feature extraction using a Darknet53-based backbone network, a feature pyramid is constructed for multi-scale feature fusion, and a tuple consisting of three feature layers is output;

[0017] The four decoupling heads of regression, intersection-over-union, classification and re-identification are used to regress, predict, classify and extract re-identification features for the tuple composed of three feature layers. The output of the regression decoupling head is a 1-dimensional vector representing the confidence score of the current bounding box, the output of the classification decoupling head is a 1-dimensional vector representing the pedestrian category, the output of the intersection-over-union decoupling head is a 4-dimensional vector representing the pedestrian bounding box, and the output of the re-identification decoupling head is a 128-dimensional vector representing the pedestrian's features. The SiamOTA strategy is used to screen positive and negative samples, and then the positive and negative samples are used to calculate the identity loss, triplet loss and center loss to generate the pedestrian bounding box, the confidence score of the bounding box, the pedestrian ID and the pedestrian's features.

[0018] Identity loss L id The calculation formula is as follows:

[0019]

[0020] Among them, N is the number of pedestrians in the current frame image, M is the number of all pedestrian IDs in the training set, i is the i-th pedestrian in the current frame image, j is the real value ID corresponding to the i-th pedestrian, I i (j) is an indicator function, which is 1 when calculating positive samples and 0 when calculating negative samples. p(j) is the probability that the ID of the i-th pedestrian is the true value ID;

[0021] Triple L tri The (i, j, k) loss calculation formula is as follows:

[0022] L tri (i, j, k) = max(a+d ij -d ik ,0),

[0023] Where a is the boundary constant, d ij is the Euclidean distance between the re-identification features of the pedestrian bounding boxes with the same ID, d ik is the Euclidean distance between the re-identification features corresponding to the bounding boxes of pedestrians with different IDs, (i, j) is the re-identification feature corresponding to the bounding boxes of pedestrians with the same ID, and (i, k) is the re-identification feature corresponding to the bounding boxes of pedestrians with different IDs;

[0024] Center loss L c The calculation formula is as follows:

[0025]

[0026] where x t is the pedestrian re-identification feature vector of the t-th image, y q is the label of the qth frame in a batch, Indicates the yth q The feature center of each pedestrian ID;

[0027] The above three losses are added together as the loss function to guide the training of the target detection network.

[0028] Preferably, obtaining the trajectory of each pedestrian and the temporal features of each trajectory includes:

[0029] Input the test set into the trained object detection network to obtain the pedestrian bounding box, confidence score and features of each frame image. The bounding box and corresponding features are obtained according to the score threshold. The bounding box with a score above the threshold is a high-scoring bounding box, and the bounding box with a score below the threshold is a low-scoring bounding box.

[0030] Kalman filtering is used to predict the new position of each trajectory in the current frame image;

[0031] The trajectory is first associated with the high-score bounding box, and the first association fuses the feature distance and the intersection-over-union distance:

[0032] The feature distance is as follows: First, the similarity between the pedestrian re-identification features of the bounding box in the current frame image and the pedestrian re-identification features in the trajectory is calculated using the Mahalanobis distance, and the Mahalanobis distance is combined with the cosine distance calculated on the pedestrian re-identification features to obtain the feature distance matrix. The specific calculation formula is as follows:

[0033] D f =λD r +(1-λ)D m ,

[0034] Where λ is the weight parameter, D r The Mahalanobis distance matrix calculated for the pedestrian re-identification features of the bounding box and the pedestrian re-identification features in the trajectory, D m The cosine distance matrix D is calculated for the pedestrian re-identification features of the bounding box and the pedestrian re-identification features in the trajectory. f It is the characteristic distance matrix obtained by weighting the Mahalanobis distance matrix and the cosine distance matrix;

[0035] The intersection-over-union (IoU) distance is calculated by using the IoU distance to calculate the distance matrix between the bounding box and the bounding box predicted by the Kalman filter. The specific calculation formula is as follows:

[0036]

[0037] Where A and B represent the bounding box of the current frame image and the bounding box predicted by the Kalman filter respectively;

[0038] After obtaining the feature distance matrix, the Mahalanobis distance is used to fuse the feature distance with the motion information to obtain the cost matrix of the Kalman filter motion information. The specific calculation formula is as follows:

[0039]

[0040] Where m is the target pedestrian, o is the pedestrian bounding box predicted by the Kalman filter, and μ m is the mean of the predicted state of the target pedestrian, ∑m is the covariance matrix, and δ is the weight parameter;

[0041] After obtaining the feature distance matrix and the intersection-of-union ratio distance matrix, the feature distance matrix and the intersection-of-union ratio distance matrix are fused through threshold screening and gate function. That is, the minimum cost of each position in the feature distance matrix and the intersection-of-union ratio distance matrix is ​​selected and placed in the corresponding position in the cost matrix to obtain the fused cost matrix. Then, the intersection-of-union ratio distance matrix is ​​fused with the bounding box confidence score to obtain the feature distance of the fused bounding box confidence score information. The specific calculation formula is as follows:

[0042] F cost (t, d) = 1-(1-D iou )×S(m, d),

[0043] Where S represents the detection score array, and d represents the detection result;

[0044] The Hungarian algorithm is then used to assign the current bounding box to the existing trajectory, obtaining the first associated matching result, unmatched bounding boxes, and unmatched trajectories. During the matching process, whenever a trajectory successfully matches the bounding box of the current frame image, the pedestrian features and long-term temporal features of the bounding box are added to the trajectory information; the calculation formula for the long-term temporal features is as follows:

[0045]

[0046] Where β is the time series feature weight, is the long-term temporal feature of the previous frame image, is the feature of the target pedestrian in the current frame image;

[0047] The unmatched tracks are associated with the unmatched bounding boxes for a second time. The second association uses only feature distance. The Mahalanobis distance is used to calculate the similarity between the unmatched track features and the pedestrian re-identification features of the unmatched bounding boxes to obtain a cost matrix. The unmatched bounding boxes are assigned to the existing unmatched tracks using the Hungarian algorithm. The bounding boxes that are still unmatched are deleted, and the tracks that are still unmatched for the second time are retained. Tracks that are still unmatched for the second time are classified as lost and deleted if they do not appear again for a certain number of frames.

[0048] Generate new trajectories for the remaining high-scoring bounding boxes. When generating new trajectories, the pedestrian features of the current frame image are added to facilitate the calculation of subsequent temporal features.

[0049] Preferably, performing similarity calculation on the trajectory feature vector of the pedestrian in the query set and the trajectory feature vectors of all pedestrians in the candidate video library, and performing accuracy calculation includes:

[0050] Input the images in the query set into the trained object detection network one by one to obtain the pedestrian feature vector of the query set;

[0051] The features corresponding to each trajectory are sorted by calculating the Euclidean distance, and the features with closer distances are averaged to obtain the pedestrian feature vector in the candidate video library. The pedestrian feature vector of the query set is then similarly calculated and sorted with all the pedestrian feature vectors of the pedestrian in the candidate video library to obtain the results of the cross-camera video pedestrian search.

[0052] A terminal device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of a cross-camera video pedestrian search method based on representation learning as described in any one of claims 1 to 5.

[0053] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor executes the steps of a method for searching pedestrians in cross-camera videos based on representation learning as described in any one of claims 1 to 5.

[0054] Beneficial effects: The present invention proposes a network that combines pedestrian detection, target tracking, and pedestrian re-identification for video data in real surveillance scenarios. It performs temporal correlation on the target pedestrian data and extracts long-term temporal features, which is conducive to cross-camera pedestrian search in real surveillance scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flow chart of a pedestrian search method across camera videos based on representation learning of the present invention;

[0056] Figure 2 This is a schematic diagram of the network structure of the video pedestrian search of the present invention;

[0057] Figure 3 is a flow chart of the pedestrian feature fusion and trajectory tracking method of the present invention;

[0058] Figure 4 This is a schematic diagram of the network structure of the backbone network Darknet53 in the present invention;

[0059] Figure 5 This is a schematic diagram of the results of the cross-camera video pedestrian search method based on representation learning of the present invention. DETAILED DESCRIPTION

[0060] The present invention will be further explained below with reference to the accompanying drawings.

[0061] like Figure 1 As shown in FIG, the pedestrian search method across camera videos based on representation learning includes the following steps:

[0062] S1: Obtain pedestrian data from cross-camera videos and divide them into training and test sets; construct a query set and a candidate video library corresponding to the query set, including the following specific steps:

[0063] S101: Obtain a cross-camera video pedestrian search dataset MTA-PS containing cross-camera video pedestrian data, perform image enhancement processing on the cross-camera video pedestrian data, and then divide the cross-camera video pedestrian data after image enhancement processing into a training set and a test set;

[0064] S102: Divide the query set and the corresponding candidate video library. For each pedestrian that appears in at least two cameras, randomly select one frame and add it to the query set. Also, add videos of the pedestrian that appear in other cameras to the query set for the pedestrian. Finally, select interference videos that do not contain the pedestrian and add them to the query set. Ensure that the query set corresponding to each pedestrian contains at least five videos.

[0065] S2: Input the training sample video data into the video pedestrian search deep learning network to learn the target bounding box, confidence score and features of the pedestrian in the image at each moment. The network structure is as follows: Figure 2 As shown, the specific steps are as follows:

[0066] S201: Use the backbone network based on Darknet53 for feature extraction. The Darknet53 flowchart is as follows Figure 4 In the Darknet53 network, the input image first passes through a convolutional layer to transform the number of input channels into the first number of output channels in the network. Then, the output tensor of this convolutional layer is input into the first "residual block" of Darknet53. In this residual block, the input tensor is sent to a series of convolutional layers and batch normalization layers. After passing through a nonlinear activation function, it is added to the input tensor through a jump connection, and finally the output tensor is sent to the next layer. After stacking multiple residual blocks, the number of channels of the output tensor increases to 1024, and the size becomes 13*13.

[0067] The original image is then scaled using a feature pyramid to obtain images of different sizes. Feature maps of different sizes are then generated for each image size. The high-level feature information is first transferred and fused through upsampling. The high-level feature map resolution is amplified by a factor of 2, maintaining consistency with the resolution of the low-level feature maps. The upsampling method uses a neighboring interpolation algorithm. The predicted feature map is then obtained through downsampling and fusion. The low-level feature map undergoes a 1x1 convolution operation to maintain the same number of channels as the high-level, upsampled feature map. Finally, the layers are fused using an addition method. The final output is a tuple consisting of three feature layers.

[0068] S202: In order to avoid interference and conflict when calculating the bounding box and extracting features, this network extracts features from the three layers of feature maps in the feature pyramid through four decoupling heads: regression, intersection-over-union, classification, and re-identification. Each decoupling head uses a 1*1 convolution and two 3*3 convolutions to obtain different results. The regression head outputs a 1-dimensional vector, representing the confidence score of the current bounding box, and the classification head outputs a 1-dimensional vector, representing the target category. There is only the pedestrian category here, so the default is 1. The intersection-over-union head outputs a 4-dimensional vector, representing the bounding box of the target, and the re-identification head outputs a 128-dimensional vector, representing the characteristics of the target pedestrian, that is, each pedestrian obtains a total of 134-dimensional feature vectors. After connecting all pedestrian feature vectors on different feature maps, the positive and negative samples are screened. First, a preliminary screening is performed, and the samples whose target center points fall in the true value box are selected as positive samples, and the rest are negative samples. The SiamOTA strategy is then used to assign labels. The cost calculation method of the true value gi and the predicted value pj is as follows:

[0069]

[0070] where λ is the balance coefficient, and is the category association loss and regression loss of the true value gi and the predicted value pj. For the true value, the first k samples with the lowest cost in the central area are selected as positive samples, and the rest are regarded as negative samples.

[0071] In order to better distinguish the characteristics of different pedestrians and increase the robustness of the network, positive and negative samples are used to calculate identity loss, triple loss, and center loss to generate pedestrian bounding boxes, confidence scores, categories, and features. The identity loss calculation formula is as follows:

[0072]

[0073] Among them, N is the number of pedestrians in the current frame image, M is the number of all pedestrian IDs in the training set, i is the i-th pedestrian in the current frame image, j is the real value ID corresponding to the i-th pedestrian, I i(j) is an indicator function, which is 1 when calculating positive samples and 0 when calculating negative samples. p(j) is the probability that the ID of the i-th pedestrian is the true value ID;

[0074] The triplet loss is calculated as follows:

[0075] L tri (i, j, k) = max(a+d ij -d ik ,0),

[0076] Where a is the boundary constant, d ij is the Euclidean distance between the re-identification features of the pedestrian bounding boxes with the same ID, d ik is the Euclidean distance between the re-identification features corresponding to the bounding boxes of pedestrians with different IDs, (i, j) is the re-identification feature corresponding to the bounding boxes of pedestrians with the same ID, and (i, k) is the re-identification feature corresponding to the bounding boxes of pedestrians with different IDs.

[0077] The formula for calculating the center loss is as follows:

[0078]

[0079] where x t is the pedestrian re-identification feature vector of the t-th sample, y q is the label of the qth frame in a batch, Indicates the yth q The feature center of each pedestrian ID category;

[0080] The above three losses are added together as the loss function to guide the training of the target detection network.

[0081] S3: If Figure 3 As shown in the figure, the pedestrian bounding box and features are input into the multi-target tracking network based on temporal feature fusion, and data association is performed on the video frames to obtain the trajectory of each pedestrian and the features of the corresponding trajectory, including the following steps:

[0082] S301: Input the video frame by frame into the pre-trained object detection network to obtain the pedestrian bounding box, confidence score and features of each frame, and obtain the bounding box and corresponding features based on the prediction score threshold;

[0083] S302: Use Kalman filtering to predict the new position of each trajectory in the current frame. Kalman filtering uses the state equation of the linear system and inputs and outputs observation data to optimally estimate the system state to predict the target position. Based on the result of the previous optimal estimate, the current estimated value is predicted. The specific calculation formula is as follows:

[0084]

[0085] Where F is the state transfer matrix, B is the control matrix, is the optimal estimate of the previous frame, u k-1 is the control input matrix of the previous moment;

[0086] S303: The trajectory is first associated with the high-resolution bounding box. The first association fuses the feature distance and the intersection-over-union distance. First, the similarity between the pedestrian re-identification features of the bounding box in the current frame and the pedestrian re-identification features predicted by the Kalman filter is calculated using the Mahalanobis distance. The Mahalanobis distance is then combined with the cosine distance calculated on the re-identification features to obtain a feature distance matrix. The specific calculation formula is as follows:

[0087] D f =λD r +(1-λ)D m

[0088] Where λ is the weight parameter, D r The Mahalanobis distance matrix calculated for the re-identification features of the bounding box and the re-identification features in the trajectory, D m The cosine distance matrix calculated for the re-identification features of the bounding box and the re-identification features in the trajectory, D f It is the characteristic distance matrix obtained by weighting the Mahalanobis distance matrix and the cosine distance matrix.

[0089] And use the intersection-in-union ratio to calculate the distance matrix between the bounding box and the bounding box predicted by the Kalman filter to obtain the intersection-in-union ratio distance. The specific calculation formula is as follows:

[0090]

[0091] Where A and B represent the bounding box of the current frame and the bounding box predicted by the Kalman filter, respectively.

[0092] After obtaining the feature distance matrix, the Mahalanobis distance is used to fuse the feature distance with the motion information to obtain the cost matrix of the Kalman filter motion information. The specific calculation formula is as follows:

[0093]

[0094] Where m is the target pedestrian, o is the pedestrian bounding box predicted by the Kalman filter, and μ m is the mean of the predicted state of the target pedestrian, ∑m is the covariance matrix, and δ is the weight parameter.

[0095] After obtaining the feature distance matrix and the intersection-of-union ratio distance matrix, the features and the intersection-of-union ratio are fused through threshold screening and gate function. That is, the minimum cost of each position in the feature distance matrix and the intersection-of-union ratio distance matrix is ​​selected and placed in the corresponding position in the cost matrix to obtain the fused cost matrix. After fusing the motion information, the intersection-of-union ratio distance is fused with the bounding box score to obtain the feature distance of the fused bounding box score information. The specific calculation formula is as follows:

[0096] F cost (t, d) = 1-(1-D iou )×S(m, d),

[0097] Where S represents the detection score array, and d represents the detection result;

[0098] The Hungarian algorithm is then used to assign the current bounding box to the existing trajectory, obtaining the first associated matching result, the unmatched bounding box, and the unmatched trajectory. In the matching result, each trajectory and each frame will include the current frame features and long-term temporal features of the target pedestrian; the calculation formula for the long-term temporal features is as follows:

[0099]

[0100] Where β is the time series feature weight, is the long-term temporal feature of the previous frame, is the feature of the target person in the current frame;

[0101] S304: The unmatched trajectory is associated with the low-score bounding box for the second time. The second association only uses the feature distance. The Mahalanobis distance is used to calculate the similarity between the unmatched trajectory features and the pedestrian re-identification features of the low-score bounding box to obtain a cost matrix. The low-score bounding box and the existing unmatched trajectory are assigned through the Hungarian algorithm to obtain the matching results of the second association, the bounding box that has not yet been matched, and the trajectory that has not yet been matched. In the matching results, the current frame features and long-term temporal features of the target pedestrian are added to each frame of each trajectory. The low-score bounding box that has not yet been matched is deleted, and the trajectory that has not yet been matched for the second time is retained. The trajectory that has not yet been matched for the second time will be classified as lost. If the trajectory does not appear again in a certain number of frames, it will be deleted;

[0102] S305: Generate new trajectories for the remaining high-scoring bounding boxes, wherein pedestrian features of the current frame are added when generating the trajectories to facilitate calculation of subsequent temporal features.

[0103] S4: Calculate the similarity and accuracy of the pedestrian trajectory feature vectors in the query set and the corresponding video library, including the following steps:

[0104] S401: Input the query set into the trained object detection network to obtain the pedestrian feature vector of the query set;

[0105] S402: Sort the features corresponding to each trajectory by calculating the Euclidean distance, calculate the average value of the features with closer distances to obtain the pedestrian feature vector in the candidate video library, and compare the pedestrian feature vector of the query set with the pedestrian feature vector in the candidate video library for similarity and sort them to obtain the results of the cross-camera video pedestrian search.

[0106] Figure 5 To demonstrate the actual test results of the method of the present invention, the query is the target pedestrian to be found, and the result of the video pedestrian search is the trajectory of the target person appearing in other cameras.

[0107] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A pedestrian search method across camera videos based on representation learning, characterized by: include: Obtain pedestrian data from cross-camera videos and divide them into training and test sets; Construct a query set and a candidate video library corresponding to the query set; Input the pedestrian data across camera videos into the object detection network to learn the bounding box of each pedestrian, the confidence score of the bounding box, and the features of each pedestrian; Obtain the trajectory of each pedestrian and the temporal features of each trajectory; Calculate the similarity between the pedestrian trajectory feature vector in the query set and all the pedestrian trajectory feature vectors in the candidate video library, and perform accuracy calculation; Obtaining the trajectory of each pedestrian and the temporal features of each trajectory includes: Input the test set into the trained object detection network to obtain the pedestrian bounding box, confidence score and features of each frame image. The bounding box and corresponding features are obtained according to the score threshold. The bounding box with a score above the threshold is a high-scoring bounding box, and the bounding box with a score below the threshold is a low-scoring bounding box. Kalman filtering is used to predict the new position of each trajectory in the current frame image; The trajectory is first associated with the high-score bounding box, and the first association fuses the feature distance and the intersection-over-union distance: The feature distance is as follows: First, the similarity between the pedestrian re-identification features of the bounding box in the current frame image and the pedestrian re-identification features in the trajectory is calculated using the Mahalanobis distance, and the Mahalanobis distance is combined with the cosine distance calculated on the pedestrian re-identification features to obtain the feature distance matrix. The specific calculation formula is as follows: D f =λD r +(1-λ)D m , Where λ is the weight parameter, D r The Mahalanobis distance matrix calculated for the pedestrian re-identification features of the bounding box and the pedestrian re-identification features in the trajectory, D m The cosine distance matrix D is calculated for the pedestrian re-identification features of the bounding box and the pedestrian re-identification features in the trajectory. f It is the characteristic distance matrix obtained by weighting the Mahalanobis distance matrix and the cosine distance matrix; The intersection-over-union (IoU) distance is calculated by using the IoU distance to calculate the distance matrix between the bounding box and the bounding box predicted by the Kalman filter. The specific calculation formula is as follows: Where A and B represent the bounding box of the current frame image and the bounding box predicted by the Kalman filter respectively; After obtaining the feature distance matrix, the Mahalanobis distance is used to fuse the feature distance with the motion information to obtain the cost matrix of the Kalman filter motion information. The specific calculation formula is as follows: Where m is the target pedestrian, o is the pedestrian bounding box predicted by the Kalman filter, μm is the mean of the target pedestrian's predicted state, ∑m is the covariance matrix, and δ is the weight parameter; After obtaining the feature distance matrix and the intersection-of-union ratio distance matrix, the feature distance matrix and the intersection-of-union ratio distance matrix are fused through threshold screening and gate function. That is, the minimum cost of each position in the feature distance matrix and the intersection-of-union ratio distance matrix is ​​selected and placed in the corresponding position in the cost matrix to obtain the fused cost matrix. Then, the intersection-of-union ratio distance matrix is ​​fused with the bounding box confidence score to obtain the feature distance of the fused bounding box confidence score information. The specific calculation formula is as follows: F cost (t,d)=1-(1-D iou )×S(m,d), Where S represents the detection score array, and d represents the detection result; The Hungarian algorithm is then used to assign the current bounding box to the existing trajectory, obtaining the first associated matching result, unmatched bounding boxes, and unmatched trajectories. During the matching process, whenever a trajectory successfully matches the bounding box of the current frame image, the pedestrian features and long-term temporal features of the bounding box are added to the trajectory information; the calculation formula for the long-term temporal features is as follows: Where B is the time series feature weight, is the long-term temporal feature of the previous frame image, is the feature of the target pedestrian in the current frame image; The unmatched tracks are associated with the unmatched bounding boxes for a second time. The second association uses only feature distance. The Mahalanobis distance is used to calculate the similarity between the unmatched track features and the pedestrian re-identification features of the unmatched bounding boxes to obtain a cost matrix. The unmatched bounding boxes are assigned to the existing unmatched tracks using the Hungarian algorithm. The bounding boxes that are still unmatched are deleted, and the tracks that are still unmatched for the second time are retained. Tracks that are still unmatched for the second time are classified as lost and deleted if they do not appear again for a certain number of frames. Generate new trajectories for the remaining high-scoring bounding boxes. When generating new trajectories, the pedestrian features of the current frame image are added to facilitate the calculation of subsequent temporal features.

2. The method for pedestrian search in cross-camera videos based on representation learning according to claim 1, characterized in that: Obtain pedestrian data from cross-camera videos and divide them into training and test sets; Constructing the query set and the candidate video library corresponding to the query set specifically includes: Obtain a cross-camera video pedestrian search dataset (MTA-PS) containing cross-camera video pedestrian data, perform image enhancement processing on the cross-camera video pedestrian data, and then divide the cross-camera video pedestrian data after image enhancement processing into a training set and a test set; Construct a query set, randomly select one frame of image for each pedestrian that appears in at least two cameras and add it to the query set, add the videos of the pedestrian that appear in other cameras to the candidate video library of the pedestrian, and select interference videos that do not contain the pedestrian to add to the candidate video library of the pedestrian, ensuring that the candidate video library of each pedestrian in the query set contains at least five videos.

3. The method for pedestrian search across camera videos based on representation learning according to claim 1, characterized in that: The pedestrian data across camera videos is fed into the object detection network to learn the bounding box of each pedestrian, the confidence score of the bounding box, and the features of each pedestrian, including: The training set is subjected to feature extraction using a Darknet53-based backbone network, a feature pyramid is constructed for multi-scale feature fusion, and a tuple consisting of three feature layers is output; The four decoupling heads of regression, intersection-over-union, classification and re-identification are used to regress, predict, classify and extract re-identification features for the tuple composed of three feature layers. The output of the regression decoupling head is a 1-dimensional vector representing the confidence score of the current bounding box, the output of the classification decoupling head is a 1-dimensional vector representing the pedestrian category, the output of the intersection-over-union decoupling head is a 4-dimensional vector representing the pedestrian bounding box, and the output of the re-identification decoupling head is a 128-dimensional vector representing the pedestrian's features. The SiamOTA strategy is used to screen positive and negative samples, and then the positive and negative samples are used to calculate the identity loss, triplet loss and center loss to generate the pedestrian bounding box, the confidence score of the bounding box, the pedestrian ID and the pedestrian's features. Identity loss L id The calculation formula is as follows: Among them, N is the number of pedestrians in the current frame image, M is the number of all pedestrian IDs in the training set, i is the i-th pedestrian in the current frame image, j is the real value ID corresponding to the i-th pedestrian, I i (j) is an indicator function, which is 1 when calculating positive samples and 0 when calculating negative samples. p(j) is the probability that the ID of the i-th pedestrian is the true value ID; Triple L tri The (i, j, k) loss is calculated as follows: L tri (i,j,k)=max(a+d ij -d ik ,0), Where a is the boundary constant, d ij is the Euclidean distance between the re-identification features of the pedestrian bounding boxes with the same ID, d ik is the Euclidean distance between the re-identification features corresponding to the bounding boxes of pedestrians with different IDs. (i, j) is the re-identification feature corresponding to the bounding boxes of pedestrians with the same ID, and (i, k) is the re-identification feature corresponding to the bounding boxes of pedestrians with different IDs. Center loss L c The calculation formula is as follows: where x t is the pedestrian re-identification feature vector of the t-th image, y q is the label of the qth frame in a batch, Indicates the yth q The feature center of each pedestrian ID; The above three losses are added together as the loss function to guide the training of the target detection network.

4. The method for pedestrian search in cross-camera videos based on representation learning according to claim 1, characterized in that: The similarity calculation is performed between the trajectory feature vector of the pedestrian in the query set and the trajectory feature vectors of all pedestrians in the candidate video library, and the accuracy calculation includes: Input the images in the query set into the trained object detection network one by one to obtain the pedestrian feature vector of the query set; The features corresponding to each trajectory are sorted by calculating the Euclidean distance, and the features with closer distances are averaged to obtain the pedestrian feature vector in the candidate video library. The pedestrian feature vector of the query set is then similarly calculated and sorted with all the pedestrian feature vectors of the pedestrian in the candidate video library to obtain the results of the cross-camera video pedestrian search.

5. A terminal device, characterized in that: The invention comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of a method for searching pedestrians in cross-camera videos based on representation learning as described in any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of a cross-camera video pedestrian search method based on representation learning as described in any one of claims 1 to 4.

Images