Cross-camera video pedestrian search method based on representation learning, terminal equipment and computer readable storage medium

Abstract

The invention discloses a cross-camera video pedestrian search method based on representation learning, terminal equipment and a computer readable storage medium, and the method comprises the following steps: obtaining cross-camera video pedestrian data, and constructing a query set and a candidate video library corresponding to the query set; inputting the video into a target detection network, and learning a bounding box of each pedestrian, a confidence score of the bounding box and features of each pedestrian; inputting the pedestrian bounding boxes and the features into a multi-target tracking network based on time sequence feature fusion, performing data association on the video frames, and obtaining the trajectory of each pedestrian and the time sequence feature of each trajectory; and performing similarity calculation on the pedestrian trajectory feature vector in the query set and all pedestrian trajectory feature vectors of the pedestrian in the candidate video library, and performing accuracy calculation. According to the invention, for a real video monitoring scene, extraction and data association are carried out on pedestrian features by using long-term time sequence features in a video, so that target pedestrians can be searched in a cross-camera scene.

Description

Technical Field

[0001] The present invention relates to the fields of target detection, pedestrian re-identification and target tracking, and specifically to a cross-camera video pedestrian search method based on representation learning, a terminal device and a computer-readable storage medium. Background Art

[0002] Pedestrian search is a combination of pedestrian detection and pedestrian re-identification. That is, given a target pedestrian, the target pedestrian can be accurately detected and identified in the uncropped surveillance images across cameras. It can answer the questions of "who is he and where is he" at the same time, greatly enhancing and improving the function and efficiency of intelligent visual surveillance.

[0003] Although pedestrian search has achieved good results on the two mainstream datasets CUHK-SYSU and PRW, the contents of these two datasets are images, which still have certain deviations from the actual video surveillance scenes and are difficult to implement in the actual surveillance video system. Therefore, video pedestrian search is also proposed on the basis of multi-camera multi-target tracking. Multi-camera multi-target tracking aims to determine the location of each person at any time from the video streams shot by multiple cameras, and video pedestrian search adds a query module on this basis to find the trajectory of the pedestrian to be queried in the video data across cameras in the entire scene. Therefore, video pedestrian search can also be considered as a combination of multi-target tracking and video pedestrian re-identification.

[0004] In terms of data sets, an important data set in the field of multi-camera multi-target tracking is DukeMTMC, which is a campus surveillance video data set proposed by Duke University in 2014 and has been used in many video field studies. However, DukeMTMC was disabled in 2019 due to privacy issues, and the research on multi-camera video tasks was also affected, including video pedestrian search. The data sets of other video tasks, such as multi-target tracking, video pedestrian re-identification, video understanding, etc., do not meet the requirements of video pedestrian search. Therefore, video pedestrian search also faces the challenge of missing data sets. In terms of features, compared with images, videos have an additional time dimension and richer spatiotemporal information, which is also not available in current image-based pedestrian search. It can also avoid the model's dependence on single-frame images and reduce the impact of the model's recognition difficulties caused by the occlusion of a single image. If you want to improve the accuracy of video pedestrian re-identification, it is inevitable to use and integrate these multi-frame features of pedestrians. Therefore, how to use these time dimension information is the core issue of video pedestrian search.

[0005] Therefore, based on the above technical problems, it is necessary to design a new cross-camera video pedestrian search method. Summary of the invention

[0006] Purpose of the invention: In view of the above-mentioned prior art, a cross-camera video pedestrian search method, terminal device and computer-readable storage medium based on representation learning are proposed to realize cross-camera video pedestrian search.

[0007] Technical solution:

[0008] A pedestrian search method for cross-camera videos based on representation learning, comprising:

[0009] Obtain pedestrian data from cross-camera videos and divide them into training sets and test sets; construct a query set and a candidate video library corresponding to the query set;

[0010] Input the pedestrian data across camera videos into the object detection network to learn each pedestrian bounding box, the confidence score of the bounding box, and the features of each pedestrian;

[0011] Obtain the trajectory of each pedestrian and the temporal characteristics of each trajectory;

[0012] The similarity between the pedestrian trajectory feature vector in the query set and all the pedestrian trajectory feature vectors of the pedestrian in the candidate video library is calculated, and the accuracy is calculated.

[0013] Preferably, obtaining pedestrian data across camera videos and dividing them into a training set and a test set; and constructing a query set and a candidate video library corresponding to the query set specifically include:

[0014] A cross-camera video pedestrian search dataset MTA-PS containing cross-camera video pedestrian data is obtained, and image enhancement processing is performed on the cross-camera video pedestrian data. Then, the cross-camera video pedestrian data after image enhancement processing is divided into a training set and a test set; a query set is constructed, and a frame of image is randomly selected for each pedestrian that appears in at least two cameras and added to the query set, and videos in which the pedestrian appears in other cameras are added to the candidate video library of the pedestrian, and interference videos that do not contain the pedestrian are selected and added to the candidate video library of the pedestrian, so as to ensure that the candidate video library of each pedestrian in the query set contains at least five videos.

[0015] Preferably, the pedestrian data across the camera videos is input into the object detection network to learn each pedestrian bounding box, the confidence score of the bounding box, and the features of each pedestrian, including:

[0016] The training set is used to extract features using a Darknet53-based backbone network, and a feature pyramid is constructed for multi-scale feature fusion, outputting a tuple consisting of three feature layers;

[0017] The four decoupling heads of regression, intersection-over-union, classification and re-identification are used to regress, predict, classify and extract re-identification features for the tuple composed of three feature layers. The output of the regression decoupling head is a 1-dimensional vector representing the confidence score of the current bounding box, the output of the classification decoupling head is a 1-dimensional vector representing the pedestrian category, the output of the intersection-over-union decoupling head is a 4-dimensional vector representing the pedestrian bounding box, and the output of the re-identification decoupling head is a 128-dimensional vector representing the characteristics of the pedestrian. The SiamOTA strategy is used to screen positive and negative samples, and then the positive and negative samples are used to calculate the identity loss, triplet loss and center loss to generate the pedestrian bounding box, the confidence score of the bounding box, the pedestrian ID and the pedestrian's features.

[0018] Identity loss L id The calculation formula is as follows:

[0019]

[0020] Where N is the number of pedestrians in the current frame image, M is the number of all pedestrian IDs in the training set, i is the i-th pedestrian in the current frame image, j is the real value ID corresponding to the i-th pedestrian, and I i (j) is an indicator function, which is 1 when calculating positive samples and 0 when calculating negative samples. p(j) is the probability that the ID of the i-th pedestrian is the true value ID;

[0021] Triple L tri The (i, j, k) loss calculation formula is as follows:

[0022] L tri (i, j, k) = max(a + d ij -d ik ,0),

[0023] Where a is the boundary constant, d ij is the Euclidean distance between the re-identification features of the bounding boxes of pedestrians with the same ID, d ik is the Euclidean distance between the re-identification features corresponding to the bounding boxes of pedestrians with different IDs, (i, j) is the re-identification feature corresponding to the bounding boxes of pedestrians with the same ID, and (i, k) is the re-identification feature corresponding to the bounding boxes of pedestrians with different IDs;

[0024] Center loss L c The calculation formula is as follows:

[0025]

[0026] where x t is the pedestrian re-identification feature vector of the t-th image, y q is the label of the qth frame in a batch, Indicates the yth q The feature center of each pedestrian ID;

[0027] The above three losses are added together as the loss function to guide the training of the target detection network.

[0028] Preferably, obtaining the trajectory of each pedestrian and the time series features of each trajectory includes:

[0029] Input the test set into the trained object detection network to obtain the pedestrian bounding box, confidence score and features of each frame image. According to the score threshold, the bounding box and the corresponding features are obtained. The bounding box with a score higher than the threshold is a high-score bounding box, and the bounding box with a score lower than the threshold is a low-score bounding box.

[0030] Kalman filtering is used to predict the new position of each trajectory in the current frame image;

[0031] The trajectory is first associated with the high-resolution bounding box, and the first association fuses the feature distance and the intersection-over-union distance:

[0032] The feature distance is as follows: first, the similarity between the pedestrian re-identification feature of the boundary box in the current frame image and the pedestrian re-identification feature in the trajectory is calculated using the Mahalanobis distance, and the Mahalanobis distance is combined with the cosine distance calculated on the pedestrian re-identification feature to obtain a feature distance matrix. The specific calculation formula is as follows:

[0033] D f =λD r +(1-λ)D m ,

[0034] Where λ is the weight parameter, D r The Mahalanobis distance matrix D is calculated for the pedestrian re-identification features of the bounding box and the pedestrian re-identification features in the trajectory. m The cosine distance matrix D is calculated for the pedestrian re-identification features of the bounding box and the pedestrian re-identification features in the trajectory. f It is the characteristic distance matrix obtained by weighting the Mahalanobis distance matrix and the cosine distance matrix;

[0035] The intersection-over-union distance is obtained by using the intersection-over-union distance to calculate the distance matrix between the bounding box and the bounding box predicted by the Kalman filter. The specific calculation formula is as follows:

[0036]

[0037] Where A and B represent the bounding box of the current frame image and the bounding box predicted by the Kalman filter respectively;

[0038] After obtaining the feature distance matrix, the Mahalanobis distance is used to fuse the feature distance with the motion information to obtain the cost matrix of the Kalman filter motion information. The specific calculation formula is as follows:

[0039]

[0040] Where m is the target pedestrian, o is the pedestrian bounding box predicted by the Kalman filter, and μ m is the mean of the predicted state of the target pedestrian, ∑m is the covariance matrix, and δ is the weight parameter;

[0041] After obtaining the feature distance matrix and the intersection-and-union ratio distance matrix, the feature distance matrix and the intersection-and-union ratio distance matrix are fused through threshold screening and gate function, that is, the minimum cost of each position in the feature distance matrix and the intersection-and-union ratio distance matrix is ​​selected and put into the corresponding position in the cost matrix to obtain the fused cost matrix. Then, the intersection-and-union ratio distance matrix is ​​fused with the bounding box confidence score to obtain the feature distance of the fused bounding box confidence score information. The specific calculation formula is as follows:

[0042] F cost (t, d) = 1-(1-D iou )×S(m, d),

[0043] Where S represents the detection score array, and d represents the detection result;

[0044] Then, the current bounding box and the existing trajectory are assigned by the Hungarian algorithm to obtain the first associated matching result, unmatched bounding box and unmatched trajectory. During the matching process, whenever the trajectory successfully matches the bounding box of the current frame image, the pedestrian features and long-term temporal features of the bounding box are added to the trajectory information; the calculation formula of the long-term temporal features is as follows:

[0045]

[0046] Where β is the time series feature weight, is the long-term temporal feature of the previous frame of image, The features of the target pedestrian in the current frame image;

[0047] The unmatched track is associated with the unmatched bounding box for the second time. The second association only uses the feature distance. The Mahalanobis distance is used to calculate the similarity between the unmatched track features and the pedestrian re-identification features of the unmatched bounding box to obtain the cost matrix. The unmatched bounding box is assigned to the existing unmatched track through the Hungarian algorithm. The unmatched bounding box is deleted and the track that is still unmatched for the second time is retained. The track that is still unmatched for the second time will be classified as lost and deleted if it does not appear again in a certain number of frames.

[0048] Generate new trajectories for the remaining high-score bounding boxes, where the pedestrian features of the current frame image are added when generating the new trajectory to facilitate the calculation of subsequent temporal features.

[0049] Preferably, performing similarity calculation on the trajectory feature vector of the pedestrian in the query set and the trajectory feature vectors of all pedestrians of the pedestrian in the candidate video library, and performing accuracy calculation includes:

[0050] Input the images in the query set into the trained object detection network one by one to obtain the pedestrian feature vector of the query set;

[0051] The features corresponding to each trajectory are sorted by calculating the Euclidean distance, and the features with closer distances are averaged to obtain the pedestrian feature vector in the candidate video library. The pedestrian feature vector of the query set is similar to all the pedestrian feature vectors of the pedestrian in the candidate video library and sorted to obtain the results of cross-camera video pedestrian search.

[0052] A terminal device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of a cross-camera video pedestrian search method based on representation learning as described in any one of claims 1 to 5.

[0053] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor executes the steps of a method for searching pedestrians in cross-camera videos based on representation learning as described in any one of claims 1 to 5.

[0054] Beneficial effects: The present invention proposes a network for joint pedestrian detection, target tracking and pedestrian re-identification for video data in real monitoring scenarios, performs time series correlation on the target pedestrian data and extracts long-term time series features, which is conducive to cross-camera pedestrian search in real monitoring scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flow chart of a method for searching pedestrians in cross-camera videos based on representation learning of the present invention;

[0056] Figure 2 It is a schematic diagram of the network structure of the video pedestrian search of the present invention;

[0057] Figure 3 is a flow chart of the pedestrian feature fusion and trajectory tracking method in the present invention;

[0058] Figure 4 It is a schematic diagram of the network structure of the backbone network Darknet53 in the present invention;

[0059] Figure 5 It is a schematic diagram of the results of the cross-camera video pedestrian search method based on representation learning of the present invention. DETAILED DESCRIPTION

[0060] The present invention will be further explained below in conjunction with the accompanying drawings.

[0061] like Figure 1 As shown, the cross-camera video pedestrian search method based on representation learning includes the following steps:

[0062] S1: Obtain pedestrian data across camera videos and divide them into training sets and test sets; construct a query set and a candidate video library corresponding to the query set, including the following specific steps:

[0063] S101: obtaining a cross-camera video pedestrian search dataset MTA-PS containing cross-camera video pedestrian data, performing image enhancement processing on the cross-camera video pedestrian data, and then dividing the cross-camera video pedestrian data after the image enhancement processing into a training set and a test set;

[0064] S102: Divide the query set and the corresponding candidate video library, randomly select one frame for each pedestrian that appears in at least two cameras and add it to the query set, add the video of the pedestrian that appears in other cameras to the query set of the pedestrian, and select interference videos that do not contain the pedestrian to add to the query set, so as to ensure that the query set corresponding to each pedestrian contains at least five videos;

[0065] S2: Input the training sample video data into the video pedestrian search deep learning network to learn the target bounding box, confidence score and features of the pedestrian in the image at each moment. The network structure is as follows: Figure 2 As shown, the specific steps are as follows:

[0066] S201: Use the Darknet53-based backbone network for feature extraction. The Darknet53 flow chart is as follows Figure 4 . In the Darknet53 network, the input image first passes through a convolutional layer to transform the number of input channels into the first number of output channels in the network. Then, the output tensor of this convolutional layer is input into the first "residual block" of Darknet53. In this residual block, the input tensor is fed into a series of convolutional layers and batch normalization layers, and after a nonlinear activation function, it is added to the input tensor through a jump connection, and finally the output tensor is sent to the next layer. After stacking multiple residual blocks, the number of channels of the output tensor increases to 1024, and the size becomes 13*13.

[0067] After that, the original image is scaled through the feature pyramid to obtain images of different sizes. Then, feature maps of different sizes are generated based on each size of image. The high-level feature information is first transferred and fused by upsampling. The high-level feature map resolution is enlarged by 2 times upsampling to keep it consistent with the resolution of the low-level feature map. The upsampling method uses the neighboring interpolation algorithm. The predicted feature map is then obtained by downsampling fusion. The bottom-level feature map is kept consistent with the high-level feature map after upsampling by a 1*1 convolution operation. Finally, it is fused by addition. Finally, a tuple result consisting of 3 feature layers is output;

[0068] S202: In order to avoid interference and conflict when calculating the bounding box and extracting features, this network extracts features from the three layers of feature maps in the feature pyramid through four decoupling heads: regression, intersection-over-union, classification, and re-identification. Each decoupling head uses a 1*1 convolution and two 3*3 convolutions to obtain different results. The output of the regression head is a 1-dimensional vector, representing the confidence score of the current bounding box, and the output of the classification head is a 1-dimensional vector, representing the target category. There is only the pedestrian category here, so the default is 1. The intersection-over-union head outputs a 4-dimensional vector, representing the bounding box of the target, and the re-identification head outputs a 128-dimensional vector, representing the characteristics of the target pedestrian, that is, each pedestrian obtains a total of 134-dimensional feature vectors. After connecting all pedestrian feature vectors on different feature maps, positive and negative samples are screened. First, a preliminary screening is performed to select samples whose target center points fall in the true value box as positive samples, and the rest are negative samples. Then, the SiamOTA strategy is used to assign labels. The cost calculation method of the true value gi and the predicted value pj is as follows:

[0069]

[0070] where λ is the balance coefficient, and is the category association loss and regression loss of the true value gi and the predicted value pj. For the true value, the first k samples with the lowest cost in the central area are selected as positive samples, and the rest are used as negative samples.

[0071] In order to better distinguish the characteristics of different pedestrians and increase the robustness of the network, positive and negative samples are used to calculate identity loss, triple loss and center loss to generate pedestrian bounding boxes, confidence scores, categories and features. The identity loss calculation formula is as follows:

[0072]

[0073] Where N is the number of pedestrians in the current frame image, M is the number of all pedestrian IDs in the training set, i is the i-th pedestrian in the current frame image, j is the real value ID corresponding to the i-th pedestrian, and I i(j) is an indicator function, which is 1 when calculating positive samples and 0 when calculating negative samples. p(j) is the probability that the ID of the i-th pedestrian is the true value ID;

[0074] The triplet loss calculation formula is as follows:

[0075] L tri (i, j, k) = max(a + d ij -d ik ,0),

[0076] Where a is the boundary constant, d ij is the Euclidean distance between the re-identification features of the bounding boxes of pedestrians with the same ID, d ik is the Euclidean distance between the re-identification features corresponding to the bounding boxes of pedestrians with different IDs, (i, j) is the re-identification feature corresponding to the bounding boxes of pedestrians with the same ID, and (i, k) is the re-identification feature corresponding to the bounding boxes of pedestrians with different IDs.

[0077] The center loss calculation formula is as follows:

[0078]

[0079] where x t is the pedestrian re-identification feature vector of the tth sample, y q is the label of the qth frame in a batch, Indicates the yth q The feature center of each pedestrian ID category;

[0080] The above three losses are added together as the loss function to guide the training of the target detection network.

[0081] S3: Figure 3 As shown in the figure, the pedestrian bounding box and features are input into the multi-target tracking network based on temporal feature fusion, and the video frames are associated with data to obtain the trajectory of each pedestrian and the features of the corresponding trajectory, including the following steps:

[0082] S301: Input the video frame by frame into the pre-trained object detection network, obtain the pedestrian bounding box, confidence score and features of each frame, and obtain the bounding box and corresponding features according to the prediction score threshold;

[0083] S302: Use Kalman filtering to predict the new position of each trajectory in the current frame. Kalman filtering uses the state equation of the linear system and inputs and outputs observation data to optimally estimate the system state to achieve the function of predicting the target position. Based on the result of the last optimal estimation, the estimated value at the current moment is predicted. The specific calculation formula is as follows:

[0084]

[0085] Where F is the state transfer matrix, B is the control matrix, is the optimal estimate of the previous frame, u k-1 is the control input matrix of the previous moment;

[0086] S303: The trajectory is first associated with the high-resolution bounding box. The first association fuses the feature distance and the intersection-over-union distance. First, the similarity between the pedestrian re-identification feature of the bounding box in the current frame and the pedestrian re-identification feature predicted by the Kalman filter is calculated using the Mahalanobis distance. The Mahalanobis distance is fused with the cosine distance calculated on the re-identification feature to obtain the feature distance matrix. The specific calculation formula is as follows:

[0087] D f =λD r +(1-λ)D m

[0088] Where λ is the weight parameter, D r The Mahalanobis distance matrix D is calculated for the re-identification features of the bounding box and the re-identification features in the trajectory. m The cosine distance matrix D is calculated for the re-identification features of the bounding box and the re-identification features in the trajectory. f It is the characteristic distance matrix obtained by weighting the Mahalanobis distance matrix and the cosine distance matrix.

[0089] And the distance matrix is ​​calculated using the intersection-and-union ratio between the bounding box and the bounding box predicted by the Kalman filter to obtain the intersection-and-union ratio distance. The specific calculation formula is as follows:

[0090]

[0091] Where A and B represent the current frame bounding box and the bounding box predicted by the Kalman filter, respectively.

[0092] After obtaining the feature distance matrix, the Mahalanobis distance is used to fuse the feature distance with the motion information to obtain the cost matrix of the Kalman filter motion information. The specific calculation formula is as follows:

[0093]

[0094] Where m is the target pedestrian, o is the pedestrian bounding box predicted by the Kalman filter, and μ m is the mean of the predicted state of the target pedestrian, ∑m is the covariance matrix, and δ is the weight parameter.

[0095] After obtaining the feature distance matrix and the intersection-over-union ratio distance matrix, the features and the intersection-over-union ratio are fused through threshold screening and gate function, that is, the minimum cost of each position in the feature distance matrix and the intersection-over-union ratio distance matrix is ​​selected and put into the corresponding position in the cost matrix to obtain the fused cost matrix. After fusing the motion information, the intersection-over-union ratio distance is fused with the bounding box score to obtain the feature distance of the fused bounding box score information. The specific calculation formula is as follows:

[0096] F cost (t, d) = 1-(1-D iou )×S(m, d),

[0097] Where S represents the detection score array, and d represents the detection result;

[0098] Then, the current bounding box and the existing trajectory are assigned by the Hungarian algorithm to obtain the first associated matching result, the unmatched bounding box and the unmatched trajectory. In the matching result, each trajectory and each frame will add the current frame features and long-term time series features of the target pedestrian; the calculation formula of the long-term time series features is as follows:

[0099]

[0100] Where β is the time series feature weight, is the long-term temporal feature of the previous frame, is the characteristics of the target person in the current frame;

[0101] S304: The unmatched trajectory is associated with the low-score bounding box for the second time. The second association only uses the feature distance. The Mahalanobis distance is used to calculate the similarity between the unmatched trajectory features and the pedestrian re-identification features of the low-score bounding box to obtain the cost matrix. The low-score bounding box and the existing unmatched trajectory are assigned through the Hungarian algorithm to obtain the matching results of the second association, the unmatched bounding box and the unmatched trajectory. In the matching results, the current frame features and long-term time series features of the target pedestrian are added to each frame of each trajectory. The unmatched low-score bounding box is deleted, and the trajectory that is still unmatched for the second time is retained. The trajectory that is still unmatched for the second time will be classified as lost, and it will be deleted if it does not appear again in a certain number of frames;

[0102] S305: Generate new trajectories for the remaining high-score bounding boxes, wherein pedestrian features of the current frame are added when generating the trajectories to facilitate calculation of subsequent temporal features.

[0103] S4: Calculate the similarity between the pedestrian trajectory feature vector in the query set and the pedestrian trajectory feature vector in the corresponding video library, and calculate the accuracy, including the following steps:

[0104] S401: Input the query set into the trained object detection network to obtain the pedestrian feature vector of the query set;

[0105] S402: Sort the features corresponding to each trajectory by calculating the Euclidean distance, calculate the average value of the features with closer distances to obtain the pedestrian feature vector in the candidate video library, and compare and sort the pedestrian feature vector in the query set with the pedestrian feature vector in the candidate video library by similarity to obtain the result of the cross-camera video pedestrian search.

[0106] Figure 5 To demonstrate the actual test effect of the method of the present invention, the query is the target pedestrian to be found, and the result of the video pedestrian search is the trajectory of the target person appearing under other cameras.

[0107] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A pedestrian search method across camera videos based on representation learning, characterized in that: include: Obtain pedestrian data from cross-camera videos and divide them into training sets and test sets; Construct a query set and a candidate video library corresponding to the query set; Input the pedestrian data across camera videos into the object detection network to learn each pedestrian bounding box, the confidence score of the bounding box, and the features of each pedestrian; Obtain the trajectory of each pedestrian and the temporal characteristics of each trajectory; The similarity between the pedestrian trajectory feature vector in the query set and all the pedestrian trajectory feature vectors of the pedestrian in the candidate video library is calculated, and the accuracy is calculated.

2. The method for searching pedestrians in cross-camera videos based on representation learning according to claim 1, characterized in that: Obtain pedestrian data from cross-camera videos and divide them into training sets and test sets; Constructing a query set and a candidate video library corresponding to the query set specifically includes: A cross-camera video pedestrian search dataset MTA-PS containing cross-camera video pedestrian data is obtained, the cross-camera video pedestrian data is subjected to image enhancement processing, and then the cross-camera video pedestrian data after image enhancement processing is divided into a training set and a test set; Construct a query set, randomly select a frame of image for each pedestrian that appears in at least two cameras and add it to the query set, add the videos of the pedestrian that appear in other cameras to the candidate video library of the pedestrian, and select interference videos that do not contain the pedestrian to add to the candidate video library of the pedestrian, ensuring that the candidate video library of each pedestrian in the query set contains at least five videos.

3. The method for searching pedestrians in cross-camera videos based on representation learning according to claim 1, characterized in that: The pedestrian data across camera videos is input into the object detection network to learn each pedestrian bounding box, the confidence score of the bounding box, and the features of each pedestrian, including: The training set is used to extract features using a Darknet53-based backbone network, and a feature pyramid is constructed for multi-scale feature fusion, outputting a tuple consisting of three feature layers; The four decoupling heads of regression, intersection-over-union, classification and re-identification are used to regress, predict, classify and extract re-identification features for the tuple composed of three feature layers. The output of the regression decoupling head is a 1-dimensional vector representing the confidence score of the current bounding box, the output of the classification decoupling head is a 1-dimensional vector representing the pedestrian category, the output of the intersection-over-union decoupling head is a 4-dimensional vector representing the pedestrian bounding box, and the output of the re-identification decoupling head is a 128-dimensional vector representing the characteristics of the pedestrian. The SiamOTA strategy is used to screen positive and negative samples, and then the positive and negative samples are used to calculate the identity loss, triplet loss and center loss to generate the pedestrian bounding box, the confidence score of the bounding box, the pedestrian ID and the pedestrian's features. Identity loss L id The calculation formula is as follows: Where N is the number of pedestrians in the current frame image, M is the number of all pedestrian IDs in the training set, i is the i-th pedestrian in the current frame image, j is the real value ID corresponding to the i-th pedestrian, and I i (j) is an indicator function, which is 1 when calculating positive samples and 0 when calculating negative samples. p(j) is the probability that the ID of the i-th pedestrian is the true value ID; Triple L tri The (i, j, k) loss calculation formula is as follows: L tri (i,j,k)=max(a+d ij -d ik ,0), Where a is the boundary constant, d ij is the Euclidean distance between the re-identification features of the bounding boxes of pedestrians with the same ID, d ik is the Euclidean distance between the re-identification features corresponding to the bounding boxes of pedestrians with different IDs, (i, j) is the re-identification feature corresponding to the bounding boxes of pedestrians with the same ID, and (i, k) is the re-identification feature corresponding to the bounding boxes of pedestrians with different IDs; Center loss L c The calculation formula is as follows: where x t is the pedestrian re-identification feature vector of the t-th image, y q is the label of the qth frame in a batch, Indicates the yth q The feature center of each pedestrian ID; The above three losses are added together as the loss function to guide the training of the target detection network.

4. The method for searching pedestrians in cross-camera videos based on representation learning according to claim 3, characterized in that: Obtaining the trajectory of each pedestrian and the temporal features of each trajectory include: Input the test set into the trained object detection network to obtain the pedestrian bounding box, confidence score and features of each frame image. According to the score threshold, the bounding box and the corresponding features are obtained. The bounding box with a score higher than the threshold is a high-score bounding box, and the bounding box with a score lower than the threshold is a low-score bounding box. Kalman filtering is used to predict the new position of each trajectory in the current frame image; The trajectory is first associated with the high-resolution bounding box, and the first association fuses the feature distance and the intersection-over-union distance: The feature distance is as follows: first, the similarity between the pedestrian re-identification feature of the boundary box in the current frame image and the pedestrian re-identification feature in the trajectory is calculated using the Mahalanobis distance, and the Mahalanobis distance is combined with the cosine distance calculated on the pedestrian re-identification feature to obtain a feature distance matrix. The specific calculation formula is as follows: D f =λD r +(1-λ)D m , Where λ is the weight parameter, D r The Mahalanobis distance matrix D is calculated for the pedestrian re-identification features of the bounding box and the pedestrian re-identification features in the trajectory. m The cosine distance matrix D is calculated for the pedestrian re-identification features of the bounding box and the pedestrian re-identification features in the trajectory. f It is the characteristic distance matrix obtained by weighting the Mahalanobis distance matrix and the cosine distance matrix; The intersection-over-union distance is obtained by using the intersection-over-union distance to calculate the distance matrix between the bounding box and the bounding box predicted by the Kalman filter. The specific calculation formula is as follows: Where A and B represent the bounding box of the current frame image and the bounding box predicted by the Kalman filter respectively; After obtaining the feature distance matrix, the Mahalanobis distance is used to fuse the feature distance with the motion information to obtain the cost matrix of the Kalman filter motion information. The specific calculation formula is as follows: Where m is the target pedestrian, o is the pedestrian bounding box predicted by the Kalman filter, and μ m is the mean of the predicted state of the target pedestrian, ∑m is the covariance matrix, and δ is the weight parameter; After obtaining the feature distance matrix and the intersection-and-union ratio distance matrix, the feature distance matrix and the intersection-and-union ratio distance matrix are fused through threshold screening and gate function, that is, the minimum cost of each position in the feature distance matrix and the intersection-and-union ratio distance matrix is ​​selected and put into the corresponding position in the cost matrix to obtain the fused cost matrix. Then, the intersection-and-union ratio distance matrix is ​​fused with the bounding box confidence score to obtain the feature distance of the fused bounding box confidence score information. The specific calculation formula is as follows: F cost (t,d)=1-(1-D iou )×S(m,d), Where S represents the detection score array, and d represents the detection result; Then, the current bounding box and the existing trajectory are assigned by the Hungarian algorithm to obtain the first associated matching result, unmatched bounding box and unmatched trajectory. During the matching process, whenever the trajectory successfully matches the bounding box of the current frame image, the pedestrian features and long-term temporal features of the bounding box are added to the trajectory information; the calculation formula of the long-term temporal features is as follows: Where β is the time series feature weight, is the long-term temporal feature of the previous frame of image, The features of the target pedestrian in the current frame image; The unmatched track is associated with the unmatched bounding box for the second time. The second association only uses the feature distance. The Mahalanobis distance is used to calculate the similarity between the unmatched track features and the pedestrian re-identification features of the unmatched bounding box to obtain the cost matrix. The unmatched bounding box is assigned to the existing unmatched track through the Hungarian algorithm. The unmatched bounding box is deleted and the track that is still unmatched for the second time is retained. The track that is still unmatched for the second time will be classified as lost and deleted if it does not appear again in a certain number of frames. Generate new trajectories for the remaining high-score bounding boxes, where the pedestrian features of the current frame image are added when generating the new trajectory to facilitate the calculation of subsequent temporal features.

5. The method for searching pedestrians in cross-camera videos based on representation learning according to claim 4, characterized in that: The similarity calculation of the pedestrian trajectory feature vector in the query set and the trajectory feature vectors of all pedestrians in the candidate video library is performed, and the accuracy calculation includes: Input the images in the query set into the trained object detection network one by one to obtain the pedestrian feature vector of the query set; The features corresponding to each trajectory are sorted by calculating the Euclidean distance, and the features with closer distances are averaged to obtain the pedestrian feature vector in the candidate video library. The pedestrian feature vector of the query set is similar to all the pedestrian feature vectors of the pedestrian in the candidate video library and sorted to obtain the results of cross-camera video pedestrian search.

6. A terminal device, characterized in that: It comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of a method for searching pedestrians in cross-camera videos based on representation learning as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of a method for searching pedestrians across camera videos based on representation learning as described in any one of claims 1 to 5.

Images