Video Person Re-identification Method Based on Trajectory Fusion in Complex Underground Space

Abstract

The video person re-identification method based on complex underground space trajectory fusion solves the problem of large-scale target occlusion in complex underground space video person re-identification; the Social‑GAN model can achieve accurate person trajectory prediction; a spatiotemporal trajectory fusion model is constructed to introduce person trajectory videos that are not affected by occlusion into the re-identification network to solve the problem of apparent visual feature extraction errors caused by occlusion, and effectively alleviate the impact of occlusion problems on re-identification performance; in addition, a trajectory fusion MARS_traj dataset is constructed, and the time frame number and spatial coordinate information are added to the MARS dataset to make it suitable for the video person re-identification method based on complex underground space trajectory fusion.

Description

Technical Field

[0001] The invention belongs to the field of image processing, and in particular relates to a video personnel re-identification method by integrating complex underground space trajectories. Background Art

[0002] Person re-identification refers to retrieving persons with the same identity from images of persons taken across multiple cameras. Depending on the input data, it can be divided into image person re-identification and video person re-identification. Compared with image person re-identification, video person re-identification contains more information, including time information and motion information between frames. With the development of video surveillance equipment, video person re-identification using time information clues has received more attention.

[0003] Although significant progress has been made in the research of video person re-identification in recent years, video re-identification in places such as complex underground spaces still faces many challenges, such as insufficient and uneven lighting, and crowded scenes leading to target occlusion, which in turn leads to huge changes in the appearance of people. Therefore, target occlusion is one of the biggest difficulties in video person re-identification in complex underground spaces.

[0004] Commonly used video person re-identification methods to solve the problem of target occlusion include attention mechanism and generative adversarial network. Attention mechanism uses attention model to select discriminative frames from video sequences to generate information-rich video representation, but it will discard partially occluded images, such as the Quality Aware Network (QAN) proposed by Liu et al., and the Attentive Spatial-Temporal Pooling Networks (ASTPN) proposed by Xu et al. Therefore, some scholars have proposed to use generative adversarial networks to reproduce the appearance representation of the occluded part, such as the Spatio-Temporal Completion Network (STCNet) proposed by Hou et al. However, generative adversarial networks can only restore the appearance of images that are occluded by a small part, while the appearance of images that are occluded over a large area is difficult to restore. Summary of the invention

[0005] The present invention combines the trajectory prediction Social-GAN model with the temporal complementary learning network (TCLNet) for video re-identification, and proposes a video personnel re-identification method with complex underground space trajectory fusion to solve the problem of large-scale target occlusion in complex underground space video personnel re-identification. First, from the perspective of time domain and space domain, the influence of external surrounding environment and internal factors such as pedestrian personality and hobbies on the moving direction and speed of pedestrian trajectory is studied, and the Social-GAN model is used to realize the accurate personnel trajectory prediction of the social attribute. Then, the proposed spatiotemporal trajectory fusion model is constructed, and the predicted spatiotemporal trajectory data of pedestrians is sent to the re-identification network for apparent visual feature extraction, so as to realize the effective combination of apparent visual features in the video sequence with personnel trajectory data, solve the apparent visual feature extraction error problem caused by occlusion, and effectively alleviate the influence of occlusion problem on re-identification performance.

[0006] The video person re-identification method based on complex underground space trajectory fusion includes the following steps:

[0007] Step 1: Create a trajectory fusion dataset MARS_traj, which includes personnel identity data and video sequences, and add time frame number and spatial coordinate information for each person on MARS_traj. The test set in MARS_traj includes the retrieval dataset query and the candidate dataset gallery;

[0008] Step 2: determine whether the search video in the search dataset query contains occluded images, input the image sequence containing occlusion into the trajectory prediction model for future trajectory prediction, and obtain the prediction set query_pred containing the predicted trajectory; if it is determined that the image sequence does not contain occlusion, no trajectory prediction is performed, and directly go to step 4 for fusion feature extraction;

[0009] Step 3, the obtained query_pred is fused with the candidate videos in the candidate data set gallery in terms of spatiotemporal trajectory to obtain a new fused video set query_TP;

[0010] Step 4: Use the video re-identification model to extract the spatiotemporal trajectory fusion features containing the apparent visual information and motion trajectory information for query_TP, perform feature distance measurement and candidate video sorting, and obtain the final re-identification performance evaluation indicators mAP and Rank-k, where mAP represents the mean average precision, Rank-k represents the probability that the CMC curve matches correctly in the first k videos in the sorted gallery, and the CMC curve (Cumulative Match Characteristic) reflects the algorithm's retrieval accuracy cumulative matching characteristics; and use the Rank-1 result as the video re-identification result.

[0011] Furthermore, in step 2, future trajectory prediction is based on the historical trajectory, which is realized through the Social GAN ​​model. The historical trajectory coordinates of known persons are used to obtain the predicted trajectory coordinates.

[0012] Furthermore, in step 3, in the spatiotemporal trajectory fusion feature, the temporal trajectory fusion is to consider the temporal continuity between the predicted trajectory and the known historical trajectory and calculate the temporal fusion loss in the time domain. As shown in formula (1):

[0013]

[0014] Among them, Δt is the frame difference between the last frame of the video sequence in the query and the first frame of the video sequence in the gallery. The frame number constant threshold T and the larger constant φ determine the temporal continuity of the frame difference Δt between the query and the gallery.

[0015] Furthermore, in step 3, in the spatiotemporal trajectory fusion feature, the spatial trajectory fusion is to consider the misalignment between the predicted trajectory and the frame number of the candidate video in the gallery, and calculate the spatial fusion loss

[0016]

[0017]

[0018] N=2,3,…,7, (2)

[0019] in, (n=9-j), p i It represents the Euclidean distance between the corresponding coordinates of the predicted trajectory sequence and the gallery candidate sequence; N represents the allowed deviation range between the predicted trajectory and the candidate video frame number.

[0020] Furthermore, in step 3, after obtaining the temporal fusion loss and the spatial fusion loss, the limited fusion loss in the temporal domain and the spatial domain of the jth video in gallery and the ith video in query_pred is calculated according to formula (3):

[0021]

[0022] Where N2 is the total number of video sequences in the gallery. According to formula (3), The smallest j value is obtained, so that the j-th video in the gallery is sent to the query_TP set for subsequent spatiotemporal trajectory fusion feature extraction.

[0023] Furthermore, in step 4, the new query set query_TP and the candidate set gallery extracted after the fusion of time and space trajectories are sent to the temporal complementary network TCLNet, and finally the temporal average pooling is used to aggregate the group features to obtain the final fused video feature vector; the temporal complementary network TCLNet uses the ResNet-50 network as the backbone network, and inserts the temporal saliency enhancement module TSB and the temporal saliency erasure module TSE into it; for T-frame continuous video, the backbone network inserted with TSB extracts features for each frame, marked as F={F1,F2,…,F T}, and then divide it into k groups, each containing N consecutive frame features C k ={F (k-1)N+1, …,F kN}, input each group into TSE, and use formula (4) to extract complementary features:

[0024] c k =TSE(F (k-1)N+1 ,…,F kN )=TSE(C k ) (4)

[0025] The cosine similarity is used to calculate the distance metric between the video feature vector A(x1, y1) in query_TP and the video feature vector B(x2, y2) in the candidate set gallery, as shown in formula (5):

[0026]

[0027] The videos in the gallery are sorted according to the distance metric, and the re-identification evaluation indicators mAP and Rank-k are calculated based on the sorting results, and the Rank-1 result is used as the video re-identification result.

[0028] The beneficial effects achieved by the present invention are as follows: a video personnel re-identification method with complex underground space trajectory fusion is proposed to solve the problem of large-scale target occlusion in complex underground space video personnel re-identification; accurate personnel trajectory prediction can be achieved through the Social-GAN model; personnel trajectory videos not affected by occlusion are introduced into the re-identification network to solve the problem of apparent visual feature extraction errors caused by occlusion, and effectively alleviate the impact of the occlusion problem on the re-identification performance; in addition, a trajectory fusion MARS_traj dataset is constructed, and time frame number and spatial coordinate information are added to the MARS dataset to make it suitable for the video personnel re-identification method with complex underground space trajectory fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of a method for re-identifying persons through video fusion of complex underground space trajectories in an embodiment of the present invention.

[0030] Figure 2 This is an example diagram of timing fusion when T=4 in an embodiment of the present invention.

[0031] Figure 3 This is an example diagram of spatial fusion when N=4 in an embodiment of the present invention.

[0032] Figure 4 This is an example diagram of sequence label modification in the MARS_traj dataset in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0034] The overall framework of the algorithm of the present invention is as follows Figure 1As shown in the figure. First, it is judged whether the retrieved video in the query dataset query contains occluded images, and the image sequence containing occlusion is input into the trajectory prediction model for future trajectory prediction. If it is judged that the image sequence does not contain occlusion, trajectory prediction is not performed, and fusion feature extraction is performed directly; secondly, the obtained predicted trajectory query_pred dataset is fused with the candidate videos in the gallery in the time domain and space domain to obtain a new fused video sequence query_TP; finally, the video re-identification model is used to extract the spatiotemporal trajectory fusion features containing apparent visual information and motion trajectory information, and the feature distance measurement and candidate video sorting are performed to obtain the final re-identification performance evaluation indicators mAP and Rank-k, where mAP represents the mean average precision (mean Average Precision), Rank-k represents the possibility that the CMC curve matches correctly in the first k videos in the sorted gallery, and the CMC curve (Cumulative Match Characteristic) reflects the cumulative matching characteristics of the algorithm's retrieval accuracy; and the Rank-1 result is used as the video re-identification result.

[0035] Personnel trajectory prediction predicts the future trajectory of a person by observing the historical trajectory information of the person. The present invention uses Social GAN ​​to achieve personnel future trajectory prediction. The 8-bit known person coordinates are input into the Social GAN ​​model for trajectory prediction, and 8 frames of predicted trajectory coordinates are obtained. From the perspective of time domain and space domain, these predicted trajectory sequences are fused and extracted with the candidate videos in the gallery.

[0036] (1) Time Trajectory Fusion

[0037] Considering the temporal continuity between the predicted trajectory and the known historical trajectory, the present invention calculates the temporal fusion loss in the time domain As shown in Formula 1:

[0038]

[0039] Wherein, Δt is the frame difference between the last frame of the video sequence in the query and the first frame of the video sequence in the gallery, and the frame constant threshold T and the larger constant φ determine the temporal continuity of the frame difference Δt between the query and the gallery. By comparing the values ​​of the frame constant T, T=4 is selected in the embodiment of the present invention. Figure 2 Indicates the selection of video sequences in the gallery when T=4.

[0040] (2) Spatial trajectory fusion

[0041] In actual scenarios, there are problems such as discontinuous frame timing between adjacent video sequences, which causes the frame number of the predicted trajectory sequence in the present invention to be misaligned with the frame number of the candidate sequence in the gallery. Therefore, the present invention considers the possible frame number error and calculates the spatial fusion loss

[0042]

[0043]

[0044] N=2,3,…,7, (2)

[0045] in (n=9-j), p i represents the Euclidean distance between the corresponding coordinates of the predicted trajectory sequence and the gallery candidate sequence, at different l N The meanings expressed in different Figure 3 shown.

[0046] In formula (2), N represents the deviation range of the number of frames allowed for the predicted trajectory sequence and the candidate sequence. Since the number of frames is fixed, too small N will reduce the flexibility of fusion matching, and too large N will increase the possibility of fusion matching errors. Therefore, in the embodiment of the present invention, when N=4 is adopted, better experimental results can be obtained.

[0047] After obtaining the temporal fusion loss and spatial fusion loss according to formulas (1) and (2), the limited fusion loss in the temporal and spatial domains of the jth video in gallery and the ith video in query_pred is calculated according to formula (3):

[0048]

[0049] Where N2 is the total number of video sequences in the gallery. According to formula (3), The smallest j value sends the jth video sequence in the gallery into the query_TP set for subsequent spatiotemporal trajectory fusion feature extraction.

[0050] The new query set query_TP extracted after the fusion of time and space trajectories and the candidate set gallery are sent to the Temporal Complementary Learning Network (TCLNet). The network uses the ResNet-50 network as the backbone network, into which the temporal saliency enhancement module (TSB) and the temporal saliency erasure module (TSE) are inserted. For T-frame continuous video, the backbone network with TSB inserted extracts features for each frame, marked as F = {F1, F2, ..., F T}, and then divide it into k groups, each containing N consecutive frame features C k ={F (k-1)N+1 ,…,F kN}, input each group into TSE, and extract complementary features using formula (4). Finally, use temporal average pooling to aggregate group features to obtain the final fused video feature vector; use cosine similarity to calculate the distance metric between the video feature vector A(x1,y1) in query_TP and the video feature vector B(x2,y2) in the candidate set gallery, as shown in formula (5), and sort the videos in gallery according to the distance metric, calculate the re-identification evaluation indicators mAP and Rank-k based on the sorting results, and use the Rank-1 result as the video re-identification result.

[0051] c k =TSE(F (k-1)N+1 ,…,F kN )=TSE(C k ) (4)

[0052]

[0053] The present invention constructs a trajectory fusion dataset MARS_traj suitable for occluded video person re-identification based on trajectory prediction. In order to test the model's ability to handle occlusion problems, the test set of MARS_traj of the present invention includes a query test set query and a candidate test set gallery, with a total of 744 person identities and 9659 video sequences. In order to verify the person trajectory prediction, the time frame number and spatial coordinate information are added to the person label for each person selected in the MARS_traj test set, such as Figure 4 In order to improve the authenticity of the trajectory, the coordinate values ​​are provided by the real trajectory prediction ETH-UCY dataset.

[0054] Based on the fusion dataset MARS_traj, the process of the re-identification method described in the present invention is as follows:

[0055] Input: dataset MARS_traj; trajectory prediction model Social GAN; video person re-identification model.

[0056] Output: mAP and rank-k.

[0057] (1) Input the spatiotemporal information of the video ID in the query dataset into the trajectory prediction model.

[0058] (2) The generator in Social GAN ​​generates possible prediction trajectories based on the input spatiotemporal information.

[0059] (3) The discriminator in SocialGAN identifies the generated prediction trajectory and obtains the query_pred that meets the prediction trajectory.

[0060] (4) Assume that the initial value i=1.

[0061] (5) Assume that the initial value j=1.

[0062] (6) According to formula (1) and formula (2), calculate the temporal fusion loss of the jth video in gallery and the predicted trajectory predi of the i-th video in query_pred: and spatial fusion loss

[0063] (7) j=j+1; repeat operation (6) until j=N2 (the number of video sequences in the MARS_traj dataset gallery).

[0064] (8) Obtain the minimum limited fusion loss according to formula (3), and assign the value j corresponding to the minimum limited fusion loss to i j .

[0065] (9) Put the video sequence in Gallery into query_TP.

[0066] (10) i=i+1; repeat operations (5)-(9) until i=N1 (the number of video sequences in the MARS_traj dataset query).

[0067] (11) Perform video fusion feature extraction on query_TP, gallery.

[0068] (12) Calculate the feature distance metric based on query_TP and gallery video features, and sort the gallery.

[0069] (13) Obtain the final re-identification performance evaluation indicators mAP and Rank-k based on the query, and use the Rank-1 result as the video re-identification result. mAP represents the mean average precision, Rank-k represents the probability that the CMC curve matches correctly in the first k videos in the sorted gallery, and the CMC curve (Cumulative Match Characteristic) reflects the algorithm's retrieval accuracy cumulative matching characteristics.

[0070] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A video person re-identification method based on complex underground space trajectory fusion, characterized by: The method comprises the following steps: Step 1: Create a trajectory fusion dataset MARS_traj, which includes personnel identity data and video sequences, and add time frame number and spatial coordinate information for each person on MARS_traj. The test set in MARS_traj includes the retrieval dataset query and the candidate dataset gallery; Step 2: determine whether the search video in the search dataset query contains occluded images, input the image sequence containing occlusion into the trajectory prediction model for future trajectory prediction, and obtain the prediction set query_pred containing the predicted trajectory; if it is determined that the image sequence does not contain occlusion, no trajectory prediction is performed, and directly go to step 4 for fusion feature extraction; Step 3, the obtained query_pred is fused with the candidate videos in the candidate data set gallery in terms of spatiotemporal trajectory to obtain a new fused video set query_TP; Step 4: Use the video re-identification model to extract the spatiotemporal trajectory fusion features containing the apparent visual information and motion trajectory information for query_TP, perform feature distance measurement and candidate video sorting, and obtain the final re-identification performance evaluation indicators mAP and Rank-k, where mAP represents the mean average precision, Rank-k represents the probability that the CMC curve matches correctly in the first k videos in the sorted gallery, and the CMC curve (Cumulative Match Characteristic) reflects the algorithm's retrieval accuracy cumulative matching characteristics; And the Rank-1 result is used as the video re-identification result.

2. The method for video person re-identification based on complex underground space trajectory fusion according to claim 1 is characterized by: In step 2, future trajectory prediction is based on the historical trajectory, which is realized through the Social GAN ​​model. The historical trajectory coordinates of known persons are used to obtain the predicted trajectory coordinates.

3. The method for video person re-identification based on complex underground space trajectory fusion according to claim 1 is characterized by: In step 3, among the spatiotemporal trajectory fusion features, the temporal trajectory fusion is to consider the temporal continuity between the predicted trajectory and the known historical trajectory and calculate the temporal fusion loss in the time domain As shown in formula (1): Among them, Δt is the frame difference between the last frame of the video sequence in the query and the first frame of the video sequence in the gallery. The frame number constant threshold T and the larger constant φ determine the temporal continuity of the frame difference Δt between the query and the gallery.

4. The method for video person re-identification based on complex underground space trajectory fusion according to claim 1 is characterized by: In step 3, in the spatiotemporal trajectory fusion feature, the spatial trajectory fusion is to consider the misalignment between the predicted trajectory and the frame number of the candidate video in the gallery, and calculate the spatial fusion loss N＝2,3,…,7, (2) in, p i It represents the Euclidean distance between the corresponding coordinates of the predicted trajectory sequence and the gallery candidate sequence; N represents the allowed deviation range between the predicted trajectory and the candidate video frame number.

5. The method for video person re-identification based on complex underground space trajectory fusion according to claim 1 is characterized by: In step 3, after obtaining the temporal fusion loss and spatial fusion loss, the limited fusion loss in the temporal and spatial domains of the jth video in gallery and the ith video in query_pred is calculated according to formula (3): Where N2 is the total number of video sequences in the gallery. According to formula (3), The smallest j value is obtained, so that the j-th video in the gallery is sent to the query_TP set for subsequent spatiotemporal trajectory fusion feature extraction.

6. The method for video person re-identification based on complex underground space trajectory fusion according to claim 1 is characterized by: In step 4, the new query set query_TP and the candidate set gallery extracted after the fusion of time and space trajectories are sent to the temporal complementary network TCLNet, and finally the temporal average pooling is used to aggregate the group features to obtain the final fused video feature vector; the temporal complementary network TCLNet uses the ResNet-50 network as the backbone network, and inserts the temporal saliency enhancement module TSB and the temporal saliency erasure module TSE into it; for T-frame continuous video, the backbone network inserted with TSB extracts features for each frame, marked as F={F1,F2,…,F T }, and then divide it into k groups, each containing N consecutive frame features C k ={F (k-1)N+1 ,…,F kN }, input each group into TSE, and use formula (4) to extract complementary features: c k =THIS(F (k-1)N+1 ,…,F kN )=TSE(C k ) (4) The cosine similarity is used to calculate the distance metric between the video feature vector A(x1, y1) in query_TP and the video feature vector B(x2, y2) in the candidate set gallery, as shown in formula (5): The videos in the gallery are sorted according to the distance metric, and the re-identification evaluation indicators mAP and Rank-k are calculated based on the sorting results, and the Rank-1 result is used as the video re-identification result.

Images