Video pedestrian re-identification method and system based on multi-frame interaction and multi-layer fusion

Abstract

The present application relates to a kind of video pedestrian re-identification method and system based on multi-frame interaction and multi-layer fusion, which comprises the following steps: 1) extract multiple pedestrian video sequences from data set, form the training data set consisting of video sequence;2) construct video pedestrian re-identification network model based on multi-frame interaction and multi-layer fusion, the video pedestrian re-identification network model mainly includes intra-frame spatial interaction module, multi-frame time sequence interaction module, multi-level local enhancement module, segmentation strategy module and multi-frame classification head combination module;By using training data set, video pedestrian re-identification network model is trained, and network model parameter is obtained;3) the video pedestrian re-identification network model trained is used to retrieve the video containing the pedestrian in test video, and video pedestrian re-identification is realized.The method and system are conducive to obtaining more stable, robust, accurate video pedestrian re-identification result.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a video pedestrian re-identification method and system based on multi-frame interaction and multi-layer fusion. Background Technology

[0002] The goal of pedestrian re-identification is to find images or video sequences of the same pedestrian across different cameras. Pedestrian re-identification has a wide range of applications, including suspect tracking and security monitoring. Pedestrian re-identification methods can be divided into image-based and video-based methods. Image-based methods, because they only use a single image to search for the target pedestrian, are sensitive to image quality issues, including occlusion, complex backgrounds, and changes in shooting angle. However, video-based pedestrian re-identification methods can extract spatiotemporal information about the pedestrian from multiple frames, resulting in richer feature representations and mitigating the impact of single-image quality issues.

[0003] Video person re-identification methods can be divided into two categories: sequence-based methods and frame-based methods. Sequence-based methods aim to directly learn spatiotemporal feature representations through 3D convolutional neural networks or recurrent neural networks. Frame-based methods are characterized by first extracting single-frame feature representations and then combining features from multiple single frames within the same video. Currently, some methods only explore intra-frame features in the single-frame feature extraction part, ignoring the temporal interactions between frames. Recently, some methods have achieved better performance by introducing inter-frame temporal interactions to explore contextual information. However, these methods mainly use convolutional neural networks as the backbone network. Although they can obtain detailed features of pedestrians from a local perspective, the limited perceptual range of convolutional neural networks means they lack the ability to acquire global dependencies.

[0004] By leveraging the advantages of self-attention mechanisms, the Transformer method can explore the relationships between spatial features globally, and this method has been introduced into the field of video person re-identification. For example, J. Wu et al. (J. Wu, L. He, W. Liu, Y. Yang, Z. Lei, T. Mei, and SZ Li, CAViT: Contextual alignment vision Transformer for video object re-identification, in Proceedings of the European Conference on Computer Vision, 2022, 549–566.) proposed the CAViT model, which uses a spatiotemporal transfer attention module for information interaction between adjacent frames. However, this method of temporal interaction between adjacent frames does not establish the contextual relationship between the currently processed frame and non-adjacent frames, ignoring the rich temporal features contained in non-adjacent frames. T. He et al. (T. He, X. Jin, X. Shen, J. Huang, Z. Chen, and XSHua, Dense interaction learning for video-based person re-identification, in Proceedings of the IEEE International Conference on...) ComputerVision, 2021, 1490–1501.) performs spatiotemporal interaction on features from all frames after pooling and compression, but this method ignores detailed information about pedestrians. T. Zhang et al. (T. Zhang, L. Wei, L. Xie, Z. Zhuang, Y. Zhang, B. Li, and Q. Tian, ​​Spatiotemporal Transformer for video-based person reidentification, arXiv preprint arXiv:2103.16469 (2021.) choose to establish the relationship between all frames in the video at once when performing spatiotemporal interaction, but this method easily ignores the relationship between pairs of frames.

[0005] Currently, some Transformer-based methods use multiple Transformer layers to extract high-level semantic features, but neglect the discovery of local detailed features of pedestrians. Some video pedestrian re-identification works attempt to combine Transformer and CNN, such as T. He et al. (T. He, X. Jin, X. Shen, J. Huang, Z. Chen, and XSHua, Dense interaction learning for video-based person re-identification, in Proceedings of the IEEE International Conference on Computer Vision, 2021, 1490–1501.) who proposed Dense Attention to fuse features from all CNN modules in the bottom and middle layers with features from the high-level self-attention modules to generate spatiotemporal features of different granularities. However, the fine-grained features used to discover information about pedestrian body parts and the low-level spatial features are not fully utilized. Summary of the Invention

[0006] The purpose of this invention is to provide a video pedestrian re-identification method and system based on multi-frame interaction and multi-layer fusion, which is beneficial to obtaining more stable, robust and accurate video pedestrian re-identification results.

[0007] To achieve the above objectives, the technical solution adopted by this invention is: a video pedestrian re-identification method based on multi-frame interaction and multi-layer fusion, comprising the following steps:

[0008] 1) Extract video sequences of multiple pedestrians from the dataset to form a training dataset consisting of video sequences;

[0009] 2) Construct a video person re-identification network model based on multi-frame interaction and multi-layer fusion. The video person re-identification network model mainly includes an intra-frame spatial interaction module, a multi-frame temporal interaction module, a multi-level local enhancement module, a segmentation strategy module, and a multi-frame classification head combination module. The video person re-identification network model is trained using a training dataset to obtain the network model parameters.

[0010] 3) The trained video pedestrian re-identification network model is used to retrieve test videos containing the pedestrian to achieve video pedestrian re-identification.

[0011] Further, in step 1), the dataset is the MARS video person re-identification dataset. Multiple video sequences of K pedestrians are extracted from the MARS video person re-identification dataset. Each pedestrian contains J video sequences, and each video sequence contains F frame images.

[0012] Further, in step 2), the video pedestrian re-identification network model includes a single-frame feature extraction part and a multi-frame combination part. The single-frame feature extraction part includes an intra-frame spatial interaction module, a multi-frame temporal interaction module, a multi-level local enhancement module, and a segmentation strategy module. The multi-frame combination part includes a multi-frame classification head combination module, a classification loss, and a triplet loss. The single-frame feature extraction part uses the intra-frame spatial interaction module as the backbone network, and sets up multi-frame temporal interaction modules and multi-level local enhancement modules in the shallow, middle, and high layers of the network model to discover long-range inter-frame interaction information and global and local information in different layers within the frame. A segmentation strategy module is set up in the high layers of the network model to discover pedestrian body part features. The working process of the video pedestrian re-identification network model is as follows:

[0013] A) First, each single-frame image in the video is processed. A sequence of blocks of a fixed size is obtained by cutting the single-frame image. Then, 2D block feature embeddings are generated through convolution and flattening operations. Classification header information, camera information embeddings, and position information embeddings are added to generate the input of the intra-frame spatial interaction module. Then, the intra-frame spatial interaction module extracts intra-frame features from the feature embeddings of multiple single-frame images in the input video based on the visual Transformer. After passing through several intra-frame spatial interaction modules, the output features of multiple single-frame images in the video are input into the multi-frame temporal interaction module in the shallow, middle, and high-level networks.

[0014] B) The multi-frame temporal interaction modules at each layer acquire the common features of the remaining frames of the same video for each single-frame feature, and then input the multiple single-frame features in the video into the multi-level local enhancement modules respectively;

[0015] C) The multi-level local enhancement modules at each layer perform multi-level feature fusion and local enhancement of pedestrian features for each single frame feature in the low, medium and high-level networks, and then input each single frame feature into the intra-frame spatial interaction module;

[0016] D) The high-level features output by the final intra-frame spatial interaction module are segmented into four types of single-frame features through the segmentation strategy module, including no segmentation, horizontal segmentation, vertical segmentation and block segmentation. Each segmented part is then input into the intra-frame spatial interaction module.

[0017] E) The multi-frame classification head combination module combines the features of each single frame image in the video to generate video sequence-level features for the video pedestrian re-identification retrieval task; the segmentation strategy features output by the multi-frame classification head combination module are subjected to triplet loss and classification loss calculation respectively, and the optimizer reduces the distance between positive samples, increases the distance between negative samples, and reduces the error of the model in classifying pedestrians.

[0018] Furthermore, in step B), F-1 multi-head inter-frame alignment attention operations and multilayer perceptron operations are used to perform temporal interaction between the current processing frame and all remaining frames in the video.

[0019] Using the frame following the current processing frame as the first interaction frame, perform multi-head inter-frame alignment attention operations and multilayer perceptron operations. The output features after the interaction continue to undergo multi-head inter-frame alignment attention operations and multilayer perceptron operations sequentially with subsequent frames until the interaction with the frame preceding the current processing frame is complete. The output feature is represented as S. r ;

[0020] First, obtain the input features, including the current processing frame. Remaining frames of the video to be interacted with Where n represents a frame in the video, r represents the r-th multi-frame temporal interaction module, and F represents the number of frames in the video; Q, K, V features are generated for each frame using a linear mapping.

[0021] The frame index sequence is represented as p n = [n, n+1, n+2, ..., n-1], representing the features obtained by interacting the current nth frame with a remaining frame in the video. Represented as:

[0022]

[0023]

[0024] in, Generated by layer normalization, subscripts n and p n [t+1] represents the nth frame and the pth frame. n The index of frame [t+1] This indicates that the current processing is of the nth frame and the pth frame. n Features obtained after the interaction of frames [t+1]; MIAA(·,·,·) represents the multi-head inter-frame alignment attention operation. LN denotes residual connections, LN denotes layer normalization, MLP(·) denotes multilayer perceptron operation, and t takes the value p. n The index values ​​corresponding to positions 0 to F-2 in the middle;

[0025] The multi-head inter-frame alignment attention operation involves Y heads, meaning that the inter-frame alignment attention operation it contains is performed Y times simultaneously. The features output by multiple inter-frame alignment attention operations are combined. The process of processing the multi-head intra-frame alignment operation between the current nth frame and a remaining frame within the same video is represented as:

[0026]

[0027]

[0028] Among them, W e It is a learnable parameter, head y Let y represent the y-th header in MIAA, and IAA(·,·,·) represent the inter-frame alignment attention operation;

[0029] The process of inter-frame alignment attention operation in multi-head intra-frame alignment is represented as follows:

[0030]

[0031] in, represents regularization, and Softmax(·) represents the normalization exponential function.

[0032] Further, in step C), the implementation method of the multi-level local enhancement module is as follows:

[0033] Using the multi-frame features S within the video that contain common pedestrian features obtained in step B), r For each single-frame feature, local enhancement operations are used to discover intra-frame local features. For the first multi-level local enhancement module in the network model, the feature M1 obtained by local enhancement operations is represented as:

[0034]

[0035] in, This indicates that the r-th multi-frame temporal interaction module retains the feature embedding of each frame after removing the classification header of each frame; Resh(·) represents the reshape operation; Conv(·) represents the local enhancement operation, which includes 3 layers of 2D convolution, batch normalization and activation functions; Indicates residual connection;

[0036] For the second and subsequent multi-level local augmentation modules within the network model, the input features are supplemented with the features output from the previous multi-level local augmentation module. First, the current layer features... and the previous layer multi-level local enhancement module feature M r-1 The layers are fused by addition, followed by local enhancement, and finally the output feature M is obtained by residual connection. r The feature M obtained by using fusion operation and local enhancement operation r Represented as:

[0037]

[0038] Among them, M r-1 This represents the output characteristics of the multi-level local enhancement module at layer r-1; Fus(·,·) represents the fusion operation.

[0039] Furthermore, in step E), the implementation method of the multi-frame classification head combination module is as follows:

[0040] The four classification headers obtained from multiple single frames include the classification header based on non-segmentation. Classification head based on horizontal segmentation Classification head based on vertical segmentation And block-based classification head Based on the segmentation strategy, the data were divided into four different groups. Each group contained classification heads for the same segmentation strategy across all frames in the video. Then, the video-level features of each group were averaged. The average features obtained for the corresponding segmentation strategy were then used as the basis for the calculation. The triplet loss is calculated; finally, multiple probability values ​​b are obtained using batch normalization and the classifier layer, and the classification loss is calculated.

[0041] The present invention also provides a video pedestrian re-identification system based on multi-frame interaction and multi-layer fusion, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-mentioned method steps.

[0042] Compared with the prior art, the present invention has the following beneficial effects: The method and system effectively utilize long-range interactive learning between frames and global and local information at different layers within the frame, which can obtain robust and discriminable pedestrian features. It can not only effectively solve the misalignment problem between different frames, but also alleviate noise problems such as pedestrian occlusion and complex background, thereby obtaining more stable, robust and accurate video pedestrian re-identification results. Attached Figure Description

[0043] Figure 1 This is an architecture diagram of the video pedestrian re-identification network model based on multi-frame interaction and multi-layer fusion in an embodiment of the present invention. Detailed Implementation

[0044] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0045] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0046] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0047] This embodiment provides a video pedestrian re-identification method based on multi-frame interaction and multi-layer fusion, including the following steps:

[0048] 1) Extract video sequences of multiple pedestrians from the dataset to form a training dataset composed of video sequences.

[0049] 2) Construct a video person re-identification network model based on multi-frame interaction and multi-layer fusion. This model mainly includes an intra-frame spatial interaction module, a multi-frame temporal interaction module, a multi-level local enhancement module, a segmentation strategy module, and a multi-frame classification head combination module. The network model parameters are obtained by training the model using a training dataset. In this embodiment, the architecture of the video person re-identification network model is as follows: Figure 1 As shown.

[0050] 3) The trained video pedestrian re-identification network model is used to retrieve test videos containing the pedestrian to achieve video pedestrian re-identification.

[0051] In step 1), the dataset is the MARS video person re-identification dataset. Multiple video sequences of K pedestrians are extracted from the MARS video person re-identification dataset. Each pedestrian contains J video sequences, and each video sequence contains F frame images.

[0052] In step 2), the video pedestrian re-identification network model includes a single-frame feature extraction part and a multi-frame combination part. The single-frame feature extraction part includes an intra-frame spatial interaction module, a multi-frame temporal interaction module, a multi-level local enhancement module, and a segmentation strategy module. The multi-frame combination part includes a multi-frame classification head combination module, classification loss, and triplet loss. The single-frame feature extraction part uses the intra-frame spatial interaction module as the backbone network, and sets up multi-frame temporal interaction modules and multi-level local enhancement modules in the shallow, middle, and high layers of the network model to discover long-range inter-frame interaction information and global and local information in different layers within the frame. A segmentation strategy module is set up in the high layers of the network model to discover pedestrian body part features. The working process of the video pedestrian re-identification network model is as follows:

[0053] A) First, each single-frame image in the video is processed by cutting the single-frame image to obtain a sequence of blocks of a fixed size. Then, 2D block feature embeddings are generated through convolution and flattening operations. Classification header information, camera information embeddings, and position information embeddings are added to generate the input of the intra-frame spatial interaction module. Then, the intra-frame spatial interaction module extracts intra-frame features from the feature embeddings of multiple single-frame images in the input video based on the visual Transformer. After passing through several intra-frame spatial interaction modules, the output features of multiple single-frame images in the video are input into the multi-frame temporal interaction module in the shallow, middle, and high-level networks.

[0054] B) The multi-frame temporal interaction modules at each layer acquire the common features of the remaining frames of the same video for each single-frame feature, and then input the multiple single-frame features in the video into the multi-level local enhancement modules respectively.

[0055] In step B), F-1 multi-head inter-frame alignment attention operations and multilayer perceptron operations are used to perform temporal interaction between the current processing frame and all remaining frames in the video.

[0056] Using the frame following the current processing frame as the first interaction frame, perform multi-head inter-frame alignment attention operations and multilayer perceptron operations. The output features after the interaction continue to undergo multi-head inter-frame alignment attention operations and multilayer perceptron operations sequentially with subsequent frames until the interaction with the frame preceding the current processing frame is complete. The output feature is represented as S. r .

[0057] First, obtain the input features, including the current processing frame. Remaining frames of the video to be interacted with Where n represents a frame in the video, r represents the r-th multi-frame temporal interaction module, and F represents the number of frames in the video; Q, K, V features are generated for each frame using a linear mapping.

[0058] The frame index sequence is represented as p n = [n, n+1, n+2, ..., n-1], where n represents a frame in the video, and the features obtained by interacting the current nth frame with the remaining frames in the video. Represented as:

[0059]

[0060]

[0061] in, Generated by layer normalization, subscripts n and p n [t+1] represents the nth frame and the pth frame. n The index of frame [t+1] This indicates that the current processing is of the nth frame and the pth frame. nFeatures obtained after the interaction of frames [t+1]; MIAA(·,·,·) represents the multi-head inter-frame alignment attention operation. LN denotes residual connections, LN denotes layer normalization, MLP(·) denotes multilayer perceptron operation, and t takes the value p. n The index values ​​corresponding to positions 0 to F-2.

[0062] The multi-head inter-frame alignment attention operation involves Y heads, meaning that the inter-frame alignment attention operation it contains is performed Y times simultaneously. The features output by multiple inter-frame alignment attention operations are combined. The process of processing the multi-head intra-frame alignment operation between the current nth frame and a remaining frame within the same video is represented as:

[0063]

[0064]

[0065] Among them, W e It is a learnable parameter, head y Let y represent the y-th header in MIAA, and IAA(·,·,·) represent the inter-frame alignment attention operation.

[0066] The process of inter-frame alignment attention operation in multi-head intra-frame alignment is represented as follows:

[0067]

[0068] in, represents regularization, and Softmax(·) represents the normalization exponential function.

[0069] C) The multi-level local enhancement modules at each layer perform multi-level feature fusion and local enhancement of pedestrian features for each single frame feature in the low, medium and high-level networks, and then input each single frame feature into the intra-frame spatial interaction module.

[0070] In step C), the implementation method of the multi-level local enhancement module is as follows:

[0071] Using the multi-frame features S within the video that contain common pedestrian features obtained in step B), r For each single-frame feature, local enhancement operations are used to discover intra-frame local features. For the first multi-level local enhancement module in the network model, the feature M1 obtained by local enhancement operations is represented as:

[0072]

[0073] in, This indicates that the r-th multi-frame temporal interaction module retains the feature embedding of each frame after removing the classification header of each frame; Resh(·) represents the reshape operation; Conv(·) represents the local enhancement operation, which includes 3 layers of 2D convolution, batch normalization and activation functions; Indicates residual connection;

[0074] For the second and subsequent multi-level local augmentation modules in the network model (r≥2), the input features are supplemented with the features output by the previous multi-level local augmentation module. First, the current layer features... and the previous layer multi-level local enhancement module feature M r-1 The layers are fused by addition, followed by local enhancement, and finally the output feature M is obtained by residual connection. r The feature M obtained by using fusion operation and local enhancement operation r Represented as:

[0075]

[0076] Among them, M r-1 This represents the output characteristics of the multi-level local enhancement module at layer r-1; Fus(·,·) represents the fusion operation.

[0077] D) The high-level features output by the final intra-frame spatial interaction module are segmented into single-frame features using the segmentation strategy module in four ways: no segmentation, horizontal segmentation, vertical segmentation, and block segmentation. Each segmented part is then input into the intra-frame spatial interaction module.

[0078] E) The multi-frame classification head combination module combines the features of each single frame image in the video to generate video sequence-level features for the video pedestrian re-identification retrieval task; the segmentation strategy features output by the multi-frame classification head combination module are subjected to triplet loss and classification loss calculation respectively, and the optimizer reduces the distance between positive samples, increases the distance between negative samples, and reduces the error of the model in classifying pedestrians.

[0079] In step E), the implementation method of the multi-frame classification head combination module is as follows:

[0080] The four classification headers obtained from multiple single frames include the classification header based on non-segmentation. Classification head based on horizontal segmentation Classification head based on vertical segmentation And block-based classification head Based on the segmentation strategy, the data were divided into four different groups. Each group contained classification heads for the same segmentation strategy across all frames in the video. Then, the video-level features of each group were averaged. The average features obtained for the corresponding segmentation strategy were then used as the basis for the calculation. The triplet loss is calculated; finally, multiple probability values ​​b are obtained using batch normalization and the classifier layer, and the classification loss is calculated.

[0081] This embodiment also provides a video pedestrian re-identification system based on multi-frame interaction and multi-layer fusion, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-mentioned method steps.

[0082] In this embodiment, the MARS dataset is used for comparative verification under the setting of searching gallery video sequences from pedestrian query video sequences. Table 1 shows the comparison results of the proposed method with other video pedestrian re-identification methods on the MARS dataset. As can be seen from Table 1, the proposed method has higher accuracy and robustness compared with other video pedestrian re-identification methods, specifically showing the best Rank-1 and mAP.

[0083] Table 1

[0084] method Rank-1 (%) mAP(%) This invention 91.09 87.55 SINet 91.00 86.20 CAViT 90.80 87.20 SGMN 90.76 85.38 ASANet-a-PMI 90.60 86.00 STMN 90.50 84.50 STRF 90.30 86.10 BiCnet-TKS 90.20 86.00 SSN3D 90.10 86.20 SGWCNN 90.00 85.70 AGW+ 87.60 83.00

[0085] In Table 1, SINet corresponds to the method proposed by S. Bai et al. (S. Bai, B. Ma, H. Chang, R. Huang, and X. Chen, Salientto-broad transition for video person re-identification, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022, pp. 7339–7348.)

[0086] CAViT corresponds to the method proposed by J. Wu et al. (J. Wu, L. He, W. Liu, Y. Yang, Z. Lei, T. Mei, and S.Z. Li, CAViT: Contextual alignment vision Transformer for video object re-identification, in Proceedings of the European Conference on Computer Vision, 2022, pp. 549–566.)

[0087] SGMN corresponds to the method proposed by C. Chen et al. (C. Chen, M. Ye, M. Qi, J. Wu, Y. Liu, and J. Jiang, Saliency and granularity: discovering temporal coherence for video-based person re-identification, IEEE Transactions on Circuits and Systems for Video Technology. 32(9)(2022)6100–6112.)

[0088] ASANet-a-PMI corresponds to the method proposed by T. Chai et al. (T. Chai, Z. Chen, A. Li, J. Chen, X. Mei, and Y. Wang, Video person re-identification using attribute-enhanced features, IEEE Transactions on Circuits and Systems for Video Technology 32(11)(2022)7951–7966.)

[0089] STMN corresponds to the method proposed by C. Eom et al. (C. Eom, G. Lee, J. Lee, and B. Ham, Video-based person re-identification with spatial and temporal memory networks, in Proceedings of the IEEE International Conference on Computer Vision, 2021, pp. 12036–12045.)

[0090] STRF corresponds to the method proposed by A. Aich et al. (A. Aich, M. Zheng, S. Karanam, T. Chen, AKR Howdhury, and Z. Wu, Spatio-temporal representation factorization for video-based person re-identification, in Proceedings of the IEEE International Conference on Computer Vision, 2021, pp. 152–162.)

[0091] BiCnet-TKS corresponds to the method proposed by R. Hou et al. (R. Hou, H. Chang, B. Ma, R. Huang, and S. Shan, BiCnet-TKS: Learning efficient spatial-temporal representation for video person re-identification, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021, pp. 2014–2023.)

[0092] SSN3D corresponds to the method proposed by X. Jiang et al.

[0093] SGWCNN corresponds to the method proposed by Y. Yao et al. (Y. Yao, X. Jiang, H. Fujita, and Z. Fang, Asparse graph wavelet convolution neural network for video-based person re-identification, Pattern Recognition. 129(2022)108708.)

[0094] AGW+ corresponds to the method proposed by M. Ye et al. (M. Ye, J. Shen, G. Lin, T. Xiang, L. Shao, and S.C. Hoi, Deep learning for person re-identification: a survey and outlook, IEEE Transactions on Pattern Analysis and Machine Intelligence 44(6)(2022)2872–2893.)

[0095] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0097] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0098] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0099] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A video pedestrian re-identification method based on multi-frame interaction and multi-layer fusion, characterized in that, The method comprises the following steps: 1) extracting a plurality of pedestrian video sequences from a data set to form a training data set composed of video sequences; 2) constructing a video pedestrian re-identification network model based on multi-frame interaction and multi-layer fusion, wherein the video pedestrian re-identification network model mainly comprises an intra-frame spatial interaction module, a multi-frame time sequence interaction module, a multi-level local enhancement module, a segmentation strategy module and a multi-frame classification head combination module; training the video pedestrian re-identification network model using the training data set to obtain network model parameters; 3) using the trained video pedestrian re-identification network model to search for videos containing the pedestrian in the test video to realize video pedestrian re-identification; The working process of the video pedestrian re-identification network model is as follows: A) first, process each single frame picture in the video, obtain a sequence of fixed size blocks by cutting the single frame picture, generate 2D block feature embedding through convolution operation and flattening operation, add classification head information, camera information embedding and position information embedding to generate the input of the intra-frame spatial interaction module, then perform intra-frame feature extraction on the feature embedding of the input multiple single frame pictures in the video based on visual Transformer through the intra-frame spatial interaction module, and after passing through several intra-frame spatial interaction modules, input the output single frame features in the video into the multi-frame time sequence interaction module at the shallow, middle and high layer networks; B) each layer of the multi-frame time sequence interaction module acquires the common features of the remaining frames of the same video based on each single frame feature, and then inputs the multiple single frame features in the video into the multi-level local enhancement module; C) each layer of the multi-level local enhancement module performs multi-level feature fusion and local enhancement of the pedestrian features on each single frame feature at the low, middle and high layer networks, and then inputs each single frame feature into the intra-frame spatial interaction module; D) the high layer features output by the last intra-frame spatial interaction module are input into the segmentation strategy module to segment the single frame features in four segmentation modes including no segmentation, horizontal segmentation, vertical segmentation and block segmentation, and each segmented part is input into the intra-frame spatial interaction module; E) the multi-frame classification head combination module combines the features of each single frame picture in the video to generate video sequence level features for the video pedestrian re-identification search task; each segmented strategy feature output by the multi-frame classification head combination module is subjected to triple loss and classification loss calculation, and an optimizer is used to reduce the distance between positive samples, increase the distance between negative samples, and reduce the error of model classification pedestrians; In step B), the current processing frame is interacted with all the remaining frames in the video using - 1 multi-head inter-frame alignment attention operation and a multi-layer perceptron operation for the temporal interaction of the current processing frame with all the remaining frames in the video; The output feature after the interaction is sequentially subjected to the multi-head inter-frame alignment attention operation and the multi-layer perception operation with the next frame, until the interaction with the previous frame of the current processing frame is completed, and the output feature representation is ; First, input features are acquired, including the current processing frame and the remaining frames of the video to be interacted , wherein represents a frame in the video, represents the th multi-frame time sequence interaction module, represents the number of frames in the video; a linear mapping is used to generate features for each frame; The frame index sequence is represented as For the current processing of the n-th frame and the features obtained by interacting with some frame remaining in the video is represented as: wherein, generated by layer normalization, subscript n and denotes the frame and the index of the frame, denotes the current processing frame and the frame after interaction; denotes the multi-head inter-frame alignment attention operation, denotes the residual connection, denotes the layer normalization, denotes the multi-layer perception operation, takes the index value corresponding to the position of 0 to -2; The multi-head inter-frame alignment attention operation includes one head, i.e., the inter-frame alignment attention operation included therein is performed simultaneously times, and combines features output by the multiple inter-frame alignment attention operations, the current processing is the frame and the multi-head intra-frame alignment operation of the remaining frame in the same video. wherein, are learnable parameters, denotes the th head in MIAA, denotes an inter-aligned attention operation; The process of the inter-frame alignment attention operation in the multi-head intra-frame alignment operation is represented as: wherein denotes a regularization, denotes a normalized exponential function.

2. The multi-frame interaction and multi-layer fusion based video pedestrian re-identification method according to claim 1, characterized in that, In step 1), the data set is a MARS video pedestrian re-identification data set, and a plurality of video sequences of a pedestrian are extracted from the MARS video pedestrian re-identification data set, each pedestrian containing video sequences, each video sequence containing frame pictures. In step 1), the data set is a MARS video pedestrian re-identification data set, and a plurality of video sequences of a pedestrian are extracted from the MARS video pedestrian re-identification data set, each pedestrian containing video sequences, each video sequence containing frame pictures. In step 1), the data set is a MARS video pedestrian re-identification data 3. The method of claim 1, wherein the method further comprises: In step 2), the video pedestrian re-identification network model comprises a single-frame feature extraction part and a multi-frame combination part, the single-frame feature extraction part comprises an intra-frame spatial interaction module, a multi-frame time sequence interaction module, a multi-level local enhancement module and a segmentation strategy module, and the multi-frame combination part comprises a multi-frame classification head combination module, a classification loss and a triplet loss; the single-frame feature extraction part takes the intra-frame spatial interaction module as a backbone network, and the multi-frame time sequence interaction module and the multi-level local enhancement module are arranged in the shallow layer, the middle layer and the high layer of the network model, so as to explore the long-range interaction information between frames and the global and local information of different layers in the frame, and the segmentation strategy module is arranged in the high layer of the network model to explore the pedestrian body part features.

4. The method of claim 3, wherein the method further comprises: In step C), the implementation method of the multi-level local enhancement module is as follows: Using the multi-frame features within the video containing common pedestrian features obtained in step B) For each single-frame feature, local enhancement operations are used to discover intra-frame local features. For the first multi-level local enhancement module in the network model, the features obtained by the local enhancement operations are used... Represented as: in, Indicates the first The multi-frame temporal interaction module retains the feature embedding of each frame after removing the classification header of each frame. This indicates a reshape operation; This represents a local enhancement operation, which consists of three layers of 2D convolution, batch normalization, and activation functions. Indicates residual connection; For the second backward multi-level local enhancement module in the network model, the input features are added with the features output by the previous multi-level local enhancement module. First, the current layer features and the features of the previous multi-level local enhancement module are fused by addition, and then local enhancement operation is performed, and finally, the output features are obtained by using residual connection , and the features obtained by using the fusion operation and the local enhancement operation are represented as: ​ wherein, denotes the output features of the multi-level local enhancement module of layer -1; denotes a fusion operation.

5. The method of claim 3, wherein the method further comprises: In step E), the implementation method of the multi-frame classification head combination module is as follows: The four classification heads of the obtained multiple single frames, including a classification head based on non-division , a classification head based on horizontal division , a classification head based on vertical division , and a classification head based on block division , are placed into four different groups according to a division strategy, each group containing classification heads of the same division strategy in all frames in the video, then the video-level features of each group are respectively averaged, and average features corresponding to the division strategy are obtained , and triplet loss calculation is performed; finally, batch normalization and a classifier layer are used to obtain multiple probability values , and classification loss calculation is performed.

6. A video pedestrian re-identification system based on multi-frame interaction and multi-layer fusion, characterized in that, The computer program instructions are stored in the memory and can be executed by the processor, and when the processor executes the computer program instructions, the method of any one of claims 1-5 can be implemented.

Images