A video pedestrian re-identification method based on multi-scale resolution alignment
By optimizing multi-branch networks and loss functions, the problem of insufficient utilization of multi-scale information in existing video pedestrian re-identification methods is solved, achieving higher accuracy and robustness.
Patent Information
- Application Number
- CN202310407317.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-14
- Publication Date
- 2025-12-02
- Estimated Expiration
- 2043-04-14
AI Technical Summary
Existing video pedestrian re-identification methods ignore potential information at different scales when processing multi-scale videos, resulting in low re-identification accuracy.
Design a hierarchical multi-branch network to achieve feature resolution matching and information exchange through multi-scale resolution alignment. Utilize effective information from multi-scale video clips, generate video-level feature vectors using global average pooling layers, and optimize the network through cross-entropy loss, triplet loss, and mutual information loss.
It effectively improves the accuracy of pedestrian re-identification in videos and achieves robustness in cross-scale information interaction and feature representation.
Smart Images
Figure CN116416649B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of pattern recognition and image retrieval, and more specifically relates to a video pedestrian re-identification method based on multi-scale resolution alignment. Background Technology
[0002] Video-based person re-identification aims to retrieve video clips of the same key target from multiple cameras. Video person re-identification technology plays a crucial role in an increasing number of surveillance-related applications. In recent years, image-based person re-identification has made significant progress in both methods and datasets. Most related research focuses on learning discriminative features from a single image. However, the performance of image-based re-identification is limited when the key target is occluded or undergoes significant visual changes. In contrast, video-based re-identification can utilize richer appearance and temporal information by using multi-frame video sequences as input.
[0003] Thanks to the development of deep convolutional neural networks, video re-identification methods have made exciting progress. However, most existing video-based models only process the appearance information of key targets at a single spatial scale. This ignores the fact that pedestrian videos are acquired at multiple scales.
[0004] Existing multi-scale feature representation learning methods can be divided into two categories:
[0005] (1) Learn features using multi-scale convolutional flow.
[0006] This method resizes all input images to a fixed resolution and feeds them into the same network with filters at different resolution levels. However, this design ignores potentially useful information at other different scales.
[0007] (2) Learn features from multi-scale inputs.
[0008] This method employs a multi-branch network to learn features from inputs at different scales. The features learned by such methods attempt to be scale-invariant, meaning that scale variations in the input object are offset by scale variations in the features. However, achieving cross-scale information exchange is not easy because different scale branches have different feature space resolutions within the same network stage. Summary of the Invention
[0009] To address the aforementioned issues, this invention designs a hierarchical multi-branch network that enables feature resolution matching across scales, facilitating cross-branch information exchange and fully utilizing effective information from pedestrian video clips at different spatial scales, thereby effectively improving the accuracy of video re-identification.
[0010] To achieve the above objectives, the present invention provides the following technical solution:
[0011] A video pedestrian re-identification method based on multi-scale resolution alignment includes the following steps:
[0012] S1. Construct multi-scale pedestrian video clips;
[0013] S2. Input video clips of different scales into a hierarchical multi-branch network to extract features;
[0014] S3. Connect the feature maps output from each scale branch in the time dimension, and use a global average pooling layer to generate the final video-level feature vector.
[0015] S4. Construct cross-entropy loss, triplet loss, and mutual information loss to optimize multi-branch networks;
[0016] S5. Repeat steps S1-S4 until the model converges;
[0017] S6. After training, feature vectors are extracted from pedestrian video clips at multiple scales using a multi-branch network. The Euclidean distance between the feature vectors of the pedestrian video clip to be retrieved and the feature vectors of the candidate pedestrian video clips is calculated. Finally, the pedestrian re-identification results are obtained by sorting them in ascending order of Euclidean distance.
[0018] Furthermore, the specific process of step S1 is as follows:
[0019] S11. Randomly extract 4 frames from each video sequence to form an original input video segment;
[0020] S12. Adjust the image in the input video clip to a fixed size;
[0021] S13. Construct multi-scale pedestrian video segments based on the original input video segments. Where V s This represents the downsampled version of the original input video segment V, and s represents the index number of the scale branch.
[0022] Furthermore, step S2 is as follows:
[0023] S21. Construct a multi-branch network, which includes three feedforward subnetwork branches. Among them B s Responsible for processing the corresponding video clip V sThe backbone network of B0 consists of four network stages of ResNet-50, namely Stage 1 to Stage 4, with each network stage consisting of multiple residual blocks. The backbone network of B1 consists of the last three network stages of ResNet-50, namely Stage 2 to Stage 4, and the backbone network of B2 consists of the last two network stages of ResNet-50, namely Stage 3 to Stage 4.
[0024] S22. In the second and third network stages of the multi-branch network, two spatial information interaction modules and one multi-scale temporal information interaction module are integrated. The specific calculation process of the spatial information interaction module is as follows:
[0025]
[0026]
[0027]
[0028]
[0029] in, and This represents the convolution operation. and X represents the matrix dimension transformation operation. si E1, E2, and E3 represent the video feature maps input to the spatial information module, while E3, E4, E5, and E6 represent the feature maps generated in the intermediate stages of the spatial information module. si The feature map output by the spatial information module represents the specific calculation process of the multi-scale temporal information interaction module.
[0030] X msti =[X0, X1, X2]
[0031]
[0032]
[0033]
[0034]
[0035] in, and This represents the convolution operation. and This represents a matrix dimension transformation operation, where X0, X1, and X2 represent feature maps of the same spatial size in the three scale branches, and [·] represents a connection in the time dimension. mstiZ1, Z2, and Z3 represent the video feature maps input to the multi-scale temporal information interaction module, respectively, and Z3 represent the feature maps generated in the middle of the multi-scale temporal information interaction module. msti This represents the feature map output by the spatial information module;
[0036] S23. Input multi-scale pedestrian video clips into a multi-branch network to extract features. The specific process is as follows:
[0037]
[0038]
[0039] Among them, Conv s Indicates that it comes from the corresponding branch B s convolutional blocks, Indicates branch B s Mid-network stage Stade s+l The extracted intermediate video feature map, where l represents the index number of the feature map.
[0040] Furthermore, in step S2, the feature maps of branches at different scales have the same spatial size in the same network stage.
[0041] Furthermore, step S3 is as follows:
[0042] S31. Connect the feature maps output from the three scale branches along the time dimension:
[0043]
[0044] in, This represents the output feature map of the first branch. This represents the output feature map of the second branch. F represents the output feature map of the third branch. all It is a tensor of size C×12×H×W, where C represents the number of channels of the tensor, H represents the height of the tensor, and W represents the width of the tensor;
[0045] S32. Generate the final video-level feature vector using a global average pooling layer:
[0046]
[0047] Among them, F output f represents the final video-level feature vector. i，j，k F represents all The vector at position (j, k, i), where j, k, and i represent the X, Y, and Z values of the three-dimensional coordinates, respectively.
[0048] Furthermore, in step S4, a loss function is constructed to optimize the multi-branch network. The loss function is as follows:
[0049]
[0050] in, The loss function representing the overall loss. Represents the cross-entropy loss function. Represents the triplet loss function. Let λ1 and λ2 represent the mutual information loss function, and let λ1 and λ2 represent the adjustment parameters.
[0051] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0052] (1) This invention proposes a video-based pedestrian re-identification framework, which learns discriminative and robust video feature representations from input video segments with different spatial resolutions, effectively improving the accuracy of video pedestrian re-identification.
[0053] (2) This invention proposes a hierarchical multi-branch network that achieves feature size matching between scales, thereby enabling information interaction between branches.
[0054] (3) The present invention designs a spatial information interaction module and a multi-scale temporal information interaction module. These two modules realize spatial and temporal information interaction between multiple specific scale branches. Attached Figure Description
[0055] Figure 1 This is a flowchart of the method of the present invention;
[0056] Figure 2 This is a schematic diagram of the multi-branch network of the present invention;
[0057] Figure 3 This is a schematic diagram of the spatial information interaction module and the multi-scale temporal information interaction module of the present invention. Detailed Implementation
[0058] The technical solutions provided by the present invention will be described in detail below with reference to specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.
[0059] This embodiment provides a video pedestrian re-identification method based on multi-scale resolution alignment, the process of which is as follows: Figure 1 As shown, the specific implementation of this method includes the following steps:
[0060] S1. Construct multi-scale pedestrian video clips, specifically including the following sub-steps:
[0061] S11. Randomly extract 4 frames from each video sequence captured by the camera to form an original input video segment.
[0062] S12. Adjust the resolution of all frame images to 256 pixels × 128 pixels.
[0063] S13. Construct multi-scale pedestrian video segments based on the original input video segments. Among them, V s This represents the downsampled version of the original input video segment V, where s represents the index number of the scale branch. The image resolution in V0 is 256 pixels × 128 pixels, the image resolution in V1 is 128 pixels × 64 pixels, and the image resolution in V2 is 64 pixels × 32 pixels.
[0064] S2. Input video clips of different scales into a hierarchical multi-branch network to extract features, specifically including the following sub-steps:
[0065] S21. Construct a multi-branch network, which includes three feedforward subnetwork branches. Among them B s Responsible for processing the corresponding video clip V s The backbone network of B0 consists of four network stages of ResNet-50, namely Stage 1 to Stage 4, with each network stage composed of multiple residual blocks. The backbone network of B1 consists of the last three network stages of ResNet-50, namely Stage 2 to Stage 4, and the backbone network of B2 consists of the last two network stages of ResNet-50, namely Stage 3 to Stage 4.
[0066] S22, Reference Figure 2 and Figure 3 ( Figure 2 The diagram shows the architecture of multiple interaction modules in the third network stage (the architecture of the second network stage in the branch network of this invention is the same as that of the third network stage). Two spatial information interaction modules and one multi-scale temporal information interaction module are integrated in the second and third network stages of the multi-branch network. The specific calculation process of the spatial information interaction module is as follows:
[0067]
[0068]
[0069]
[0070]
[0071] in, and This represents the convolution operation. and X represents the matrix dimension transformation operation. si E1, E2, and E3 represent the video feature maps input to the spatial information module, while E3, E4, E5, and E6 represent the feature maps generated in the intermediate stages of the spatial information module. si This represents the feature map output by the spatial information module. The specific calculation process of the multi-scale temporal information interaction module is as follows:
[0072] X msti =[X0, X1, X2]
[0073]
[0074]
[0075]
[0076]
[0077] in, and This represents the convolution operation. and This represents a matrix dimension transformation operation, where X0, X1, and X2 represent feature maps of the same spatial size in the three scale branches, and [·] represents a connection in the time dimension. msti Z1, Z2, and Z3 represent the video feature maps input to the multi-scale temporal information interaction module, respectively, and Z3 represent the feature maps generated in the middle of the multi-scale temporal information interaction module. msti This represents the feature map output by the spatial information module.
[0078] In this invention, feature maps of branches with different scales in the same network stage have the same spatial size, such as the feature map output by the third network stage in branch 0 and the feature map output by the third network stage in branch 1.
[0079] S23. Input multi-scale pedestrian video clips into a multi-branch network to extract features. The specific process is as follows:
[0080]
[0081]
[0082] Among them, Conv s Indicates that it comes from the corresponding branch B s convolutional blocks, Indicates branch B s Mid-network stage s+l The extracted intermediate video feature map, where l represents the index number of the feature map.
[0083] S3. Concatenate the feature maps output from each scale branch along the time dimension, and use a global average pooling layer to generate the final video-level feature vector. This includes the following sub-steps:
[0084] S31. Connect the feature maps output from the three scale branches along the time dimension:
[0085]
[0086] in, This represents the output feature map of the first branch. This represents the output feature map of the second branch. F represents the output feature map of the third branch. all It is a tensor of size C×12×H×W, where C represents the number of channels of the tensor, H represents the height of the tensor, and W represents the width of the tensor;
[0087] S32. Generate the final video-level feature vector using a global average pooling layer:
[0088]
[0089] Among them, F output f represents the final video-level feature vector. i，j，k F represents all The vector at position (j, k, i), where j, k, and i represent the X, Y, and Z values of the three-dimensional coordinates, respectively;
[0090] S4. Construct a loss function to optimize the multi-branch network;
[0091]
[0092] in, The loss function representing the overall loss. Represents the cross-entropy loss function. Represents the triplet loss function. Let λ1 and λ2 represent the mutual information loss function, and let λ1 and λ2 represent the adjustment parameters.
[0093] S5. Repeat steps S1-S4 until the model converges;
[0094] S6. After training, feature vectors are extracted from pedestrian video clips at multiple scales using a multi-branch network. The Euclidean distance between the feature vectors of the pedestrian video clip to be retrieved and the feature vectors of the candidate pedestrian video clips is calculated. Finally, the pedestrian re-identification results are obtained by sorting them in ascending order of Euclidean distance.
[0095] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention.
Claims
1. A video pedestrian re-identification method based on multi-scale resolution alignment, characterized in that, Includes the following steps: S1. Construct multi-scale pedestrian video clips; S2. Input video clips of different scales into a hierarchical multi-branch network to extract features; The process is as follows: S21. Construct a multi-branch network, which includes three feedforward subnetwork branches. Among them B s The backbone network of B0, which is responsible for processing the corresponding video segment Vs, consists of four network stages of ResNet-50, namely Stage 1 to Stage 4. Each network stage consists of multiple residual blocks. The backbone network of B1 consists of the last three network stages of ResNet-50, namely Stage 2 to Stage 4. The backbone network of B2 consists of the last two network stages of ResNet-50, namely Stage 3 to Stage 4. S22. In the second and third network stages of the multi-branch network, two spatial information interaction modules and one multi-scale temporal information interaction module are integrated. The specific calculation process of the spatial information interaction module is as follows: in, and This represents the convolution operation. and X represents the matrix dimension transformation operation. si E1, E2, and E3 represent the video feature maps input to the spatial information module, while E3, E4, E5, and E6 represent the feature maps generated in the intermediate stages of the spatial information module. si The feature map output by the spatial information module represents the specific calculation process of the multi-scale temporal information interaction module. X msti [X0,X1,X2] in, and This represents the convolution operation. and This represents a matrix dimension transformation operation, where X0, X1, and X2 represent feature maps of the same spatial size in the three scale branches, and [·] represents a connection in the time dimension. msti Z1, Z2, and Z3 represent the video feature maps input to the multi-scale temporal information interaction module, respectively, and Z3 represent the feature maps generated in the middle of the multi-scale temporal information interaction module. msti This represents the feature map output by the spatial information module; S23. Input multi-scale pedestrian video clips into a multi-branch network to extract features. The specific process is as follows: Among them, Conv s Indicates that it comes from the corresponding branch B s convolutional blocks, Indicates branch B s Mid-network stage s+l The extracted intermediate video feature map, where l represents the index number of the feature map; S3. Connect the feature maps output from each scale branch in the time dimension, and use a global average pooling layer to generate the final video-level feature vector. S4. Construct cross-entropy loss, triplet loss, and mutual information loss to optimize multi-branch networks; S5. Repeat steps S1-S4 until the model converges; S6. After training, feature vectors are extracted from pedestrian video clips at multiple scales using a multi-branch network. The Euclidean distance between the feature vectors of the pedestrian video clip to be retrieved and the feature vectors of the candidate pedestrian video clips is calculated. Finally, the pedestrian re-identification results are obtained by sorting them in ascending order of Euclidean distance.
2. The video pedestrian re-identification method based on multi-scale resolution alignment according to claim 1, characterized in that, The specific process of step S1 is as follows: S11. Randomly extract 4 frames from each video sequence to form an original input video segment; S12. Adjust the image in the input video clip to a fixed size; S13. Construct multi-scale pedestrian video segments based on the original input video segments. Where V s This represents the downsampled version of the original input video segment V, and s represents the index number of the scale branch.
3. The video pedestrian re-identification method based on multi-scale resolution alignment according to claim 1, characterized in that, In step S2, the feature maps of branches at different scales have the same spatial size in the same network stage.
4. The video pedestrian re-identification method based on multi-scale resolution alignment according to claim 1, characterized in that, The process of step S3 is as follows: S31. Connect the feature maps output from the three scale branches along the time dimension: in, This represents the output feature map of the first branch. This represents the output feature map of the second branch. F represents the output feature map of the third branch. all It is a tensor of size C×12×H×W, where C represents the number of channels of the tensor, H represents the height of the tensor, and W represents the width of the tensor; S32. Generate the final video-level feature vector using a global average pooling layer: Among them, F output f represents the final video-level feature vector. i,j,k F represents all The vector at position (j,k,i), where j, k, and i represent the X, Y, and Z values of the three-dimensional coordinates, respectively.
5. The video pedestrian re-identification method based on multi-scale resolution alignment according to claim 1, characterized in that, In step S4, a loss function is constructed to optimize the multi-branch network. The loss function is as follows: in, The loss function representing the overall loss. Represents the cross-entropy loss function. Represents the triplet loss function. Let λ1 and λ2 represent the mutual information loss function, and let λ1 and λ2 represent the adjustment parameters.