Video pedestrian re-identification method based on multi-scale fusion

Abstract

This invention provides a video pedestrian re-identification method based on multi-scale fusion, comprising: constructing a 3DCABlock architecture for shallow pedestrian feature extraction, outputting a global shallow feature map of pedestrians; constructing a 3DISBlock architecture for deep pedestrian feature extraction, obtaining a pedestrian depth representation map with strong discriminativeness; constructing an Arattention module for pedestrian feature correction, obtaining a more effective pedestrian depth feature map; and constructing an ATC module for pedestrian feature expansion, obtaining the final pedestrian feature information. This invention can extract richer pedestrian semantic features and multi-frame spatiotemporal features at multiple scales, and effectively retrieve pedestrian identities with a relatively small number of parameters, solving the problems of overly complex network models and poor real-time performance in multi-frame feature extraction for video pedestrian re-identification.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer vision, and particularly relates to a video person re-identification method based on multi-scale fusion. BACKGROUND

[0002] With the rapid development of artificial intelligence and deep learning technology, the urban security system is accelerating towards the direction of intelligent evolution. Person re-identification (Reid) technology, as a core means of cross-camera tracking of specific targets, has shown great application value in public security fields such as cracking down on criminal activities and searching for lost people. In the criminal investigation scene, this technology can realize the cross-regional track of criminals through the accurate matching of multi-dimensional biological characteristics of suspects; in the aspect of people's livelihood guarantee, it can carry out real-time feature comparison in the city's million-camera network based on the historical image data of lost people, and provide key technical support for finding the elderly with dementia, abducted children and other emergency events. The current mainstream static image Reid method relies on 2D convolution network to extract single-frame features, although it plays a role in basic identity recognition, but in actual security scenes, it often causes insufficient stability of feature representation due to target environmental light interference and cross-camera perspective differences and other problems. Compared with this, the dynamic video Reid technology based on 3D neural network can effectively capture valuable deep-level identity markers such as individual motion patterns and fine-grained features by analyzing the spatiotemporal information such as gait features and motion trajectories between consecutive frames, significantly improving the recognition robustness of suspects and lost people in complex situations.

[0003] The spatiotemporal modeling capability of video Reid technology brings a double breakthrough to urban security: in the aspect of crime early warning, by constructing a spatiotemporal trajectory graph across cameras, it can realize real-time early warning of potential criminal behaviors such as abnormal wandering and tailing tracking; in the field of personnel search, combined with the time and space analysis of geographic information system, it can quickly delineate the possible activity area of lost people. However, the existing technology still faces two major challenges: the static method lacks temporal information, resulting in insufficient feature stability, while the dynamic method has high computational complexity, affecting real-time performance. This technical bottleneck restricts the construction of city-level visual perception network, and it is urgent to balance the efficiency and accuracy of spatiotemporal feature fusion.

[0004] Pedestrian re-identification method based on static single-frame image: Zhang Lei et al. proposed a multi-branch cooperative full-scale network, which improved the lightweight feature extraction network OSNet, and used the high-dimensional reconstruction of OSNet to represent the feature table of re-identification. Its high dimension is obtained by stacking four cooperative branches, namely global branch, local branch, correlation branch and contrast branch, to obtain multi-scale features with strong correlation for re-identification, so as to accurately search pedestrians. However, this method only learns the single-frame pedestrian features, lacks temporal information, is sensitive to large view angle changes, and cannot be applied to pedestrian re-identification in complex conditions.

[0005] Pedestrian re-identification method based on dynamic video: He Zhimin et al. proposed a pedestrian re-identification method based on long and short term relationship network, which improved the Transformer structure, deeply mined the correlation information of short and long time series through cross attention structure, obtained pedestrian features with time series, and obtained better search results. However, due to the complexity of the transformer structure, the parameter amount is large, the real-time performance of the search is poor, and the real-time search of re-identification cannot be solved.

[0006] In summary, in view of the problems and deficiencies of the prior art, it is very meaningful to disclose a new pedestrian re-identification method based on deep learning. SUMMARY

[0007] Therefore, the application discloses a video pedestrian re-identification method and system based on multi-scale fusion to solve the problems of complex network model and poor real-time performance in multi-frame feature extraction

[0008] The technical scheme provided by the application is a video pedestrian re-identification method based on multi-scale fusion, which comprises the following steps:

[0009] S1: constructing a 3DCABlock architecture for pedestrian shallow feature extraction, inputting a feature map into the 3DCABlock architecture, and outputting a global shallow feature map of the pedestrian through a mixed 3D convolution in the architecture and a cross attention mechanism in the Transformer;

[0010] S2: constructing a 3DISBlock architecture for pedestrian deep feature extraction network, using a stacked multi-branch fusion structure to extract features from the input global shallow feature map of the pedestrian, and obtaining a pedestrian deep representation map with strong discriminability, which contains global feature information of multi-frame pedestrian features;

[0011] S3: constructing an Arattention module of a pedestrian feature correction module, extracting features from the pedestrian deep representation map, and obtaining a more effective pedestrian deep feature map;

[0012] S4: Construct the ATC module for pedestrian feature expansion, and obtain the final pedestrian feature information based on the more effective pedestrian depth feature map mentioned above.

[0013] Preferably, S1 includes:

[0014] Preliminary extraction of the feature map is performed using 3D CNN convolution to obtain the global shallow features of the human in the feature map;

[0015] The relationships between frames in the feature map are obtained through cross-attention in the Transformer.

[0016] Preferably, the pedestrian deep feature extraction network 3DISBlock architecture in S2 extracts features from the input global shallow feature map of the pedestrian, including:

[0017] The global shallow feature map of pedestrians is removed by a 3D small kernel convolution module to remove background interference from the feature map;

[0018] 3D depthwise separable convolutional feature blocks are formed by alternately stacking 3×3×3 and 1×1×1 convolutional kernels;

[0019] A multi-branch structure is constructed based on multiple 3D depthwise separable convolutional feature blocks. Four parallel feature extraction paths with different receptive fields are constructed in the spatiotemporal dimension to extract features under four different receptive fields: short-range fine-grained features, mid-range semantic associations, and long-range spatiotemporal dependencies.

[0020] The features from the four different receptive fields are weighted using DYNMGate, and then weighted and fused to obtain the optimal fusion result, which is used as the pedestrian depth feature.

[0021] Preferably, the Arattention module of the pedestrian feature correction module in S3 is composed of a cascaded channel attention submodule and a spatial attention submodule;

[0022] The channel attention submodule uses dual-path 3D adaptive pooling to capture channel dimension statistical features and generates channel importance weights through a bottleneck structure fully connected layer; the spatial attention submodule aggregates mean and maximum features along the channel dimension and fuses spatiotemporal context information through 3D convolution to generate a spatial saliency mask.

[0023] The channel attention submodule and the spatial attention submodule respectively reweight the input features in the channel dimension and refocus them in the spatial dimension, and respectively output enhanced features that retain temporal continuity and spatial discriminativeness;

[0024] The weights of each feature are obtained by using DYNMGate, and then weighted and fused together with the residual of the original feature map to obtain more effective pedestrian depth features.

[0025] Preferably, the pedestrian feature expansion ATC module in S4 obtains three reduced-dimensional feature maps from the feature map output in S3 through multi-branch allocation;

[0026] The spatiotemporal importance of each pixel is obtained by convolving the three reduced-dimensional feature maps with a size of 1×1×1. The pedestrian feature map with background interference removed is obtained by 3D normalization and ReLU activation function.

[0027] Meanwhile, the feature map channels of the three reduced dimensions are reduced to 0.125 of the original, resulting in three branches with reduced channels. The three branches are then convolved with kernel sizes of 7×1×1, 1×7×1, and 1×1×7, respectively, to deeply mine cross-dimensional features. These features are then concatenated with the pedestrian feature map that has had its background interference removed to obtain the final pedestrian feature information.

[0028] This invention also provides a video pedestrian re-identification system based on multi-scale fusion, comprising:

[0029] The 3DCABlock module for shallow pedestrian feature extraction is used to extract shallow pedestrian features.

[0030] The 3DISBlock module of the pedestrian deep feature extraction network is used to extract deep features of pedestrians;

[0031] The Arattention module of the pedestrian feature correction module is used to enhance the expressive power of deep pedestrian features;

[0032] The ATC module for pedestrian feature expansion is used to expand and supplement multi-dimensional information of pedestrian features, remove background interference, and improve the ability to represent pedestrian features.

[0033] This invention first uses 3DCABlock to perform preliminary extraction of multi-frame input information through cross-referencing, addressing the issue of limited feature information in a single frame. Then, along multiple 3DCABlock architectures, the large-kernel 3D convolution operation is divided into multiple small-kernel 3D convolutions stacked together. The DYNMGate dynamic 3D gating mechanism is used to fuse multi-scale, multi-frame pedestrian features, outputting global feature information for multi-frame pedestrians, thus solving the problem of large parameter counts in ordinary 3D convolution operations. The output pedestrian features are then fed into an ARattention mechanism to enhance the network's extraction of effective pedestrian features. Finally, an ATC network is used to extract multi-frame features along multiple dimensions, supplementing global information with local information to obtain the final feature extraction result. This method effectively addresses the problems of redundant and complex 3D feature extraction network structures and shallow feature extraction by lightweight 3D networks, significantly improving the accuracy of video pedestrian re-identification.

[0034] This invention proposes an ISNet method and system for video pedestrian re-identification based on multi-scale fusion. This method can extract pedestrian features at multiple scales and richer pedestrian semantic features from multiple frames, and effectively retrieve pedestrian identities with a small number of parameters. It solves the problems of overly complex network models and poor real-time performance in multi-frame feature extraction in video pedestrian re-identification.

[0035] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit the disclosure of the present invention. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is an overall framework diagram of the video pedestrian re-identification method based on multi-scale fusion provided in the embodiments of the present invention;

[0039] Figure 2 A schematic diagram of the 3DCAblock structure for pedestrian shallow feature extraction provided in an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of the 3DISBlock architecture for pedestrian depth feature extraction network provided in the embodiments of the present invention;

[0041] Figure 4 A schematic diagram of the Arattention module structure of the pedestrian feature correction module provided in the embodiments of the present invention;

[0042] Figure 5 A schematic diagram of the pedestrian feature expansion ATC module structure provided in an embodiment of the present invention;

[0043] Figure 6 The pedestrian retrieval results are shown in the embodiment of the present invention. Detailed Implementation

[0044] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of systems consistent with some aspects of the invention as detailed in the appended claims.

[0045] To effectively address the issues of redundant and complex 3D feature extraction network structures and shallow feature extraction in lightweight 3D networks, this implementation scheme first provides a video pedestrian re-identification method based on multi-scale fusion, such as... Figures 1-6 As shown, specifically, it includes the following steps:

[0046] S1: Construct a 3DCAblock (3D Convolution And Cross Attentionblock) architecture for shallow pedestrian feature extraction. This architecture uses a hybrid CNN-Transformer to perform preliminary extraction of the feature map through 3D CNN convolutions, initially extracting the global shallow features of pedestrians from the feature map;

[0047] Then, cross-attention from the Transformer is used to examine the relationships between frames in the feature map.

[0048] By combining these two approaches, we can enhance our understanding of global feature information and optimize temporal correlation, leading to a better understanding of the global semantic information and spatiotemporal correlation of the feature maps. This step uses this architecture as the global shallow feature extraction part for pedestrians; the output is the global shallow feature map of the pedestrians.

[0049] Preferably, such as Figure 2 As shown, it includes the following:

[0050] S11: The 3DCAblock network uses a large 1×7×7 kernel to convolve the input feature map, obtaining its shallow spatial feature information. Through this type of large convolution kernel, the network can effectively capture information from a wider range of local regions, thereby extracting more detailed image features.

[0051] S12: Separate the temporal dimension of the low-dimensional information obtained in S11, dividing its 4 frames into 6 two-frame feature sequences x. i The dimension of each sequence is N = CHW, where C is the number of channels, H is the feature map height, and W is the feature map width.

[0052] Then, take one set of sequences as the query matrix Q for cross-attention, and the other set as the key matrix K and value matrix V, calculate the points between the query and the key, and scale them;

[0053] The similarity matrix is ​​normalized using the softmax function. The attention weights for each query on different keys are then weighted and summed to obtain the interaction feature sequence, which is calculated as follows:

[0054]

[0055] After constructing temporal correlation information through cross-attention, the frame order is aggregated and reshaped into T×C×H×W. Through convolution of the temporal dimension, shallow features with temporal correlation consistent with the input are obtained.

[0056] The obtained shallow features are subjected to 3D max pooling of size 1×3×3 to retain the most significant spatial features, and finally the global shallow features are obtained.

[0057] S2: Construct the 3DISBlock (3D Interaction and Scale Block) architecture for pedestrian deep feature extraction. The 3DISBlock network architecture uses a three-dimensional small kernel convolutional module to suppress background noise in the initial features and effectively enhances the saliency of the semantic features of the pedestrian subject through local receptive field constraints.

[0058] Based on this, a heterogeneous feature extraction module based on depthwise separable convolution is constructed. By alternately stacking 3×3×3 and 1×1×1 convolution kernels to form a parameter-efficient hierarchical structure, the exponential expansion of the receptive field is effectively achieved.

[0059] The multi-branch heterogeneous architecture constructs four parallel feature extraction paths with different receptive fields in the spatiotemporal dimension, respectively capturing short-range fine-grained features, mid-range semantic associations, and long-range spatiotemporal dependencies.

[0060] To optimize the feature fusion process, a dynamic multi-gated weighting mechanism, DYNMGate, is introduced. Through learnable channel-spatiotemporal attention joint modeling, it adaptively generates three-dimensional weight tensors for each branch feature, and finally generates a highly discriminative pedestrian depth representation through weighted feature fusion.

[0061] Preferably, such as Figure 3 As shown, it includes:

[0062] S21: The pedestrian feature map obtained from S1 is subjected to a 1×1×1 three-dimensional convolution, followed by 3D normalization and ReLU activation function to remove the influence of background noise.

[0063] S22: Stack four 3×3×3 and 1×1×1 convolutional kernels respectively to construct pedestrian features at four different scales after removing background noise interference from the shallow pedestrian features;

[0064] S23: DYNMGate pools the features under four different receptive fields, obtains their respective weights through two fully connected layers and ReLU activation function and through Sigmoid, and then fuses the information at each scale in a weighted manner.

[0065] S24: Multi-scale fusion information is aggregated through a 1×1×1 three-dimensional volume and 3D normalization to finally obtain a pedestrian depth representation with strong discriminative power.

[0066] S3: Construct the Arattention (All Road Dynamic Attention Combination) module for pedestrian feature correction. This module consists of cascaded channel attention and spatial attention. It enhances the spatiotemporal feature representation capability through dynamic feature recalibration. Its channel attention submodule uses dual-path 3D adaptive pooling to capture channel-dimensional statistical features and generates channel importance weights through a bottleneck-structured fully connected layer. The spatial attention submodule aggregates mean and maximum features along the channel dimension and fuses spatiotemporal context information through 3D convolution to generate a spatial saliency mask.

[0067] The two sub-modules respectively reweight the input features in the channel dimension and refocus them in the spatial dimension, outputting enhanced features that retain temporal continuity and spatial discriminability. Finally, the weights of each feature are obtained through DYNMGate in S2, and then weighted and fused and connected to the residual of the original feature map, which dynamically improves the spatiotemporal correlation modeling ability of pedestrian features in video sequences.

[0068] The Arattention module of the pedestrian feature correction module employs multiple attention fusion methods to reduce feature map shifts caused by background changes or different viewpoints, enhancing the deep feature map enhancement model's ability to express pedestrian features: Deep features are subjected to spatial information aggregation and channel refocusing. Spatial information aggregation involves performing mean and max pooling operations on each channel of the input feature map to obtain the spatial importance of each location. A 3D convolutional layer further processes spatial attention, applying the calculated spatial weighting coefficients to the original input x to adjust the features at each spatial location. Channel refocusing first performs global average pooling and max pooling, which aggregate the input features differently. The pooled features are then processed by an MLP to calculate the weighting coefficients for each channel. Finally, the weighting coefficients are calculated using the Sigmoid function and applied to the original input x for channel weighting. The dual-branch feature maps obtain their respective weights through DYNMGate, resulting in weighted fusion and residual connection with the original feature map to obtain more effective pedestrian depth features.

[0069] S4: Construct the ATC (Attention Transfer and Compensation) module for pedestrian feature expansion. This module uses multi-branch allocation to obtain feature maps with three reduced dimensions. These maps are then passed through three different single-dimensional large-kernel convolutions to supplement multi-dimensional information. After removing background interference from the complete feature map, the output pedestrian feature information is obtained, ultimately leading to accurate pedestrian re-identification and retrieval images. Figure 6 This is the pedestrian re-identification retrieval map of the present invention.

[0070] like Figure 4 As shown, preferably, it includes:

[0071] S41: The spatiotemporal importance of each pixel is obtained by convolution of size 1×1×1 through the complete feature map, and the pedestrian feature map with background interference removed is obtained by 3D normalization and ReLU activation function.

[0072] S42: Reduce the full feature map channels to 0.125 of the original to obtain three branches with reduced channels. Perform convolutions of 7×1×1, 1×7×1, and 1×1×7 kernel sizes on the three branches respectively to deeply mine cross-dimensional features. Concatenate the features obtained in S41 to obtain the final pedestrian feature information.

[0073] Finally, the video pedestrian re-identification network model employs a combination of labeled smooth cross-entropy loss and triplet loss as constraints, with the two losses and the total loss denoted as L, respectively. triplet L CE The calculation of L is as shown in the following formula.

[0074] L triplet =max(d(a,p)-d(a,n)+margin,0)

[0075]

[0076] L = L triplet +L CE

[0077] The above method is based on the ISNet video pedestrian re-identification system, which is a multi-scale fusion system. Its network model consists of four parts: a 3DCABlock architecture for shallow pedestrian feature extraction, a 3DISBlock architecture for deep pedestrian feature extraction, an Arattention module for pedestrian feature correction, and an ATC module for pedestrian feature expansion. First, the 3DCABlock architecture extracts temporally correlated shallow features of pedestrians, preparing for the extraction of deep pedestrian features. Then, the 3DISBlock architecture constructs multi-scale branches using stacked depthwise separable convolutional residual blocks and employs the DYNMGate dynamic 3D gating mechanism to fuse spatiotemporal context information. Subsequently, the Arattention module in the pedestrian feature correction module refocuses features through a spatial-channel dual-dimensional attention mechanism, combined with dynamic gating residual connections to enhance feature robustness. Finally, the ATC module for pedestrian feature expansion captures cross-dimensional features using multi-dimensional large-kernel convolutions and concatenates them with background-suppressed features to generate the final pedestrian representation.

[0078] Through verification, it has been found that this method effectively solves key problems in pedestrian re-identification, such as large number of parameters, complex structure, and high sensitivity to changes in viewpoint, through progressive processing of shallow spatiotemporal modeling, multi-scale deep feature fusion, dynamic attention correction, and large receptive field feature expansion, and effectively improves the accuracy of pedestrian re-identification retrieval.

[0079] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these changes and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A video pedestrian re-identification method based on multi-scale fusion, characterized in that, The method comprises the following steps: S1: constructing a 3D CABlock architecture for pedestrian shallow feature extraction, inputting a feature map into the 3D CABlock architecture, and outputting a global shallow feature map of the pedestrian through a mixed 3D convolution in the architecture and a cross-attention mechanism in the Transformer; S2: constructing a 3DISBlock architecture for pedestrian deep feature extraction network, adopting a stacked multi-branch fusion structure to extract features of the input global shallow feature map of the pedestrian, and obtaining a pedestrian deep representation map with strong discriminability, which contains global feature information of multiple frames of pedestrian features; S3: constructing an Arattention module of a pedestrian feature correction module, extracting features of the pedestrian deep representation map, and obtaining a more effective pedestrian deep feature map; S4: constructing an ATC module for pedestrian feature expansion, and obtaining final pedestrian feature information based on the more effective pedestrian deep feature map; The 3DISBlock architecture for pedestrian deep feature extraction network in S2 extracts features of the input global shallow feature map of the pedestrian, which comprises the following steps: The global shallow feature map of the pedestrian is removed from the background interference by a three-dimensional small kernel convolution module; A 3D depth separable convolution feature block is formed by alternately stacking 3*3*3 and 1*1*1 convolution kernels; Based on the plurality of 3D depth separable convolution feature blocks, a multi-branch structure is jointly constructed, four parallel feature extraction paths with different receptive fields are constructed in the space-time dimension, and four features under different receptive fields are extracted: short-range fine-grained features, medium-range semantic association and long-range space-time dependence; The four features under different receptive fields are weighted and fused by DYNMGate to obtain the optimal fusion result as the pedestrian deep feature; The Arattention module of the pedestrian feature correction module in S3 is composed of a channel attention submodule and a spatial attention submodule in cascade; The channel attention submodule adopts a double-path three-dimensional adaptive pooling to capture channel dimension statistical features, and generates channel importance weights through a bottleneck structure full connection layer; the spatial attention submodule aggregates mean and maximum value features along the channel dimension, fuses space-time context information through a three-dimensional convolution, and generates a spatial saliency mask; The channel attention submodule and the spatial attention submodule respectively re-weight the channel dimension and re-focus the spatial dimension of the input feature, and respectively output enhanced features that retain time continuity and spatial discriminability; The weights of the respective features are obtained by DYNMGate, and the weighted fusion is connected to the original feature map residual, to obtain a more effective pedestrian deep feature; The ATC module for pedestrian feature expansion in S4 obtains three feature maps with reduced dimensions from the feature map output from S3 through multi-branch distribution; The three feature maps with reduced dimensions are obtained through a convolution with a size of 1*1*1 to obtain the space-time importance of each pixel, and a pedestrian feature map without background interference is obtained through 3D normalization and a Relu activation function; At the same time, the three reduced dimension feature map channels are reduced to 0.125 of the original, obtaining three reduced channel branches, and the three branches are respectively subjected to convolution with a convolution kernel size of 7x1x1, 1x7x1 and 1x1x7, deep mining cross-dimension features, and splicing with the above pedestrian feature map removing background interference, obtaining final pedestrian feature information.

2. The multi-scale fusion based video person re-identification method according to claim 1, characterized in that, Comprise: S1 comprises: Preliminary extraction of the feature map through the 3D CNN convolution, obtaining the global shallow features of the person in the feature map; Obtaining the relationship between the frames in the feature map through the cross attention in the Transformer.

3. A multi-scale fusion based video pedestrian re-identification system, characterized in that, The system is used for realizing the multi-scale fusion based video pedestrian re-identification method of claim 1, and comprises: A 3D CABlock module for shallow feature extraction of pedestrians, used for extracting shallow features of pedestrians; A 3D ISBlock module for deep feature extraction network of pedestrians, used for extracting deep features of pedestrians; An Arattention module of a pedestrian feature correction module, used for enhancing the expression ability of the deep features of pedestrians; An ATC module for pedestrian feature expansion, used for expanding and supplementing the multi-dimensional information of the pedestrian features, removing background interference and improving the expression ability of the pedestrian features.

Images