Parallel multilevel feature fusion network pedestrian re-identification system

Abstract

The invention discloses a parallel multilevel feature fusion network pedestrian re-identification system, which takes OSnet as a basic feature extraction model and comprises a hierarchical feature integrator, a spatial feature reconstructor and a deep feature and shallow feature regularization module. According to the method, the recognition accuracy of the system under different resolution conditions can be improved, and a high recognition rate can be kept even in the face of complex conditions such as blurring, illumination change or camera difference.

Description

Technical Field

[0001] The present invention relates to the field of pedestrian recognition, and in particular to a parallel multi-level feature fusion network pedestrian re-recognition system. Background Art

[0002] In all fields of the information-based and intelligent modern society, video surveillance has become a core tool for urban public security monitoring, traffic management, and security defense in public places such as shopping malls and banks. However, faced with a large amount of surveillance video, when it is necessary to search for a specific target from it, the supervisors have to rely on the naked eye to observe frame by frame, which makes tracking the target pedestrians time-consuming and error-prone, and there is also the risk of missing key information. To solve this problem, pedestrian re-identification technology based on computer vision technology came into being. This advanced technology can accurately identify the identity of the same pedestrian from different camera perspectives, greatly improving the monitoring efficiency and the accuracy of tracking targets.

[0003] The features of pedestrians in images are usually highly similar and have low variance due to their small differences, which makes it difficult for the system to distinguish different pedestrians at the same resolution. However, the pedestrian images captured by actual surveillance cameras often have different resolutions, so a more effective feature extraction method is needed. In existing pedestrian re-identification methods, when processing images of different resolutions, researchers usually optimize both high-resolution and low-resolution images at the same time, which results in the model being able to extract information well when processing high-resolution images, but having difficulty effectively capturing key details when processing low-resolution images. Although using super-resolution methods to generate high-resolution images from low-resolution images can improve image quality, it will also modify the original image information, which directly leads to inconsistent features of the same pedestrian in images of different resolutions. This phenomenon is called "resolution bias."

[0004] When processing pedestrian image features, existing person re-identification methods often focus too much on optimizing the model's attention to the query image (query), but fail to fully consider the consistency of the activation value area between the query image and the reference gallery (gallery). We call this phenomenon of not fully considering whether the activation area between the query image and the reference gallery corresponds to each other "attention distribution difference".

[0005] In existing person re-identification tasks, establishing a robust person re-identification framework requires fully capturing the key information in the image. The main problems faced include: (1) how to capture common features with identifiable features in images of different resolutions and finely control the feature fusion process to reduce the impact of resolution bias, thereby improving the performance and generalization ability of the model in complex visual tasks; (2) how to optimize the attention distribution to ensure that the activation value area between the query image and the reference gallery remains consistent, so that the model can enhance the richness of the feature representation of the key area and improve the model's processing ability in the person re-identification task. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention proposes a parallel multi-level feature fusion network person re-identification system. The technical solution of the present invention is implemented as follows: a parallel multi-level feature fusion network person re-identification system, including a hierarchical feature integrator, a spatial feature reconstructor, and a deep feature and shallow feature regularization module.

[0007] The hierarchical feature integrator obtains the feature map x from the input image processed by OSnet by expanding its channel to twice its original value through convolution operation, which is denoted as For the feature map x 0 Split on the channel dimension, dividing it into two parts: a subset with a higher number of channels and a subset with lower channel count Where C+C′=2C, C>C′. According to formula (1), the feature map p is split in the channel dimension to obtain a set of feature maps

[0008] S i =C / 2 (n-i+1) ,where 1≤i≤n (1)

[0009] Where n=4 is the number of times the feature map p is segmented. i As the input of subsequent independent branches; in each branch, the feature map p i With the feature map q i-1 The new feature map q is obtained by fusion through formula (2) i ,

[0010]

[0011] Among them, φ comb is a composite function. i As the output of the current branch and the input of the next branch; q i Perform subsequent feature processing in the current branch, as shown in formula (3),

[0012]

[0013] Among them, BN is batch normalization, ReLU is the rectified linear unit activation function, and conv is deep separable convolution. The point convolution kernel of the deep separable convolution is The depth convolution kernel is For the i-th layer branch, its processing module Φ conv The number is related to the sequence number of the branch, and its processing module Φ conv The number of is set to ni. If ni = 0, its processing module Φ conv The number of is set to 1. For the last branch, the processing module Φ conv The deep separable convolution uses the point convolution kernel to q i Perform convolution operation; the result of all processing modules of the i-th layer branch is Ψ i As shown in formula (4):

[0014]

[0015] After all branches are processed, each branch result Ψ i Sum the sum, perform batch normalization BN on the added result, then add the OSNet output feature map x and the batch normalization result and perform instance normalization IN, as shown in formula (5),

[0016]

[0017] The spatial feature reconstructor adopts a feature map reconstruction strategy to reorganize the output feature map x of OSNet to obtain three spatial feature maps. The reorganization process is as follows: the feature map x is divided equally in its width and height dimensions, and each dimension is divided into μ equal parts. Therefore, the feature map x is divided into m=μ×μ small blocks, and the size of each small block is (H / μ, W / μ, C), where H, W and C represent the height, width and number of channels of the feature map, respectively. In each small block, it is divided into three equal parts in the channel dimension to obtain three sub-blocks. Further, each small block is evenly divided into three sub-blocks. From each small block, a sub-block is extracted and superimposed along the channel dimension to form a new feature map with full-image spatial information.

[0018] The scale dot product attention module further enhances the representation of features by calculating the correlation between local spatial features;

[0019] The deep feature and shallow feature regularization module regularizes the features output by the hierarchical feature integrator and the spatial feature reconstructor using a dot product operation.

[0020] Furthermore, the hierarchical feature integrator transforms the feature map p i With the feature map q i-1 The new feature map q is obtained by fusion through formula 2 i When , the steps also include: setting the convolution kernel K i For input q i-1 Perform convolution operation to adjust q i-1 The number of channels. In each iteration i, a trainable convolution kernel K is used i and weight W i to guide this interaction.

[0021] Furthermore, the spatial feature reconstructor divides each block into three equal parts in the channel dimension, into three sub-blocks of size (H / μ, W / μ, C / 3).

[0022] Furthermore, the hierarchical feature integrator and spatial feature reconstructor use dot product operations to regularize deep and shallow features. This approach allows the model to fuse deep and shallow feature information while retaining deep visual information. Through regularization, the model can use different gradient flows to evaluate the strength of the relationship between features. This helps the model take advantage of the mutual influence between features.

[0023] The beneficial effect of the present invention is that, compared with the prior art, the present invention can improve the recognition accuracy of the system under different resolution conditions, and can maintain a high recognition rate even in complex situations such as blur, lighting changes or camera differences. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a structural diagram of a parallel multi-level feature fusion network pedestrian re-identification system of the present invention;

[0025] Figure 2 It is a schematic diagram of the structure of the hierarchical feature integrator in the present invention;

[0026] Figure 3 It is a schematic diagram of the structure of the spatial feature reconstructor in the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] See also Figure 1The present invention discloses a parallel multi-level feature fusion network pedestrian re-identification system, which mainly includes a level feature integrator 1, a spatial feature reconstructor 2 and a deep feature and shallow feature regularization module 3.

[0029] The hierarchical feature integrator 1 processes feature maps of different depths independently by applying convolution in different branches, effectively capturing the differences between deep features and establishing fine connections between them. Figure 2 As shown in the figure, the hierarchical feature integrator first uses a convolution operation to expand the channel of the feature map x obtained by the input image processed by OSnet to twice its original size, which is recorded as Then, for the feature map x 0 Split the channel dimension into two complementary parts: a subset with a higher number of channels and a subset with lower channel count Where C+C'=2C, C>C'. Then, the hierarchical feature integrator divides the feature map p in the channel dimension according to formula (1) to obtain a set of feature maps

[0030] S i =C / 2 (n-i+1) ,where 1≤i≤n (1)

[0031] In a specific embodiment of the present invention, n=4 means that p is segmented 4 times, and each feature map p i As the input of subsequent independent branches. This segmentation operation allows us to extract a more refined feature subset from the original feature map, providing a basis for multi-level feature representation for subsequent feature processing and deep learning.

[0032] In each branch, the feature map p i With the feature map q i-1 Through the composite function Φ comb (As shown in formula (2)) is fused to obtain a new feature map q i .q i As the output of the current branch and the input of the subsequent operation and the next branch. comb In the previous branch, the input q i-1 and the current branch p i There is a problem of inconsistent number of channels. Therefore, the convolution kernel K is set i For input q i-1 Perform convolution operation to adjust q i-1 The number of channels. In each iteration i, there is a set of trainable convolution kernels K i and weight W ito guide this interaction, which enhances the model's ability to capture and express complex features.

[0033]

[0034] In formula (2), for the feature map p i and feature map q i-1 The dot product operation can enhance the intrinsic connection between the two feature maps at the level, so that the subsequent operations of the branch can capture the similarity of the features; at the same time, the feature map q i-1 It integrates the hierarchical features of the previous branches so that subsequent branches can share and integrate hierarchical features, thereby enhancing the expressive power of the features.

[0035] By applying customized convolutions of different depths in different branches of the attention module, multi-scale feature extraction can be achieved. This hierarchical feature processing strategy enables the model to establish fine connections between features of different depths, so that it can simultaneously learn features from basic to complex. This ability is crucial for accurately identifying and understanding objects of different sizes and complexities in images.

[0036] In addition, the composite function Φ comb The output q i Subsequent feature processing is required in the current branch. A subsequent processing module Φ conv The operations included are: deep separable convolution, batch normalization (BN), and ReLU activation function, which can be used to reduce the number of parameters, as shown in formula (3).

[0037]

[0038] Among them, the point convolution kernel of the deep separable convolution is The depth convolution kernel is

[0039] For the last branch, the deep separation of the processing module can be convolved using the point convolution kernel q i For the i-th layer branch, its processing module Φ conv The number is related to the sequence number of the branch, and its processing module Φ conv The number of is set to ni. If ni = 0, its processing module Φ conv The number of is set to 1. The result of all processing modules of the i-th layer branch is Ψ i As shown in formula (4):

[0040]

[0041] Finally, after all branches are processed, each branch result Ψ iThen, the sum is performed. Then, the batch normalization operation BN is performed on the added result. Then, the output feature map x of OSNet is added to the batch normalization result and then instance normalization IN is performed. The whole process is shown in formula (5):

[0042]

[0043] By combining the feature maps Ψ of different branches or layers i Adding can integrate feature information from different levels or paths and enhance the model's understanding of the input data. Adding it to the input feature map x helps the gradient flow in the network, especially in deep networks, which helps alleviate the problem of gradient disappearance or explosion.

[0044] The structure of spatial feature reconstructor 2 is as follows Figure 3 As shown in the figure, the spatial feature reconstructor optimizes the attention distribution so that the model can better capture and utilize contextual information, thereby performing better in processing the correspondence between the query image (query) and the reference gallery (gallery).

[0045] The spatial feature reconstructor first uses the feature map reconstruction strategy to reorganize the output feature map x of OSnet to obtain three spatial feature maps. Since the feature map x is extracted from the convolutional neural network, it contains rich spatial and channel information. In order to analyze these features more carefully, it is necessary to first divide the feature map equally. The division here is based on the parameter μ, that is, each dimension is divided into μ equal parts. In this way, the feature map is divided into m = μ × μ small blocks, each of which has a size of (H / μ, W / μ, C), where H, W and C represent the height, width and number of channels of the feature map, respectively. In a specific embodiment of the present invention, each small block is divided into 3 equal parts in the dimension of the channel, divided into three sub-blocks of size (H / μ, W / μ, C / 3). This subdivision allows the model to capture features at a finer level, which helps to identify local features of pedestrians, such as clothing texture, color distribution, etc.

[0046] Then, the spatial feature reconstructor 2 further extracts a sub-block from each small block and stacks them into a feature map with full-image spatial information in the dimension of the channel. This step helps the model capture global context information while maintaining local features. In the corresponding embodiment, by using the reconstruction strategy, a total of 3 feature maps are reconstructed, each of which contains different spatial and channel information, increasing the expressive power of the model.

[0047] Afterwards, these feature maps are then input into the scaled dot product attention module, which further enhances the representation of features by calculating the correlation between local spatial features. In the corresponding embodiment, the three feature maps are input into the scaled dot product attention module to calculate the correlation between local spatial features. This process enables the model to focus on local areas, thereby improving the efficiency of the self-attention mechanism.

[0048] The feature map reconstruction strategy enables the feature map to focus on local spatial features while also gathering global spatial information. In addition, when calculating the correlation of spatial features, the attention model can obtain complete spatial context information, thereby focusing more on the salient features of pedestrians.

[0049] Deep features usually contain more abstract semantic information, while shallow features contain richer spatial details; deep features help capture the overall appearance and posture of pedestrians, while shallow features can provide more information about local textures and edges. In order to fuse deep hierarchical features with shallow spatial features and keep the features between the two levels consistent, the present invention uses a deep feature and shallow feature regularization module 3 to regularize the deep features and shallow features through a dot product operation.

[0050] By regularizing the dot product operation of deep features and shallow features, the model can use different gradient flows to evaluate the strength of the association between features, so that the model can take advantage of the mutual influence between features and effectively fuse deep features with shallow features, thereby improving the performance of pedestrian re-identification tasks.

[0051] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A parallel multi-level feature fusion network pedestrian re-identification system, characterized by: It includes hierarchical feature integrator, spatial feature reconstructor and deep feature and shallow feature regularization modules, among which, The hierarchical feature integrator obtains the feature map x from the input image processed by OSnet by expanding its channel to twice its original value through convolution operation, which is denoted as The feature map x0 is split in the channel dimension and divided into two parts: a subset with a higher number of channels and a subset with lower channel count Where C+C′=2C, C>C′. According to formula (1), the feature map p is split in the channel dimension to obtain a set of feature maps p={p i |1≤i≤n}, S i =C / 2 (n-i+1) ,where 1≤i≤n (1) Among them, n = 4 is the number of times the feature map p is segmented, and each feature map p i As the input of subsequent independent branches; in each branch, the feature map p i With the feature map q i-1 The new feature map q is obtained by fusion through formula (2) i , Among them, Φ comb is a composite function, q i As the output of the current branch and the input of the next branch; q i Perform subsequent feature processing in the current branch, as shown in formula (3), Among them, BN is batch normalization, ReLU is the rectified linear unit activation function, conv is the deep separable convolution, and the point convolution kernel of the deep separable convolution is The depth convolution kernel is For the i-th layer branch, its processing module Φ conv The number is related to the sequence number of the branch, and its processing module φ conv The number of is set to ni, if ni = 0, its processing module Φ conv The number of is set to 1. For the last branch, the processing module Φ conv The deep separable convolution uses the point convolution kernel to q i Perform convolution operation; the result of all processing modules of the i-th layer branch is Ψ i As shown in formula (4): After all branches are processed, each branch result Ψ i Sum the sum, perform batch normalization BN on the added result, then add the OSNet output feature map x and the batch normalization result and perform instance normalization IN, as shown in formula (5), The spatial feature reconstructor adopts a feature map reconstruction strategy to reorganize the output feature map x of OSNet to obtain three spatial feature maps. The reorganization process is as follows: the feature map x is divided equally in its width and height dimensions, and each dimension is divided into μ equal parts. The feature map x is divided into m = μ × μ small blocks, and the size of each small block is (H / μ, W / μ, C), where H, W and C represent the height, width and number of channels of the feature map respectively; in each small block, it is divided into three equal parts in the channel dimension to obtain three sub-blocks, and a sub-block is extracted from each small block and superimposed along the channel dimension to form a new feature map with global spatial information. Finally, three feature maps with full-image spatial information are obtained, and the feature maps with full-image spatial information are input into the scale dot product attention module, and the scale dot product attention module further enhances the representation of features by calculating the correlation between local spatial features; The deep feature and shallow feature regularization module regularizes the deep features and shallow features output by the hierarchical feature integrator and the spatial feature reconstructor using a dot product operation.

2. The parallel multi-level feature fusion network person re-identification system according to claim 1, characterized in that: The hierarchical feature integrator transforms the feature map p i With the feature map q i-1 The new feature map q is obtained by fusion through formula (2) i When , the steps include: setting the convolution kernel K i For input q i-1 Perform convolution operation to adjust q i-1 The number of channels is , and in each iteration i, a trainable convolution kernel K is used i and weight W i to guide this interaction.

3. The parallel multi-level feature fusion network person re-identification system according to claim 1, characterized in that: The spatial feature reconstructor divides each block into three equal parts in the channel dimension, into three sub-blocks of size (H / μ, W / μ, C / 3).

Images