Pedestrian re-identification method based on attention mechanism and bimodal feature fusion

By using the self-attention module to integrate RGB and grayscale features in the pedestrian re-identification technology, and combining ResNetCPA and ResNetPA methods, the problem of color inconsistency in pedestrian images caused by changes in lighting and camera parameters is solved, and the accuracy of pedestrian re-identification is improved.

CN119007246APending Publication Date: 2024-11-22BEIJING INFORMATION SCI & TECH UNIV

Patent Information

Application Number
CN202411091496.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing pedestrian re-identification technology causes inconsistent images of the same pedestrian under changes in lighting and camera parameters, and the colors of different pedestrians are similar, resulting in a decrease in recognition accuracy.

Method used

The self-attention module is used to fuse RGB modal features, gray-scale modal features and RGB and gray-scale dual-modal features. The attention mechanism is introduced in the convolutional neural network through methods such as ResNetCPA and ResNetPA to automatically pay attention to pedestrian images. important feature areas to improve the recognition performance of network models.

Benefits of technology

Effectively utilize RGB and grayscale image information to improve the accuracy of pedestrian re-identification, especially in scenarios with poor lighting or similar colors of pedestrian clothes, the recognition performance of the model is significantly improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007246A_ABST
    Figure CN119007246A_ABST
Patent Text Reader

Abstract

The invention relates to a pedestrian re-identification method in the technical field of electronics. The pedestrian re-identification task aims at finding out a target pedestrian under camera devices which are not overlapped and have different visual angles. In actual application, due to interference problems of different degrees existing in illumination conditions, camera shooting visual angles, image resolution, environment backgrounds and the like, a pedestrian re-identification task has a great challenge. The invention provides a pedestrian re-identification method based on attention mechanism and bimodal feature fusion in order to solve the problems that the colors of the same pedestrian image are inconsistent and the colors of different pedestrian images are similar due to the influence of illumination and camera parameters. According to the method, the bimodal features of the RGB mode and the gray mode are fused, the features of the pedestrian image are captured more comprehensively by using the color, edge structure and other information of the pedestrian image, and an attention mechanism is introduced, so that the model can automatically pay attention to important feature regions in the pedestrian image, and the recognition performance of the network model is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Electronic technology:

[0002] Description: The present invention relates to the field of electronic technology, and in particular to a pedestrian re-identification method based on attention mechanism and bimodal feature fusion. Background Art

[0003] Person Re-identification (Re-ID) is an important research direction in the field of computer vision. It is mainly used to determine whether pedestrians appearing in different surveillance videos belong to the same pedestrian. Research in this field is of great significance for application scenarios such as video surveillance, tracking, and personnel search.

[0004] The principle of pedestrian re-identification is mainly based on feature extraction and matching. First, the pedestrian image is preprocessed through computer vision technology, including image segmentation, feature extraction and other operations to obtain the feature representation of the pedestrian. Then, these features are matched with the existing pedestrian database to achieve pedestrian re-identification.

[0005] Application areas include security monitoring, which is used to achieve cross-camera tracking and retrieval, helping the police to quickly locate and track; intelligent transportation can be used for vehicle monitoring and traffic flow statistics and analysis to improve traffic efficiency and safety; human-computer interaction can be applied in interactive entertainment scenarios, similar to Kinnect human-computer interaction, and key point detection technology is very valuable; intelligent cognition of emergencies in public places, such as theft and gang fights, can take intelligent measures after the camera recognizes such behavior, such as automatic alarm, which has great social value; in social networks, pedestrian re-identification can be used to automatically tag people in photos to improve the user experience of social networks. Summary of the invention

[0006] The purpose of the present invention is to provide a novel pedestrian re-identification method to solve the problem that the color of the same pedestrian image is inconsistent and the color of different pedestrian images is similar due to the influence of lighting and camera parameters in the prior art. The present invention adopts a self-attention module to fuse RGB modal features, grayscale modal features, and RGB and grayscale bimodal features, so as to more effectively utilize the image information of RGB modality and grayscale modality.

[0007] The effectiveness of the RestNetCPA and RestNetPA proposed in this invention and the multi-stream fusion module using the attention mechanism at the same time are proved, so that the model can automatically focus on the important feature areas in the pedestrian image, thereby improving the recognition performance of the network model.

[0008] First, the basic convolutional neural network is trained on a large dataset. The trained basic convolutional neural network can extract the feature information of the image well. Then, the RGB image and the grayscale image are respectively input into the convolutional residual neural network that has been trained and added with different attention mechanisms to form the RGB modal flow and the grayscale modal flow, thereby obtaining the RGB image features and the grayscale image features. The bimodal feature splicing flow is formed by splicing and fusion at the feature layer to obtain the bimodal fusion feature. Finally, the feature splicing result obtained by the RGB modal flow, the grayscale modal flow and the bimodal feature splicing flow is used as the final feature representation of the model. During the training process, the model optimizes the loss function training for each branch to improve the generalization ability of each branch. At the same time, the global loss function is also used to optimize the final bimodal features. The proposed network model is tested on three public datasets: Market1501, CUHK03 and DukeMTMC-ReID, and the effectiveness of the model is proved.

[0009] Since RGB and grayscale images are complementary, that is, RGB images can provide rich color information, and grayscale images can provide more comprehensive texture, contour, and structure information. At the same time, grayscale images can reduce the interference of color changes and other problems to the model to a certain extent, especially in poor lighting conditions or when pedestrians' clothes are similar in color. Therefore, the use of a more appropriate grayscale image conversion algorithm is crucial to improving the accuracy of model recognition.

[0010] In order to more effectively utilize pedestrian images in grayscale mode, this paper proposes a dual-modal pedestrian re-identification method based on deep convolutional neural network and attention mechanism. The overall network structure is as follows: Figure 1 As shown. The deep method proposed in the present invention is divided into four branch modules, namely, RGB modal flow module, grayscale modal flow mode, dual-modal splicing flow module and multi-stream fusion mode. The overall process is described as follows: First, the pedestrian image in RGB mode is converted into a pedestrian image in grayscale mode according to the grayscale image conversion algorithm, and the image in RGB mode and the image in grayscale mode are respectively sent to the RGB modal flow module and the grayscale modal flow module to obtain the pedestrian features under RGB mode and the pedestrian features under grayscale mode. Secondly, in order to better fuse the RGB modal features and the grayscale modal features, the pedestrian features under RGB mode and the pedestrian features under grayscale mode are spliced ​​in the dual-modal splicing flow module. Finally, the pedestrian features obtained by the RGB modal flow module, the grayscale modal flow mode and the dual-modal splicing flow module are fused in the multi-stream fusion module, and the attention mechanism is used to obtain pedestrian features with richer color, texture and structure information, thereby improving the recognition accuracy of the pedestrian re-identification model.

[0011] The RGB modality flow module and the grayscale modality flow module use the ResNetCPA and ResNetPA methods to extract features from pedestrian images in RGB and grayscale modalities, respectively. The main idea of ​​the ResNetCPA and ResNetPA methods is to coordinate the deep convolutional neural network and the attention mechanism, making full use of the advantages of the convolutional neural network and the attention mechanism, that is, not only can the local features of pedestrians be obtained, but also the corresponding relationships can be established globally. Specifically, the multi-scale pedestrian features are extracted hierarchically using the deep convolutional neural network to capture the local information of the pedestrian image, while the attention mechanism can process the pedestrian features of different scales extracted by the convolutional neural network in a global range, thereby establishing a more effective global relationship. The RGB modality flow module, the grayscale modality flow module, and the dual-modality splicing flow module all use global average pooling (GAP) to reduce the dimension of the features and the fully connected layer (FC) and form the feature expression of each module. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is the overall network structure diagram of the present invention;

[0013] Figure 2 This is the ResNetPA structure diagram;

[0014] Figure 3 This is the PosAttention graph of the improved self-attention mechanism;

[0015] Figure 4 This is the ResNetCPA structure diagram;

[0016] Figure 5 It is the structure diagram of RGA-C; DETAILED DESCRIPTION

[0018] Step 1: Grayscale image conversion algorithm

[0019] The conversion of RGB modality to grayscale modality image is actually the change of human eye's perception of color to human eye's perception of brightness. Therefore, the idea of ​​the grayscale image conversion algorithm used in the present invention is: the human eye's perception of light is uneven, among which the perception of green is greater than red, and red is greater than blue. This means that when processing images, this perception difference should be considered and each color should be given a different weight. Based on this, the present invention sets weights so that image processing is more in line with the human eye's perception of light. Therefore, when processing RGB images, weights can be assigned to the three colors of red, green, and blue according to the characteristics of human eye perception to better reflect the visual perception of the human eye. The specific conversion algorithm is as follows:

[0020] Gray(a,b)=R(a,b)×0.299+G(a,b)×0.587+B(a,b)×0.114

[0021] Gray(a,b) represents the pixel size of row a and column b in the grayscale image, and R, G and B represent the red, green and blue color channels of the RGB image respectively. The present invention transforms the grayscale image format into three channels with the same value to meet the needs of the network model.

[0022] Step 2: ResNetPA method design

[0023] This paper proposes the ResNetPA attention method, which mainly uses the self-attention module to improve the deep convolutional neural network to extract pedestrian features. Figure 2 As shown. First, the basic convolutional neural network is trained on a large dataset. The trained basic convolutional neural network can well extract the feature information of the image. The pedestrian image is input into the trained convolutional neural network. Then, each residual block in the baseline convolutional network is connected to self-attention, and the last three residual blocks are feature fused. The residual block fusion mainly performs linear interpolation operations on the 4th and 5th convolutional layers to normalize the obtained features. In addition, 1×1 convolution is used in the 3rd and 4th convolutional layers to achieve dimensional unification, and finally the three features of the same size are fused.

[0024] Transformer has been proven to achieve better results than traditional convolutional neural networks in the field of image processing. Therefore, inspired by the Transformer feature extraction method, self-attention mechanism and position encoding are added to each residual block of the convolutional neural network to form a model structure ResNetPA that combines convolutional neural network and attention mechanism. ResNetPA mainly uses ResNet50 as the benchmark network. The main idea is to add an improved self-attention mechanism to each residual block of ResNet50. ResNetPA not only realizes the adjustment of features in the local spatial area of ​​pedestrian images, making the network model pay more attention to important feature information in the image, but also enhances the processing of multi-scale features in the original ResNet50 model, that is, the importance of each residual block is determined by the attention mechanism, so as to make full use of feature information of different scales, while reducing the problem of too many parameters and calculations of Transformer when extracting features.

[0025] In order to compensate for the loss of feature position and other information when only using the self-attention method in the image processing process, the present invention combines relative position encoding with the traditional self-attention method. This method not only considers the context feature information of the pedestrian image, but also pays attention to the mutual importance of features at different positions in the pedestrian image. Therefore, the self-attention mechanism method in the ResNetPA method proposed in the present invention adopts the self-attention mechanism with relative position encoding.

[0026] Given an input sequence x=(x1,…,x n ), which contains a total of n input elements, then a new sequence SA containing n elements can be obtained through the attention mechanism. n ), the input sequence x is linearly transformed to obtain Q, K and V, then the original self-attention mechanism is expressed as shown in the formula

[0027] Q = xW Q ,K=xW K ,V=xW V

[0028]

[0029] Among them, W Q , W K , w V are linear transformation matrices respectively; d represents the vector dimension of Q, K, V, SA i Represents the input element x i The self-attention output.

[0030] According to the self-attention method, relative position encoding is added on this basis, and the improved self-attention mechanism is shown in the formula.

[0031]

[0032] Among them, PA i represents the output of the improved attention method, P ij is the relative position encoding, and They represent the vector representation of two edges in the relative position representation. The specific implementation process is as follows Figure 3 shown.

[0033] Figure 3 In the example, H×W×C is the dimension of the input feature x, where H, W, and C represent the length, width, and number of channels of the feature, respectively. x undergoes three linear transformation matrices W Q , W K and W VThey are transformed into Q, K and V respectively. The output results of the two branches Q and K are respectively transformed in dimension and multiplied to obtain the corresponding weight coefficients, and then the relative position code P is added on the basis of the weight coefficients. K , and then add the weighted result of the position code through the softmax function and add the relative position code P V The final output is the result of the PosAttention method.

[0034] The RestNetPA method proposed in the present invention adds the PosAttention method to the residual block of ResNet50. The specific network structure design is shown in Table 1, where each curly bracket is a residual block.

[0035] Table 1 Network structure of RestNetPA

[0036]

[0037]

[0038] Step 3: ResNetCPA method design

[0039] In order to more comprehensively extract pedestrian features in RGB mode, the ResNetCPA method is proposed, which adds a channel attention method to the ResNetPA mentioned in the previous module to enhance the representation ability of pedestrian features. The structure diagram of this method is shown in Figure 4 shown.

[0040] Most existing pedestrian re-identification methods use local convolution to learn attention to extract important pedestrian features, but often ignore the mining of more comprehensive feature information in a global mode. Based on the above problems, the present invention introduces a global channel attention mechanism to enable the model to extract more sufficient pedestrian features. Traditional channel attention usually reduces the dimension of each channel when performing attention mechanism calculations. Although this method can learn the functional relationship between channels, it takes a long time to capture the relevant attention relationship, and only focusing on local relationships may have a negative impact on the correlation results between channels. Therefore, the present invention adopts a global channel attention method (ChannelRelation-Aware Global Attention, RGA-C), which aims to capture the relationship between global feature information and each channel in the feature representation. The RGA-C structure diagram is shown in the figure. Figure 5 shown.

[0041] Given a feature map X of size H×W×C, in order to clearly represent the relationship between channels, each channel is regarded as a feature point. Therefore, the feature map X has a total of C feature points, and each feature point is represented as x i , where i = 1,…,C. Then the relationship between the two channels R i,j The mathematical expression of is shown in the formula.

[0042]

[0043] where θ and are two functions of embedding feature nodes, which are used to transform x i and x j Mapped to a new feature representation space and thus converted into a new feature representation, both are mapped through 1×1 convolution, batch normalization (BN) and ReLU activation function.

[0044] In addition, in order to learn the relationship between channels more comprehensively, this method also adds its own channel feature information, and uses the global feature information and local attention information of the channel, so the two are combined to obtain the spatial relationship perception feature y′ i , as shown in the formula.

[0045]

[0046] in and φ represent the channel features themselves x i and the global channel relation r i The embedding function mainly converts x i and r i Mapped to the same feature domain, both embedding functions are mapped through a 1×1 convolution, batch normalization (BN) and ReLU activation function. pool(.) represents the global average pooling operation along the feature dimension to reduce the dimension to 1. Finally, in order to extract richer feature information, a modeling function is designed to obtain the channel attention value att of the i-th channel feature. i , as shown in the formula.

[0047] att i =Sigmoid(ω2ReLU(ω1y′ i ))

[0048] Where ω1 and ω2 are matrices obtained by 1×1 convolution and batch normalization (BN), which are used to control the channel dimension. ω1 shrinks the channel dimension at a predetermined reduction ratio, and ω2 converts the channel dimension to 1.

[0049] The RGA-C attention method obtains more important feature information in pedestrian images by paying comprehensive and effective attention to the channel dimension, thereby enhancing the expression ability of the network model and improving performance. Therefore, the attention method ResNetCPA proposed in the present invention combines the channel attention RGA-C method with the previously proposed ResNetPA method to further improve the feature extraction of the RGB modality. Specifically, the RGA-C method is further added to the residual block of ResNet50. The network structure design diagram of ResNetPA and ResNetCPA is shown in Table 2, and each residual block is wrapped in brackets.

[0050] Table 2 Schematic diagram of network structure design of ResNetPA and ResNetCPA

[0051]

[0052] Step 4: Loss function design

[0053] The loss function of the method proposed in the present invention mainly uses two loss functions: ID loss and triple loss. The cross entropy classification loss function is mainly ID loss to distinguish pedestrians by comparing the differences between feature vectors. The specific definition is as follows:

[0054]

[0055] Where B represents the number of batches for training, S represents the number of pedestrian identity categories, logp(s|x i ) represents the sample x i The probability of predicting the Sth class is δ(.) is the Dirac function, δ(y i -s) represents the pedestrian sample x i Belongs to y i Probability distribution of individual pedestrian classes.

[0056] The main purpose of the triplet loss function is to find sample pairs with high similarity in the same given batch of pedestrian data, to increase the distance between pedestrian images of the same category with low similarity, and to decrease the distance between pedestrian images of different categories with high similarity, thereby encouraging the same pedestrian images to be close to each other and different pedestrian images to be far away from each other, thus achieving the effect of clustering images of the same category. M pedestrians are randomly selected, and each pedestrian has K images. Then the number of images in a batch during training is N = M × K, and the mathematical expression of the triplet loss function is as follows:

[0057]

[0058] in represents the t-th image of the m-th type of pedestrian, Representation and Another image of the same pedestrian (positive sample), Representation and An image of pedestrians of different types (negative samples). And a represents the interval, that is, and The minimum difference between the two. D(.,.) represents the Euclidean distance.

[0059] Then the overall loss function used by each branch is:

[0060] L all (X,Y)=L cross (X,Y)+θL tri (X,Y)

[0061] Where λ is a hyperparameter used to balance the relationship between the ID loss function and the triplet loss function. In the present invention, independent loss functions are added to the grayscale flow branch and the RGB branch, and a global loss is added after the bimodal feature fusion, so as to better improve the performance of the network model.

Claims

1. A pedestrian re-identification method based on attention mechanism and bimodal feature fusion, the pedestrian re-identification method comprising the following steps: S1: converting the pedestrian image in RGB mode into a pedestrian image in gray mode according to the grayscale image conversion algorithm; S2: The RGB modality flow module and the grayscale modality flow module use the ResNetCPA and ResNetPA methods to extract features from pedestrian images in the RGB modality and grayscale modality, respectively, to obtain pedestrian features in the RGB modality and pedestrian features in the grayscale modality; S3: The obtained features are spliced ​​in the bimodal splicing flow module; S4: Fuse the pedestrian features obtained by the RGB modality flow module, the grayscale modality flow module, and the bimodal splicing flow module; S5: Design loss function and use two loss functions: ID loss and triplet loss.

2. The pedestrian re-identification method according to claim 1, characterized in that: The deep method proposed in the present invention is divided into four branch modules, namely, RGB modal flow module, grayscale modal flow module, dual-modal splicing flow module and multi-stream fusion modality.

3. The grayscale pedestrian image according to claim 1, characterized in that: When processing RGB images, weights can be assigned to the three colors of red, green, and blue according to the characteristics of human eye perception to better reflect the visual perception of the human eye. The specific conversion algorithm is as follows: Gray(a,b)=R(a,b)×0.299+G(a,b)×0.587+B(a,b)×0.114 4. The ResNetPA attention method of claim 1, wherein: The self-attention module improves the deep convolutional neural network to extract pedestrian features. First, the basic convolutional neural network is trained on a large dataset. The trained basic convolutional neural network can well extract the feature information of the image. The pedestrian image is input into the trained convolutional neural network. Then, each residual block in the baseline convolutional network is connected to self-attention, and the last three residual blocks are fused. The residual block fusion mainly performs linear interpolation operations on the 4th and 5th convolutional layers to normalize the obtained features. In addition, 1×1 convolution is used in the 3rd and 4th convolutional layers to achieve dimensional unification, and finally the three features of the same size are fused.

5. The ResNetCPA attention method of claim 1, wherein: A global channel attention mechanism is introduced to enable the model to extract more comprehensive pedestrian features. Traditional channel attention usually reduces the dimension of each channel when performing attention mechanism calculation. Although this method can learn the interaction relationship between channels, it takes a long time to capture the relevant attention relationship, and only focusing on local relationships may have a bad impact on the correlation results between channels. Therefore, the present invention adopts a global channel attention method (Channel Relation-Aware Global Attention, RGA-C), which aims to capture the relationship between global feature information and each channel in feature representation.

6. The RGA-C attention method of claim 1, wherein: Its own channel feature information is added, and the global feature information and local attention information of the channel are utilized at the same time. Given a feature map X of size H×W×C, in order to clearly represent the relationship between channels, each channel is regarded as a feature point. Therefore, the feature map X has a total of C feature points, and each feature point is represented as x i , where i = 1,…,C. Then the relationship between the two channels R i,j The mathematical expression of is shown in the formula. where θ and are two functions of embedding feature nodes, which are used to transform x i and x j Mapped to a new feature representation space and thus converted into a new feature representation, both are mapped through 1×1 convolution, batch normalization (BN) and ReLU activation function. In addition, in order to learn the relationship between channels more comprehensively, this method also adds its own channel feature information, and uses the global feature information and local attention information of the channel, so the two are combined to obtain the spatial relationship perception feature y i ′, as shown in the formula. in and ψ represent the channel feature itself x i and the global channel relation r i The embedding function mainly converts x i and r i Mapped to the same feature domain, both embedding functions undergo a 1×1 convolution, batch normalization (BN), and ReLU activation function to complete the mapping. The RGA-C attention method obtains more important feature information in pedestrian images by paying comprehensive and effective attention to the channel dimension, thereby enhancing the expression ability of the network model and improving performance. Therefore, the attention method ResNetCPA proposed in this method combines the channel attention RGA-C method with the previously proposed ResNetPA method to further improve the feature extraction of RGB modality. Specifically, the RGA-C method is further added to the residual block of ResNet50.

Citation Information

Patent Citations

  • Cross-modal pedestrian re-identification method based on sample center loss function

    CN113569639A

  • Cross-modal pedestrian re-identification method based on channel recombination and attention mechanism

    CN115100678A

  • Cross-modal pedestrian re-identification method based on spectrum perception and attention mechanism

    CN116798070A

  • Transform-based cross-modal pedestrian re-identification method

    CN116863501A

Cited By

  • Domain generalization personnel re-identification method and system based on multi-modal fusion and structure perception enhancement

    CN121747157A