Occluded pedestrian re-identification method based on cross attention feature fusion
By using the cross attention feature fusion method in pedestrian re-identification, the problem of pedestrian recognition misleading in the case of occlusion is solved, and the precise matching and local information matching ability of occlusion pedestrian re-identification are improved.
Patent Information
- Application Number
- CN202411257736.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-09
- Publication Date
- 2025-01-10
AI Technical Summary
The prior art is difficult to deal with occlusion in pedestrian re-identification, which leads to misleading recognition and difficulty in model training, especially when models trained in centralized pedestrian images without occlusion cannot fit pedestrian recognition.
Using a method based on cross attention feature fusion, local features of pedestrian images are extracted through Transformer network blocks, and the self-attention mechanism and cross attention module are integrated to enhance the network's ability to match local information and adapt to pedestrian re-identification under occlusion. .
It realizes accurate matching of pedestrian re-identification tasks under occlusion, enhances the network's ability to match local information, reduces blocked information learning, and improves recognition accuracy.
Smart Images
Figure CN119296132A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to an occluded pedestrian re-identification method based on cross-attention feature fusion. Background Art
[0002] Pedestrian re-identification refers to identifying images or video sequences of the same pedestrian based on their appearance features in multiple non-overlapping camera fields of view. This is a computer vision task with important application value and challenges. In recent years, with the in-depth development of deep learning, neural networks have shown astonishing results in different fields. Similarly, this has greatly promoted the rapid development of pedestrian re-identification. With the development of science and technology and the popularization of street cameras, pedestrian re-identification has become more important. It can provide effective technical support for the fields of intelligent security, video surveillance, criminal investigation, etc.
[0003] However, due to the complexity of the real world, matching people with different identities is still a challenging task, and pedestrian re-identification still faces many challenges. Such as posture, privacy, cross-scene, occlusion and other issues. At the same time, because the activity of people is uncertain, the target person is likely to be occluded by various objects in the real world. The occluders may be non-target pedestrians or non-pedestrian objects, which will greatly increase the difficulty of the re-identification task. Specifically, when the target pedestrian to be identified is partially obscured by an occluder, the recognition network is likely to mistake the occluder for part of the target pedestrian, thereby misleading the recognition network to varying degrees. For example, when a human body is obscured by a green mailbox, the network will think that the green mailbox is part of the target person, and thus pay more attention to the green mailbox, resulting in failure to identify the target person.
[0004] However, in the process of model training, the model is often easily affected by occlusions, and the model trained on a set of pedestrian images without occlusions is difficult to fit pedestrian recognition under occlusion, resulting in the trained model being unable to complete the re-identification task under occlusion. Summary of the invention
[0005] In view of the shortcomings of the prior art, the present invention provides an occluded pedestrian re-identification method based on cross-attention feature fusion. The present invention establishes connections between local parts during model training and obtains the connection information between local parts, thereby enhancing the network's ability to match local information to adapt to pedestrian images under occlusion and achieve accurate matching of occluded pedestrian re-identification tasks.
[0006] The technical means adopted by the present invention are as follows:
[0007] A method for re-identifying an occluded person based on cross-attention feature fusion, comprising the following steps:
[0008] A pedestrian image is obtained from a street view surveillance video, wherein the pedestrian image contains multiple pedestrians, and a backbone network is used to extract features of the pedestrian image, thereby obtaining a global feature vector of the image; the backbone network is configured to include a plurality of serially connected Transformer blocks, wherein the Transformer block is composed of a multi-head self-attention mechanism module, a normalization module, and a fully connected layer;
[0009] Performing a pooling operation on the acquired global feature vector using an average pooling layer, wherein the pooling operation is used to reduce the dimension of the global feature vector;
[0010] Segmenting the global features after the pooling operation, and segmenting the global features into several partial feature vectors according to the channel dimension, so as to obtain partial level features;
[0011] Extract attention from part-level features so that each part-level feature focuses on its own important area;
[0012] Using a cross-attention module to cross-fuse the partial-level features after attention extraction, the cross-fusion is used to make the partial-level features after attention extraction learn the feature information of the adjacent parts, so as to obtain auxiliary features after cross-fusion;
[0013] After the cross-fused partial-level feature vectors are normalized, similarity is calculated with the features of the queried pedestrian, thereby achieving occluded pedestrian re-identification.
[0014] Furthermore, the step of extracting features by the Transformer block includes:
[0015] Firstly, the pedestrian features of the pedestrian image are extracted through the multi-head self-attention mechanism, and the three vectors q, k, and v required by the self-attention mechanism are obtained through linear mapping;
[0016] Then, a new feature vector is obtained through matrix multiplication, and then passed through a fully connected layer and a normalization layer. After normalization, a residual connection is performed with the previous vector.
[0017] Finally, the global feature vector is obtained after a linear layer.
[0018] Furthermore, the whole process of obtaining the partial-level feature vector can be expressed as:
[0019] p 0,1,2,3 =attr(G(f))
[0020] Among them, attr() represents the self-attention module, G(·) represents the adaptive average pooling layer, f represents the global feature vector, and p 0,1,2,3It means that the global feature is divided into four partial features, numbered 0, 1, 2, and 3 respectively.
[0021] Furthermore, the self-attention module includes multiple 3*3 convolutional layers, batch normalization and Relu activation functions, wherein the input dimension of the 3*3 convolutional layer is consistent with the output dimension, the kernel size is 3, the step size is 1, and the image padding is 1. The global feature vector after average pooling first enters the 3*3 convolutional layer, then performs batch normalization again, and finally enters the Relu activation function.
[0022] Further, the cross-attention module is used to cross-fuse the partial-level features after attention extraction, and the cross-fusion is used to make the partial-level features after attention extraction learn the feature information of the adjacent parts, so as to obtain the auxiliary features after cross-fusion, including obtaining the auxiliary features after cross-fusion according to the following calculation:
[0023]
[0024] Where N is the number of divided parts.
[0025] Furthermore, the cross-attention module includes a plurality of self-attention layers and linear layers. The cross-attention module receives auxiliary features and part-level features, cross-fuses the two, and obtains a cross-fused image part-level feature vector F:
[0026]
[0027] Among them, ω(·) represents the linear mapping of the linear layer, represents matrix multiplication, softmax(·) is the softmax activation function, and p j Represents the part-level features after attention extraction, j = 0, 1, 2, 3.
[0028] Furthermore, the entire cross-attention module is supervised and trained using a triple loss function, where the loss function of the triple loss is:
[0029]
[0030] in, represents the Euclidean distance of the positive sample, represents the Euclidean distance of negative samples, and m represents the margin.
[0031] The Euclidean distance calculation formula is:
[0032]
[0033] in represents the kth eigenvector, represents the kth true value, K represents the number of samples, Indicates a wide range of The Euclidean distance between samples i and j in .
[0034] Furthermore, the backbone network is supervised and trained using the ID loss function, which is defined as:
[0035]
[0036] Among them, W i is the weight matrix of sample i, y i represents the true value corresponding to sample i, F i represents the image part-level feature vector corresponding to sample i, and M represents the total number of samples.
[0037] Compared with the prior art, the present invention has the following advantages:
[0038] The present invention uses a Transformer network block to process pedestrian images. The Transform network divides the image into multiple partial-level feature blocks and embeds the position vector, and then performs feature extraction. Then, the processed overall feature vector is pooled through an adaptive pooling layer, and the self-attention mechanism is used to enhance the features after pooling. After the features are enhanced, they are divided into multiple partial local feature vectors. Thereafter, the cross-attention feature fusion module proposed by the present invention is used to perform a new cross-fusion on the segmented local features, thereby obtaining new fused features, and the module is supervised by triple loss, which enables the information between the local parts to achieve an information fusion effect, so that the network no longer learns occluded local information, but information that is related between local parts, thereby enhancing the network's ability to match local information, so as to adapt to pedestrian images under occlusion, and achieve accurate matching of occluded pedestrian re-identification tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0040] Figure 1 This is a flow chart of an occluded pedestrian re-identification method based on cross-attention feature fusion in the present invention.
[0041] Figure 2 The following is a detailed execution flow of the occluded pedestrian re-identification method based on cross-attention feature fusion in the embodiment.
[0042] Figure 3 This is a diagram of the network model structure for executing the occluded pedestrian re-identification method based on cross-attention feature fusion in the embodiment. DETAILED DESCRIPTION
[0043] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0044] like Figure 1-3 As shown, the present invention provides an occluded pedestrian re-identification method based on cross-attention feature fusion, comprising the following steps:
[0045] S1. Pedestrian images are captured through street view surveillance videos, and then the backbone network is used to extract features from these pedestrian images to obtain the global feature vector of the image.
[0046] The pedestrian image contains multiple pedestrians, and the backbone network is used to extract features from the pedestrian image, thereby obtaining the global feature vector of the image. In the specific implementation, the pedestrian image containing pedestrians is processed, and the specific target to be queried is called query, and other pedestrians are gallery. The backbone network is set to include multiple transformer blocks connected in series, and the transformer block is composed of a multi-head self-attention mechanism module, a normalization module, and a feedforward neural network. The feedforward neural network is not Figure 3 The fully connected layer in .
[0047] Among them, the steps of feature extraction through the Transformer block include:
[0048] First, the pedestrian features of the pedestrian image vector are extracted through a multi-head self-attention mechanism, and the three vectors q, k, and v required by the self-attention mechanism are obtained through linear mapping. The pedestrian image vector here is the pedestrian image vector obtained after pre-processing such as image enhancement.
[0049] Then, by performing matrix multiplication on the three vectors q, k, and v mentioned above, a new feature vector is obtained, which is then passed through a fully connected layer and a normalization layer. After normalization, it is residually connected with the previous vector.
[0050] Finally, the global feature vector is obtained after a linear layer.
[0051] S2. Perform a pooling operation on the obtained global feature vector using an average pooling layer, and the pooling operation is used to reduce the dimension of the global feature. The average pooling layer is used to perform a pooling operation on it. This step effectively reduces the channel dimension, thereby obtaining a more refined and representative global feature.
[0052] In a specific embodiment, the average pooling layer uses adaptive average pooling to simplify the channel dimension of the global features. Specifically, fixed dimension pooling is performed through the average pooling layer, so that the subsequent segmentation module can more effectively segment the global features and obtain the corresponding partial feature vectors.
[0053] S3, splitting the global features after the pooling operation, and splitting the global features into several partial feature vectors according to the channel dimension, so as to obtain partial-level features.
[0054] The global features processed by the adaptive average pooling layer are finely segmented and divided into multiple partial feature vectors according to the channel dimension to obtain fine features at the partial level. The specific segmentation action is completed by continuous learning of the neural network. In this application, the global features are divided into four parts. S4. The partial-level features are subjected to attention extraction, so that each partial-level feature pays more attention to its more important area and obtains more important information.
[0055] Specifically, the partial-level feature vector is extracted through the self-attention mechanism, so that the vector can focus on the important feature area itself and weaken the area where the pedestrian is blocked. The whole process of obtaining the partial-level feature vector in the present invention can be expressed as:
[0056] p 0,1,2,3 =attr(G(f))
[0057] Where attr(·) represents the self-attention module, G(·) represents the adaptive average pooling layer, f is the global feature vector, and p 0,1,2,3 It means that the global feature is divided into four partial features, numbered 0, 1, 2, and 3 respectively.
[0058] Furthermore, the self-attention mechanism module is mainly composed of multiple 3*3 convolutional layers, batch normalization and Relu activation function. The input dimension of the 3*3 convolutional layer is consistent with the output dimension, the kernel size is 3, the step size is 1, and the image padding is 1. The global feature vector after average pooling first enters the 3*3 convolutional layer, then performs batch normalization again, and finally enters the Relu activation function.
[0059] S5. Use the cross-attention module to cross-fuse the partial-level features after attention extraction, and the cross-fusion is used to make the partial-level features after attention extraction learn the feature information of the adjacent parts, so as to obtain the auxiliary features after cross-fusion. Use the cross-attention module to cross-fuse multiple partial-level features, so that the features not only contain their own important information, but also learn the feature information of the adjacent parts, so as to obtain the partial-level feature vector of the image after cross-fusion.
[0060] In a specific embodiment, the global feature is divided into equal proportions according to the channel dimension through a neural network. Specifically, the original global feature dimension is [b, 242, 768], which is divided into four [b, 768], where b is the batch size. In this embodiment, four 768-dimensional partial feature vectors are obtained, and then the four partial feature vectors are fused by addition, and the obtained fused vector is used as an auxiliary feature, where the fusion of the partial feature vectors into an auxiliary vector can be expressed as:
[0061]
[0062] After obtaining the auxiliary features, the main purpose is to attach the auxiliary features to the main features in an auxiliary form, so that the features have the characteristics of connection and interaction, which will ensure that the final vector is no longer occluded. The main features here refer to the partial-level features after attention extraction. Using the cross-attention fusion module, the auxiliary features are integrated into the four partial feature vectors, thereby obtaining four local vectors that are connected between local parts and are no longer occluded. It can be specifically expressed as:
[0063]
[0064] Among them, ω(·) represents the linear mapping of the linear layer, represents matrix multiplication, softmax(·) is the softmax activation function, cp i is the auxiliary feature, p j is the part-level feature after attention extraction, j = 0, 1, 2, 3.
[0065] Furthermore, this embodiment uses a triple loss function to supervise the training of the entire cross-attention module. The triple loss fine-tunes the cross-attention fusion module by shortening the similar distance between the positive sample and the feature and pushing the distance between the feature and the negative example, so that the cross-attention fusion module can better extract favorable regional features and effectively integrate them into some features. The loss function of the triple loss is:
[0066]
[0067] in, represents the Euclidean distance of the positive sample, represents the Euclidean distance of negative samples, and m represents the margin. The Euclidean distance is a commonly used method for calculating similarity in computer vision, and its formula is:
[0068]
[0069] in represents the kth eigenvector, represents the kth true value K represents the number of samples, Indicates a wide range of The Euclidean distance between samples i and j in .
[0070] In a specific embodiment, since the feature vector obtained by the entire backbone network will be supervised by the ID loss during the training process, after obtaining the final feature vector, the final feature vector is optimized using BNNeck, thereby using the ID loss for supervision, and the ID loss is defined as:
[0071]
[0072] Among them, W i is the weight matrix of sample i, y i represents the true value corresponding to sample i, F i represents the image part-level feature vector corresponding to sample i, and M represents the total number of samples.
[0073] S6. The obtained fused feature vector is normalized and finally used for identification of occluded pedestrian re-identification, so as to obtain the final output of the recognition result.
[0074] In a specific embodiment, batch normalization is used in the backbone network to normalize the vector, and the final feature vector after fusion is normalized using layer normalization. The normalized feature vector is similar to the feature vector of the target pedestrian to be identified, and the occluded pedestrian is re-identified based on the calculation result.
[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for occluded pedestrian re-identification based on cross-attention feature fusion, characterized in that: The following steps are involved: A pedestrian image is obtained from a street view surveillance video, wherein the pedestrian image contains multiple pedestrians, and a backbone network is used to extract features of the pedestrian image, thereby obtaining a global feature vector of the image; the backbone network is configured to include a plurality of serially connected Transformer blocks, wherein the Transformer block is composed of a multi-head self-attention mechanism module, a normalization module, and a fully connected layer; Performing a pooling operation on the acquired global feature vector using an average pooling layer, wherein the pooling operation is used to reduce the dimension of the global feature vector; Segmenting the global features after the pooling operation, and segmenting the global features into several partial feature vectors according to the channel dimension, so as to obtain partial level features; Extract attention from part-level features so that each part-level feature focuses on its own important area; Using a cross-attention module to cross-fuse the partial-level features after attention extraction, the cross-fusion is used to make the partial-level features after attention extraction learn the feature information of the adjacent parts, so as to obtain auxiliary features after cross-fusion; After the cross-fused partial-level feature vectors are normalized, similarity is calculated with the features of the queried pedestrian, thereby achieving occluded pedestrian re-identification.
2. The method for re-identifying pedestrians under occlusion based on cross-attention feature fusion according to claim 1, characterized in that: The steps of feature extraction in the Transformer block include: Firstly, the pedestrian features of the pedestrian image are extracted through the multi-head self-attention mechanism, and the three vectors q, k, and v required by the self-attention mechanism are obtained through linear mapping; Then, a new feature vector is obtained through matrix multiplication, and then passed through a fully connected layer and a normalization layer. After normalization, a residual connection is performed with the previous vector. Finally, the global feature vector is obtained after a linear layer.
3. The method for re-identifying pedestrians under occlusion based on cross-attention feature fusion according to claim 2, characterized in that: The whole process of obtaining the partial-level feature vector can be expressed as: p 0,1,2,3 =attr(G(f)) Where attr(·) represents the self-attention module, G(·) represents the adaptive average pooling layer, f represents the global feature vector, and p 0,1,2,3 It means that the global feature is divided into four partial features, numbered 0, 1, 2, and 3 respectively.
4. The method for re-identifying an occluded pedestrian based on cross-attention feature fusion according to claim 3, characterized in that: The self-attention module includes multiple 3*3 convolutional layers, batch normalization and Relu activation function, where the input dimension and output dimension of the 3*3 convolutional layer are consistent, the kernel size is 3, the step size is 1, and the image padding is 1. The global feature vector after average pooling first enters the 3*3 convolutional layer, then performs batch normalization again, and finally enters the Relu activation function.
5. The method for re-identifying an occluded pedestrian based on cross-attention feature fusion according to claim 3, characterized in that: The cross-attention module is used to cross-fuse the partial-level features after attention extraction, and the cross-fusion is used to make the partial-level features after attention extraction learn the feature information of the adjacent parts, so as to obtain the auxiliary features after cross-fusion, including obtaining the auxiliary features after cross-fusion according to the following calculation: Where N is the number of divided parts.
6. The method for re-identifying pedestrians under occlusion based on cross-attention feature fusion according to claim 5, characterized in that: The cross-attention module includes multiple self-attention layers and linear layers. The cross-attention module receives auxiliary features and part-level features, cross-fuses the two to obtain a cross-fused image part-level feature vector F: Among them, ω(·) represents the linear mapping of the linear layer, represents matrix multiplication, softmax(·) is the softmax activation function, and p j Represents the part-level features after attention extraction, j = 0, 1, 2, 3.
7. The method for re-identifying pedestrians under occlusion based on cross-attention feature fusion according to claim 6, characterized in that: The entire criss-cross attention module is supervised for training using a triple loss function, where the loss function of the triple loss is: in, represents the Euclidean distance of the positive sample, represents the Euclidean distance of negative samples, and m represents the margin. The Euclidean distance calculation formula is: in represents the kth eigenvector, represents the kth true value, K represents the number of samples, Indicates a wide range of The Euclidean distance between samples i and j in .
8. The method for re-identifying an occluded person based on cross-attention feature fusion according to claim 7, characterized in that: The backbone network is supervised and trained using the ID loss function, which is defined as: Among them, W i is the weight matrix of sample i, y i represents the true value corresponding to sample i, F i represents the image part-level feature vector corresponding to sample i, and M represents the total number of samples.