Pose-guided occluded pedestrian re-identification method and device based on cross-attention feature fusion

Abstract

This invention belongs to the field of pedestrian re-identification technology and relates to a pose-guided cross-attention feature fusion algorithm for occluded pedestrian re-identification. First, a foreground-aware channel attention mechanism and a key feature space cross-attention fusion mechanism guide the backbone network to perceive pedestrian appearance information. Second, leveraging the robustness of confidence levels of human key points, a mask for predefined skeletal regions is generated through mathematical modeling. Finally, an interaction matrix is ​​established between the local skeletal features of the pedestrian and the feature maps output by the backbone network. Under the constraint of the spatial attention mask, the semantic alignment capability of the local skeletal features is enhanced through the fusion of local contextual information. Experimental results show that the proposed algorithm has a significant performance improvement compared to existing methods.

Description

Technical Field

[0001] This invention belongs to the field of pedestrian re-identification technology, and in particular to a pose-guided cross-attention feature fusion algorithm and device for occluded pedestrian re-identification. Background Technology

[0002] Pedestrian re-ID aims to identify the same person from non-overlapping camera views. It has attracted increasing attention due to its potential applications in intelligent surveillance, such as pedestrian retrieval, cross-camera tracking, and anomaly detection. Although research on person re-ID has been ongoing for about a decade, changes in lighting, pedestrian occlusion, and background variations still make matching in overlapping scenes extremely challenging. Among these adverse factors, occlusion is one of the key issues, significantly impacting the performance of re-identification.

[0003] The core task of occluded pedestrian re-identification is to study how to decouple occlusion information from pedestrian appearance information in images and construct effective pedestrian representations to find target pedestrians based on their similarity. Currently, mainstream methods typically utilize advanced pedestrian pose estimation models to assist re-identification models in locating and extracting features from the unoccluded regions of pedestrians. For example, the PGFA network filters occlusion information from the keypoint heatmap output by the pose estimation network and the feature map output by the backbone network to obtain local features corresponding to each keypoint. IVPE integrates pose information into the re-identification framework, designing a pose mask feature branch that allows the model to focus only on the pedestrian's body region, employing pose embedding information to construct pedestrian features with discriminative information. To consider the influence of occluded region information while focusing on visible information, Wang constructed a pose-guided feature disentangling model. This method models and describes both occluded and unoccluded information separately to enhance the learning of recognition features. These methods utilize the semantic information and spatial location information of pedestrian pose keypoints to achieve pedestrian region localization, improving re-identification performance to some extent.

[0004] However, these methods still have two shortcomings:

[0005] First, simply extracting visible features from the output feature map of the backbone network cannot decouple occlusion information. Because the receptive field of spatial units in the feature map expands as the network deepens, the receptive field of units in non-occluded areas will cover occluded areas, causing occlusion information to be transmitted to areas that were originally unoccluded.

[0006] Second, using a pedestrian feature extraction method similar to mask filtering or semantic alignment on the feature map output by the backbone network limits the backbone network's ability to perceive and make decisions about occlusion information. This method only updates some neurons in the network, thus limiting the network's learning of another part of the neurons related to occlusion and background information. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a pose-guided, cross-attention feature fusion algorithm and apparatus for occluded pedestrian re-identification. This algorithm, based on feature fusion, combines pose information with the backbone network and local feature extraction network in the re-identification network, improving the network's ability to perceive and make decisions regarding occlusion information and to extract robust features of pedestrians.

[0008] The technical solution adopted by this invention to solve the technical problem is:

[0009] This method first constructs a visual appearance region attention module, employing a multi-layer pose information soft fusion approach to drive the backbone network to learn occlusion information in intermediate layers. It also mitigates the problem of occlusion information diffusion through an inhibition mechanism, unlike mainstream pose estimation methods that only filter high-level features. Secondly, this invention innovatively generates predefined skeletal region masks using mathematical modeling, leveraging human keypoint information. Compared to masks generated by semantic segmentation, this method provides pixel-level visibility confidence and exhibits stronger stability in low-resolution pedestrian image conditions. Finally, an interaction matrix is ​​established between the pedestrian's skeletal local features and the features output by the backbone network. Through the fusion of local contextual information, pedestrian retrieval features are learned.

[0010] The first aspect of this invention is to provide a pose-guided cross-attention feature fusion algorithm for occluded pedestrian re-identification. First, a foreground-aware channel attention mechanism and a key feature space cross-attention fusion mechanism are used to guide the backbone network to perceive pedestrian appearance information. Second, the robustness of the confidence of human key points is utilized to generate a mask for a predefined skeletal region through mathematical modeling. Finally, an interaction relationship matrix is ​​established for the local skeletal features of the pedestrian and the feature map output by the backbone network. Under the constraint of the spatial attention mask, high-level semantically aligned pedestrian retrieval features are generated through the fusion of local contextual information.

[0011] A second aspect of the present invention is to provide an apparatus for implementing the above-described algorithm, comprising:

[0012] The visual appearance area attention module guides the backbone network to perceive pedestrian appearance information through the channel attention mechanism of foreground perception and the cross-attention fusion mechanism of key feature space.

[0013] The skeletal region modeling module utilizes the robustness of confidence in human key points to generate masks for predefined skeletal regions through mathematical modeling.

[0014] The cross-attention fusion module of skeleton modulation establishes an interaction matrix for the local skeletal features of pedestrians and the feature maps output by the backbone network. Under the constraint of spatial attention mask, it generates high-level semantically aligned pedestrian retrieval features through the fusion of local contextual information.

[0015] Furthermore, the visible appearance region attention module consists of two or more modules, which are embedded in different layers of the backbone network, including a channel attention submodule for foreground perception and a cross-attention fusion submodule for key feature space.

[0016] Furthermore, the foreground-aware channel attention submodule encodes the features of the foreground region using a keypoint heatmap, calculates the attention score of each channel for the foreground portion, and then performs channel weighting on the feature map input to the Visual Appearance Region Attention (VARA) module to suppress the feature responses of channels related to occlusion and background noise.

[0017] Furthermore, the algorithm for the foreground-aware channel attention submodule is as follows:

[0018] First, the key point heatmap F k The channel signal vector is obtained by global average pooling downsampling of the features corresponding to the non-zero regions. This process can be represented as:

[0019] V c =AvgPool(F in ⊙F k )

[0020] In the formula, ⊙ represents the element-wise product of feature maps; AvgPool(·) represents the global average pooling operation;

[0021] Secondly, connect two different convolutional layers {C in / r,1×1,1} and {C in ,1×1,1}, and add ReLU and Sigmoid activation functions after each convolutional layer, and in the parameters of the two convolutional layers ({C in / r,1×1,1} and {C in In the 1×1,1) setting, the first term represents the number of output channels, the second term represents the size of the convolution kernel, the third term represents the stride of the convolution kernel, and the channel attention vector. The calculation process is expressed as follows:

[0022] A c = sigmoid(W2relu(W1V) c))

[0023] in, and These represent the weight parameters of the two convolutional layers; r is the dimension decay ratio.

[0024] Finally, the generated channel score vector A c The feature map F is applied to the module input in a product manner. in Generate channel-weighted feature maps

[0025] Furthermore, the algorithm for the key feature space cross-attention fusion submodule is as follows:

[0026] First, feature vectors are extracted from the heatmaps of 14 key points, forming a sparse feature map. The feature map obtained by weighting the output of the foreground perception submodule Consider it as a dense feature map;

[0027] F d The channels are reduced in dimensionality using a 1×1 convolutional layer and then flattened in spatial plane to obtain the input for the attention mechanism. Similarly, F s After two independent operations as described above, we obtain and Q d V s and K s These are represented as query vector, value vector, and key vector, respectively, where t represents the dimension decay rate.

[0028] Finally, a 1×1 convolutional layer is used to restore the number of channels in the cross-attention module output feature map to C. in and resize it to H in ×W in ×C in / t, then with feature map F d Make residual connections and input them into the next convolutional layer of the ResNet network.

[0029] Furthermore, the skeletal region modeling module uses keypoint coordinates and confidence scores to model the features of the skeletal region between keypoints, generating a soft mask with pixel confidence scores for the human skeleton, and using it to extract local features of the skeleton.

[0030] Furthermore, the specific algorithm for the skeletal region modeling module is as follows:

[0031] First, the skeletal regions of pedestrians need to be predefined;

[0032] Secondly, each predefined bone region is modeled separately to obtain the corresponding weight mask. The weight mask M for a specific local bone region is represented as:

[0033] r=α*exp(γ(-(sinθ) 2 ))l AO

[0034]

[0035]

[0036] Where θ represents the angle between line segments AB and CO, l AO denoted by AO, α and γ represent the overall radiation scale expansion coefficient and the lateral compression coefficient of the radiation scale, respectively, and I(·,·) represents the confidence level of the current local skeletal region;

[0037] After obtaining the local feature mask, when extracting the local features corresponding to each skeletal region, the output feature maps of Conv.B1, VARA1, and Conv.B4 layers are fused to obtain the feature map F'∈R. 4H×4W×C ;

[0038] M of each layer of mask i ∈R 4H×4W×1 By multiplying the corresponding pixels of each feature map F', the local feature maps of the skeletal region are obtained. Then, a global average pooling is performed to obtain a local feature vector with C channels. This process can be represented as:

[0039]

[0040] In the formula, pool(·) represents the global average pooling operation.

[0041] Furthermore, the skeleton-modulated cross-attention fusion module (SMCF module) comprises a multi-head self-attention submodule and a skeleton-modulated cross-attention submodule (SMC). The input to the module is the flattened feature map G∈R output by the backbone network. HW×C Local features of the skeleton * ∈R 14×C Attention weight map F generated by the skeletal mask w ∈R 14×HW G and f * After dimensionality reduction through 1×1 convolutional layers, we obtain G. in ∈R HW×C' and C′ is the number of filters in a 1×1 convolutional layer;

[0042] Local feature map of skeleton The input is first fed into a multi-head self-attention module, where the self-attention mechanism performs a global analysis on the feature map and uncovers the relationships between features; then, the multi-head self-attention outputs the result. The query matrix is ​​input to the cross-attention submodule (SMC) of the skeleton modulation; the key matrix and value matrix are given by G. in Provided; the computational process of the cross-attention submodule for skeletal modulation is represented as follows:

[0043]

[0044] In the formula, and These represent the weights of the three fully connected layers, d w Indicates the scale factor.

[0045] The attention weight map F w The cross-attention map generated in the cross-attention submodule of bone modulation is multiplied element-wise to constrain the fusion process at the local spatial level. During alignment and fusion, each bone feature only focuses on the relevant region corresponding to its mask. Finally... After passing through residual connections, a normalization layer, and a fully connected layer, the fused local skeletal features are obtained.

[0046] A third aspect of the present invention is to provide an electronic device, including a memory and a processor; wherein:

[0047] Memory: Used to store instructions that can be executed by the processor;

[0048] Processor: The processor is configured to perform the following: guide the backbone network to perceive pedestrian appearance information through a foreground-aware channel attention mechanism and a key feature space cross-attention fusion mechanism; utilize the robustness of the confidence of human key points to generate a mask for a predefined skeletal region through mathematical modeling; establish an interaction relationship matrix for the local skeletal features of the pedestrian and the feature map output by the backbone network; and generate high-level semantically aligned pedestrian retrieval features through the fusion of local contextual information under the constraint of the spatial attention mask.

[0049] The advantages and positive effects of this invention are:

[0050] 1. This algorithm proposes a visual appearance region attention module, which is organically combined with the backbone network. Through the channel attention mechanism of foreground perception and the cross-attention fusion mechanism of key feature space, it guides the backbone network to perceive pedestrian appearance information, improves the model's ability to perceive and identify occlusion information, and alleviates the problem of occlusion information diffusion caused by the increase of the network's receptive field.

[0051] 2. This algorithm utilizes the robustness of human keypoint confidence through a skeletal region modeling module to generate predefined skeletal region masks using mathematical modeling. Compared to masks generated by semantic segmentation, this method can provide pixel-level visibility confidence and exhibits stronger stability under low-resolution pedestrian image conditions. It compensates for some information loss in pedestrian appearance regions between keypoints in the heatmap output by the pose estimation model.

[0052] 3. The cross-attention fusion module of the skeleton modulation in this algorithm uses the local region weight map provided by the skeleton mask to constrain the cross-attention fusion process between the features output by the backbone network and the local features, thereby realizing the local feature discrimination enhancement and high-level semantic feature alignment. Attached Figure Description

[0053] Figure 1 This is a diagram of the PCRF network structure.

[0054] Figure 2 Here is a detailed structural diagram of the VARA module;

[0055] Figure 3 Example of a pedestrian keypoint heatmap output by the ViTPose model;

[0056] Figure 4 This is a schematic diagram of the predefined skeletal region division;

[0057] Figure 5 A schematic diagram of the skeletal region modeling;

[0058] Figure 6 Here is a diagram of the SMCF module structure;

[0059] Figure 7 Visualization results of a local skeletal region mask;

[0060] Figure 8 This is a feature heatmap for visualization purposes in this invention. Detailed Implementation

[0061] The present invention will be further described in detail below through specific embodiments. The following embodiments are merely descriptive and not limiting, and should not be used to limit the scope of protection of the present invention.

[0062] The PCRF network of this invention mainly consists of three parts: a visual appearance region attention module (embedded in the backbone network), a skeleton region modeling module, and a skeleton-modulated cross-attention fusion module.

[0063] First, the visual appearance region attention module achieves the organic integration of pose information and multi-layer backbone network. Through the channel attention mechanism of foreground perception and the cross-attention fusion mechanism of key feature space, it guides the backbone network to perceive pedestrian appearance information, improves the model's ability to perceive and identify occlusion information, and alleviates the problem of occlusion information diffusion caused by the increase of network receptive field.

[0064] Secondly, the skeletal region modeling module leverages the robustness of confidence levels in human key points to generate masks for predefined skeletal regions through mathematical modeling. Compared to masks generated by semantic segmentation, the method of this invention can provide pixel-level visibility confidence and exhibits stronger stability under low-resolution pedestrian image conditions.

[0065] Finally, the cross-attention fusion module of skeleton modulation establishes an interaction matrix for the local skeletal features of pedestrians and the feature maps output by the backbone network. Under the constraint of the spatial attention mask, high-level semantically aligned pedestrian retrieval features are generated through the fusion of local contextual information.

[0066] To enhance the backbone network's attention to visible regions in occluded images, mainstream methods often fuse pose features directly with the backbone network's output features. However, since the receptive field of a feature map spatial unit expands with network depth, the receptive field of a non-occluded region in the backbone network's output layer can cover the occluded region. This causes occlusion information to be transmitted to previously unoccluded regions, preventing the algorithm from decoupling occlusion information. To address this, this invention proposes a pose-semantic information-guided Visible Appearance Region Attention (VARA) module and embeds it into ResNet-50. VARA is embedded in different layers of the ResNet-50 backbone network, constructing an attention-guided pose information soft fusion mechanism. This drives the backbone network to learn the differences between occlusion and pedestrian information in intermediate layers and adaptively suppress the expression of occlusion information. Compared to mainstream pose estimation-based methods, this module, through a multi-layer soft fusion mechanism, effectively alleviates the problem of occlusion information diffusion caused by the expansion of the receptive field.

[0067] The VARA module comprises a foreground-aware channel attention submodule and a key feature spatial cross-attention fusion submodule. Since pose keypoint information provides relatively reliable spatial relationships between local regions of the human body, employing a channel-first, then spatial attention approach helps suppress identity-irrelevant channels and improves the reliability of spatial feature fusion. Figure 1 As shown, the Conv.B2 and Conv.B3 layers of the backbone network are connected to two VARA modules in a serial manner. Figure 2The detailed internal structure of the VARA module is shown below. The structure and principles of the two sub-modules are explained in detail below.

[0068] Traditional channel attention mechanisms typically use Global Average Pooling (GAP) to compress the spatial dimension of the feature map to obtain channel signal vectors, where the weights of feature information distributed across the spatial dimension are equal. Clearly, the channel signal vectors obtained through GAP are not an optimal choice for representing channel importance. If the influence of each spatial pixel is the same, the attention mechanism may be affected by cluttered background or occlusion information, failing to provide accurate information for modeling the independence between channels. To address this, a foreground-aware channel attention submodule is proposed. Its core is to first encode the features of the foreground region using a keypoint heatmap, calculate the attention score for each channel regarding the foreground portion, and then perform channel weighting on the VARA input feature map to suppress the feature responses of channels related to occlusion and background noise.

[0069] A ResNet-50 network pre-trained on the ImageNet dataset was used as the backbone of the CNN, with a simple modification: the average pooling layer and fully connected layer at the tail were removed to obtain a feature map F∈R with spatial information. H×W×C Here, H, W, and C represent height, width, and number of channels, respectively. The internal structure of the ResNet-50 network can be divided into five parts, defined as conv1, and conv.B1, conv.B2, conv.B3, and conv.B4, which consist of Bottleneck residual blocks. The latter four parts contain 3, 4, 6, and 3 Bottleneck residual blocks, respectively, and each residual block contains 3 convolutional layers. With the image resolution of the backbone network input set to 256×128, the feature map output by the backbone CNN has a resolution of 8×4.

[0070] Define the feature map of the VARA input as follows: H in W in and C in Let represent the height, width, and number of channels of the input feature map, respectively; define the input heatmap containing 14 keypoints as . The spatial dimensions of the key point heatmap are consistent with the spatial dimensions of a single channel in the input feature map. Firstly, as... Figure 2 As shown, the key point heatmap F k The channel signal vector is obtained by global average pooling downsampling of the features corresponding to the non-zero regions. This process can be represented as:

[0071] Vc =AvgPool(F in ⊙F k )

[0072] In the formula, ⊙ represents element-wise multiplication of feature maps; AvgPool(·) represents global average pooling operation. Next, two different convolutional layers {C} are connected. in / r,1×1,1} and {C in The convolutional layer is defined as {1×1,1}, and ReLU and Sigmoid activation functions are added after each convolutional layer to achieve a Squeeze-and-Excitation attention mechanism. In the parameter settings of the convolutional layer, the first term represents the number of output channels, the second term represents the size of the convolutional kernel, and the third term represents the stride of the convolutional kernel. Channel attention vector. The calculation process is expressed as follows:

[0073] A c = sigmoid(W2 relu(W1 V) c ))

[0074] in, and These represent the weight parameters of the two convolutional layers; the dimension decay ratio r is set to 8. Finally, the generated channel score vector A... c The feature map F is applied to the module input in a product manner. in Generate channel-weighted feature maps Compared to directly applying global average pooling to feature maps, this method uses prior semantic knowledge of key points to filter irrelevant information distributed in space. Driven by the channel attention mechanism, different convolutional kernels of the convolutional layer (one kernel corresponds to the feature map of one channel) can adaptively decide the importance of the extracted patterns, thereby improving the model's ability to perceive outliers.

[0075] The combination of channel and spatial attention mechanisms has been widely used to improve the discriminative power of original feature maps. The first step in traditional spatial attention is often average pooling along the channel direction of the feature map, followed by flattening the resulting 2D features into a feature vector. However, the average pooling process in this operation ignores the importance differences between different channels of the feature map. To address this, a key feature space cross-attention fusion submodule is proposed. This module enhances the model's attention to the context-visible region at the spatial level without channel compression, thereby further enhancing the input feature map.

[0076] First, feature vectors are extracted from the heatmaps of 14 key points, forming a sparse feature map. The feature map obtained by weighting the output of the previous submodule It is considered a dense feature map. To achieve linear transformation of vector features while reducing computational cost during attention calculation, F... d The channels are reduced in dimensionality using a 1×1 convolutional layer and then flattened in spatial plane to obtain the input for the attention mechanism. Similarly, F s After two independent operations as described above, we obtain and Q d V s and K s Let be the query vector, value vector, and key vector, respectively, and t represent the dimensionality decay ratio. The cross-attention mechanism mines the correlation between sparse and dense features, effectively integrating the importance representations of different local features for each pixel region, thereby enhancing the expression of pedestrian appearance features in the original feature map. Finally, a 1×1 convolutional layer restores the number of channels in the cross-attention module's output feature map to C. in and resize it to H in ×W in ×C in / t, then with feature map F d Make residual connections and input them into the next convolutional layer of the ResNet network.

[0077] The output of a human pose estimation network typically includes the defined keypoint coordinates, confidence scores, and their corresponding heatmaps. Most methods use the keypoint heatmap as a mask for human body parts to extract local features of pedestrians or utilize semantic segmentation results from pedestrian images to align to more detailed semantic regions. However, on the one hand, the radiation range of the pedestrian keypoint heatmap differs somewhat from the pedestrian's appearance region, making the former method's extraction of pedestrian features incomplete (ViTPose model, where the sum of the radiation range of the pedestrian keypoint heatmap does not cover the complete pedestrian appearance region, and some extractable appearance information still exists between adjacent keypoints). Figure 3 On the other hand, when the image resolution is low, the accuracy of the semantic mask in the latter method is easily affected.

[0078] To address this, a Skeletal Area Modeling (SAM) strategy is proposed. This strategy uses keypoint coordinates and confidence scores to model the features of the skeletal region between keypoints, generating a soft mask with pixel confidence scores for the human skeleton, and using it to extract local features of the skeleton.

[0079] First, the skeletal regions of the pedestrian need to be predefined. Based on the human keypoint settings of the CrowdPose dataset, the method delineates the following pedestrian skeletal regions: upper arm, lower arm, upper leg, lower leg, torso, and head. Considering that the torso occludes a large portion of the body and is susceptible to occlusion, it is further subdivided into four sub-regions: selecting the midpoints of the four sides constituting the torso, and connecting these midpoints to divide the torso into four sub-regions: upper left torso, upper right torso, lower left torso, and lower right torso. This results in 14 skeletal regions (where the upper arm, lower arm, upper leg, and lower leg each contain left and right parts), as shown below. Figure 4 As shown. Each sub-region of the torso is composed of the midpoints of its top and bottom edges and the midpoints of its left and right edges. The confidence of the midpoints is calculated by linear interpolation, while other skeletal regions are composed of two corresponding keypoints.

[0080] Secondly, each predefined skeletal region is modeled separately to obtain the corresponding weight mask. For example... Figure 5 As shown, keypoints A and B at both ends of a certain skeletal region are defined as A and B, respectively, point C (i,j) is an arbitrary point in the image plane space, and point O (c x ,c y Let A be the midpoint of AB, and let the confidence levels of points A and B be δ. A and δ B The weight mask M for a specific local skeletal region can be represented as:

[0081] r=α*exp(γ(-(sinθ) 2 ))l AO

[0082]

[0083]

[0084] Where θ represents the angle between line segments AB and CO, l AO denoted by AO, r represents the radiation range, α and γ represent the overall radiation scale expansion coefficient and the transverse (perpendicular to line segment AB) compression coefficient of the radiation scale, respectively, and I(·,·) represents the confidence level of the current local skeletal region. When calculating this confidence level, (δ) needs to be considered. A +δ BThe result of () / 2 is compared with the threshold ρ. If it exceeds the threshold, the result is retained; otherwise, it is compressed to 0.5 of the original value. The threshold ρ = 0.7 is the optimal threshold obtained through debugging. It should be noted that since the details of the mask region can only be revealed at high resolution, the spatial layer size of the mask modeling is set to 4H×4W, consistent with the spatial layer size of the Conv.B1 layer output feature map. Following the above method, 14 predefined local skeletal regions are modeled, and the final generated mask map is represented as M∈R. 4H×4W×14 This adaptive method of generating skeletal masks can effectively suppress responses from occluded or background regions and achieve a smooth transition between responses from human and non-human regions, resulting in more complete semantics of the extracted local features.

[0085] After obtaining the local feature mask, the most intuitive approach to extracting local features for each skeletal region is to directly multiply the mask and the feature map output by the backbone network. However, this method has certain drawbacks. The reason is that the details of the mask region can only be revealed at high resolution, resulting in a larger size than the output size of the backbone network. If the feature map is upsampled to fit the mask image size, the pixel features corresponding to the local mask region have high similarity but contain less detailed information. Conversely, if the mask image is downsampled to fit the backbone network output size, the local mask region becomes very coarse due to size compression, severely reducing the effectiveness of pose cues.

[0086] Therefore, to enhance the representation of local feature details while maintaining the mask image size, this invention employs a multi-layer fusion approach to alleviate this problem. Specifically, the output feature maps of Conv.B1, VARA1, and Conv.B4 layers are fused to obtain feature map F'∈R. 4H×4W×C The fused feature map integrates high-level semantic information and shallow-level detail information, making up for the shortcomings of directly upsampling the feature map output by the backbone network.

[0087] Next, local feature extraction is performed, and the predefined skeletal region is modeled to obtain the corresponding mask image M∈R. 4H×4W×14 . Make each layer of mask M i ∈R 4H×4W×1 By multiplying the corresponding pixels of each feature map F', the local feature maps of the skeletal region are obtained. Then, a global average pooling is performed to obtain a local feature vector with C channels. This process can be represented as:

[0088]

[0089] In the formula, pool(·) represents the global average pooling operation. The mask modeling process considers the key point coordinates and their confidence levels. Therefore, when there is an occlusion in a certain bone region, the response of the mask pixels corresponding to the occluded part is low, which can suppress the expression of occlusion information in local features.

[0090] Although the generation of local skeletal features considers the confidence level and spatial radiation characteristics of human keypoints, the skeletal modeling process does not include relevant trainable parameters and relies on the spatial location information of keypoints output by the human pose estimation model, thus exhibiting poor flexibility. Furthermore, the skeletal mask only provides pixel reliability information (i.e., visibility information) within a part's range, but not information on the discriminative strength of pixel features within that range. Therefore, the discriminative power of the extracted local skeletal features is not fully exploited.

[0091] To address the two issues mentioned above, a cross-attention fusion module (SMCF) based on skeleton modulation is proposed. This module incorporates part of the Transformer decoder structure and utilizes a cross-attention mechanism to dynamically capture the semantic correlation between the features output by the backbone network and the local features of the skeleton. Since the local features of the skeleton represent the features of a region containing several spatial pixels, the alignment and fusion process can be simplified to a "one-bone-to-multiple-pixels" problem.

[0092] Specifically, such as Figure 6 As shown, the SMCF module contains a multi-head self-attention submodule and a skeleton-modulated cross-attention (SMC) submodule. The input to the module is the flattened feature map G∈R output by the backbone network. HW×C Local features of the skeleton * ∈R 14×C Attention weight map F generated by the skeletal mask w ∈R 14×HW Due to the input features G and f * Since the dimension is relatively large, it is reduced by passing each element through a 1×1 convolutional layer to obtain G. in ∈R HW ×C' and C′ represents the number of filters in a 1×1 convolutional layer. (Skeleton local feature map) The input is first processed by a multi-head self-attention module. Here, the self-attention mechanism performs a global analysis on the feature map and uncovers the relationships between features (such as the symmetrical structure of pedestrians, the differences between upper and lower body features, etc.). Then, the multi-head self-attention module outputs the results. The query matrix is ​​input into the SMC submodule. Additionally, the key and value matrices are derived from G.in Provided. The calculation process of the SMC submodule can be represented as follows:

[0093]

[0094] In the formula, and These represent the weights of the three fully connected layers, d w The scaling factor is used to avoid the vanishing gradient problem caused by softmax when the attention map elements are too large. The fusion process should avoid a particular patch in a query focusing on visual regions unrelated to it; this is achieved by scaling the attention weight map F... w Element-wise multiplication with the cross-attention map generated in SMC constrains the fusion process at the local spatial level. During alignment fusion, each bone feature focuses only on the relevant region corresponding to its mask, allowing the network to learn high-level semantic features for alignment more accurately, thus enhancing the discriminative power of the original bone's local features. Finally, After passing through residual connections, a normalization layer, and a fully connected layer, the fused local skeletal features are obtained.

[0095] The proposed network is deployed in the PyTorch framework, and we use a single A100 (40G) graphics card for training and testing. A ResNet50 network pre-trained on the ImageNet dataset is used as the backbone, with the fully connected layers at the end (outputting 1000 classes) removed, resulting in a backbone output feature resolution of 8×4. Pose information is generated by the advanced pedestrian pose estimation model ViTPose, including the location coordinates, confidence scores, and keypoint heatmaps for 14 keypoints. To accelerate the training of the re-identification model, pose information is generated offline. During training, the input image size is uniformly scaled to 256×128. For the triplet loss, a hard positive example and a hard negative example are selected from each sample in the mini-batch obtained after PK sampling to form a triplet, with a margin set to 0.3. The ID loss function is used to calculate the loss for local branches, while the ID loss and triplet loss are used to calculate the loss for global local branches. The entire training process adopts an end-to-end training method, with the learning rate initialized to 3.5×10⁻⁶. -4 The learning rate will decay to 3.5 × 10⁻⁶ at the 40th, 70th, and 100th iterations, respectively. -5 3.5×10 -6 and 3.5×10 -7 The model converged after 150 iterations. During the inference phase of the model, all local and global features output by the network are concatenated along the channel direction to obtain a one-dimensional feature vector, which serves as the re-identification representation of the pedestrian.

[0096] ablation experiment

[0097] First, to verify the effectiveness of each module in the method of this invention, ablation studies were conducted on the Occluded-DukeMTMC, Partial-ReID, and Partial-iLIDS datasets. The experimental results are shown in Table 1. "Baseline" indicates a method that learns pedestrian local features using only the multiplication of keypoint heatmaps and feature maps. VARA, SAM, and SMCF represent the appearance region attention module, the skeleton region modeling module, and the cross-attention fusion module for skeleton modulation, respectively.

[0098] Table 1 Ablation experiments (%) on Included-DukeMTMC, Partial-ReID, and Partial-iLIDS

[0099]

[0100] The experimental results in the first two groups of Table 1 show that embedding VARA into the Baseline network improves the model's R-1 accuracy by 5.3% and mAP by 4.4%, indicating that introducing keypoint masks in the intermediate layers of the network has a positive effect on the perception of occlusion and pedestrian discriminative features. Furthermore, the R-1 accuracy of the third group of experiments significantly exceeds that of the Baseline model by 7.1%, demonstrating that pedestrian local features extracted solely through keypoint masks may lose some appearance information. The SAM modeling method proposed in this invention not only generates complete local region masks for pedestrians but also considers the appearance shape and keypoint confidence of each local region in greater detail, thus avoiding the introduction of occlusion information and background noise. The fourth group of experimental results shows that the combination of the VARA and SAM modules further improves the model's performance. This phenomenon may be because the occlusion information in the feature map output by the ResNet backbone network with the VARA module is suppressed to some extent, and the more precise localization of the skeletal region by SAM makes the extracted local features more discriminative. The final two sets of experiments show that the local region constraints provided by the skeletal mask allow local features to be fused with higher resolution and stronger semantic information within a certain spatial range, which is beneficial for aligning the semantics of local features and improving their robustness. Furthermore, similar experimental results were obtained on the Partial-ReID and Partial-iLIDS datasets. Therefore, the PCRF algorithm proposed in this invention is effective for occluded person re-identification tasks.

[0101] Furthermore, to investigate the impact of the application position of the VARA module in the middle layer of the backbone network on the model's performance, a variant of the VARA network was validated on the Occluded-DukeMTMC dataset. The experimental results are shown in Table 2. Here, "P" indicates that the Conv.X layer and the VARA module are connected in parallel, and "S" indicates that the Conv.X layer and the VARA module are connected in serial order (Conv.X layer first, VARA module second). "DenseVARA" indicates that the VARA module is applied to all Conv.X layers, i.e., a densely connected mode. PCRF w / o VARA represents the model obtained by removing the VARA module using the method of this invention, serving as the baseline network for this experimental study.

[0102] Table 2. Evaluation of VARA module application at different locations in the PCRF network (%)

[0103]

[0104]

[0105] The results in Table 2 show that when the VARA module is only applied to the low-level feature extraction layer Conv.B1, the performance improvement of the serial connection method is only slight, while the performance of the parallel connection method decreases. Similarly, when the VARA module is only applied to the high-level layers, the performance improvement of both connection methods is not significant. Conversely, when the VARA module is only applied to the Conv.B3 layer, the model performance improvement is more significant, with the parallel and serial connection models improving R-1 accuracy by 3.2% and 2.8%, respectively. This may be because the semantics learned from low-level features are weak, making it difficult for the attention mechanism in the VARA module to extract identity-related cues; the feature resolution of the high-level output is only 8×4, resulting in overly coarse keypoint masks and a significant decrease in the effectiveness of pose cues. In addition, attempting to embed the VARA module densely into the backbone network inevitably leads to high GPU resource consumption. Under limited hardware resources, the performance decline is likely due to the small batch size. Finally, the model achieves optimal performance when the VARA module is added to both Conv.B2 and Conv.B3 layers. By comparing the performance of serial and parallel connection models under the same conditions, it can be seen that the serial connection method is generally better than the parallel connection method. This may be because in the parallel method, occlusion information is passed to the higher layers along the convolutional modules of the original backbone network; while in the parallel method, the feature map output by the upper layer is suppressed to a certain extent after passing through the attention submodule of the foreground perception channel, so the noise information passed to the downstream is smaller.

[0106] Comparative experiment

[0107] To provide an objective evaluation of the pose-guided cross-attention feature fusion occluded person re-identification model proposed in this invention, the present invention is compared with current mainstream occluded person re-identification methods on the large-scale occluded person re-identification dataset Occluded-DukeMTMC and the complete person re-identification datasets Market-1501 and DukeMTMC-ReID, as shown in Tables 3 and 4. The methods compared are all typical methods in the field of person re-identification and representative methods in the past three years. The experimental results are excerpted from the original papers in journals or conferences.

[0108] Table 3 compares the results with mainstream methods on the Occluded-DukeMTMC (%)

[0109]

[0110]

[0111] [1]Focus on the visible regions: semantic-guided alignment model for occluded person re-identification[J]. Sensors, 2020, 20(16):4431.

[0112] [2]Semantic-aware occlusion-robust network for occluded person re-identification[J]. IEEE Transactions on Circuits and Systems for VideoTechnology, 2020, 31(7):2764-2778.

[0113] [3]Semantic-guided shared feature alignment for occluded person re-Identification[C]. Asian Conference on Machine Learning, 2020:17-32.

[0114] [4]Pose-guided visible part matching for occluded person ReID[C] / / Proceedings of the IEEE / CVF conference on computer vision andpatternrecognition.2020:11744-11752.

[0115] [5]J.Miao,Y.Wu,P.Liu,Y.Ding,andY.Yang,“Pose-guided feature alignmentfor occluded person re-identification,[C]in Proc.ICCV,Oct.2019,pp.542–551

[0116] [6]Pose-Guided Part-Based Adaptive Pyramid Features for OccludedPerson Reidentification[J].Mathematical Problems in Engineering,2020,2020:1-8.

[0117] [7]PAFM:pose-drive attention fusion mechanism for occluded person re-identification[J].Neural Computing andApplications,2022,34(10):8241-8252.

[0118] [8]High-order information matters:Learning relation and topology foroccluded person re-identification[C].Proceedings of the IEEE / CVF conferenceon computer vision and pattern recognition,2020:6449-6458.

[0119] [9]Pose-guided inter-and intra-part relational transformer foroccluded person re-identification[C].Proceedings ofthe 29thACM InternationalConference on Multimedia,2021:1487-1496.

[0120]

[10] Dual attention-based method for occluded person re-identification[J].Knowledge-Based Systems,2021,212:106554.

[0121]

[11] Hou R,MaB,Chang H,et al.Feature completion for occludedperson re-identification[J].IEEE Transactions on Pattern Analysis and MachineIntelligence,2021,44(9):4894-4912.

[0122]

[12] Wang S,Huang B,Li H,et al.Key point-aware occlusion suppressionand semantic alignment for occluded person re-identification[J].InformationSciences,2022,606:669-687.

[0123]

[13] Wang H,Chen X,Liu C.Pose-guided part matching network viashrinking and reweighting for occludedperson re-identification[J].ImageandVision Computing,2021,111:104186.

[0124]

[14] Occluded Person Re-Identification via Defending Against AttacksFrom bstacles[J].IEEE Transactions on Information Forensics and Security,2022,18:147-161.

[0125]

[15] Diverse part discovery:Occluded person re-identification withpart-aware transformer[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition.2021:2898-2907.

[0126]

[16] TransReID:Transformer-based Object Re-Identification[C]IEEEInternational Conference on ComputerVision 2021,1-10.

[0127]

[17] Pose-guided feature disentangling for occluded person re-identification based on transformer,[C]in Proc.AAAI Conference.Artificial.Intelligence.,2022,pp.2540–2549

[0128]

[18] Multidirection and Multiscale Pyramid in Transformer for Video-Based Pedestrian Retrieval[J]IEEE Transactions on Industrial Informatics(2022):18

[0129]

[19] Learning disentangled representation implicitly via transformer for occluded person reidentification, [J] IEEE Transaction on Multimedia, 2022, (07).

[0130] Table 3 shows the occluded person re-identification models based on person parsers (or semantic segmentation) and pose estimation, respectively, in the first and second groups. The table shows that the proposed PCRF model achieves 70.6% R-1 accuracy and 60.9% mAP on the Occluded-DukeMTMC dataset, surpassing most existing occluded person re-identification methods based on external semantics. Compared to the methods in the first group, the PCRF model of this invention improves R-1 accuracy and mAP by at least 4.1% and 2.6%, respectively, indicating that the method of generating a skeletal mask based on pose key points in this invention is superior to the semantic mask generated by the parser from the original image. This may be because the input image resolution of the person re-identification model varies widely, while the person parser is a fine-grained hierarchical perceptual task, easily affected by low-resolution images and severe occlusion problems. Conversely, the pose estimation model relies on high-level semantic features of local regions, which can output confidence information for severely occluded local areas of the human body. Although there are some errors in the spatial location prediction of keypoints in this region, the extracted local features are more robust because the keypoint confidence information provides stable occlusion perception cues for modeling the skeletal semantic region. The experimental results of the second group of methods in the table show that the proposed PCRF method significantly outperforms PGFA and PPAPN methods in R-1, indicating that using only keypoint heatmaps to extract local features has certain limitations. Compared with RFCnet, the method of this invention has a more significant advantage in R-1 and mAP accuracy. RFCnet and the method of this invention are similar in that they both embed pose semantic information into the backbone network. RFCnet encodes pose information to obtain the coordinates of feature map segmentation and then introduces a semantic segmentation mask to extract features of the visible region; while the method of this invention can model and generate the visible region using only pose information, giving it a greater advantage in terms of low algorithm complexity. Furthermore, it is important to note that these pose estimation-based methods all introduce pose information after the backbone network, making feature extraction overly reliant on the pose estimation model, which has certain limitations. In contrast, the VARA module proposed in this invention organically combines the attention mechanism with the backbone network, providing occlusion perception cues to the model in a soft-fusion manner during the image feature extraction stage, thus alleviating the problem of over-reliance on external semantic models.

[0131] Table 4 Comparison with Mainstream Methods on Market1501 and DukeMTMC-ReID (%)

[0132]

[0133]

[0134]

[20] Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline)[C]. Proceedings of the European conference on computer vision

[0135] (ECCV), 2018: 480 - 496.

[0136]

[21] Circle loss: A unified perspective of pair similarity optimization[C]. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2020: 6398 - 6407.

[0137]

[22] Interaction-and-aggregation network for person re-identification[C]. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2019: 9317 - 9326.

[0138]

[23] Pyramidal person re-identification via multi-loss dynamic training[C]. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2019: 8514 - 8522.

[0139]

[24] Salience-guided cascaded suppression network for person re-identification[C].

[0140] Proceedings of the IEEE / CVF conference on computer vision and patternrecognition,2020:3300-3310.

[0141]

[25] Attention-aligned network for person re-identification[J].IEEETransactions on Circuitsand Systems forVideo Technology,2020,31(8):3140-3153.

[0142]

[26] Learning Feature Recovery Transformer for Occluded Person Re-Identification[J]IEEETransactions on Image Processing(2022):31

[0143] Table 4 evaluates the proposed method and some mainstream methods on a complete pedestrian re-identification dataset. As shown in the table, the proposed method achieves R-1 accuracy of 95.5% and 90.8% on the two datasets, respectively, essentially surpassing the performance of almost all pose estimation-based occluded pedestrian re-identification models. In unoccluded environments, the pose estimation results show high confidence. This leading result is likely due to the PCRF method's more detailed localization of pedestrian regions and more complete extraction of appearance features, thus avoiding the loss of some discriminative appearance features. Compared to pose estimation-based SRNet, the proposed PCRF method shows a significant advantage on the DukeMTMC-ReID dataset, improving R-1 accuracy and mAP by 3.5% and 5.3%, respectively. This dataset is relatively large and contains a certain number of occluded images; therefore, this result also demonstrates the robustness of the proposed method in complex environments. Of course, the proposed PCRF method still cannot surpass models specifically designed for complete pedestrian re-identification tasks (such as SCSNAANet), but its performance is competitive. Furthermore, the SAM strategy proposed in this invention, as a decoupled local semantic localization algorithm, can be combined with these state-of-the-art works to further improve the model's performance. In summary, the proposed PCRF method has strong generalization ability in complete pedestrian scenarios.

[0144] Visualization Analysis

[0145] To better demonstrate the advantages of the proposed method, a visualization of the mask image output by the Skeletal Region Modeling (SAM) module in the Local Feature Extraction stage is shown. For example... Figure 7 As shown, (1) to (6) represent the six occluded pedestrian image samples tested, and (a) to (d) represent the original pedestrian image, the keypoint visualization result output by the pose estimation model ViTPose, the keypoint heatmap output by the pose estimation model ViTPose, and the skeleton mask map output by the SAM module of this invention, respectively. Since there are many predefined skeleton regions and considering the need to more intuitively highlight the contrast between the response levels of the visible and occluded areas of the pedestrian, the masks of the 14 local skeleton regions generated by the SAM module are merged into a complete pedestrian skeleton mask map.

[0146] Figure 7Several sets of results show that the SAM module can achieve relatively complete pedestrian appearance region localization. Compared with keypoint heatmaps, the method of this invention enhances the attention to the visible area while suppressing the response to occlusions and background areas, and compensates for some information loss of pedestrian appearance regions between keypoints in the pose estimation output heatmap. In addition, in the process of defining local skeletal regions, because the torso of the pedestrian is finely divided, even in the case of severe occlusion, the local features associated with the torso, which have more discriminative information, still occupy a large proportion in the final generated pedestrian representation. Therefore, it has certain advantages in terms of semantic alignment and feature richness.

[0147] When faced with occlusion, the coordinates of key points predicted by the pose estimation model for the occluded parts lack certain reliability. For example, in test sample (1), when the lower body of a pedestrian is obscured by a car, even humans find it difficult to determine the true positions of the knees and ankles. Therefore, existing methods for feature extraction based on key point heatmaps have relatively poor effectiveness in terms of local features. However, in this case, the confidence of the key points output by the pose estimation model depends on the high-level semantic features of the local region, which is a reflection of the visibility of the pedestrian's appearance in that region, and therefore has certain value in perceiving occlusion interference. The SAM module utilizes this reliable information and assigns visibility attributes to each local skeletal region through threshold filtering. As can be seen from the figure, the response of the mask output by the method of this invention in the occluded local region is lower than the response of the corresponding region of the key point heatmap, which proves that the key point heatmap has certain limitations in providing effective pedestrian information. Furthermore, it can be seen from the results of test samples (3) and (5) in the figure that under the condition that the pose estimation results have a certain difference, the mask map output by the method of the present invention can achieve a relatively stable output, indicating that the SAM module does not have a strong dependence on the external model.

[0148] Overall, the SAM module can effectively locate local skeletal regions and suppress occlusion information, and still has good robustness and stability even when the pose estimation detection results have certain errors.

[0149] Figure 8 A visualized feature heatmap of the present invention is provided. It can be seen from the figure that the present invention can effectively focus on the visible area of ​​pedestrians, improve the model's perception of pedestrian appearance information and the model's ability to perceive and identify occlusion information.

[0150] This invention proposes a pose-guided cross-attention feature fusion algorithm for occluded pedestrian re-identification based on pose information. The algorithm comprises three modules: VARA, SAM, and SMCF. The VARA module is organically integrated with the backbone network, guiding it to perceive pedestrian appearance information through a foreground-aware channel attention mechanism and a key feature space cross-attention fusion mechanism. This enhances the model's ability to perceive and discriminate occlusion information and mitigates the problem of occlusion information diffusion caused by the increase in the network's receptive field. The SAM module models local skeletal regions of the human body. The SMCF module utilizes the local region weight map provided by the skeletal mask to constrain the cross-attention fusion process between the features output by the backbone network and local features, thereby constructing a complete local feature set and compensating for information loss in some pedestrian appearance regions between key points in the heatmap output by the pose estimation model. The SMCF module, using the local region weight map provided by the skeletal mask, constrains the cross-attention fusion process between the features output by the backbone network and local features, achieving enhanced local feature discriminativeness and alignment of high-level semantic features. Related ablation experiments demonstrate the effectiveness of each module in this invention. Furthermore, compared with existing occluded person re-identification methods based on external semantic cues, the method of this invention achieves an R-1 accuracy of 70.6% and an mAP accuracy of 60.9% on the large occluded dataset Occluded-DukeMTMC, demonstrating the effectiveness of the proposed method in occluded scenarios.

[0151] The present invention further provides an electronic device, including a memory and a processor; wherein:

[0152] Memory: Used to store instructions that can be executed by the processor;

[0153] Processor: The processor is configured to perform the following: guide the backbone network to perceive pedestrian appearance information through a foreground-aware channel attention mechanism and a key feature space cross-attention fusion mechanism; utilize the robustness of the confidence of human key points to generate a mask for a predefined skeletal region through mathematical modeling; establish an interaction relationship matrix for the local skeletal features of the pedestrian and the feature map output by the backbone network; and generate high-level semantically aligned pedestrian retrieval features through the fusion of local contextual information under the constraint of the spatial attention mask.

[0154] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the inventive concept, and these all fall within the protection scope of the present invention.

Claims

1. A pose-guided cross-attention feature fusion method for occluded pedestrian re-identification, characterized in that, Step S1: guiding the backbone network to realize the perception of pedestrian appearance information through a foreground-aware channel attention mechanism and a key feature space cross-attention fusion mechanism; Step S2: generating a mask of a predefined skeletal region in a mathematical modeling manner by using the robustness of the confidence of human key points; Step S3: establishing an interaction relationship matrix for the skeletal local features of the pedestrian and the feature map output by the backbone network, and generating pedestrian retrieval features with high-level semantic alignment by fusing local context information under the constraint of the spatial attention mask; The foreground-aware channel attention mechanism is: First, the non-zero regions of the keypoint heat map F k are down-sampled by global average pooling of the features corresponding to the non-zero regions to obtain a channel signal vector This process is represented as: V c = AvgPool(F in ⊙ F k ) In the formula, ⊙ represents element-wise multiplication of feature maps; AvgPool(·) represents a global average pooling operation; Secondly, two different convolutional layers {C in / r,1×1,1} and {C in / r,1×1,1} are connected, and ReLU and Sigmoid activation functions are added after each convolutional layer. In the parameter setting of the two layers of convolutional layers {C in / r,1×1,1} and {C in / r,1×1,1}, the first item represents the number of output channels, the second item represents the size of the convolution kernel, and the third item represents the step of the convolution kernel; the calculation process of the channel attention vector is represented as: A c = sigmoid(W2relu(W1V c )) wherein, and denote the weight parameters of two convolutional layers, respectively; r is the dimension decay rate. Finally, the generated channel score vector A c acts on the feature map F of the module input in a multiplicative way in , generating a channel-weighted feature map The key feature space cross-attention fusion mechanism is: First, the corresponding feature vector is extracted according to the heat map of 14 key points, and the composition is a sparse feature map And the output of the foreground perception submodule is weighted to obtain a feature map is regarded as a dense feature map; F d Dimensionality reduction of channels by a 1x1 convolutional layer and flattening in the spatial dimension to get the input to the cross-attention mechanism Similarly, F s After two independent operations, we get and where Q d , V s and K s are the query vector, value vector and Key vector, respectively, and t represents the dimension attenuation ratio; Finally, a 1×1 convolutional layer is used to restore the number of channels in the cross-attention module output feature map to C. in and resize it to H in ×W in ×C in / t, then with feature map F d Perform residual connections and input them into the next convolutional layer of the ResNet network; The internal structure of the Resnet network is divided into five parts, which are defined as conv1, and conv.B1, conv.B2, conv.B3 and conv.B4 composed of Bottleneck, wherein the last four parts contain 3, 4, 6 and 3 Bottleneck residual blocks respectively, and each residual block contains three convolutional layers; The specific method of step S2 is: Firstly, the skeletal region of the pedestrian needs to be predefined; Secondly, each predefined bone region is modeled respectively to obtain a corresponding weight mask, defining the key points at both ends of a certain bone region as A and B, C point (i, j) as an arbitrary position point in the image plane space, O point (c x , c y ) as the midpoint of AB, the confidence of A and B points as δ A and δ B respectively, and the weight mask M of a certain local bone region is expressed as: r = a * exp(γ(-(sin θ) 2 )) l AO where θ represents the included angle between the line segment AB and CO, l AO represents the length of the line segment AO, r represents the radiation range, α and γ respectively represent the radiation overall scale expansion coefficient and the lateral extrusion coefficient of the radiation scale, I(·,·) represents the confidence of the current local bone region, and in calculating the confidence, the result of (δ A +δ B ) / 2 is compared with a threshold value ρ, if it exceeds the threshold value, the result is retained, otherwise the result is compressed to 0.5 of the original, wherein the threshold value ρ=0.7 is the optimal threshold value obtained by debugging. After obtaining the local feature mask, when extracting the local features corresponding to each bone region, the output feature maps of the Conv. Bl, VARA1 and Conv. B4 layers are fused to obtain a feature map F' e R 4H×4W×C ; The mask M of each layer is removed i ∈R 4H×4W×1 Respectively and the feature map F' corresponding pixel product, get the local feature map of the bone region After another global average pooling, the local feature vector with C channels is obtained The process is represented as: In the formula, pool(·) represents a global average pooling operation; Step S3, the specific method is: input the flattened feature map G output by the backbone network ∈R HW×C , the local feature f of the skeleton * ∈R 14×C and the attention weight map F generated by the skeleton mask w ∈R 14×HW , G and f * respectively pass through a 1×1 convolution layer for dimension reduction to obtain G in ∈R HW×C′ and C′ is the number of filters of the 1×1 convolution layer; Skeletal local feature map will be input into a multi-head self-attention module, at which time the self-attention mechanism performs global analysis on the feature map and excavates the mutual relationship between the features; then, the output result of the multi-head self-attention is input as a query matrix into a skeletal modulation cross-attention submodule; a key matrix and a value matrix are provided by C in ; the calculation process of the skeletal modulation cross-attention submodule is represented as: wherein and denote the weights of the three fully connected layers, d w denotes a scale factor, The attention weight map F w The multiplication of the corresponding elements of the cross-attention map generated in the cross-attention submodule modulated by the skeleton constrains the fusion process at the local spatial level, and each skeleton feature only focuses on the relevant area corresponding to its mask when aligning and fusing, and finally, After residual connection, a normalization layer and a fully connected layer, the fused local features of the skeleton are obtained 2. An apparatus for implementing the method of claim 1, characterized by It includes: A visual appearance region attention module that guides the backbone network to realize the perception of pedestrian appearance information through a foreground-aware channel attention mechanism and a key feature space cross-attention fusion mechanism; A skeletal region modeling module that generates a mask of a predefined skeletal region in a mathematical modeling manner by using the robustness of the confidence of human key points; A skeletal modulated cross-attention fusion module that establishes an interaction relationship matrix for the skeletal local features of the pedestrian and the feature map output by the backbone network, and generates pedestrian retrieval features with high-level semantic alignment by fusing local context information under the constraint of the spatial attention mask.

3. The apparatus of claim 2, wherein, The visual appearance region attention module is more than two, which are embedded in different layers of the backbone network and contain foreground-aware channel attention submodules and key feature space cross-attention fusion submodules.

4. The apparatus of claim 2, wherein, The foreground-aware channel attention submodule encodes the features of the foreground region by means of the key point heat map, calculates the attention scores of each channel related to the foreground part, and then performs channel weighting on the feature map input by the visual appearance region attention module to suppress the feature responses of the channels related to occlusion and background noise.

5. The apparatus of claim 2, wherein, The skeletal region modeling module models the features of the skeletal region between key points by using key point coordinates and confidence, generates a soft mask with pixel confidence scores for the skeleton of the human body, and uses it to extract local features of the skeleton.

6. The apparatus of claim 2, wherein, The skeletal modulated cross-attention fusion module contains a multi-head self-attention submodule and a skeletal modulated cross-attention submodule.

7. An electronic device, comprising: It includes a memory and a processor; wherein: Memory: for storing instructions executable by the processor; The processor is configured to execute the method of claim 1.

Images