Dressing pedestrian re-identification method based on pedestrian soft semantic information explicit-implicit learning

Abstract

The invention belongs to the field of computer vision, and discloses a clothes changing pedestrian re-identification method based on pedestrian soft semantic information explicit-implicit learning. The method comprises the following steps: constructing a pedestrian head data set; extracting initial soft semantic information; front camera visual angle head information representation is extracted and enhanced; clothing area characterization is inhibited; the comprehensive loss function is minimized; and testing the model effect. According to the method, pedestrian soft semantic information in an original image is fully mined, a multi-level progressive feature enhancement module and a region enhancement module are designed, effective pedestrian soft semantic features are adaptively extracted in different levels of a model network in stages through a hierarchical representation dynamic mechanism, and the influence of camera view angle changes on recognition efficiency is reduced; according to the method, pedestrian soft semantic representation is enhanced in an explicit-implicit adversarial learning mode, clothing related area representation is inhibited, and the model is guided to fully learn specific pedestrian identity information. According to the model method provided by the invention, test results on a plurality of data sets have relatively high recognition accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and relates to a method for re-identifying pedestrians who have changed clothes based on explicit and implicit learning of pedestrian soft semantic information. Background Art

[0002] In recent years, person re-identification (Re-ID) has become increasingly important in the field of intelligent security. This technology aims to address the challenge of matching the identities of specific individuals in surveillance systems distributed across time and space. While current research based on deep learning frameworks has achieved breakthroughs, it generally relies too heavily on clothing features, assuming that pedestrians maintain a fixed attire for short periods of time. However, real-world scenarios present numerous complexities: ordinary people periodically change their clothing, and criminal suspects frequently change their attire to evade tracking systems. These changing attire scenarios can cause traditional algorithms that rely on color and texture features to degrade in performance when deployed. Therefore, achieving accurate identification of pedestrians in clothing-changing scenarios has become a research hotspot in the field of Re-ID.

[0003] At present, the main challenge of clothing-changing pedestrian re-identification (CC-ReID) is how to ensure that the model always focuses on the identity area that is not related to clothing, while effectively discarding the interference of clothing-related areas. In order to solve the above problems, many methods for clothing-changing pedestrian re-identification have been deeply studied, which can be mainly divided into two categories: (1) methods based on feature decoupling; (2) methods based on hard semantic representation learning. The method based on feature decoupling mainly uses adversarial generative networks (Humayun AI, Balestriero R, Baraniuk R.Polarity sampling:Quality and diversity control ofpre-trained generative networks via singular values[C] / / Proceedings of theIEEE / CVF Conference on Computer Vision and Pattern Recognition.2022) to separate clothing-related and clothing-irrelevant features. However, using adversarial generative networks for feature separation is a time-consuming process, and the images generated by them have a negative impact on the performance of the model due to the lack of corresponding real labels. The method based on hard semantic representation learning guides the model to learn invariant pedestrian identity features by inputting auxiliary information related to identity and independent of clothing into the model, such as pedestrian gait (JinX, He T, Zheng K, et al. Cloth-changing person re-identification from a single image with gait prediction and regularization[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2022.), body shape (Hong P, Wu T, Wu A, et al. Fine-grained shape-appearance mutual learning for cloth-changing person re-identification[C] / / Proceedings of the IEEE / CVF conferenceon computer vision and pattern recognition.2021.), etc.Although this method has low training costs and is effective to a certain extent, it ignores two important factors: first, existing methods lack the ability to adapt to complex scenarios such as clothing diversity (such as switching between winter and summer clothing), sudden changes in perspective, and switching between dynamic video streams and static image sets; second, the identity features extracted by existing methods exhibit static propagation characteristics at the network level. As the network level deepens, indiscriminate learning at different levels may lead to perceptual bias in the features of irrelevant areas of the final clothing.

[0004] To address the poor adaptability to dynamic scenes and rigid representation learning problems of existing methods, this paper proposes an implicit and explicit learning method for re-identifying people who have changed clothing, guided by their soft semantic information. This method aims to fully exploit the soft semantic information of the human body contained in the original RGB image, guiding the model to gradually focus on feature areas unrelated to clothing, such as the foreground contour of the human body, limb joints, and the head, in a gradual and in-depth manner. By leveraging the dynamic hierarchical representation mechanism of human soft semantic information, the method autonomously selects and focuses on specific biometric representations at different network layers, achieving the complete preservation of identity features within the neural network hierarchical architecture, effectively solving the problem of re-identifying people who have changed clothing. Summary of the Invention

[0005] To fully leverage the soft semantic information inherent in the original image, this paper proposes an implicit and explicit learning method for clothing-changing person re-identification (RID) based on soft semantic information to address the clothing interference issue in current clothing-changing scenarios. First, the original pedestrian image and its corresponding cropped head image are fed into the multi-level progressive feature enhancement module and identity consistency module designed in this paper to extract initial soft semantic information. The multi-level progressive feature enhancement module guides the model to adaptively shift its focus region at different network levels, from the foreground contour to the limb joints, and finally to the head in the deep network, thereby gradually mining fine-grained identity information. In the identity consistency module, to mitigate the impact of camera viewpoint changes on model performance, the present invention designs a region enhancement module to enhance the geometric consistency of the soft semantic information of the head. Then, to further eliminate the interference of clothing information in the extracted soft semantic information, the present invention designs an explicit enhancement module and an implicit suppression module. The semantic matrix obtained from the human parsing network is used to separate clothing-related and identity-related mask maps. This assists the multi-level progressive feature enhancement module in explicitly enhancing the soft semantic representation and utilizes an adversarial penalty mechanism to suppress clothing-related region representations. Finally, in the training phase, the above modules are combined through a joint learning framework, and in the testing phase, only the backbone feature extraction network is retained and the corresponding modules are discarded, which reduces the computational and time costs and improves efficiency.

[0006] The technical solution of the present invention:

[0007] A method for re-identifying pedestrians after changing clothes based on explicit and implicit learning of pedestrian soft semantic information includes the following steps:

[0008] Step 1: Use the human body parsing network to obtain the human body structure semantic matrix based on the human body region labels;

[0009] The pre-trained SCHPNet human parsing network (Li P, Xu Y, Wei Y, et al. Self-correction for human parsing [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020) is used to generate a human structure semantic matrix Y. The human structure semantic matrix Y contains 7 labels, namely background, head, torso, upper arm, lower arm, thigh, and lower leg, corresponding to the pixel values 0 to 6 in the human structure semantic matrix Y. The corresponding 7 human structure mask images are obtained by separating the human structure semantic matrix Y according to the labels.

[0010] Step 2: Use the prediction box information of the target detection framework to accurately crop the pedestrian head area;

[0011] Deploy a recognition framework based on the YOLOv5 architecture to accurately locate the pedestrian head region in the original pedestrian image and crop the pedestrian head image. Finally, the original pedestrian image and the extracted pedestrian head image are paired and combined to construct an enhanced multimodal training dataset.

[0012] Step 3: The overall model consists of three branches: the original image branch composed of the ResNet50 network, the multi-level progressive feature enhancement module branch, and the identity consistency module branch. In addition, the explicit enhancement module cooperates with the original image branch to obtain the identity invariant feature F ID , the implicit suppression module cooperates with the original image branch to obtain the clothing sensitive feature F C ,The identity consistency module branch includes the region enhancement module;

[0013] The human body structure semantic matrix is used to obtain the pedestrian black image corresponding to the original pedestrian image, and then fed into the multi-level progressive feature enhancement module to extract the human foreground contour, limb joints, and initial soft semantic information of the head. The specific implementation process of the multi-level progressive feature enhancement module is as follows:

[0014] Using the human body structure semantic matrix Y, the pixel values of the clothing-related areas in the original pedestrian image are set to 0 to obtain the corresponding pedestrian black clothing image; then the human body structure mask images separated by the human body parsing network are recombined to obtain the human foreground contour mask image M silh , limb joint mask map M limand head mask M head , and then perform feature enhancement in the three layers of the ResNet50 network; specifically: First, the stage 1 of the ResNet50 network is not processed, and the pedestrian black clothing image is obtained after the stage 2 of the ResNet50 network. Where B represents the feature identifier extracted by the multi-level progressive feature enhancement module, and then two consecutive 1×1 convolutional layers are used to Reduce the dimension to 1 dimension; then connect the ReLU layer and Sigmoid layer after these two 1×1 convolution layers for parameter optimization to obtain the attention heat map H 2 ; Then use the mean square loss function MSE to construct M silh With H 2 The loss constraint between 2 and Perform element-wise multiplication at the corresponding spatial position to obtain the enhanced features in stage 2 The specific process is expressed as:

[0015]

[0016] Among them, Conv 1×1 Indicates the dimensionality reduction operation using a 1×1 convolution kernel, represents the MSE loss function under stage 2 feature enhancement, represents element-wise multiplication, || || 2 represents the L2 norm;

[0017] Similarly, the features after stage 2 enhancement After stage 3, the feature map is obtained Use the same dimensionality reduction method to reduce the dimensionality to obtain H 3 , and use the mean square loss function to construct H 3 With M lim The loss constraint between them guides H 3 To fit the attention distribution of the human limb joint area, finally H 3 and Perform element-wise multiplication at the corresponding spatial position to obtain the enhanced features in stage 3 Then the enhanced features in stage 3 After stage 4, the feature map is obtained The same method is used to obtain the attention heat map H 4 and the enhanced features of stage 4 The specific implementation process is as follows:

[0018]

[0019]

[0020] Therefore, the total loss function under the multi-level progressive feature enhancement module is:

[0021]

[0022] Among them, L MSE represents the comprehensive MSE loss, and denote the MSE losses of the 2nd, 3rd and 4th stages respectively;

[0023] Step 4: Send the pedestrian head image to the identity consistency module, and use the region enhancement module to optimize the overall model's attention distribution on pedestrian head images from different camera perspectives. The specific implementation process of the identity consistency module is as follows:

[0024] During the training process, the input pedestrian head image is sent to the identity consistency module to obtain the feature representation H 4,l , where 4 represents the features after the 4th convolutional block in the ResNet50 network, l represents the low-level features; the low-level features H under the front camera and back camera perspectives are 4,l After the generalized average pooling layer and the fully connected layer, the mid-level feature and high-level feature representations are obtained respectively; the specific implementation process is as follows:

[0025]

[0026] Where f represents the front camera view, n represents the back camera view, m and h represent the middle and high levels respectively; GeMP represents the generalized average pooling layer, and FC represents the fully connected layer;

[0027] Then, the front camera view feature map H f As the input of the region enhancement module, it is enhanced at the channel attention level and the spatial attention level respectively; specifically, at the channel attention level, H f It is sent to the GeMP layer for generalized average pooling, followed by two convolutional layers Conv down and Conv up Reducing and increasing the dimension respectively; At the spatial attention level, a convolutional layer is used to f Dimensionality reduction is performed, and average pooling and maximum pooling operations are performed along the channel at the same time; finally, the Sigmoid activation function is used for optimization, and the optimized representations are compared with H f Perform element-wise multiplication to get and After adding the two together, we get the final enhanced frontal camera perspective head feature. The specific implementation process is as follows:

[0028]

[0029] Among them, REA represents the regional enhancement module, max represents the maximum pooling operation, avg represents the average pooling operation, σ1 represents the Sigmoid activation function, c and s represent the channel level and spatial level respectively, ⊕ represents the element level addition, Conv down and Conv up Represent the convolution operations of dimensionality reduction and dimensionality increase respectively;

[0030] Finally, the maximum mean difference loss function MMDLoss is used to The feature map F extracted from the original image branch based on the original image of the pedestrian r Perform loss constraint, where F r Pedestrian appearance features F extracted from the original image branch R After processing by the generalized average pooling layer, the specific implementation process is as follows:

[0031]

[0032] Among them, μ represents mean calculation, σ represents variance calculation;

[0033] Step 5: Further use the human body structure semantic matrix to separate identity-related and clothing-related mask images. The explicit enhancement module and implicit suppression module are used to guide the overall model to adaptively learn identity-related information and penalize clothing-related region representations. The specific implementation process of the explicit enhancement module and implicit suppression module is as follows:

[0034] First, the head mask obtained from the human body structure semantic matrix Y is used as the auxiliary input of the explicit enhancement module, and the upper arm mask, torso mask and thigh mask are combined as the auxiliary input of the implicit suppression module; further, in order to achieve the pedestrian appearance feature F R Separate the head, torso and limbs structural features, and design a multi-region feature parsing mechanism based on spatial attention guidance; specifically, the pedestrian appearance feature F R ∈R H×W×C Input into the convolution classification module composed of convolution kernel and softmax activation function, and output the multidimensional attention weight map M∈R H×W×P , where the dimensional parameters are defined as: height H, width W, number of channels C, attention weight map dimension P = 7, pixel values represent background 0, head 1, torso 2, upper arm 3, lower arm 4, thigh 5 and calf 6 respectively; the mathematical representation of this mechanism is as follows:

[0035] M=SoftMax(Conv 1×1 (F R )) (17)

[0036] Among them, Conv 1×1 Indicates the dimensionality reduction operation using a 1×1 convolution kernel; for the attention weight matrix M p The pth channel of its matrix element Quantify the confidence that the spatial coordinate (i, j) belongs to the p-th type of human structure area, and the probability value is constrained to the unit interval [0,1] by the sigmoid function;

[0037] Finally, in order to enhance the accuracy of attention weight mapping in distinguishing human body structure areas, a semantic alignment loss function L is designed. SA Drive spatial attention distribution optimization, its mathematical form is defined as:

[0038]

[0039] Where N is the number of batch samples, M(h,w) and Y(h,w) represent the spatial attention weight matrix and the probability value of the human body structure area at the coordinate (h,w), respectively;

[0040] Aiming at the generalization error of the human body parsing network, a dynamic attention optimization mechanism is introduced - through the cross entropy supervision loss function L ID The driver parameter update makes the generated attention distribution adaptive to clothing changes rather than being restricted by the static constraints of the pre-trained parsing network; finally, the explicit enhancement module performs attention modulation operations to transform the appearance features F R and the optimized head attention probability map M head Multiply to generate identity invariant feature F ID , its mathematical expression is:

[0041] F ID =F R ×M head (19)

[0042] Among them, M head ∈R H×W×1 is the head attention probability map, × represents the tensor element-level product operation; considering the clothing-independent features F extracted by the multi-level progressive feature enhancement module b and identity invariant feature F id Shared latent identity space, where F b The clothing-independent features F extracted by the multi-level feature enhancement module branch B After processing by the generalized average pooling layer, F id is the identity invariant feature F ID After processing the generalized average pooling layer, the identity invariant loss function L is designed. II To achieve cross-modal feature alignment, its mathematical expression is:

[0043]

[0044] Among them, μ represents mean calculation, σ represents variance calculation;

[0045] Then the implicit suppression module aggregates the attention probability distribution of the trunk, upper arms and thighs and combines it with the pedestrian appearance representation F R Perform channel weighted fusion to generate clothing sensitive features F C , the mathematical process is described as:

[0046] F C =F R ×(M arms +M torso +M legs ) (twenty one)

[0047] Among them, the human body structure area attention probability map M is defined arms ∈R H×W×1 , M torso ∈R H×W×1 With M legs ∈R H×W×1 Corresponding to the upper arm, torso and thigh areas respectively; + means element-by-element superposition;

[0048] Secondly, by constructing a robust clothing feature classifier, the discriminative learning of clothing attributes is realized based on clothing category labels; the total number of clothing categories is determined by the union cardinality of unique clothing items corresponding to all pedestrian identities in the enhanced multimodal training dataset, thereby completing the automatic labeling of pedestrian clothing labels; the training goal of the clothing feature classifier is to minimize the clothing classification loss L C , its mathematical expression is:

[0049]

[0050] Among them, N represents the batch sample size, N C Represents the cardinality of clothing categories in the training dataset, θ j ∈R d is the learnable parameter vector of the j-th clothing discriminator, and the hyperparameter γ is used to adjust the smoothness of the probability distribution; represents the clothing features of the i-th pedestrian image, Represents the clothing label of the i-th pedestrian image; by minimizing the clothing classification loss L C , in the feature extraction stage, the overall model's ability to discriminate clothing attributes is suppressed, making the optimized clothing discriminator unable to effectively distinguish the clothing categories of the same identity and cross-clothing samples, ultimately forcing the overall model to discard the interference of clothing-related information and strengthen its focus on identity information;

[0051] Step 6: Implement weighted optimization based on the comprehensive loss function and use the backpropagation algorithm to complete the end-to-end parameter update of the model;

[0052] Based on the comprehensive loss function, weighted optimization is implemented and the back-propagation algorithm is used to complete the end-to-end parameter update of the model:

[0053] L=L ID +L tri +λ1L MMD +λ2L MSE +L II +L SA +L C (twenty three)

[0054] Among them, L represents the comprehensive loss function, L ID represents the cross entropy supervision loss function, L tri represents the triple loss function; λ1 and λ2 represent L MMD and L MSE The weight parameter of

[0055] Step 7: Input the independent test set into the optimized overall model to evaluate its cross-scenario classification accuracy.

[0056] Considering the computational and time costs of the model in practical applications, during testing, the modules used in training were discarded, and only the trained ResNet50 backbone network was used to extract pedestrian image features. Three different clothing change test datasets were used as input to the backbone network, and model performance was evaluated using mean average precision (mAP) and cumulative matching characteristics (CMC). Test results show that the model maintains high accuracy in pedestrian classification even when pedestrians change clothing across scenes.

[0057] Beneficial effects of the present invention: Unlike traditional methods for extracting rigid hard semantic features, the present invention utilizes the human body structure region mask map separated by the human body parsing network to fully explore the soft semantic information contained in the original pedestrian image, designs a multi-level progressive feature enhancement module, and guides the model to focus on different identity feature regions of the image at different levels in the deep network space through a hierarchical representation dynamic mechanism, and extracts effective soft semantic features in stages adaptively. Secondly, considering the impact of camera perspective changes on pedestrian image recognition, the present invention designs a region enhancement module to enhance the geometric consistency of soft semantic information in the head region. In addition, in order to eliminate the influence of clothing noise factors in the image on the extracted soft semantic information, the present invention designs an explicit enhancement module and an implicit suppression module. On the basis of enhancing the soft semantic representation, an adversarial penalty mechanism is adopted to suppress the clothing-related region representation, forcing the model to learn identity features unrelated to clothing. Finally, the present invention is tested on three different clothing-changing datasets, and the corresponding modules are discarded during the test phase. The test results show that the model still has a high accuracy rate for pedestrian classification in clothing-changing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flow chart of the method for re-identifying pedestrians with changed clothes based on explicit and implicit learning guided by pedestrian soft semantic information of the present invention;

[0059] Figure 2 This is a motivation diagram for extracting soft semantic information and explicit and implicit learning in the present invention, where (a) is a schematic diagram for extracting soft semantic information, and (b) is a schematic diagram for explicit and implicit learning;

[0060] Figure 3 This is the overall framework structure diagram of the soft semantics-guided implicit learning proposed in the present invention;

[0061] Figure 4 This is a structural diagram of the multi-level progressive feature enhancement module proposed by the present invention;

[0062] Figure 5 This is a structural diagram of the regional enhancement module proposed in the present invention. DETAILED DESCRIPTION

[0063] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The present invention includes but is not limited to the following embodiments.

[0064] like Figure 1 As shown in FIG, the present invention provides a method for re-identifying pedestrians after changing clothes based on explicit and implicit learning of pedestrian soft semantic information. The specific implementation process is as follows:

[0065] (1) Using the human body parsing network to obtain the human body structure semantic matrix based on the human body region labels;

[0066] The pre-trained SCHPNet human parsing network is used to generate the human semantic matrix Y, which contains 7 labels: background, head, torso, upper arm, lower arm, thigh, and lower leg, corresponding to the area with median value 0 to 6 in Y. The human structure semantic matrix Y is separated according to the labels to obtain the corresponding 7 human structure mask images;

[0067] (2) Using the prediction box information of the target detection framework to achieve accurate cropping of the pedestrian head area;

[0068] Deploy a recognition framework based on the YOLOv5 architecture to accurately locate the pedestrian head region in the original pedestrian image and crop the pedestrian head image. Finally, the original pedestrian image and the extracted pedestrian head image are paired and combined to construct an enhanced multimodal training dataset.

[0069] (3) The overall model consists of three branches: the original image branch composed of the ResNet50 network, the multi-level progressive feature enhancement module branch, and the identity consistency module branch. In addition, the explicit enhancement module cooperates with the original image branch to obtain the identity invariant feature F ID , the implicit suppression module cooperates with the original image branch to obtain the clothing sensitive feature F C ,The identity consistency module branch includes the region enhancement module;

[0070] The human body structure semantic matrix is used to obtain the pedestrian black image corresponding to the original pedestrian image, and then fed into the multi-level progressive feature enhancement module to extract the human foreground contour, limb joints, and initial soft semantic information of the head. The specific implementation process of the multi-level progressive feature enhancement module is as follows:

[0071] The value corresponding to each position of the human semantic matrix Y generated in step 1 represents the category label of the position, for example, the label corresponding to the background position is 0, the label corresponding to the head position is 1, etc. The present invention sets the pixel value of the clothing-related area in the original image to 0 to obtain the corresponding black clothing image, and then recombines the human body structure mask images separated by the human body parsing network to obtain the human foreground contour mask image M silh , limb joint mask map M lim and head mask M head , and then perform feature enhancement in the three layers of the ResNet50 network; specifically: First, the stage 1 of the ResNet50 network is not processed, and the pedestrian black clothing image is obtained after the stage 2 of the ResNet50 network. Where B represents the feature identifier extracted by the multi-level progressive feature enhancement module, and then two consecutive 1×1 convolutional layers are used to Reduce the dimension to 1 dimension; then connect the ReLU layer and Sigmoid layer after these two 1×1 convolution layers for parameter optimization to obtain the attention heat map H2 ; Then use the mean square loss function MSE to construct M silh With H 2 The loss constraint between 2 and Perform element-wise multiplication at the corresponding spatial position to obtain the enhanced features in stage 2 The specific process is expressed as:

[0072]

[0073] Among them, Conv 1×1 Indicates the dimensionality reduction operation using a 1×1 convolution kernel, represents the MSE loss function under stage 2 feature enhancement, represents element-wise multiplication, || || 2 represents the L2 norm;

[0074] Similarly, the features after stage 2 enhancement After stage 3, the feature map is obtained Use the same dimensionality reduction method to reduce the dimensionality to obtain H 3 , and use the mean square loss function to construct H 3 With M lim The loss constraint between them guides H 3 To fit the attention distribution of the human limb joint area, finally H 3 and Perform element-wise multiplication at the corresponding spatial position to obtain the enhanced features in stage 3 Then the enhanced features in stage 3 After stage 4, the feature map is obtained The same method is used to obtain the attention heat map H 4 and the enhanced features of stage 4 The specific implementation process is as follows:

[0075]

[0076] Therefore, the total loss function under the multi-level progressive feature enhancement module is:

[0077]

[0078] Among them, L MSE represents the comprehensive MSE loss, and denote the MSE losses of the 2nd, 3rd and 4th stages respectively;

[0079] (4) The pedestrian head image is sent to the identity consistency module, and the regional enhancement module therein optimizes the overall model's attention distribution on the pedestrian head image from different camera perspectives. The specific implementation process of the identity consistency module is as follows:

[0080] During the training process, the input pedestrian head image is sent to the identity consistency module to obtain the feature representation H 4,l , where 4 represents the features after the 4th convolutional block in the ResNet50 network, l represents the low-level features; the low-level features H under the front camera and back camera perspectives are 4,l After the generalized average pooling layer and the fully connected layer, the mid-level feature and high-level feature representations are obtained respectively; the specific implementation process is as follows:

[0081]

[0082] Where f represents the front camera view, n represents the back camera view, m and h represent the middle and high levels respectively; GeMP represents the generalized average pooling layer, and FC represents the fully connected layer;

[0083] Then, the front camera view feature map H f As the input of the region enhancement module, it is enhanced at the channel attention level and the spatial attention level respectively; specifically, at the channel attention level, H f It is sent to the GeMP layer for generalized average pooling, followed by two convolutional layers Conv down and Conv up Reducing and increasing the dimension respectively; At the spatial attention level, a convolutional layer is used to f Dimensionality reduction is performed, and average pooling and maximum pooling operations are performed along the channel at the same time; finally, the Sigmoid activation function is used for optimization, and the optimized representations are compared with H f Perform element-wise multiplication to get and After adding the two together, we get the final enhanced frontal camera perspective head feature. The specific implementation process is as follows:

[0084]

[0085]

[0086] Among them, REA represents the regional enhancement module, max represents the maximum pooling operation, avg represents the average pooling operation, σ1 represents the Sigmoid activation function, c and s represent the channel level and spatial level respectively, ⊕ represents the element level addition, Conv down and Conv up Represent the convolution operations of dimensionality reduction and dimensionality increase respectively;

[0087] Finally, the maximum mean difference loss function MMDLoss is used to The feature map F extracted from the original image branch based on the original image of the pedestrian r Perform loss constraint, where F r Pedestrian appearance features F extracted from the original image branch R After processing by the generalized average pooling layer, the specific implementation process is as follows:

[0088]

[0089] Among them, μ represents mean calculation, σ represents variance calculation;

[0090] (5) The human body structure semantic matrix is further used to separate identity-related and clothing-related mask images. The explicit enhancement module and implicit suppression module are used to guide the overall model to adaptively learn identity-related information and penalize clothing-related region representations. The specific implementation process of the explicit enhancement module and implicit suppression module is as follows:

[0091] First, the head mask obtained from the human body structure semantic matrix Y is used as the auxiliary input of the explicit enhancement module, and the upper arm mask, torso mask and thigh mask are combined as the auxiliary input of the implicit suppression module; further, in order to achieve the pedestrian appearance feature F R Separate the head, torso and limbs structural features, and design a multi-region feature parsing mechanism based on spatial attention guidance; specifically, the pedestrian appearance feature F R ∈R H×W×C Input into the convolution classification module composed of convolution kernel and softmax activation function, and output the multidimensional attention weight map M∈R H×W×P , where the dimensional parameters are defined as: height H, width W, number of channels C, attention weight map dimension P = 7, pixel values represent background 0, head 1, torso 2, upper arm 3, lower arm 4, thigh 5 and calf 6 respectively; the mathematical representation of this mechanism is as follows:

[0092] M=SoftMax(Conv 1×1 (F R )) (40)

[0093] Among them, Conv 1×1 Indicates the dimensionality reduction operation using a 1×1 convolution kernel; for the attention weight matrix M p The pth channel of its matrix element Quantify the confidence that the spatial coordinate (i, j) belongs to the p-th type of human structure area, and the probability value is constrained to the unit interval [0,1] by the sigmoid function;

[0094] Finally, in order to enhance the accuracy of attention weight mapping in distinguishing human body structure areas, a semantic alignment loss function L is designed. SA Drive spatial attention distribution optimization, its mathematical form is defined as:

[0095]

[0096] Where N is the number of batch samples, M(h,w) and Y(h,w) represent the spatial attention weight matrix and the probability value of the human body structure area at the coordinate (h,w), respectively;

[0097] Aiming at the generalization error of the human body parsing network, a dynamic attention optimization mechanism is introduced - through the cross entropy supervision loss function L ID The driver parameter update makes the generated attention distribution adaptive to clothing changes rather than being restricted by the static constraints of the pre-trained parsing network; finally, the explicit enhancement module performs attention modulation operations to transform the appearance features F R and the optimized head attention probability map M head Multiply to generate identity invariant feature F ID , its mathematical expression is:

[0098] F ID =F R ×M head (42)

[0099] Among them, M head ∈R H×W×1 is the head attention probability map, × represents the tensor element-level product operation; considering the clothing-independent features F extracted by the multi-level progressive feature enhancement module b and identity invariant feature F id Shared latent identity space, where F b The clothing-independent features F extracted by the multi-level feature enhancement module branch B After processing by the generalized average pooling layer, F id is the identity invariant feature F ID After processing the generalized average pooling layer, the identity invariant loss function L is designed. II To achieve cross-modal feature alignment, its mathematical expression is:

[0100]

[0101] Among them, μ represents mean calculation, σ represents variance calculation;

[0102] Then the implicit suppression module aggregates the attention probability distribution of the trunk, upper arms and thighs and combines it with the pedestrian appearance representation F R Perform channel weighted fusion to generate clothing sensitive features F C , the mathematical process is described as:

[0103] F C =F R ×(M arms +M torso +M legs ) (44)

[0104] Among them, the human body structure area attention probability map M is defined arms ∈R H×W×1 , M torso ∈R H×W×1 With M legs ∈R H×W×1 Corresponding to the upper arm, torso and thigh areas respectively; + means element-by-element superposition;

[0105] Secondly, by constructing a robust clothing feature classifier, the discriminative learning of clothing attributes is realized based on clothing category labels; the total number of clothing categories is determined by the union cardinality of unique clothing items corresponding to all pedestrian identities in the enhanced multimodal training dataset, thereby completing the automatic labeling of pedestrian clothing labels; the training goal of the clothing feature classifier is to minimize the clothing classification loss L C , its mathematical expression is:

[0106]

[0107] Among them, N represents the batch sample size, N C Represents the cardinality of clothing categories in the training dataset, θ j ∈R d is the learnable parameter vector of the j-th clothing discriminator, and the hyperparameter γ is used to adjust the smoothness of the probability distribution; represents the clothing features of the i-th pedestrian image, Represents the clothing label of the i-th pedestrian image; by minimizing the clothing classification loss L C , in the feature extraction stage, the overall model's ability to discriminate clothing attributes is suppressed, making the optimized clothing discriminator unable to effectively distinguish the clothing categories of the same identity and cross-clothing samples, ultimately forcing the overall model to discard the interference of clothing-related information and strengthen its focus on identity information;

[0108] (6) Implement weighted optimization based on the comprehensive loss function and use the back-propagation algorithm to complete the end-to-end parameter update of the model;

[0109] L=L ID +L tri +λ1L MMD +λ2L MSE +L II +L SA +L C (46)

[0110] Among them, L represents the comprehensive loss function, L ID represents the cross entropy supervision loss function, L tri Represents the triple loss function. λ1 and λ2 represent L MMD and L MSE The weight parameter of .

[0111] (7) Input the independent test set into the optimized model to evaluate its cross-scenario classification accuracy;

[0112] Taking into account the computational and time costs of the model in application, during the testing process, the modules used in the training phase were discarded, and only the trained ResNet50 backbone network was used to extract pedestrian image features. Three different clothing-changing test datasets were used as inputs to the backbone network, and the mean average precision (mAP) and cumulative matching features (CMC) were used to evaluate the model performance.

Claims

1. A method for re-identifying pedestrians after changing clothes based on implicit and explicit learning of pedestrian soft semantic information, characterized by: The following steps are involved: Step 1: Use the human body parsing network to obtain the human body structure semantic matrix based on the human body region labels; The pre-trained SCHPNet human parsing network is used to generate the human body structure semantic matrix Y, which contains 7 labels: background, head, torso, upper arm, lower arm, thigh, and lower leg, corresponding to the pixel values 0 to 6 in the human body structure semantic matrix Y. The human body structure semantic matrix Y is separated according to the labels to obtain the corresponding 7 human body structure mask images. Step 2: Use the prediction box information of the target detection framework to accurately crop the pedestrian head area; Deploy a recognition framework based on the YOLOv5 architecture to accurately locate the pedestrian head region in the original pedestrian image and crop the pedestrian head image. Finally, the original pedestrian image and the extracted pedestrian head image are paired and combined to construct an enhanced multimodal training dataset. Step 3: The overall model consists of three branches: the original image branch composed of the ResNet50 network, the multi-level progressive feature enhancement module branch, and the identity consistency module branch. In addition, the explicit enhancement module cooperates with the original image branch to obtain the identity invariant feature F ID , the implicit suppression module cooperates with the original image branch to obtain the clothing sensitive feature F C ,The identity consistency module branch includes the region enhancement module; The human body structure semantic matrix is used to obtain the pedestrian black image corresponding to the original pedestrian image, and then fed into the multi-level progressive feature enhancement module to extract the human foreground contour, limb joints and initial soft semantic information of the head; Step 4: Send the pedestrian head image to the identity consistency module, and use the region enhancement module in it to optimize the overall model's attention distribution on pedestrian head images from different camera perspectives; Step 5: Further utilize the human body structure semantic matrix to separate identity-related and clothing-related mask maps, and guide the overall model to adaptively learn identity-related information and penalize clothing-related region representations through the explicit enhancement module and implicit suppression module respectively; Step 6: Implement weighted optimization based on the comprehensive loss function and use the backpropagation algorithm to complete the end-to-end parameter update of the model; Step 7: Input the independent test set into the optimized overall model to evaluate its cross-scenario classification accuracy.

2. The method for re-identifying pedestrians after changing clothes based on explicit and implicit learning of pedestrian soft semantic information according to claim 1 is characterized in that: The specific implementation process of the multi-level progressive feature enhancement module is as follows: Using the human body structure semantic matrix Y, the pixel values of the clothing-related areas in the original pedestrian image are set to 0 to obtain the corresponding pedestrian black clothing image; then the human body structure mask images separated by the human body parsing network are recombined to obtain the human foreground contour mask image M silh , limb joint mask map M lim and head mask M head , and then perform feature enhancement in the three layers of the ResNet50 network; specifically: First, the stage 1 of the ResNet50 network is not processed, and the pedestrian black clothing image is obtained after the stage 2 of the ResNet50 network. Where B represents the feature identifier extracted by the multi-level progressive feature enhancement module, and then two consecutive 1×1 convolutional layers are used to Dimensionality reduction to 1 dimension; Then, after these two 1×1 convolutional layers, ReLU layers and Sigmoid layers are connected to optimize parameters to obtain the attention heat map H 2 ; Then use the mean square loss function MSE to construct M silh With H 2 The loss constraint between 2 and Perform element-wise multiplication at the corresponding spatial position to obtain the enhanced features in stage 2 The specific process is expressed as: Among them, Conv 1×1 Indicates the dimensionality reduction operation using a 1×1 convolution kernel, represents the MSE loss function under stage 2 feature enhancement, represents element-wise multiplication, |||| 2 represents the L2 norm; Similarly, the features after stage 2 enhancement After stage 3, the feature map is obtained Use the same dimensionality reduction method to reduce the dimensionality to obtain H 3 , and use the mean square loss function to construct H 3 With M lim The loss constraint between them guides H 3 To fit the attention distribution of the human limb joint area, finally H 3 and Perform element-wise multiplication at the corresponding spatial position to obtain the enhanced features in stage 3 Then the enhanced features in stage 3 After stage 4, the feature map is obtained The same method is used to obtain the attention heat map H 4 and the enhanced features of stage 4 The specific implementation process is as follows: Therefore, the total loss function under the multi-level progressive feature enhancement module is: Among them, L MSE represents the comprehensive MSE loss, and denote the MSE losses of the 2nd, 3rd and 4th stages respectively.

3. The method for re-identifying pedestrians after changing clothes based on explicit and implicit learning of pedestrian soft semantic information according to claim 1 is characterized in that: The specific implementation process of the identity consistency module is as follows: During the training process, the input pedestrian head image is sent to the identity consistency module to obtain the feature representation H 4,l , where 4 represents the features after the 4th convolutional block in the ResNet50 network, and l represents the low-level features; Low-level features H under the front camera and back camera views 4,l After the generalized average pooling layer and the fully connected layer, the mid-level feature and high-level feature representations are obtained respectively; the specific implementation process is as follows: Where f represents the front camera view, n represents the back camera view, m and h represent the middle and high levels respectively; GeMP represents the generalized average pooling layer, and FC represents the fully connected layer; Then, the front camera view feature map H f As the input of the region enhancement module, it is enhanced at the channel attention level and the spatial attention level respectively; specifically, at the channel attention level, H f It is sent to the GeMP layer for generalized average pooling, followed by two convolutional layers Conv down and Conv up Respectively reduce and increase its dimension; at the spatial attention level, use a convolutional layer to H f Dimensionality reduction is performed, and average pooling and maximum pooling operations are performed along the channel at the same time; finally, the Sigmoid activation function is used for optimization, and the optimized representations are compared with H f Perform element-wise multiplication to get and After adding the two together, we get the final enhanced frontal camera perspective head feature. The specific implementation process is as follows: Among them, REA represents the regional enhancement module, max represents the maximum pooling operation, avg represents the average pooling operation, σ1 represents the Sigmoid activation function, c and s represent the channel level and spatial level respectively. Table element-level addition, Conv down and Conv up Represent the convolution operations of dimensionality reduction and dimensionality increase respectively; Finally, the maximum mean difference loss function MMDLoss is used to The feature map F extracted from the original image branch based on the original image of the pedestrian r Perform loss constraint, where F r Pedestrian appearance features F extracted from the original image branch R After being processed by the generalized average pooling layer, the specific implementation process is as follows: Among them, μ represents mean calculation and σ represents variance calculation.

4. The method for re-identifying pedestrians after changing clothes based on explicit and implicit learning of pedestrian soft semantic information according to claim 1 is characterized in that: The specific implementation process of the explicit enhancement module and the implicit suppression module is as follows: First, the head mask obtained from the human body structure semantic matrix Y is used as the auxiliary input of the explicit enhancement module, and the upper arm mask, torso mask and thigh mask are combined as the auxiliary input of the implicit suppression module; further, in order to achieve the pedestrian appearance feature F R Separate the head, torso and limbs structural features, and design a multi-region feature parsing mechanism based on spatial attention guidance; specifically, the pedestrian appearance feature F R ∈R H×W×C Input into the convolution classification module composed of convolution kernel and softmax activation function, and output the multidimensional attention weight map M∈R H×W×P , where the dimensional parameters are defined as: height H, width W, number of channels C, attention weight map dimension P = 7, pixel values represent background 0, head 1, torso 2, upper arm 3, lower arm 4, thigh 5 and calf 6 respectively; the mathematical representation of this mechanism is as follows: M=SoftMax(Conv 1×1 (F R )) (17) Among them, Conv 1×1 Indicates the dimensionality reduction operation using a 1×1 convolution kernel; for the attention weight matrix M p The pth channel of its matrix element Quantify the confidence that the spatial coordinate (i, j) belongs to the p-th type of human structure area, and the probability value is constrained to the unit interval [0,1] by the sigmoid function; Finally, in order to enhance the accuracy of attention weight mapping in distinguishing human body structure areas, a semantic alignment loss function L is designed. SA Drive spatial attention distribution optimization, its mathematical form is defined as: Where N is the number of batch samples, M(h,w) and Y(h,w) represent the spatial attention weight matrix and the probability value of the human body structure area at the coordinate (h,w), respectively; Aiming at the generalization error of the human body parsing network, a dynamic attention optimization mechanism is introduced - through the cross entropy supervision loss function L ID The driver parameter update makes the generated attention distribution adaptive to clothing changes rather than being restricted by the static constraints of the pre-trained parsing network; finally, the explicit enhancement module performs attention modulation operations to transform the appearance features F R and the optimized head attention probability map M head Multiply to generate identity invariant feature F ID , its mathematical expression is: F ID =F R ×M head (19) Among them, M head ∈R H×W×1 is the head attention probability map, × represents the tensor element-level product operation; considering the clothing-independent features F extracted by the multi-level progressive feature enhancement module b and identity invariant feature F id Shared latent identity space, where F b The clothing-independent features F extracted by the multi-level feature enhancement module branch B After processing by the generalized average pooling layer, F id is the identity invariant feature F ID After processing the generalized average pooling layer, the identity invariant loss function L is designed. II To achieve cross-modal feature alignment, its mathematical expression is: Among them, μ represents mean calculation, σ represents variance calculation; Then the implicit suppression module aggregates the attention probability distribution of the trunk, upper arms and thighs and combines it with the pedestrian appearance representation F R Perform channel weighted fusion to generate clothing sensitive features F C , its mathematical process is described as: F C =F R ×(M arms +M torso +M legs ) (21) Among them, the human body structure area attention probability map M is defined arms ∈R H×W×1 , M torso ∈R H×W×1 With M legs ∈R H×W×1 Corresponding to the upper arm, torso and thigh areas respectively; + means element-by-element superposition; Secondly, by constructing a robust clothing feature classifier, the discriminative learning of clothing attributes is realized based on clothing category labels; the total number of clothing categories is determined by the union cardinality of unique clothing items corresponding to all pedestrian identities in the enhanced multimodal training dataset, thereby completing the automatic labeling of pedestrian clothing labels; the training goal of the clothing feature classifier is to minimize the clothing classification loss L C , its mathematical expression is: Among them, N represents the batch sample size, N C Represents the cardinality of clothing categories in the training dataset, θ j ∈R d is the learnable parameter vector of the j-th clothing discriminator, and the hyperparameter γ is used to adjust the smoothness of the probability distribution; represents the clothing features of the i-th pedestrian image, Represents the clothing label of the i-th pedestrian image; by minimizing the clothing classification loss L C , in the feature extraction stage, the overall model's ability to discriminate clothing attributes is suppressed, making the optimized clothing discriminator unable to effectively distinguish the clothing categories of the same identity and cross-clothing samples, and ultimately forcing the overall model to discard the interference of clothing-related information and strengthen its attention to identity information.

5. The method for re-identifying pedestrians after changing clothes based on explicit and implicit learning of pedestrian soft semantic information according to claim 1 is characterized in that: Based on the comprehensive loss function, weighted optimization is implemented and the back-propagation algorithm is used to complete the end-to-end parameter update of the model: L=L ID +L tri +θ1L MMD +λ2L MSE +L II +L SA +L C (23) Among them, L represents the comprehensive loss function, L ID represents the cross entropy supervision loss function, L tri represents the triple loss function; λ1 and λ2 represent L MMD and L MSE The weight parameter of .

Images