A training method, device, equipment and medium for pedestrian recognition model

Abstract

The present invention relates to a training method, device, equipment and medium for a pedestrian recognition model, belonging to the field of artificial intelligence technology, and the method includes: constructing a semantic feature encoder integrated with an attention mechanism; extracting semantic features of a sample data set through the semantic feature encoder; parsing the sample data set to obtain body shape features; obtaining KL divergence loss and feature difference loss based on the semantic features and the body shape features; performing weighted calculation based on at least KL divergence loss and feature difference loss to obtain model loss parameters; training a pedestrian recognition model to be trained according to the model loss parameters until the training converges to obtain a trained pedestrian recognition model. The scheme of the present invention extracts the semantic features of the original pedestrian more fully by integrating a pixel-by-pixel attention mechanism into the backbone network, and suppresses the interference of clothing-related information on the model by learning the identity information of the pedestrian through the body shape branch, so that the model pays attention to deep-level clothing-irrelevant feature information.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a training method, device, equipment and medium for a pedestrian recognition model. Background Art

[0002] At present, the traditional closed-world pedestrian re-identification algorithm has achieved satisfactory performance. However, the performance of the closed-world pedestrian re-identification model in the application of new data domains is unsatisfactory, because it does not pay attention to the impact of large differences between images across time, cameras and regions on model reasoning. In the open world of real life, the pedestrian samples captured by the camera have a large time span, and as time goes by, the problem of pedestrian changing clothes is introduced, that is, the pedestrian's clothing will change to a certain extent.

[0003] In crowded places, there are also a large number of individuals with similar visual appearance. Traditional appearance-based pedestrian re-identification methods will inevitably be misleading, resulting in a decline in model performance. Summary of the invention

[0004] In view of this, it is necessary to provide a training method, device, equipment and medium for a pedestrian recognition model to solve the problem that the appearance-based pedestrian re-identification method will inevitably mislead and thus the performance of the model will deteriorate.

[0005] In order to solve the above problems, the present invention provides a training method for a pedestrian recognition model, comprising: Construct a semantic feature encoder that incorporates an attention mechanism; Extracting semantic features of the sample data set by the semantic feature encoder; Parsing the sample data set to obtain body shape features; According to the semantic features and the body shape features, a KL divergence loss and a feature difference loss are obtained; At least perform weighted calculation based on KL divergence loss and feature difference loss to obtain model loss parameters; The pedestrian recognition model to be trained is trained according to the model loss parameter until the training converges to obtain a trained pedestrian recognition model.

[0006] Optionally, obtaining a KL divergence loss and a feature difference loss according to the semantic feature and the body shape feature specifically includes: Calculating the first identity prediction probability of the semantic feature by the first identity classifier of the pedestrian recognition model; Calculating the second identity prediction probability of the body shape feature by using the second identity classifier of the pedestrian recognition model; wherein the first identity classifier and the second identity classifier share a weight; A KL divergence loss and a feature difference loss are calculated according to the first identity prediction probability and the second identity prediction probability.

[0007] Optionally, it also includes: Obtaining clothing contrast loss and difficulty triplet loss according to the semantic features; Calculating an identity cross entropy loss based on the first identity prediction probability and the second identity prediction probability; The weighted calculation is performed at least based on the KL divergence loss and the feature difference loss to obtain the model loss parameter, which specifically includes: A weighted calculation is performed according to the clothing contrast loss, the difficult triplet loss, the identity cross entropy loss, the KL divergence loss and the feature difference loss to obtain a model loss parameter.

[0008] Optionally, parsing the sample data set to obtain body shape features specifically includes: Parsing the sample data set to obtain a human body mask image with human body parts annotated; Inputting the human body mask image into a convolutional layer for feature transformation to obtain image features; Inputting the image features into a multi-head self-attention layer to fuse shape features of semantic information from different fields of view to obtain fused features; The fused features are input into a feed-forward network composed of fully connected layers for processing to obtain body shape features.

[0009] Optionally, constructing a semantic feature encoder incorporating an attention mechanism specifically includes: Obtain the pre-trained original feature extraction network; Performing pixel-by-pixel attention between each convolution block of the original feature extraction network to obtain an original feature extraction network incorporating the pixel-by-pixel attention; The original feature extraction network with pixel-by-pixel attention mechanism is used as the semantic feature encoder with attention mechanism.

[0010] Optionally, the processing of the pixel-by-pixel attention mechanism between each convolution block of the original feature extraction network specifically includes: Obtaining a first feature output by a current convolutional block of the original feature extraction network; The first feature is processed sequentially by depthwise separable convolution, instance normalization and a first activation function to obtain a second feature; The second feature is sequentially processed by deformable convolution, layer normalization and second activation function to obtain an attention weight matrix of the same dimension as the first feature; Multiplying the first feature by the attention weight matrix point by point to obtain a third feature; The third feature is used as the input feature of the next convolution block.

[0011] Optionally, it also includes: Encoding the image to be retrieved by the semantic feature encoder of the trained pedestrian recognition model to obtain a pedestrian image feature vector of the image to be retrieved; Calculating the similarity between the pedestrian image feature vector and the image feature vectors of each image in the image data set to obtain a similarity calculation result; According to the similarity calculation result, the recognition result of the target pedestrian in the image to be retrieved is obtained.

[0012] The present invention also provides a training device for a pedestrian recognition model, comprising: a construction module, an extraction module, a parsing module, a first calculation module, a second calculation module and a training module; The building module is used to build a semantic feature encoder that incorporates an attention mechanism; The extraction module is used to extract the semantic features of the sample data set through the semantic feature encoder; The parsing module is used to parse the sample data set to obtain body shape features; The first calculation module is used to obtain KL divergence loss and feature difference loss according to the semantic features and the body shape features; The second calculation module is used to perform weighted calculation based on at least KL divergence loss and feature difference loss to obtain a model loss parameter; The training module is used to train the pedestrian recognition model to be trained according to the model loss parameter until the training converges to obtain the trained pedestrian recognition model.

[0013] Another technical solution of the present invention to solve the above technical problem is an electronic device, including a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in a training method for a pedestrian recognition model in any of the above-mentioned solutions.

[0014] Another technical solution of the present invention to solve the above technical problem is a computer-readable storage medium for storing computer-readable programs or instructions, which, when executed by a processor, can implement the steps in a training method for a pedestrian recognition model of any of the above solutions.

[0015] By applying the technical solution provided in the embodiment of the present application, the semantic features of the original pedestrians are more fully extracted by integrating the pixel-by-pixel attention mechanism into the backbone network, and the identity information of the pedestrians is learned through the body shape branch to suppress the interference of clothing-related information on the model, so that the model focuses on deep-level clothing-independent feature information. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flowchart of a training method for a pedestrian recognition model provided in an embodiment of the present invention; Figure 2 is a flowchart of the steps of calculating KL divergence loss and feature difference loss provided in an embodiment of the present invention; Figure 3 is a flowchart of steps for calculating body shape features provided in an embodiment of the present invention; Figure 4 is a schematic diagram of the training model structure provided in an embodiment of the present invention; Figure 5 is a flowchart of the process steps of a semantic feature encoder incorporating an attention mechanism provided in an embodiment of the present invention; Figure 6 is a flowchart of the steps of the pixel-by-pixel attention mechanism processing provided in an embodiment of the present invention; Figure 7 is a flowchart of the steps of the pedestrian recognition method provided in an embodiment of the present invention; Figure 8 It is a flowchart of the steps of a multi-modal clothing-changing pedestrian re-identification method integrating an attention mechanism provided in an embodiment of the present invention; Fig. 9 is a structural block diagram of a training device for a pedestrian recognition model provided in an embodiment of the present invention; Fig.10 It is a hardware structure block diagram of an electronic device provided in each embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Reference Figure 1 As shown, a flowchart of a method for training a pedestrian recognition model is shown, which can be applied to artificial intelligence, such as Figure 1 As shown, the following steps may be specifically included: Step S101, constructing a semantic feature encoder incorporating an attention mechanism; In this embodiment, a pre-trained original feature extraction network is obtained, and a pixel-by-pixel attention mechanism is processed between each convolution block of the original feature extraction network to obtain an original feature extraction network integrated with the pixel-by-pixel attention mechanism; the original feature extraction network integrated with the pixel-by-pixel attention mechanism is used as a semantic feature encoder integrated with the attention mechanism; In this embodiment, the pre-trained original feature extraction network can be a ResNet-50 network pre-trained on ImageNet as the initial backbone network. ; For example, adding a pixel-by-pixel attention mechanism to the initial backbone network may include the following steps: For the output features of the current convolution block , for the features Perform a depth-separable convolution operation to obtain features after the number of channels is transformed , and Perform instance normalization operation to obtain standard features after the normalized features are output through the GELU activation function ; , Among them, DSP represents depth-wise separable convolution, IN represents instance normalization, and GELU represents GELU activation function.

[0019] Standardized features Perform deformable convolution to obtain features , and After layer normalization and Sigmoid activation function output, the original feature is obtained Attention weight matrices of the same dimension , ， Among them, LN represents layer normalization, Sigmoid represents Sigmoid activation function, and DC represents deformable convolution.

[0020] As another example, the deformable convolution is implemented as follows: , Among them, K is the size of the convolution kernel, p is the corresponding position in the feature map, Indicates the preset offset, is the preset weight, represents the learnable offset parameter, represents the learnable weight adjustment parameter. With the original features The features obtained by element-by-element multiplication are used as the input features of the next convolution block. ; , in, Represents an element-wise multiplication operation.

[0021] Follow the above steps to build the initial backbone network The same process is performed between adjacent convolution blocks to obtain the backbone network with attention mechanism. As a semantic feature encoder.

[0022] Step S102, extracting semantic features of the sample data set through the semantic feature encoder; In this embodiment, the semantic features may include the most comprehensive features, such as body shape features, clothing features, etc. Input into the semantic feature encoder to obtain the semantic features of the sample .

[0023] Step S103, parsing the sample data set to obtain body shape features; In this embodiment, the existing human body analysis model is used to analyze the data set samples. Parse and obtain a human body mask image with clearly marked human body parts ; The mask image The convolutional layer input to the shape attention module performs feature transformation. , after the feature transformation of the convolutional layer, the feature ; Will The multi-head self-attention layer is input into the shape attention module to obtain the shape features that integrate the semantic information of different fields of view, and then input it into the feedforward network composed of fully connected layers to obtain the features. , For body shape characteristics.

[0024] The characteristics As the final body shape feature information, it is used for the calculation of subsequent model loss; For example, the existing human parsing model can be a Mask R-CNN model: Mask R-CNN can simultaneously implement target detection, bounding box regression, and mask segmentation. In the task of labeling key nodes of the human body, each key node of the human body can be regarded as a different target, and then Mask R-CNN is used to detect and label it to obtain a human mask image.

[0025] Step S104, obtaining KL divergence loss and feature difference loss according to the semantic features and the body shape features; In this embodiment, the first identity prediction probability of the semantic feature is calculated by the first identity classifier of the pedestrian recognition model; the second identity prediction probability of the body shape feature is calculated by the second identity classifier of the pedestrian recognition model; wherein the first identity classifier and the second identity classifier share weights; and the KL divergence loss and the feature difference loss are calculated based on the first identity prediction probability and the second identity prediction probability.

[0026] For example, the first identity classifier may be a classifier , the second identity classifier can be ; Semantic features And body shape characteristics Input to the classifier with shared weights and In the above example, we get the identity prediction probability of the two features. and .

[0027] Based on predicted probability and Calculating the KL divergence loss can specifically include: , in, The feature distribution representing the semantic features, distribution of traits representing body shape characteristics; Indicates the number of Batchsize data samples; i = 1. The KL divergence of the two features is added as the overall KL divergence loss: , As another example, according to the predicted probability and Calculating feature difference loss may specifically include: calculating feature difference loss of two different modal features to alleviate the impact of clothing-related features on identity reasoning: , in, represents the identity prediction probability given by the classifier corresponding to the semantic feature, Indicates bilinear pooling of two features. Indicates the identity prediction probability given by the classifier corresponding to the body shape feature; Represents the true label of the sample; Indicates Batch size; Indicates physical characteristics; Represents semantic features.

[0028] Step S105: Perform weighted calculation based on at least the KL divergence loss and the feature difference loss to obtain a model loss parameter; In this embodiment, a weighted calculation is performed based on clothing contrast loss, difficult triplet loss, identity cross entropy loss, KL divergence loss, and feature difference loss to obtain a model loss parameter; For example, the overall loss calculation of model training specifically includes: , in, represents the feature difference loss, represents the KL divergence loss, represents the hard triplet loss, represents clothing contrast loss, is the KL balance coefficient, is the characteristic difference balance coefficient, is the balance coefficient of the difficult triplet, is the clothing contrast balance coefficient. The model is optimized according to the overall loss value, and the semantic feature encoder is used after the model training reaches convergence. Calculate the feature similarity between the images to be retrieved and perform pedestrian identity inference.

[0029] Step S106: Train the pedestrian recognition model to be trained according to the model loss parameter until the training converges to obtain a trained pedestrian recognition model.

[0030] The technical solution of the embodiment of the present application more fully extracts the semantic features of the original pedestrians by integrating the pixel-by-pixel attention mechanism into the backbone network, and suppresses the interference of clothing-related information on the model by learning the identity information of the pedestrians through the body shape branch, so that the model focuses on deep-level clothing-independent feature information.

[0031] The technical solution of the embodiment of the present application is suitable for pedestrian recognition scenarios with a large time span and scenarios where pedestrians actively change clothes to avoid retrieval. By extracting clothing-independent differential features of pedestrian samples as key information for pedestrian retrieval, it still has strong stability in dealing with pedestrian recognition tasks in different scenarios.

[0032] In one example, referring to Figure 2 As shown, a flowchart of the steps for calculating KL divergence loss and feature difference loss is shown, as Figure 2 As shown, the following steps may be specifically included: Step S201: Calculate the first identity prediction probability of the semantic feature through the first identity classifier of the pedestrian recognition model; in this embodiment, the first identity classifier can be a classifier ; Step S202: Calculate the second identity prediction probability of the body shape feature through the second identity classifier of the pedestrian recognition model; wherein the first identity classifier and the second identity classifier share weights; in this embodiment, the second identity classifier can be ; Step S203: Calculate the KL divergence loss and the feature difference loss according to the first identity prediction probability and the second identity prediction probability.

[0033] In this embodiment, the classifier is obtained and classifier , the semantic features And body shape characteristics Input to the classifier with shared weights and In the above example, we get the identity prediction probability of the two features. and .

[0034] For example, according to the predicted probability and Calculating the KL divergence loss can specifically include: , in, The feature distribution representing the semantic features, distribution of traits representing body shape characteristics; Indicates the number of Batchsize data samples; i = 1. The KL divergence of the two features is added as the overall KL divergence loss: , As another example, according to the predicted probability and Calculating feature difference loss may specifically include: calculating feature difference loss of two different modal features to alleviate the impact of clothing-related features on identity reasoning: , in, represents the identity prediction probability given by the classifier corresponding to the semantic feature, Indicates bilinear pooling of two features. Indicates the identity prediction probability given by the classifier corresponding to the body shape feature; Represents the true label of the sample; Indicates Batch size; Indicates physical characteristics; Represents semantic features.

[0035] In one example, a clothing contrast loss and a difficult triplet loss are obtained according to the semantic features; an identity cross entropy loss is calculated according to the first identity prediction probability and the second identity prediction probability; In the example, by semantic features The clothing contrast loss can be calculated using the following formula: , in, Indicates the batch size. Indicates the number of garments, represents the eigenvector corresponding to the center of mass of the garment, is the temperature coefficient, It means the features are obtained after the feature transformation of the convolutional layer.

[0036] As another example, the difficult triplet loss of semantic features is calculated: , in, Represents the difficult positive sample features of the anchor box sample, represents the difficult negative samples of anchor box samples, represents the Euclidean distance between samples, represents the edge coefficient, Indicates the maximum value operation. Indicates Batchsize, Represents semantic features.

[0037] In one example, the identity cross entropy loss of two features under a shared weight classifier is calculated based on the identity prediction probability of the sample: , , in, represents the classifier's predicted probability of the sample's identity, represents the true label of the sample, Represents the number of pedestrian identities, and the identity cross entropy loss of two different modal features is added for model optimization: , in, represents the semantic feature identity cross entropy loss, represents the body shape feature identity cross entropy loss.

[0038] In one example, the weighted calculation is performed at least based on the KL divergence loss and the feature difference loss to obtain the model loss parameters, specifically including: performing weighted calculation according to the clothing contrast loss, the difficult triplet loss, the identity cross entropy loss, the KL divergence loss and the feature difference loss to obtain the model loss parameters.

[0039] For example, the overall loss calculation of model training specifically includes:

[0040] in, represents the feature difference loss, represents the KL divergence loss, represents the hard triplet loss, represents clothing contrast loss, is the KL balance coefficient, is the characteristic difference balance coefficient, is the balance coefficient of the difficult triplet, is the clothing contrast balance coefficient. The model is optimized according to the overall loss value, and the semantic feature encoder is used after the model training reaches convergence. Calculate the feature similarity between the images to be retrieved and perform pedestrian identity inference.

[0041] In one example, referring to Figure 3 As shown, a flowchart of the steps of the method for calculating body shape features is shown, such as Figure 3 As shown, the following steps may be specifically included: Step S301, parsing the sample data set to obtain a human body mask image with human body part annotations; In this embodiment, the existing human body analysis model is used for the data set samples Parse and obtain a human body mask image with clearly marked human body parts .

[0042] For example, Figure 4 As shown in the figure, after parsing the human body image, a human body mask image is obtained that distinguishes and annotates different parts of the human body. For example, the head area, upper body area, arms area, legs area and feet area are distinguished and annotated respectively. The human body mask image is input into the body shape attention module for processing to obtain the feature , where the body shape attention module includes: convolutional layer, multi-head attention and feedforward network. Input to the classifier with shared weights The identity prediction probability is obtained by processing . Based on identity prediction probability and Compute the identity cross entropy loss, KL divergence loss, and feature discrepancy loss of the model.

[0043] Step S302: input the human body mask image into a convolutional layer for feature transformation to obtain image features; In this embodiment, the mask image The convolutional layer input to the shape attention module performs feature transformation. , after the feature transformation of the convolutional layer, the feature ; Step S303: input the image features into a multi-head self-attention layer to fuse shape features of semantic information of different fields of view to obtain fused features; Step S304: Input the fusion features into a feed-forward network composed of fully connected layers for processing to obtain body shape features. The multi-head self-attention layer is input into the shape attention module to obtain the shape features that integrate the semantic information of different fields of view, and then input it into the feedforward network composed of fully connected layers to obtain the features. , For body shape characteristics.

[0044] For example, parsing the sample data set specifically includes: the existing human body parsing model analyzes the data set sample Parse and obtain a human body mask image with clearly marked human body parts ; The mask image The convolutional layer input to the shape attention module performs feature transformation. , after the feature transformation of the convolutional layer, the feature ;Will The multi-head self-attention layer is input into the shape attention module to obtain the shape features that integrate the semantic information of different fields of view, and then input it into the feedforward network composed of fully connected layers to obtain the features. , is a body shape feature. The final body shape feature information is used for the calculation of subsequent model losses. In one scenario, the existing human body parsing model can be a Mask R-CNN model.

[0045] In one example, referring to Figure 5 As shown, a flowchart of the steps of the process of integrating the semantic feature encoder with the attention mechanism is shown, as Figure 5 As shown, the following steps may be specifically included: Step S501, obtain a pre-trained original feature extraction network; in this embodiment, the pre-trained original feature extraction network can be a ResNet-50 network pre-trained on ImageNet as the initial backbone network ; Step S502, performing pixel-by-pixel attention mechanism processing between each convolution block of the original feature extraction network to obtain the original feature extraction network incorporating the pixel-by-pixel attention mechanism; In this embodiment, adding a pixel-by-pixel attention mechanism to the initial backbone network may specifically include: For the output features of the current convolution block , for the features Perform a depth-separable convolution operation to obtain features after the number of channels is transformed , and Perform instance normalization operation to obtain standard features after the normalized features are output through the GELU activation function ; , Among them, DSP represents depth-wise separable convolution, IN represents instance normalization, and GELU represents GELU activation function.

[0046] Standardized features Perform deformable convolution to obtain features , and After layer normalization and Sigmoid activation function output, the original feature is obtained Attention weight matrices of the same dimension , , Among them, LN represents layer normalization, Sigmoid represents Sigmoid activation function, and DC represents deformable convolution.

[0047] For example, the deformable convolution is implemented as: , Among them, K is the size of the convolution kernel, p is the corresponding position in the feature map, Indicates the preset offset, is the preset weight, represents the learnable offset parameter, represents the learnable weight adjustment parameter. With the original features The features obtained by element-by-element multiplication are used as the input features of the next convolution block. ; , in, Represents an element-wise multiplication operation.

[0048] Follow the above steps to build the initial backbone network The same process is performed between adjacent convolution blocks to obtain the backbone network with attention mechanism. As a semantic feature encoder.

[0049] Step S503: Use the original feature extraction network that incorporates the pixel-by-pixel attention mechanism as a semantic feature encoder that incorporates the attention mechanism.

[0050] In one example, referring to Figure 6 As shown, a flowchart of the pixel-by-pixel attention mechanism processing process is shown, as Figure 6 As shown, the following steps may be specifically included: Step S601, obtaining the first feature output by the current convolution block of the original feature extraction network; in this embodiment, the first feature is the feature output by the current convolution block ; Step S602: The first feature is processed by depthwise separable convolution, instance normalization and first activation function in sequence to obtain a second feature; in this embodiment, the second feature is a standard feature. .

[0051] In this embodiment, for the features output by the current convolution block, , for the features Perform a depth-separable convolution operation to obtain features after the number of channels is transformed , and Perform instance normalization operation to obtain standard features after the normalized features are output through the GELU activation function ; , Among them, DSP represents depth-wise separable convolution, IN represents instance normalization, and GELU represents GELU activation function.

[0052] Step S603: The second feature is processed by deformable convolution, layer normalization and second activation function in sequence to obtain an attention weight matrix with the same dimension as the first feature. In this embodiment, the attention weight matrix is ​​the same as the original feature. Attention weight matrices of the same dimension .

[0053] Step S604: multiply the first feature by the attention weight matrix point by point to obtain a third feature; in this embodiment, the third feature is an input feature .

[0054] Step S605: Use the third feature as the input feature of the next convolution block.

[0055] For example, the standardized features Perform deformable convolution to obtain features , and After layer normalization and Sigmoid activation function output, the original feature is obtained Attention weight matrices of the same dimension , , Among them, LN represents layer normalization, Sigmoid represents Sigmoid activation function, and DC represents deformable convolution.

[0056] For example, the deformable convolution is implemented as: , Among them, K is the size of the convolution kernel, p is the corresponding position in the feature map, Indicates the preset offset, is the preset weight, represents the learnable offset parameter, Represents a learnable weight adjustment parameter.

[0057] As another example, With the original features The features obtained by element-by-element multiplication are used as the input features of the next convolution block. ; , in, Represents an element-wise multiplication operation.

[0058] Follow the above steps to build the initial backbone network The same process is performed between adjacent convolution blocks to obtain the backbone network with attention mechanism. As a semantic feature encoder.

[0059] In one example, referring to Figure 7 As shown, a flowchart of the steps of the pedestrian recognition method is shown, such as Figure 7 As shown, the following steps may be specifically included: Step S701, encoding the image to be retrieved by using the semantic feature encoder of the trained pedestrian recognition model to obtain a pedestrian image feature vector of the image to be retrieved; Step S702: Calculate the similarity between the pedestrian image feature vector and the image feature vector of each image in the image data set to obtain a similarity calculation result; in this embodiment, the image data set may be a gallery data set.

[0060] Step S703: Obtain recognition results of the target pedestrian in the image to be retrieved according to the similarity calculation result.

[0061] In this embodiment, the similarity between image vectors can be calculated by a cosine similarity calculation method, and the cosine similarity measures their similarity by calculating the cosine value of the angle between two vectors. The formula is as follows: , Among them, A and B are two eigenvectors, and The cosine similarity value ranges from -1 to 1, where 1 means the vectors are in exactly the same direction, 0 means the vectors are orthogonal, and -1 means the vectors are in completely opposite directions.

[0062] In one example, referring to Figure 8 As shown, a flowchart of the steps of a multi-modal clothing-changing pedestrian re-identification method integrating an attention mechanism is shown, as Figure 8 As shown, the following steps may be specifically included: Construct a semantic feature encoder with an attention mechanism, using ResNet-50 pre-trained on ImageNet as the original feature extraction network , add a pixel-by-pixel attention mechanism module between each convolution block of the ResNet-50 network, and integrate the backbone network with the pixel-by-pixel attention mechanism As a feature extractor for the input image, the semantic features of the sample are obtained ; In this example, ResNet-50 pre-trained on ImageNet is used as the initial backbone network. , add a pixel-by-pixel attention mechanism to the initial backbone network. Specifically, for the features output by the current convolutional block , for the features Perform a depth-separable convolution operation to obtain features after the number of channels is transformed , and Perform instance normalization operation to obtain standard features after the normalized features are output through the GELU activation function ; , Among them, DSP represents depth-wise separable convolution, IN represents instance normalization, and GELU represents GELU activation function.

[0063] Standardized features Perform deformable convolution to obtain features , and After layer normalization and Sigmoid activation function output, the original feature is obtained Attention weight matrices of the same dimension , , Among them, LN represents layer normalization, Sigmoid represents Sigmoid activation function, DC represents deformable convolution, and the implementation method of deformable convolution is: , Among them, K is the size of the convolution kernel, p is the corresponding position in the feature map, Indicates the preset offset, is the preset weight, represents the learnable offset parameter, represents the learnable weight adjustment parameter. With the original features The features obtained by element-by-element multiplication are used as the input features of the next convolution block. ; , in, Represents an element-wise multiplication operation.

[0064] The same process is performed between adjacent convolutional blocks of the backbone network in the same way as above, and the backbone network with attention mechanism is obtained. As a semantic feature encoder for feature extraction, the dataset samples Input into the semantic feature encoder to obtain the semantic features of the sample .

[0065] The original pedestrian image is parsed using the existing human body parsing model to obtain a mask image with marked areas of the body. The mask image is input into the body shape attention module to extract body shape features. , the feature For model training: In this embodiment, the existing human body analysis model is used to analyze the data set samples. Parse and obtain a human body mask image with clearly marked human body parts ; The mask image The convolutional layer input to the shape attention module performs feature transformation. , after the feature transformation of the convolutional layer, the feature ; Will The multi-head self-attention layer is input into the shape attention module to obtain the shape features that integrate the semantic information of different fields of view, and then input it into the feedforward network composed of fully connected layers to obtain the features. , the feature As the final body shape feature information, it is used for the calculation of subsequent model loss; For semantic features , calculate the clothing contrast loss and difficult triplet loss of the feature for subsequent model updates: In this embodiment, for the semantic feature Calculate its clothing contrast loss for subsequent model updates; , in, Indicates the batch size. Indicates the number of garments, represents the eigenvector corresponding to the center of mass of the garment, is the temperature coefficient.

[0066] Similarly, the difficult triplet loss of semantic features is calculated: , in, Represents the difficult positive sample features of the anchor box sample, represents the difficult negative samples of anchor box samples, represents the Euclidean distance between samples, represents the edge coefficient, Indicates the maximum value operation.

[0067] Semantic features And body shape characteristics Input to the identity classifier with shared weights and In the above example, the identity prediction probabilities of the two features are calculated. and , the identity cross entropy loss, KL divergence loss and feature difference loss of the model are calculated based on the identity prediction probability, and are weighted with the loss in step 3 to finally serve as the overall training loss of the model until the training converges and is used for reasoning of clothing-changing pedestrian re-identification.

[0068] In this embodiment, the semantic feature And body shape characteristics Input to the classifier with shared weights and In the above example, we get the identity prediction probability of the two features. and ; The method of calculating the identity cross entropy loss of two features under a classifier with shared weights based on the identity prediction probability of the sample has been recorded in the previous article and will not be repeated here.

[0069] Similarly, the KL divergence loss is calculated based on the identity prediction probability of the sample's semantic features and body shape features: , in, The feature distribution representing the semantic features, The feature distribution of the body shape feature is represented by the sum of the KL divergence of the two features as the overall KL divergence loss: , Then the feature difference loss of two different modal features is calculated to alleviate the impact of clothing-related features on identity reasoning: , in, represents the identity prediction probability given by the classifier corresponding to the semantic feature, Indicates bilinear pooling of two features. Represents the identity prediction probability given by the classifier corresponding to the body shape feature.

[0070] Finally, the overall loss of model training is: , in, represents the feature difference loss, represents the KL divergence loss, represents the hard triplet loss, represents clothing contrast loss, is the balance coefficient. The model is optimized according to the overall loss value, and the semantic feature encoder is used after the model training reaches convergence Calculate the feature similarity between the images to be retrieved and perform pedestrian identity inference.

[0071] For example, Figure 4 The human body image is input into the semantic feature encoder to extract image features and obtain semantic features. , the clothing contrast loss and the difficult triplet loss of the feature are calculated for subsequent model updates.

[0072] Semantic features Input to the shared weights identity classifier Processed in to obtain the identity prediction probability .

[0073] After parsing the human body image, a human body mask image is obtained that distinguishes and annotates different parts of the human body. For example, the head area, upper body area, arms area, legs area, and feet area are distinguished and annotated respectively. The human body mask image is input into the body shape attention module for processing to obtain the feature , where the body shape attention module includes: convolutional layer, multi-head attention and feedforward network. Input to the classifier with shared weights The identity prediction probability is obtained by processing .

[0074] Predicting probability based on identity and Compute the identity cross entropy loss, KL divergence loss, and feature discrepancy loss of the model.

[0075] The technical solution of the embodiment of the present application enhances the feature extraction capability of the feature encoder by incorporating a pixel-by-pixel attention mechanism into the backbone network to obtain richer deep semantic information in the sample, thereby reducing the interference of shallow clothing-related features on identity prediction. At the same time, the model is promoted to learn clothing-independent features such as body shape of the sample based on the multimodal features of the human body parsing mask image and the body shape attention module, and the knowledge of the learned identity invariant features is transferred to the semantic feature branch through the KL divergence loss to suppress the misleading of clothing-related information. Finally, the feature difference loss in the loss function is used to calculate the difference between multimodal features to reduce the impact of clothing bias in the model reasoning process.

[0076] Reference Fig. 9 As shown, a structural block diagram of a training device 900 of a pedestrian recognition model according to an embodiment of the present invention is shown. Fig. 9 As shown, the device may specifically include the following modules: A construction module 901 is used to construct a semantic feature encoder incorporating an attention mechanism; An extraction module 902, configured to extract semantic features of a sample data set through the semantic feature encoder; The parsing module 903 is used to parse the sample data set to obtain body shape features; A first calculation module 904 is used to obtain a KL divergence loss and a feature difference loss according to the semantic feature and the body shape feature; A second calculation module 905 is used to perform weighted calculation based on at least KL divergence loss and feature difference loss to obtain a model loss parameter; The training module 906 is used to train the pedestrian recognition model to be trained according to the model loss parameter until the training converges to obtain the trained pedestrian recognition model.

[0077] In one example, the first calculation module 904 is specifically configured to calculate the first identity prediction probability of the semantic feature by using the first identity classifier of the pedestrian recognition model; Calculating the second identity prediction probability of the body shape feature by using the second identity classifier of the pedestrian recognition model; wherein the first identity classifier and the second identity classifier share a weight; A KL divergence loss and a feature difference loss are calculated according to the first identity prediction probability and the second identity prediction probability.

[0078] In one example, the first calculation module 904 is further specifically configured to obtain clothing contrast loss and difficult triplet loss according to the semantic features; Calculating an identity cross entropy loss based on the first identity prediction probability and the second identity prediction probability; The second calculation module 905 is further specifically used to perform weighted calculation according to the clothing contrast loss, the difficult triplet loss, the identity cross entropy loss, the KL divergence loss and the feature difference loss to obtain a model loss parameter.

[0079] In one example, the parsing module 903 is further specifically used to parse the sample data set to obtain a human body mask image with human body part annotations; Inputting the human body mask image into a convolutional layer for feature transformation to obtain image features; Inputting the image features into a multi-head self-attention layer to fuse shape features of semantic information from different fields of view to obtain fused features; The fused features are input into a feed-forward network composed of fully connected layers for processing to obtain body shape features.

[0080] In one example, the construction module 901 is specifically used to obtain a pre-trained original feature extraction network; Performing pixel-by-pixel attention between each convolution block of the original feature extraction network to obtain an original feature extraction network incorporating the pixel-by-pixel attention; The original feature extraction network with pixel-by-pixel attention mechanism is used as the semantic feature encoder with attention mechanism.

[0081] In one example, the construction module 901 is further specifically used to obtain a first feature output by a current convolution block of the original feature extraction network; The first feature is processed sequentially by depthwise separable convolution, instance normalization and a first activation function to obtain a second feature; The second feature is sequentially processed by deformable convolution, layer normalization and second activation function to obtain an attention weight matrix of the same dimension as the first feature; Multiplying the first feature by the attention weight matrix point by point to obtain a third feature; The third feature is used as the input feature of the next convolution block.

[0082] In one example, the device further includes the following modules: a pedestrian recognition module, configured to encode the image to be retrieved by using the semantic feature encoder of the trained pedestrian recognition model to obtain a pedestrian image feature vector of the image to be retrieved; Calculating the similarity between the pedestrian image feature vector and the image feature vectors of each image in the image data set to obtain a similarity calculation result; According to the similarity calculation result, the recognition result of the target pedestrian in the image to be retrieved is obtained.

[0083] The training device for a pedestrian recognition model provided in the above embodiment can implement the technical solution described in the above embodiment of a training method for a pedestrian recognition model. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above embodiment of a training method for a pedestrian recognition model, which will not be repeated here.

[0084] like Fig.10 As shown, the present invention also provides an electronic device 1000. The electronic device 1000 includes a processor 1001, a memory 1002 and a display 1003. Fig.10 Only some components of the electronic device 1000 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0085] In some embodiments, the memory 1002 may be an internal storage unit of the electronic device 1000, such as a hard disk or memory of the electronic device 1000. In other embodiments, the memory 1002 may also be an external storage device of the electronic device 1000, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. equipped on the electronic device 1000.

[0086] Furthermore, the memory 1002 may include both an internal storage unit of the electronic device 1000 and an external storage device. The memory 1002 is used to store application software installed in the electronic device 1000 and various data.

[0087] In some embodiments, the processor 1001 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 1002, such as a training method for a pedestrian recognition model in the present invention.

[0088] In some embodiments, the display 1003 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 1003 is used to display information on the electronic device 1000 and to display a visual user interface. The components 1001-1003 of the electronic device 1000 communicate with each other via a system bus.

[0089] In some embodiments of the present invention, when the processor 1001 executes the training program of the pedestrian recognition model in the memory 1002, the following steps may be implemented: constructing a semantic feature encoder incorporating an attention mechanism; The semantic features of the sample data set are extracted by the semantic feature encoder; the sample data set is parsed to obtain body shape features; KL divergence loss and feature difference loss are obtained according to the semantic features and the body shape features; weighted calculation is performed at least based on the KL divergence loss and the feature difference loss to obtain model loss parameters; the pedestrian recognition model to be trained is trained according to the model loss parameters until the training converges to obtain a trained pedestrian recognition model.

[0090] It should be understood that: when the processor 1001 executes the training program of the pedestrian recognition model in the memory 1002, in addition to the above functions, other functions can also be implemented. For details, please refer to the description of the corresponding method embodiment above.

[0091] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 1000 mentioned, and the electronic device 1000 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic devices may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 1000 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0092] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, a training method for a pedestrian recognition model provided by the above methods is implemented, the method comprising: constructing a semantic feature encoder incorporating an attention mechanism; The semantic features of the sample data set are extracted by the semantic feature encoder; the sample data set is parsed to obtain body shape features; KL divergence loss and feature difference loss are obtained according to the semantic features and the body shape features; weighted calculation is performed at least based on the KL divergence loss and the feature difference loss to obtain model loss parameters; the pedestrian recognition model to be trained is trained according to the model loss parameters until the training converges to obtain a trained pedestrian recognition model.

[0093] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0094] The above is a detailed introduction to the training method, device, equipment and medium of a pedestrian recognition model provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A training method for a pedestrian recognition model, characterized in that: include: Construct a semantic feature encoder that incorporates an attention mechanism; Extracting semantic features of the sample data set by the semantic feature encoder; Parsing the sample data set to obtain body shape features; According to the semantic features and the body shape features, a KL divergence loss and a feature difference loss are obtained; At least perform weighted calculation based on KL divergence loss and feature difference loss to obtain model loss parameters; The pedestrian recognition model to be trained is trained according to the model loss parameter until the training converges to obtain a trained pedestrian recognition model.

2. The method for training a pedestrian recognition model according to claim 1, characterized in that: The step of obtaining the KL divergence loss and the feature difference loss according to the semantic features and the body shape features specifically includes: Calculating the first identity prediction probability of the semantic feature by the first identity classifier of the pedestrian recognition model; Calculating the second identity prediction probability of the body shape feature by using the second identity classifier of the pedestrian recognition model; wherein the first identity classifier and the second identity classifier share a weight; A KL divergence loss and a feature difference loss are calculated according to the first identity prediction probability and the second identity prediction probability.

3. The training method of a pedestrian recognition model according to claim 2, characterized in that: Also includes: Obtaining clothing contrast loss and difficulty triplet loss according to the semantic features; Calculating an identity cross entropy loss based on the first identity prediction probability and the second identity prediction probability; The weighted calculation is performed at least based on the KL divergence loss and the feature difference loss to obtain the model loss parameter, which specifically includes: A weighted calculation is performed according to the clothing contrast loss, the difficult triplet loss, the identity cross entropy loss, the KL divergence loss and the feature difference loss to obtain a model loss parameter.

4. The method for training a pedestrian recognition model according to claim 1, characterized in that: The analyzing the sample data set to obtain body shape features specifically includes: Parsing the sample data set to obtain a human body mask image with human body parts annotated; Inputting the human body mask image into a convolutional layer for feature transformation to obtain image features; Inputting the image features into a multi-head self-attention layer to fuse shape features of semantic information from different fields of view to obtain fused features; The fused features are input into a feed-forward network composed of fully connected layers for processing to obtain body shape features.

5. The method for training a pedestrian recognition model according to claim 1, characterized in that: The construction of a semantic feature encoder incorporating an attention mechanism specifically includes: Obtain the pre-trained original feature extraction network; Performing pixel-by-pixel attention between each convolution block of the original feature extraction network to obtain an original feature extraction network incorporating the pixel-by-pixel attention; The original feature extraction network with pixel-by-pixel attention mechanism is used as the semantic feature encoder with attention mechanism.

6. The method for training a pedestrian recognition model according to claim 5, characterized in that: The processing of the pixel-by-pixel attention mechanism between each convolution block of the original feature extraction network specifically includes: Obtaining a first feature output by a current convolutional block of the original feature extraction network; The first feature is processed sequentially by depthwise separable convolution, instance normalization and a first activation function to obtain a second feature; The second feature is sequentially processed by deformable convolution, layer normalization and second activation function to obtain an attention weight matrix of the same dimension as the first feature; Multiplying the first feature by the attention weight matrix point by point to obtain a third feature; The third feature is used as the input feature of the next convolution block.

7. A method for training a pedestrian recognition model according to any one of claims 1 to 6, characterized in that: Also includes: Encoding the image to be retrieved by the semantic feature encoder of the trained pedestrian recognition model to obtain a pedestrian image feature vector of the image to be retrieved; Calculating the similarity between the pedestrian image feature vector and the image feature vectors of each image in the image data set to obtain a similarity calculation result; According to the similarity calculation result, the recognition result of the target pedestrian in the image to be retrieved is obtained.

8. A training device for a pedestrian recognition model, characterized in that: include: A construction module, an extraction module, a parsing module, a first calculation module, a second calculation module and a training module; The building module is used to build a semantic feature encoder that incorporates an attention mechanism; The extraction module is used to extract the semantic features of the sample data set through the semantic feature encoder; The parsing module is used to parse the sample data set to obtain body shape features; The first calculation module is used to obtain KL divergence loss and feature difference loss according to the semantic features and the body shape features; The second calculation module is used to perform weighted calculation based on at least KL divergence loss and feature difference loss to obtain a model loss parameter; The training module is used to train the pedestrian recognition model to be trained according to the model loss parameter until the training converges to obtain the trained pedestrian recognition model.

9. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the training method of a pedestrian recognition model described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the training method of a pedestrian recognition model described in any one of claims 1 to 7.

Images