A person re-identification method based on salient feature extraction

Abstract

The present invention discloses a person re-identification method based on salient feature extraction, comprising the following steps: S1, constructing a network model based on VIT; S2, inputting an image containing a pedestrian into a salient feature extraction module for pedestrian feature extraction, generating a binary mask containing only the pedestrian image; S3, inputting the image containing the pedestrian into the VIT module for position encoding to obtain an output sequence; S4, inputting the output sequence and the binary mask into a salient feature enhancement module for feature decoding to obtain an attention feature; S5, inputting the attention feature into a classifier for classification, and training the VIT-based network model by calculating identity loss and metric loss; S6, inputting real-time pedestrian images into the trained VIT-based network model for feature extraction, and performing person re-identification through metric sorting. The present invention extracts pedestrian salient features, assigns higher attention to the salient features, and effectively improves the accuracy and robustness of person re-identification.

Description

Technical Field

[0001] The present invention belongs to the technical field of pedestrian re-identification in computer vision, and specifically relates to a pedestrian re-identification method based on salient feature extraction. Background Art

[0002] Computer vision is the study of how machines can "see." It involves the collaboration between cameras and computers to replace the human eye in visual tasks such as identifying and measuring objects in the physical world. The ultimate goal is to replace human decision-making. Person re-identification utilizes computer vision techniques to determine whether a specific person exists in an image or video sequence, and is therefore considered a sub-problem of image retrieval. Given an image of a person captured by a surveillance device, finding images of that person across all other devices can be widely used in intelligent video surveillance, smart security, and other fields.

[0003] At present, deep learning-based methods dominate the field of pedestrian re-identification. Training a deep learning model for pedestrian re-identification can be divided into the following steps. First, a group of pedestrian images in the training set are processed and input into the deep neural network. After a large number of calculations, a set of feature vectors representing pedestrians are obtained. The vectors are then substituted into the designed loss function to calculate the loss value. Then, the model parameters are updated through gradient backpropagation and optimizer. It can be seen that it is crucial to design a network model that effectively extracts pedestrian features. Pedestrian re-identification methods can be divided into the following categories: (1) Pedestrian re-identification methods based on global images, which extract pedestrian features from the image through layers of convolutional neural networks. The method achieved good results at the beginning, but due to the characteristics of convolutional neural networks, its resolution will gradually decrease during the downsampling process, which makes the network easily ignore local discriminative features. In addition, this method is easily disturbed by occlusion, background, and human posture during feature extraction, thus affecting the discrimination rate; (2) Pedestrian re-identification methods based on image segmentation or posture estimation models. Both methods can extract pedestrian information from background information and then hand it over to the subsequent pedestrian re-identification model. However, this method is highly dependent on the pedestrian information extracted in the first stage, and there is a risk that the discriminative fine-grained features on the pedestrians will be regarded as interference and classified as background information, resulting in the loss of key pedestrian information. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to propose a pedestrian re-identification method based on salient feature extraction in response to the above-mentioned deficiencies in the prior art.

[0005] The present invention can be achieved by adopting the following technical solutions:

[0006] A pedestrian re-identification method based on salient feature extraction, the pedestrian re-identification method comprising the following steps:

[0007] S1. Constructing a VIT-based network model, wherein the VIT-based network model includes a salient feature extraction module, a VIT module, a salient feature enhancement module, and a classifier;

[0008] S2. Input the image containing pedestrians into the salient feature extraction module to extract pedestrian features and generate a binary mask containing only the pedestrian image;

[0009] S3, input the image containing pedestrians into the VIT module for position encoding to obtain an output sequence;

[0010] S4. Input the above output sequence and the binary mask into the salient feature enhancement module for feature decoding to obtain attention features;

[0011] S5. Input the attention features into the classifier for classification, and train the VIT-based network model by calculating the identity loss and the triplet metric loss and repeating steps S2 to S5.

[0012] S6. Input the real-time collected pedestrian images into the trained VIT-based network model to extract features, and realize pedestrian re-identification through measurement sorting.

[0013] Furthermore, the working process of the salient feature extraction module in step S2 is as follows:

[0014] The input image containing pedestrians is divided into a non-binary saliency probability map containing only pedestrian areas and background areas. The non-binary saliency probability map consists of N different pixels. The pixel value α of the i-th pixel is i Normalized to the interval [0, 1], i = 1, 2…, N, and then the pixel α of the i-th pixel in the non-binary saliency probability map is calculated according to the following equation i By comparing the threshold d α Converted into a binary mask consisting of only 0 and 1 values, the formula is as follows:

[0015]

[0016] The use of the salient feature extraction module can effectively extract the pedestrian area in the image containing pedestrians in the form of a binary mask, thereby facilitating the effective enhancement of the features of the pedestrian area in a targeted manner.

[0017] Furthermore, the working process of the VIT module in step S3 is as follows:

[0018] Define an input image I∈R containing pedestrians H×W×C , where H represents the height of the input image, W represents the width of the input image, and C represents the number of channels of the input image. A convolution kernel with a length and width of P and a stride of S is used to linearly map the image I to obtain a sequence of N dimensions D D is determined by the number of convolution kernels, i = 1, 2…, N, where:

[0019]

[0020] Then generate a sequence The length of the learnable CLS tag is equal to the sequence Combining, and integrating spatial information into the combination through learnable position encoding, the output sequence z0 is expressed as follows:

[0021]

[0022] Where x cls ∈R 1×D Represents the cls tag of image I, E pos ∈R (N+1)×D Represented as positional encoding.

[0023] The VIT module can be used to divide the image into 8 regions, encode the position of each region and generate classification labels, providing a classification basis for the salient feature enhancement module.

[0024] Furthermore, the working process of the salient feature enhancement module in step S4 is as follows:

[0025] The output sequence z0 is subjected to multi-head self-attention operation and multi-layer perception operation. The formula for calculating the output sequence is as follows:

[0026] z′ l =MSA(LN(z l-1 ))+z l-1

[0027] z l =MLP(LN(z′ l ))+z′ l

[0028] In the formula, LN(·) represents the inter-layer normalization operation, MSA(·) represents the multi-head attention operation, MLP(·) represents the multi-layer perception operation, and z l-1 represents the output sequence of the l-1 layer, z l represents the output sequence of layer l, z′ l represents z l-1 With z l The output sequence of the intermediate layers between them;

[0029] The multi-head self-attention operation is composed of multiple self-attention operations. The calculation formula for a single self-attention is as follows:

[0030]

[0031] where x scor Represents the attention score matrix, Q, K, V are z l-1 The query value, the value to be matched, and the input feature value contained in ;

[0032] definition is a row in the attention score matrix belonging to the class label, and in each self-attention head, find the maximum attention score x of the class vector for each small patch max , according to the maximum value of the attention score x max To modify the attention score of the class label, the calculation formula is as follows:

[0033]

[0034] where d θ is the coefficient that controls the maximum value to be added. yes The i-th row in m i is the i-th value in the binary mask flattened into a 1-dimensional vector, i = 1, 2…, N.

[0035] The salient feature enhancement module mainly assigns a stronger attention score to the pedestrian area image in the image based on the binary mask of the pedestrian image extracted by the salient feature extraction module, thereby making the network model pay more attention to the pedestrian image area rather than the background area.

[0036] Furthermore, the working process of the classifier in step S5 is as follows:

[0037] The anchor sample input by the VIT-based network model is recorded as I i , the positive sample is I j , the negative sample is I k , respectively record the labels of the three samples as y i 、y j 、y k ,f(I) is the output feature of the classifier, and the triple metric loss L tri (θ) and identity loss L entropy (θ) are respectively expressed as:

[0038]

[0039]

[0040] Where θ is the model parameter of the VIT-based network model, r is the target difference between the distance between the negative sample pairs and the distance between the positive sample pairs when the network model converges, and p i is the probability value of the input image after classification by the network model;

[0041] The target optimization function is expressed as:

[0042] L total (θ)=L tri (θ)+L entropy (θ),

[0043] By minimizing the objective function L total (θ), iteratively optimize the VIT-based network model until convergence.

[0044] Furthermore, the process of step S6 is as follows:

[0045] The real-time collected pedestrian images are input into the VIT-based network model to obtain pedestrian classification features, the Euclidean distance between the features is calculated to construct a similarity matrix, and the retrieval candidate column results are obtained by sorting according to the size of the Euclidean distance.

[0046] The present invention has the following advantages and effects compared to the prior art:

[0047] (1) This paper proposes a VIT-based network model for pedestrian re-identification, which effectively improves the recognition accuracy of the pedestrian re-identification method.

[0048] (2) The VIT-based network model proposed in the present invention can extract the image features of pedestrians to generate a binary mask image, and pay higher attention to the pedestrian area in the entire image based on the binary mask image. While avoiding focusing attention on the background area, it also does not ignore the influence of detailed features in the pedestrian area, effectively improving the performance of pedestrian re-identification, and can be widely used in the field of image recognition technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0050] Figure 1 This is a flowchart of a pedestrian re-identification method based on salient feature extraction disclosed in the present invention;

[0051] Figure 2 It is a structural diagram of the VIT-based network model disclosed in the present invention. DETAILED DESCRIPTION

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0053] Example

[0054] like Figure 1 As shown, this embodiment discloses a pedestrian re-identification method based on salient feature extraction, comprising the following steps:

[0055] S1. Constructing a VIT-based network model, wherein the VIT-based network model includes a salient feature extraction module, a VIT module, a salient feature enhancement module, and a classifier;

[0056] S2. Input the image containing pedestrians into the salient feature extraction module to extract pedestrian features and generate a binary mask containing only the pedestrian image;

[0057] The working process of the salient feature extraction module in step S2 of this embodiment is as follows:

[0058] The input image containing pedestrians is divided into a non-binary saliency probability map containing only pedestrian areas and background areas. The non-binary saliency probability map consists of N different pixels. The pixel value α of the i-th pixel is i Normalized to the interval [0, 1], i = 1, 2…, N, and then the pixel α of the i-th pixel in the non-binary saliency probability map is calculated according to the following equation i By comparing the threshold d α Converted into a binary mask consisting of only 0 and 1 values, the formula is as follows:

[0059]

[0060] In this example, we use the U pre-trained on the medium-scale training set 2 net, and threshold the pixel intensity d α It is set to 0.8. In addition, we also consider the case where no mask is found or the mask is damaged, in which case we adjust the intensity threshold to 0.2.

[0061] The use of the salient feature extraction module can effectively extract the pedestrian area in the image containing pedestrians in the form of a binary mask, thereby facilitating the effective enhancement of the features of the pedestrian area in a targeted manner.

[0062] S3, input the image containing pedestrians into the VIT module for position encoding to obtain an output sequence;

[0063] The working process of the VIT module in step S3 of this embodiment is as follows:

[0064] Define an input image I∈R containing pedestrians H×W×C , where H represents the height of the input image, W represents the width of the input image, and C represents the number of channels of the input image. A convolution kernel with a length and width of P and a stride of S is used to linearly map the image I to obtain a sequence of N dimensions D D is determined by the number of convolution kernels, i = 1, 2…, N, where:

[0065]

[0066] Then generate a sequence The length of the learnable CLS tag is equal to the sequence Combining, and integrating spatial information into the combination through learnable position encoding, the output sequence z0 is expressed as follows:

[0067]

[0068] Where x cls ∈R 1×D Represents the cls tag of image I, E pos ∈R (N+1)×D Represented as positional encoding.

[0069] The VIT module can be used to divide the image into 8 regions, encode the position of each region and generate classification labels, providing a classification basis for the salient feature enhancement module.

[0070] S4. Input the above output sequence and the binary mask into the salient feature enhancement module for feature decoding to obtain attention features;

[0071] The working process of the salient feature enhancement module in step S4 of this embodiment is as follows:

[0072] The output sequence z0 is subjected to multi-head self-attention operation and multi-layer perception operation. The formula for calculating the output sequence is as follows:

[0073] z′ l =MSA(LN(z l-1 ))+z l-1

[0074] z l =MLP(LN(z′ l ))+z′ l

[0075] In the formula, LN(·) represents the inter-layer normalization operation, MSA(·) represents the multi-head attention operation, MLP(·) represents the multi-layer perception operation, and z l-1 represents the output sequence of the l-1 layer, z l represents the output sequence of layer l, z′ l represents z l-1 With z l The output sequence of the intermediate layers between them;

[0076] The multi-head self-attention operation is composed of multiple self-attention operations. The calculation formula for a single self-attention is as follows:

[0077]

[0078] where x scor Represents the attention score matrix, Q, K, V are z l-1 The query value, the value to be matched, and the input feature value contained in ;

[0079] definition is a row in the attention score matrix belonging to the class label, and in each self-attention head, find the maximum attention score x of the class vector for each small patch max , according to the maximum value of the attention score x max To modify the attention score of the class label, the calculation formula is as follows:

[0080]

[0081] where d θ is the coefficient that controls the maximum value to be added. yes The i-th row in m i is the i-th value in the binary mask flattened into a 1-dimensional vector, i = 1, 2…, N. In this example, d θ Set to 0.3.

[0082] The salient feature enhancement module mainly assigns a stronger attention score to the pedestrian area image in the image based on the binary mask of the pedestrian image extracted by the salient feature extraction module, thereby making the network model pay more attention to the pedestrian image area rather than the background area.

[0083] S5. Input the attention features into the classifier for classification, and train the VIT-based network model by calculating the identity loss and the triplet metric loss and repeating steps S2 to S5.

[0084] The working process of the classifier in step S5 of this embodiment is as follows:

[0085] The anchor sample input by the VIT-based network model is recorded as Ii , the positive sample is I j , the negative sample is I k , respectively record the labels of the three samples as y i 、y j 、y k ,f(I) is the output feature of the classifier, and the triple metric loss L tri (θ) and identity loss l entropy (θ) are respectively expressed as:

[0086]

[0087]

[0088] Where θ is the model parameter of the VIT-based network model, r is the target difference between the distance between the negative sample pairs and the distance between the positive sample pairs when the network model converges, and p i is the probability value of the input image after classification by the network model;

[0089] The target optimization function is expressed as:

[0090] L total (θ)=L tri (θ)+L entropy (θ),

[0091] By minimizing the objective function L total (θ), iteratively optimize the VIT-based network model until convergence.

[0092] S6. Input the real-time collected pedestrian images into the trained VIT-based network model to extract features, and realize pedestrian re-identification through measurement sorting.

[0093] Step S6 of this embodiment specifically includes:

[0094] The real-time collected pedestrian images are input into the VIT-based network model to obtain pedestrian classification features, the Euclidean distance between the features is calculated to construct a similarity matrix, and the retrieval candidate column results are obtained by sorting according to the size of the Euclidean distance.

[0095] In summary, this embodiment discloses a person re-identification method using salient feature extraction. This method segments the pedestrian image region in an input image and generates a binary mask image. To prevent the network from focusing on the background area in the image, this method uses the binary mask image to guide the network's attention to the pedestrian image region, while also not neglecting other regions with discriminative features, thereby enhancing the network's ability to characterize pedestrian features. This method uses images from the Market1501 dataset, a standard dataset for person re-identification, as input images. After classification and ranking by a VIT-based network model, it achieves a mean average precision (mAP) of 88.6% and a Rank-1 accuracy of 95.2%. mAP represents mean average precision, and Rank-1 represents the average accuracy of the top retrieved images. As shown in Table 1, compared to a method containing only the VIT module, the pedestrian recognition method in this embodiment improves mAP by 2.1% and Rank-1 by 0.8ms.

[0096] Table 1. Statistics of indicators of different pedestrian recognition methods

[0097] Methods mAP (%) Rank-1 (ms) VIT 86.5 94.4 Our 88.6 95.2

[0098] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A pedestrian re-identification method based on salient feature extraction, characterized in that: The pedestrian re-identification method comprises the following steps: S1. Constructing a VIT-based network model, wherein the VIT-based network model includes a salient feature extraction module, a VIT module, a salient feature enhancement module, and a classifier; S2. Input the image containing pedestrians into the salient feature extraction module to extract pedestrian features and generate a binary mask containing only the pedestrian image; S3, input the image containing pedestrians into the VIT module for position encoding to obtain an output sequence; S4. Input the above output sequence and the binary mask into the salient feature enhancement module for feature decoding to obtain attention features; S5. Input the attention features into the classifier for classification, and train the VIT-based network model by calculating the identity loss and the triplet metric loss and repeating steps S2 to S5. S6. Input the real-time collected pedestrian images into the trained VIT-based network model to extract features, and realize pedestrian re-identification through measurement sorting.

2. The pedestrian re-identification method based on salient feature extraction according to claim 1, characterized in that: The working process of the salient feature extraction module in step S2 is as follows: The input image containing pedestrians is divided into a non-binary saliency probability map containing only pedestrian areas and background areas. The non-binary saliency probability map consists of N different pixels. The pixel value α of the i-th pixel is i Normalized to the interval [0, 1], i = 1, 2…, N, and then the pixel α of the i-th pixel in the non-binary saliency probability map is calculated according to the following equation i By comparing the threshold d α Converted into a binary mask consisting of only 0 and 1 values, the formula is as follows:

3. The pedestrian re-identification method based on salient feature extraction according to claim 2, characterized in that: The working process of the VIT module in step S3 is as follows: Define an input image I∈R containing pedestrians H×W×C , where H represents the height of the input image, W represents the width of the input image, and C represents the number of channels of the input image. A convolution kernel with a length and width of P and a stride of S is used to linearly map the image I to obtain a sequence of N dimensions D D is determined by the number of convolution kernels, i = 1, 2, ..., N, where: Then generate a sequence The length of the learnable CLS tag is equal to the sequence Combining, and integrating spatial information into the combination through learnable position encoding, the output sequence z0 is expressed as follows: Where x cls ∈R 1×D Represents the cls tag of image I, E pos ∈R (N+1)×D Represented as positional encoding.

4. The pedestrian re-identification method based on salient feature extraction according to claim 3 is characterized in that: The working process of the salient feature enhancement module in step S4 is as follows: The output sequence z0 is subjected to multi-head self-attention operation and multi-layer perception operation. The formula for calculating the output sequence is as follows: z′ l =MSA(LN(z l-1 ))+z l-1 With l =MLP(LN(z′ l ))+z′ l In the formula, LN(·) represents the inter-layer normalization operation, MSA(·) represents the multi-head attention operation, MLP(·) represents the multi-layer perception operation, and z l-1 represents the output sequence of the l-1 layer, z l represents the output sequence of layer l, z′ l represents z l-1 With z l The output sequence of the intermediate layers between them; The multi-head self-attention operation is composed of multiple self-attention operations. The calculation formula for a single self-attention is as follows: where x scor Represents the attention score matrix, Q, K, V are z l-1 The query value, the value to be matched, and the input feature value contained in ; definition is a row in the attention score matrix belonging to the class label, and in each self-attention head, find the maximum attention score x of the class vector for each small patch max , according to the maximum value of the attention score x max To modify the attention score of the class label, the calculation formula is as follows: where d θ is the coefficient that controls the maximum value to be added. yes The i-th row in m i is the i-th value in the binary mask flattened into a 1-dimensional vector, i = 1, 2, ..., N.

5. The pedestrian re-identification method based on salient feature extraction according to claim 4, characterized in that: The working process of the classifier in step S5 is as follows: The anchor sample input by the VIT-based network model is recorded as I i , the positive sample is I j , the negative sample is I k , respectively record the labels of the three samples as y i 、y j 、y k , f(I) is the output feature of the classifier, and the triple metric loss L tri (θ) and identity loss L entropy (θ) are respectively expressed as: Where θ is the model parameter of the VIT-based network model, r is the target difference between the distance between the negative sample pairs and the distance between the positive sample pairs when the network model converges, and p i is the probability value of the input image after classification by the network model; The target optimization function is expressed as: L total (θ)=L tri (θ)+L entropy (i), By minimizing the objective function L total (θ), iteratively optimize the VIT-based network model until convergence.

6. The pedestrian re-identification method based on salient feature extraction according to claim 5, characterized in that: The process of step S6 is as follows: The real-time collected pedestrian images are input into the VIT-based network model to obtain pedestrian classification features, the Euclidean distance between the features is calculated to construct a similarity matrix, and the retrieval candidate column results are obtained by sorting according to the size of the Euclidean distance.

Images