A hybrid attention-based pedestrian re-identification method

Abstract

The present disclosure relates to a hybrid attention-based pedestrian re-identification method. The method comprises: preprocessing a picture dataset by a multi-scale data enhancement algorithm to obtain a preprocessed picture dataset; wherein the picture dataset is a dataset obtained for pedestrian re-identification; constructing a hybrid attention network model, adjusting the constructed hybrid attention network model, and obtaining a pedestrian re-identification model; inputting pedestrian pictures in the preprocessed picture dataset into the pedestrian re-identification model for training, and obtaining a trained pedestrian re-identification model; inputting a to-be-identified picture and a to-be-identified video into the trained pedestrian re-identification model, and outputting an identification result. The pedestrian re-identification model obtained by the method can improve the efficiency and accuracy of pedestrian re-identification.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to a pedestrian re-identification method based on hybrid attention. Background Technology

[0002] With the improvement of computer hardware and software performance, computer vision technology has embarked on a fast track of development. However, many problems still exist in the field of pedestrian re-identification. Typically, video surveillance equipment in public areas is installed at a high position, resulting in a relatively large distance between the camera and pedestrians, making it difficult to obtain clear facial features. Therefore, pedestrian re-identification technology has become a popular research direction, especially in public safety issues such as criminal investigation and missing children, where there is an urgent need to solve the problem of pedestrian search in public areas.

[0003] In related technologies, traditional pedestrian re-identification methods rely on manual feature design, which cannot adapt to complex environments with large amounts of data. In recent years, deep learning, represented by convolutional neural networks, has achieved great success in the field of pedestrian re-identification. However, the extracted visual features are prone to getting trapped in local optima and ignoring global relationships, resulting in low recognition accuracy.

[0004] Therefore, it is necessary to provide a new technical solution to improve one or more of the problems existing in the above solutions.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this disclosure is to provide a hybrid attention-based person re-identification method, thereby overcoming, at least to some extent, one or more problems caused by the limitations and defects of related technologies.

[0007] A pedestrian re-identification method based on hybrid attention provided in an embodiment of this disclosure includes:

[0008] The image dataset is preprocessed using a multi-scale data augmentation algorithm to obtain the preprocessed image dataset; wherein, the image dataset is the dataset obtained for pedestrian re-identification;

[0009] A hybrid attention network model is constructed, and the constructed hybrid attention network model is adjusted to obtain a pedestrian re-identification model;

[0010] The pedestrian images in the preprocessed image dataset are input into the pedestrian re-identification model for training, and the trained pedestrian re-identification model is obtained.

[0011] Input the image and video to be identified into the trained pedestrian re-identification model, and output the recognition result.

[0012] In embodiments of this disclosure, the step of constructing a hybrid attention network model and adjusting the constructed hybrid attention network model includes:

[0013] A residual network is selected as the basic network structure, and a hybrid attention module is added to the last three bottleneck blocks of the residual network to obtain the adjusted hybrid attention network model.

[0014] In embodiments of this disclosure, after the steps of constructing a hybrid attention network model and adjusting the constructed hybrid attention network model, the method further includes:

[0015] The adjusted hybrid attention network model is then subjected to auxiliary information encoding to obtain the pedestrian re-identification model.

[0016] In the embodiments of this disclosure, the auxiliary information encoding is constructed from the domain information and visual information in the image dataset.

[0017] In embodiments of this disclosure, the step of inputting the preprocessed pedestrian images from the image dataset into the pedestrian re-identification model for training to obtain the trained pedestrian re-identification model includes:

[0018] The parameters of the pedestrian re-identification model are adjusted using a triplet loss function for hard samples; wherein the triplet includes a positive sample, a negative sample, and the original sample.

[0019] In embodiments of this disclosure, the loss function is as follows:

[0020]

[0021] A represents the feature vector of the anchor point, P represents the feature vector of the positive sample, N represents the feature vector of the negative sample, and d is the Euclidean distance. Therefore, d(A,P) represents the Euclidean distance between the positive sample and the anchor point, and d(A,N) represents the Euclidean distance between the negative sample and the anchor point. max refers to the maximum margin between the two distances, and min refers to the minimum margin between the two distances. The plus sign in the lower right corner of the parentheses indicates that when the value of the expression is greater than or equal to zero, the loss value is the value of the current expression, and when the value of the expression is less than zero, the loss value is zero.

[0022] In embodiments of this disclosure, the step of inputting the preprocessed pedestrian images from the image dataset into the encoded hybrid attention network model for training to obtain a trained pedestrian re-identification model includes:

[0023] The convergence of the hybrid attention network model with encoding is determined based on the value and curve of the loss function.

[0024] In embodiments of this disclosure, the step of inputting the preprocessed pedestrian images from the image dataset into the encoded hybrid attention network model for training to obtain a trained pedestrian re-identification model includes:

[0025] When the loss function does not change significantly during a preset number of training iterations, the hybrid attention network model with encoding is considered to have converged, and training is stopped, resulting in the trained pedestrian re-identification model.

[0026] In embodiments of this disclosure, after the step of obtaining the trained person re-identification model, the method further includes:

[0027] When the encoded hybrid attention network model converges, the performance of the converged hybrid attention model is judged based on the first matching rate and the average precision.

[0028] In embodiments of this disclosure, the step of inputting the image to be identified and the video to be identified into the trained pedestrian re-identification model and outputting the recognition result includes:

[0029] When the video to be identified is input into the trained pedestrian re-identification model, the image of each frame of the video is extracted, and the similarity between the pedestrian and the pedestrians in the database is determined by calculating the cosine similarity of each frame of the image and sorted, and the recognition result is output.

[0030] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0031] In one embodiment of this disclosure, the above method is used to preprocess the image dataset using a multi-scale data augmentation algorithm to obtain a preprocessed image dataset; simultaneously, a hybrid attention network model is constructed and adjusted to obtain a pedestrian re-identification model; the pedestrian re-identification model is trained using the preprocessed image dataset to obtain a trained pedestrian re-identification model, which facilitates the subsequent input of the identification image and the video to be identified into the trained pedestrian re-identification model, thereby improving the efficiency and accuracy of pedestrian re-identification.

[0032] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0034] Figure 1 This schematically illustrates a flowchart of the steps of a hybrid attention-based person re-identification method in an exemplary embodiment of this disclosure;

[0035] Figure 2 This schematically illustrates a pedestrian re-identification flowchart in an exemplary embodiment of the present disclosure;

[0036] Figure 3 This schematic diagram illustrates the structure of a hybrid attention module in an exemplary embodiment of this disclosure.

[0037] Figure 4 This schematic diagram illustrates the structure of a hybrid attention network model in an exemplary embodiment of this disclosure.

[0038] Figure 5 This schematically illustrates a flowchart of the training process for a hybrid attention network model in an exemplary embodiment of this disclosure.

[0039] Figure 6 This schematically illustrates a pedestrian re-identification flowchart for video data in an exemplary embodiment of the present disclosure. Detailed Implementation

[0040] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0041] This example implementation first provides a pedestrian re-identification method based on hybrid attention. (Reference) Figure 1 As shown, the method may include:

[0042] Step S101: Preprocess the image dataset using a multi-scale data augmentation algorithm to obtain the preprocessed image dataset; wherein, the image dataset is the dataset obtained for pedestrian re-identification.

[0043] Step S102: Construct a hybrid attention network model and adjust the constructed hybrid attention network model to obtain a pedestrian re-identification model.

[0044] Step S103: Input the pedestrian images from the preprocessed image dataset into the pedestrian re-identification model for training to obtain the trained pedestrian re-identification model.

[0045] Step S104: Input the image and video to be identified into the trained pedestrian re-identification model and output the recognition result.

[0046] Using the above method, a multi-scale data augmentation algorithm is used to preprocess the image dataset to obtain a preprocessed image dataset. At the same time, a hybrid attention network model is constructed and adjusted to obtain a person re-identification model. The preprocessed image dataset is then used to train the person re-identification model to obtain a trained person re-identification model. This facilitates the subsequent input of recognition images and videos to be identified into the trained person re-identification model, thereby improving the efficiency and accuracy of person re-identification.

[0047] Below, we will refer to Figures 1 to 6 The steps of the method described above in this example embodiment will be explained in more detail.

[0048] In step S101, the image dataset is preprocessed using a multi-scale data augmentation algorithm to obtain the preprocessed image dataset; wherein, the image dataset is the dataset obtained for person re-identification. Specifically, the image datasets Market-1501, CUHK03-np, and DukeMTMC-reID for person re-identification are obtained. The images are categorized by ID number according to the naming rules of each dataset, and augmentation enhancements such as rotation and flipping are performed. Color enhancement is performed using the MSR algorithm, and the resolution of the enhanced images is uniformly adjusted to 286×128.

[0049] The core idea of ​​the Retinex algorithm is to minimize the impact of incident components on pedestrian features in pedestrian images and to analyze the color attributes contained in the image, thereby achieving the goal of image augmentation for pedestrian datasets. The computation process of the multi-scale Retinex (MSR) data augmentation algorithm is as follows:

[0050]

[0051] Where R(a,b) represents the reflectivity of the object, an intrinsic attribute of the image, I(a,b) is the original image data, n represents the number of multi-scales in the MSR algorithm, and after multiple verifications, n=3 achieves the best results, ω i G represents the weighting coefficient. i (a, b) is usually taken as a Gaussian low-pass filter, and its calculation formula is:

[0052]

[0053] In the formula, λ i is the gain constant, and batch represents the total number of input images in one round. The MSR algorithm processes each RGB channel separately. Processing each channel individually can easily introduce new noise, leading to local image distortion and negatively impacting the overall quality of the pedestrian image, resulting in poor visual quality. Therefore, the MSR algorithm can improve this problem.

[0054] It's worth mentioning that Market-1501, CUHK03-np, and DukeMTMC-reID are currently the most commonly used person re-identification datasets.

[0055] In step S102, a hybrid attention network model is constructed and adjusted to obtain a person re-identification model. Specifically, as follows... Figure 3 As shown, a hybrid attention network model is constructed, which mainly consists of a parallel self-attention mechanism and a depthwise separable convolution. The constructed hybrid attention network model is then adjusted to obtain the adjusted hybrid attention network model, which is the pedestrian re-identification model, for subsequent use.

[0056] It's important to note that attention mechanisms sift through a large amount of information, focusing on a small amount of important data while ignoring most of the less important information. Self-attention mechanisms, a variation of attention, reduce reliance on external information, making them better at capturing correlations within data or features to solve long-distance dependency problems. Self-attention mechanisms share weights across channels, which reduces computational complexity and parameters, but its drawback is poor modeling ability for channel-level information.

[0057] Depthwise convolution uses one kernel to assist one channel, unlike regular convolution, where computation is performed independently on each channel. Its computation is entirely within a 2D feature map. The number of kernels is the same as the previous layer, and the number of output feature maps is the same as the number of channels in the input layer. Depthwise convolution shares connection weights spatially, which reduces parameter complexity and increases network capability.

[0058] It should also be noted that, based on the idea that self-attention and depthwise convolution should be complementary and can provide each other with additional information during the learning process, a parallel design is proposed to simultaneously model spatial and dimensional information to enhance the model's ability to extract robust features.

[0059] In step S103, the pedestrian images from the preprocessed image dataset are input into the pedestrian re-identification model for training, resulting in a trained pedestrian re-identification model. Specifically, the pedestrian re-identification model is trained using the preprocessed image dataset of pedestrian images to obtain a trained pedestrian re-identification model. This allows the trained pedestrian re-identification model to be used subsequently to identify the images to be identified.

[0060] In step S104, the image and video to be identified are input into the trained pedestrian re-identification model, and the identification result is output. Specifically, after the pedestrian re-identification model is trained, the image and video to be identified are input into the trained pedestrian re-identification model to obtain the identification structure, thereby improving the efficiency and accuracy of pedestrian re-identification.

[0061] Optionally, in some embodiments, the step of constructing a hybrid attention network model and adjusting the constructed hybrid attention network model includes:

[0062] A residual network was selected as the basic network structure. A hybrid attention module was added to the last three bottleneck blocks of the residual network to obtain the adjusted hybrid attention network model. Specifically, the hybrid attention network model was adjusted by selecting a residual network (ResNet50) as the basic network structure. Further, the ResNet50 network used was a residual network pre-trained on ImageNet. The further adjusted network structure included inserting the hybrid attention module into the bottleneck block of the 5th layer of the ResNet network. The input was the output from the first four layers of the ResNet50 network. The output after hybrid attention processing was an N-dimensional feature concatenation from depthwise convolution and self-attention, where N is a hyperparameter.

[0063] In one embodiment, the ResNet50 residual network pre-trained on ImageNet is modified by inserting a hybrid attention layer into its fifth layer. The output of the self-attention mechanism is concatenated with the output of the depthwise convolution to obtain N-dimensional features. After dimensionality upscaling via a 1×1 convolution, the final image features are obtained through a linear layer, a global average normalization layer, and a ReLU activation function layer. The network structure is as follows. Figure 3 As shown.

[0064] Figure 3 Each layer performs the following functions: The input is a pedestrian image. Each of the first to fifth convolutional layers downsamples the input image by a factor of 2. The input image features are (64*128*3), the fourth layer outputs the image features (4*8*1024), and the fifth layer outputs the image features (2*4*2048). Feature concatenation: The features of the fourth layer output image and the fifth layer output image are concatenated to form a 3072-dimensional feature.

[0065] Optionally, in some embodiments, the step of constructing a hybrid attention network model and adjusting the constructed hybrid attention network model further includes:

[0066] The adjusted hybrid attention network model is then augmented with auxiliary information encoding to obtain the pedestrian re-identification model. Specifically, the auxiliary information encoding includes camera encoding and viewpoint encoding. By embedding these auxiliary information encodings, the model is optimized to learn invariant non-visual features.

[0067] In one embodiment, the auxiliary information encoding is constructed from domain information and visual information in the image dataset. Domain information includes, for example, different camera labels, and viewpoint information includes, for example, different camera viewpoint labels.

[0068] Optionally, in some embodiments, the step of inputting the preprocessed pedestrian images from the image dataset into the pedestrian re-identification model for training to obtain the trained pedestrian re-identification model includes:

[0069] The parameters of the pedestrian re-identification model are adjusted using a triplet loss function for hard samples; wherein the triplet includes a positive sample, a negative sample, and the original sample.

[0070] Optionally, in some embodiments, the loss function is as follows:

[0071]

[0072] A represents the feature vector of the anchor point, P represents the feature vector of the positive sample, N represents the feature vector of the negative sample, and d is the Euclidean distance. Therefore, d(A,P) represents the Euclidean distance between the positive sample and the anchor point, and d(A,N) represents the Euclidean distance between the negative sample and the anchor point. max refers to the maximum margin between the two distances, and min refers to the minimum margin between the two distances. The plus sign in the lower right corner of the parentheses indicates that when the value of the expression is greater than or equal to zero, the loss value is the value of the current expression, and when the value of the expression is less than zero, the loss value is zero.

[0073] Specifically, the hard sample mining triplet loss refers to the following: For each batch, multiple pedestrians are selected, and for each pedestrian, multiple images are randomly selected. For each image, a triplet is formed by selecting the feature vector A of the target sample, the feature vector P of the positive sample least similar to the target sample, and the feature vector N of the negative sample most similar to the target sample. Here, the target sample is also called the original sample or anchor point.

[0074] Optionally, in some embodiments, the step of inputting the preprocessed pedestrian images from the image dataset into the pedestrian re-identification model for training to obtain a trained pedestrian re-identification model includes:

[0075] The convergence of the encoded hybrid attention network model is determined based on the value and curve of the loss function. Specifically, the enhanced pedestrian image is input into the newly constructed pedestrian re-identification model for training. The parameters of the pedestrian re-identification model are optimized by minimizing the triplet loss function. The number of nodes to be randomly lost in each training iteration is set, the number of training iterations for all samples in the training set is set, the learning rate is set, and an optimizer is selected. The convergence of the loss function is determined based on the loss curve. When the loss function value does not change significantly after a certain number of training iterations (e.g., 5, 10, etc.), the encoded hybrid attention network model is considered to have converged, training is stopped, and the trained pedestrian re-identification model is obtained. The training process is as follows: Figure 5 As shown.

[0076] Optionally, in some embodiments, the step of inputting the preprocessed pedestrian images from the image dataset into the pedestrian re-identification model for training to obtain a trained pedestrian re-identification model includes:

[0077] In one embodiment, when the loss function does not change significantly during a preset number of training iterations, the hybrid attention network model is considered to have converged, and training is stopped to obtain the trained pedestrian re-identification model.

[0078] Specifically, if the loss function value is close to stable during the preset number of training iterations, that is, when the change is not significant, it means that the hybrid attention network model is converging. Furthermore, when the hybrid attention network model is converging, training of the hybrid attention network model is stopped, meaning that the trained pedestrian re-identification model has been obtained.

[0079] Optionally, in some embodiments, after obtaining the trained person re-identification model, the method further includes:

[0080] When the encoded hybrid attention network model converges, the performance of the converged hybrid attention model is judged based on the first matching rate and the average precision. Specifically, when the hybrid attention model converges, the performance of the converged hybrid attention model can be judged based on the first matching rate and the average precision, that is, the performance of the trained person re-identification model can be judged based on the first matching rate and the average precision. The calculation of the first matching rate and the average precision can be referred to existing technologies, and will not be elaborated further in this embodiment.

[0081] Optionally, in some embodiments, the step of inputting the image and video to be identified into the trained pedestrian re-identification model and outputting the recognition result includes:

[0082] When the video to be identified is input into the trained pedestrian re-identification model, the image of each frame of the video is extracted, and the similarity between the pedestrian and the pedestrians in the database is determined by calculating the cosine similarity of each frame of the image and sorted, and the recognition result is output.

[0083] Specifically, the video sequence includes candidate images of pedestrians to be queried. When extracting pedestrians, the time and location of their appearance are recorded in the database. The cosine similarity can be calculated using existing techniques, which will not be elaborated upon in this embodiment.

[0084] It should be noted that the judgment criterion uses cosine similarity. The cosine distance between each pedestrian in the candidate image extracted from the video and the pedestrian image is calculated. Pedestrians in the images are sorted by distance, which represents the degree of similarity between the two images. The higher the similarity, the higher the probability that the two images belong to the same person. Finally, the time information and specific location of each pedestrian appearing in the video are output sequentially, completing the pedestrian re-identification.

[0085] It should also be noted that the pedestrian re-identification system based on video information is constructed as shown in the attached figure. Figure 6 As shown, it can be summarized into the following steps:

[0086] (1) Data acquisition: Obtaining raw video from surveillance cameras.

[0087] (2) Pedestrian bounding box generation: Pedestrians are cropped from the collected data by manual annotation or auxiliary algorithms (such as pedestrian detection and target tracking), requiring that pedestrians occupy most of the space in the cropped image.

[0088] (3) Training data annotation: add camera number, viewpoint label, pedestrian label, time of the current video frame and other information to the image name.

[0089] (4) Model training: The labeled data is fed into the designed deep learning model so that the model can learn from the data how to identify the hidden expression patterns of different pedestrians.

[0090] (5) Pedestrian retrieval: Apply the trained model to the test scenario to verify its actual effect.

[0091] The following experiment will further illustrate this disclosure.

[0092] Based on the steps of the pedestrian re-identification method described above, the recognition performance was verified on three image datasets: Market-1501, CUHK03-np, and DukeMTMC-reID. Performance test results show that the method of this invention achieves Rank-1 accuracy of 93.9%, 80.0%, and 87.7% on the three image datasets, and mAP of 91.6%, 79.3%, and 75.6%, respectively. Rank-1 represents the accuracy of the first image in the recognition results, also known as the first matching rate, and mAP is the mean precision, which is the sum and average of the average precisions across multi-classification tasks. The recognition accuracy and first matching rate of this invention are both excellent.

[0093] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A pedestrian re-identification method based on hybrid attention, characterized in that, The method includes: The image dataset is preprocessed using a multi-scale data augmentation algorithm to obtain the preprocessed image dataset; wherein, the image dataset is the dataset obtained for pedestrian re-identification; A hybrid attention network model is constructed, and the constructed hybrid attention network model is adjusted to obtain a pedestrian re-identification model; The pedestrian images in the preprocessed image dataset are input into the pedestrian re-identification model for training, and the trained pedestrian re-identification model is obtained. Input the image and video to be identified into the trained pedestrian re-identification model, and output the recognition result; The step of constructing a hybrid attention network model and adjusting the constructed hybrid attention network model includes: A residual network is selected as the basic network structure, and a hybrid attention module is added to the last three bottleneck blocks of the residual network to obtain the adjusted hybrid attention network model. The step of constructing a hybrid attention network model and adjusting the constructed hybrid attention network model further includes: The adjusted hybrid attention network model is further encoded with auxiliary information to obtain the pedestrian re-identification model. The auxiliary information encoding is constructed from the domain information and visual information in the image dataset.

2. The pedestrian re-identification method based on hybrid attention according to claim 1, characterized in that, The step of inputting the preprocessed pedestrian images from the image dataset into the pedestrian re-identification model for training, and obtaining the trained pedestrian re-identification model, includes: The parameters of the pedestrian re-identification model are adjusted using a triplet loss function for hard samples; wherein the triplet includes a positive sample, a negative sample, and the original sample.

3. The pedestrian re-identification method based on hybrid attention according to claim 1, characterized in that, The loss function is as follows: （1） A represents the feature vector of the anchor point, P represents the feature vector of the positive sample, N represents the feature vector of the negative sample, and d is the Euclidean distance. Therefore, d(A,P) represents the Euclidean distance between the positive sample and the anchor point, and d(A,N) represents the Euclidean distance between the negative sample and the anchor point. max refers to the maximum margin between the two distances, and min refers to the minimum margin between the two distances. The plus sign in the lower right corner of the parentheses indicates that when the value of the expression is greater than or equal to zero, the loss value is the value of the current expression, and when the value of the expression is less than zero, the loss value is zero.

4. The pedestrian re-identification method based on hybrid attention according to claim 1, characterized in that, The step of inputting the preprocessed pedestrian images from the image dataset into the encoded hybrid attention network model for training to obtain a trained pedestrian re-identification model includes: The convergence of the hybrid attention network model with encoding is determined based on the value and curve of the loss function.

5. The pedestrian re-identification method based on hybrid attention according to claim 4, characterized in that, The step of inputting the preprocessed pedestrian images from the image dataset into the encoded hybrid attention network model for training to obtain a trained pedestrian re-identification model includes: When the loss function does not change significantly during a preset number of training iterations, the hybrid attention network model with encoding is determined to have converged, and training is stopped, resulting in the trained pedestrian re-identification model.

6. The pedestrian re-identification method based on hybrid attention according to claim 5, characterized in that, After obtaining the trained person re-identification model, the following steps are also included: When the encoded hybrid attention network model converges, the performance of the converged hybrid attention model is judged based on the first matching rate and the mean of the average precision.

7. The pedestrian re-identification method based on hybrid attention according to claim 1, characterized in that, The step of inputting the image and video to be identified into the trained pedestrian re-identification model and outputting the recognition result includes: When the video to be identified is input into the trained pedestrian re-identification model, the image of each frame of the video is extracted, and the similarity between the pedestrian and the pedestrians in the database is determined by calculating the cosine similarity of each frame of the image and sorted, and the recognition result is output.

Images