A pedestrian re-identification method and device integrating multi-head self-attention and convolution

Abstract

The application discloses a pedestrian re-identification method and device integrating multi-head self-attention and convolution. The proposed double-layer multi-head self-attention (DL-MHSA) divides a feature matrix by a grid, and decomposes the MHSA into two layers, the first layer only calculates the features in a single grid, and the second layer calculates the overall features on the basis of the grid, and the two-step operation reduces the calculation complexity; the DL-MHSA is introduced into the OSNet, the features of different scales and global features are fused, the model can obtain more recognizable features for the input pictures, and the pedestrian re-identification performance is improved; and the model is pre-trained by using a large-scale unlabeled pedestrian dataset, the model obtains high-level semantics beneficial to the task, the performance of subsequent fine-tuning training on a labeled dataset is improved, and the generalization capability of the model is stronger.

Description

Technical Field

[0001] This application relates to the field of intelligent video surveillance technology, and in particular to a pedestrian re-identification method and device. Background Technology

[0002] In recent years, public safety has become a growing concern. Driven by practical needs, the number of surveillance cameras in daily life and production is increasing. Researchers have conducted extensive studies on personnel re-identification (re-ID) within intelligent public safety monitoring systems.

[0003] Some biometric features, such as facial features and gait, are easier to identify than visual appearance. However, due to camera resolution and the practical processing complexities of open-world surveillance systems, purely visual appearance is often more feasible.

[0004] Pedestrian re-identification aims to determine the identity of a person in a given detected image by calculating the similarity to images in a given image library (a set of candidate objects). Considering variations in lighting, pose, and viewpoint, the same person can look significantly different in different surveillance cameras. Furthermore, different people can appear very similar in certain situations due to differences in clothing and body shape. Therefore, re-ID with significant intra-class and minor inter-class variations is more challenging than ordinary classification tasks.

[0005] Thanks to the feature extraction capabilities of Convolutional Neural Networks (CNNs), re-ID research has made significant progress. Several CNN-based supervised learning methods have achieved excellent results. In recent years, the Transformer architecture, which has achieved great success in Natural Language Processing (NLP), has attracted increasing attention from scholars researching how to apply it to the 2D / 3D structures of visual data.

[0006] In pursuit of performance, existing technologies often employ models with a large number of parameters, which presents challenges for practical deployment, especially at the edge: high storage pressure on devices and long inference times. Most existing models prioritize accuracy over computational complexity, which is another aspect that needs to be considered in practical deployment.

[0007] Meanwhile, the inventors also noted that existing person re-identification networks use ImageNet pre-training. While this method can obtain low-level information such as texture and boundaries, the high-level information it obtains is the ability to identify objects (ImageNet is an object recognition dataset), which is quite different from the person re-identification task. In a broad sense, person re-identification only involves pedestrians; the model needs to learn to identify pedestrian identities. Considering variations in lighting, posture, and perspective, the same person can look very different under different surveillance cameras. Furthermore, in some cases, different people can look very similar due to differences in clothing, body type, and perspective. Therefore, re-ID, with its large intra-class differences and small inter-class variations, is more challenging than normal classification tasks. The domain gap between the ImageNet dataset and the person re-identification dataset means that only low-level information and some high-level information (pedestrian location, etc.) in the pre-training mode are useful for the actual person re-identification task; information beneficial for person re-identification needs to be obtained through retraining. If the person re-identification dataset used for training is small, it is difficult for the model to fully acquire useful information. Summary of the Invention

[0008] To address the issues of high computational complexity, long inference time, and unsatisfactory model performance in existing technologies, this application provides a pedestrian re-identification method and apparatus that integrates multi-head self-attention and convolution.

[0009] Firstly, a person re-identification method integrating multi-head self-attention and convolution includes:

[0010] Acquire the surveillance video image data to be tested;

[0011] The surveillance video image data to be tested is input into the trained DM-OSNet model. The DM-OSNet model is a two-layer multi-head self-attention neural network based on OSNet's multi-scale feature dynamic fusion. The two-layer multi-head self-attention (DL-MHSA) divides the input feature matrix into a grid of a set size, first calculating the features within a single grid region, and then calculating the overall features using the grid as a unit to reduce computational complexity. The two-layer multi-head self-attention (DL-MHSA) is then integrated into the bottleneck of OSNet, replacing the convolutional flow that perceives local receptive fields in OSNet with a self-attention flow that extracts global information about relevant features to capture long-range dependencies and global information.

[0012] Based on the computation results of the DM-OSNet model, the pedestrian re-identification results are output.

[0013] Secondly, it provides corresponding virtual devices, including various program modules.

[0014] Thirdly, a computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the above-described method.

[0015] Fourthly, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0016] The present invention has at least the following beneficial effects:

[0017] The proposed dual-layer multi-head self-attention (DL-MHSA) divides the feature matrix into a grid and decomposes MHSA into two layers. The first layer calculates features within a single grid, and the second layer calculates overall features based on the grid. This two-step operation reduces computational complexity. DL-MHSA is introduced into the lightweight pedestrian re-identification model OSNet, which integrates multiple scales. By adding a new global feature stream to the existing blocks that aggregate features at different scales, DL-MHSA fuses features at different scales and global features, enabling the model to obtain more discriminative features from the input image and improving pedestrian re-identification performance.

[0018] This invention pre-trains the model using a large-scale unlabeled pedestrian dataset (unsupervised learning is pre-trained on the unlabeled dataset LUPerson, rather than ImageNet), enabling the model to acquire high-level semantics that are beneficial to the task. This improves the performance of subsequent fine-tuning training on labeled datasets and makes the model more generalizable. Attached Figure Description

[0019] Figure 1 This is a schematic diagram illustrating the principle and structure of Multi-Head Self-Attention (MHSA).

[0020] Figure 2 This is a schematic diagram illustrating the principle of a two-layer multi-head self-attention system.

[0021] Figure 3 This is a schematic diagram of the bottleneck of the DM-OSNet model in one embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram comparing the computational complexity of an embodiment of the present invention with that of the prior art. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] The proposed network improves upon the lightweight convolutional network model OSNet by integrating the Transformer's Multi-Head Self-Attention (MHSA) module to achieve multi-scale local and global information fusion, thereby enhancing model performance. To maintain the model's lightweight nature and low computational complexity, this embodiment divides MHSA into two layers, overcoming its high computational complexity through a two-step operation. Specifically:

[0025] MHSA is used to extract global information and obtain discriminative features from pedestrian images at a larger scale. However, unlike convolution operations, MHSA requires computation across the entire feature matrix (feature matrix multiplication), resulting in higher computational complexity and impacting model inference time. To reduce computational complexity, this invention proposes dividing the feature matrix into a grid and decomposing MHSA into two layers. The first layer computes features within a single grid, while the second layer computes the overall features based on the grid. This two-step operation reduces computational complexity and is termed DL-MHSA.

[0026] OSNet is a network specifically designed for pedestrian re-identification. It mainly focuses on fine-grained information at different scales (such as the patterns on a pedestrian's shoes and clothing). However, its performance is not optimal. This embodiment found that a new global feature stream can be added to the existing blocks that aggregate features at different scales. By aggregating global and local information, the performance can be effectively improved.

[0027] Furthermore, existing pedestrian re-identification networks, which are pre-trained on ImageNet, cannot learn the fine-grained information required for pedestrian re-identification in advance. This embodiment proposes to pre-train on an unlabeled dataset and then train on a labeled dataset to improve the model's generalization ability and further enhance its performance.

[0028] When the amount of data is large, using clustering to generate pseudo-labels is often extremely computationally expensive in terms of space and time. This embodiment proposes to use the geographic information of the original data to divide the large-scale unlabeled pedestrian dataset and then cluster it, which can reduce the computation time and space consumption of clustering without reducing performance.

[0029] Specific examples are explained below:

[0030] 1. Two-layer multi-head self-attention (DL-MHSA)

[0031] MHSA is an important attention module in Transformer. MHSA first obtains the query Q, keyword K, and value V by applying three sets of projections to the input feature map X∈R. H×W×C H, W, and C are the height, width, and feature dimensions of X.

[0032] They are then divided into multiple parts. Each part, including Q, K, V, and subsequent processing, represents a head, with Q, K, and V mapped from the original C dimension to... Dimension, where k represents the number of heads, and each head requires Q. i K i and V i These mappings, along with subsequent operations, constitute the head of attention generation.

[0033] The self-attention of each head can be represented as:

[0034]

[0035] Finally, the self-attention of multiple heads is concatenated to form an MHSA. The simplified structure of an MHSA is as follows: Figure 1 As shown. It should be noted that several headers are omitted here.

[0036] Considering the matrix multiplication operation of equation (1), the computational complexity of MHSA is affected by the values ​​of H and W.

[0037] The time complexity of MHSA can be expressed as O((H×W)). 2 ).

[0038] To reduce the time and space complexity of MHSA, we designed a two-layer MHSA, namely DL-MHSA. First, in the first layer, we divide the input feature map X using a grid of size G×G. By projecting the input X, we can obtain Q. i K i and V i for:

[0039]

[0040] Applying equation (1), we can obtain:

[0041]

[0042]

[0043] When we compute each head, we can obtain the feature map of the first layer:

[0044] attention1=F(head0, head1,..head k )+X (5)

[0045] In equation (5), F(.) represents the connection operation on multiple head inputs.

[0046] Taking a 256x128 pedestrian image as an example for pedestrian re-identification, if G is set to 8, each 32x16 pixel region is set as a grid. Here, the concept of a grid is not the same as that of a convolutional kernel, which has kernel size and stride. The matrix multiplication work of multi-head attention is reduced by using grid units, thus minimizing the computational cost. Convolution is only performed after these two steps. The choice of G considers the size of the input feature map's H and W, and it should be divisible. A value of 4 is also acceptable; in this embodiment, 8 is chosen.

[0047] In the second layer, we also map attention1 to Q. i K i and V i However, K and V are downsampled only by using average pooling, as shown in the following formula:

[0048]

[0049]

[0050]

[0051]

[0052] We can obtain the final attention as:

[0053] attention2=F(head0, head1,..head k +attention1 (10)

[0054] By performing two steps, the computational complexity is reduced from the original O((H×W)). 2 The value has been reduced to the current O(H×W×G) 2 While still obtaining self-attention from the input feature map, we... Figure 2 The simplified structure of the DL-MHSA is shown in the image. For clarity, we will only show the header.

[0055] 2. OSNet adds self-attention Transformer flow.

[0056] OSNet is a comprehensive feature learning network designed specifically for person re-identification tasks. It achieves this by designing a residual block composed of multiple convolutional streams, each of which detects features at a certain scale, and then dynamically fuses these features through a unified aggregation gate.

[0057] OSNet achieves different receptive field sizes by stacking multiple 3×3 convolutions in each convolutional stream. For example, two 3×3 convolutions can be used to achieve a 5×5 receptive field. In the original OSNet, 3×3, 5×5, 7×7, and 9×9 receptive fields are aggregated in the residual block.

[0058] Previous studies have added multiple branches to OSNet to improve model performance. Among these new branches, there is certainly a global branch, highlighting the importance of global information for the re-ID task. CNN-based models are more concerned with aggregating local information because they do not easily obtain global information. To obtain global information, convolutional networks need to stack convolutional layers. In contrast, Transformer-based models have an inherent ability to acquire global information. Our technique integrates DL-MHSA into the bottleneck of OSNet, enabling it to acquire the ability to integrate global information.

[0059] The main structure of the OSNet residual bottleneck consists of residual blocks with extended dimensions and a unified aggregation gate. The residual blocks perceive multi-scale features through multiple parallel convolutional streams with different receptive fields. The unified aggregation gate then aggregates these features to capture a wide range of scales. We replace the convolutional streams that perceive local receptive fields with self-attention streams that extract global information about the features, and then provide fine-grained fusion of the input features through the unified aggregation gate.

[0060] Compared to convolutional operations, self-attention modules use a weighted averaging operation based on the input feature context. The attention module can adaptively focus on different regions and capture more features. In this invention, we propose replacing the convolutional stream with a 9×9 receptive field with a two-layer MHSA to capture long-range dependencies and global information, such as... Figure 3 As shown. Suppose the input feature map of our improved bottleneck block is x, and the bottleneck residual is... The acquisition of can be represented as:

[0061]

[0062] Where Ft represents the convolutional flow with a receptive field of (2t+1)×(2t+1), and G represents the unified aggregation gate's processing of features. ⊙ represents the Hadamard product. H(x) represents our proposed two-layer multi-head self-attention. Note that, for clarity, Equation 11 omits the 1×1 convolution operation.

[0063] 3. Pre-training on a large-scale unlabeled pedestrian count dataset

[0064] Previous proposed person re-identification networks were pre-trained on ImageNet, which is unsuitable for person re-identification tasks due to the significant domain gap between the datasets used for person re-identification and ImageNet. Our proposal uses LUPerson for pre-training on the re-ID task.

[0065] LUPerson is a large-scale unlabeled pedestrian dataset with 4 million unlabeled pedestrian images and over 200,000 people. It was created by downloading videos from video websites and then automatically cropping pedestrians using a YOLOv5 object detector.

[0066] The unlabeled dataset is clustered using the DBSCAN density clustering algorithm to generate pseudo-labels. These pseudo-labels are then used as supervisory information to pre-train the network, enabling it to learn advanced semantics that aid in pedestrian re-identification.

[0067] Because DBSCAN requires calculating the distance to each feature point, and every certain training epoch requires regenerating features and performing clustering again, clustering consumes a large proportion of the training time, significantly delaying our pre-training process. Since the original videos in the LUPerson dataset are retrieved from video websites by country and city, we propose dividing them by country and city, performing clustering and generating pseudo-labels separately within each division. This greatly reduces the time spent on clustering and accelerates our pre-training process. Furthermore, given the extremely low probability of the same person appearing in videos from different countries and cities, our proposed division method does not significantly degrade clustering performance during pre-training.

[0068] It can be seen that this embodiment has the following advantages:

[0069] (1) It reduces the computational complexity of the multi-head self-attention mechanism and accelerates inference time.

[0070] (2) Performance was improved while maintaining the model's lightweight nature. First, we integrated multi-head self-attention and convolutional operations to fuse global and local information, enabling the model to obtain more discriminative features from the input image and improving pedestrian re-identification performance. Second, we pre-trained the model using a large-scale unlabeled pedestrian dataset, allowing it to acquire high-level semantics beneficial to this task. This improved the performance of subsequent fine-tuning on labeled datasets, resulting in stronger generalization ability.

[0071] (3) Other methods to improve model performance while keeping the model lightweight include transfer learning and distillation learning.

[0072] Table 1

[0073]

[0074] Table 1 shows the comparison results of the method of this embodiment with other methods on a public dataset. It can be seen that the method of this embodiment has better or similar performance compared with other methods, and at the same time, the method of this embodiment has a smaller number of parameters and computational cost.

[0075] Figure 4 This paper compares the number of model parameters and the number of floating-point operations required for the method in this embodiment with some general methods. It can be seen that the DM-OSNet in this embodiment has a smaller model size and lower computational requirements than other model backbones.

[0076] In one embodiment, a pedestrian re-identification device integrating multi-head self-attention and convolution is also provided, comprising the following program modules:

[0077] The image data acquisition module is used to acquire the image data of the surveillance video under test.

[0078] The AI ​​computing module is used to input the surveillance video image data to be tested into the trained DM-OSNet model for computation. The DM-OSNet model is a two-layer multi-head self-attention neural network based on OSNet's multi-scale feature dynamic fusion. The two-layer multi-head self-attention (DL-MHSA) divides the input feature matrix into a grid of a set size and decomposes MHSA into two layers. The first layer only calculates the features within a single grid, and the second layer calculates the overall features using the grid as a unit. This two-step operation reduces computational complexity. The DL-MHSA is then integrated into the bottleneck of OSNet, replacing the convolutional stream that perceives local receptive fields in OSNet with a self-attention stream that extracts global information about relevant features, thereby capturing long-range dependencies and global information.

[0079] The result output module is used to output the pedestrian re-identification result based on the computation result of the DM-OSNet model.

[0080] For specific details regarding the modules in this device, please refer to the description of the pedestrian re-identification method embodiment integrating multi-head self-attention and convolution above, which will not be repeated here. Each module in this device is stored in software form in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.

[0081] In one embodiment, a computer device is also provided, which can be a server or a terminal. The computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in the above embodiments. The DM-OSNet model and its related data described above can be stored locally or accessed via a network connection.

[0082] In one embodiment, a computer-readable storage medium is also provided, on which a computer program is stored relating to all or part of the processes in the methods of the above embodiments.

[0083] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0084] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A person re-identification method integrating multi-head self-attention and convolution, characterized in that, include: Acquire the surveillance video image data to be tested; The surveillance video image data to be tested is input into the DM-OSNet model that has been trained. The DM-OSNet model is a two-layer multi-head self-attention neural network based on OSNet's multi-scale feature dynamic fusion. The two-layer multi-head self-attention divides the input feature matrix into a grid of a set size, first calculating features within a single grid region, and then calculating the overall features using the grid as a unit to reduce computational complexity. Furthermore, the two-layer multi-head self-attention is integrated into the bottleneck of OSNet, replacing the convolutional stream that perceives local receptive fields in OSNet with a self-attention stream that extracts global information about relevant features, thereby capturing long-range dependencies and global information. Based on the computation results of the DM-OSNet model, output the pedestrian re-identification results; The calculation process of the two-layer multi-head self-attention includes: First, the input feature map is input in the first layer. use The grid size is separated; input via projection. ,get , and for: H, W, and C represent the height, width, and feature dimension of X, respectively; Q represents the query condition; K represents the keyword; and V represents the value. Q, K, V, and each subsequent part represent a header. , and These represent the query condition, keyword, and value for the i-th header, respectively; Head i Self-attention for each head; When calculating each head, the feature mapping of the first layer is: In the above formula, This represents a connection operation across multiple head inputs; On the second layer Mapped to , and However, K and V are downsampled only by using average pooling, as shown in the following formula: The final attention obtained is:

2. The pedestrian re-identification method integrating multi-head self-attention and convolution as described in claim 1, characterized in that, The training process of the DM-OSNet model is as follows: first, the model is pre-trained using a large-scale unlabeled pedestrian dataset to enable the model to acquire high-level semantics that are beneficial to the task, and then fine-tuned and trained on a labeled dataset.

3. The pedestrian re-identification method integrating multi-head self-attention and convolution according to claim 2, characterized in that, The large-scale unlabeled pedestrian dataset is LUPerson.

4. The pedestrian re-identification method integrating multi-head self-attention and convolution as described in claim 1, characterized in that, The computational process of integrating two-layer multi-head self-attention into the bottleneck of OSNet, wherein it is assumed that the input feature map of the bottleneck block is The bottleneck residual is , : in Describes a convolutional flow, whose receptive field is G represents the unified aggregation gate's handling of features. It represents the Hadamah accumulation. This indicates a two-layer multi-head self-attention mechanism.

5. A pedestrian re-identification device integrating multi-head self-attention and convolution, characterized in that, Includes the following program modules: The image data acquisition module is used to acquire the image data of the surveillance video under test. The AI ​​computing module is used to input the surveillance video image data to be tested into the trained DM-OSNet model for computation. The DM-OSNet model is a two-layer multi-head self-attention neural network based on OSNet's multi-scale feature dynamic fusion. The two-layer multi-head self-attention divides the input feature matrix into a grid of a set size, first calculating the features within a single grid region, and then calculating the overall features using the grid as a unit to reduce computational complexity. The two-layer multi-head self-attention is then integrated into the bottleneck of OSNet, replacing the convolutional flow that perceives the local receptive field in OSNet with a self-attention flow that extracts global information about the relevant features, in order to capture long-range dependencies and global information. The result output module is used to output the pedestrian re-identification result based on the calculation result of the DM-OSNet model; The calculation process of the two-layer multi-head self-attention includes: First, the input feature map is input in the first layer. use The grid size is separated; input via projection. ,get , and for: H, W, and C represent the height, width, and feature dimension of X, respectively; Q represents the query condition; K represents the keyword; and V represents the value. Q, K, V, and each subsequent part represent a header. , and These represent the query condition, keyword, and value for the i-th header, respectively; Head i Self-attention for each head; When calculating each head, the feature mapping of the first layer is: In the above formula, This represents a connection operation across multiple head inputs; On the second layer Mapped to , and However, K and V are downsampled only by using average pooling, as shown in the following formula: The final attention obtained is:

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images