A pedestrian re-identification method and system for unmanned aerial vehicle (UAV) from an aerial perspective

Abstract

The present invention belongs to the field of machine vision technology, and discloses a method and system for pedestrian re-identification under an aerial perspective applied to a drone. The method comprises: obtaining a target pedestrian image and a data set of images to be searched under an aerial perspective; extracting features of the target pedestrian image and each image to be searched using a pre-trained drone pedestrian re-identification network; judging whether there is an image of the target pedestrian in the image data set to be searched according to the similarity between the features of the target pedestrian image and each image to be searched; wherein the drone pedestrian re-identification network is a network with a plurality of parallel branches and a depthwise separable convolution block as a basic unit. On the basis of achieving lightweight to the greatest extent, it is also possible to obtain multi-scale features of pedestrians in the image, which is helpful to enhance the model performance.

Description

Technical Field

[0001] The present invention belongs to the field of machine vision technology, and in particular relates to a method and system for pedestrian re-identification from an aerial perspective applied to an unmanned aerial vehicle. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] The goal of pedestrian re-identification (Re-ID) is to find pedestrians with the same identity in multiple cameras that are not connected to each other. Currently, most research on pedestrian re-identification focuses on searching images taken by ground cameras. However, traditional ground cameras, such as surveillance cameras installed indoors and traffic monitoring cameras installed at outdoor intersections, have limited field of view because they cannot move.

[0004] Drones can automatically cruise and take photos of a certain area according to artificial regulations. They are not restricted by the environment and are extremely convenient. Some scholars have applied drones to pedestrian re-identification. However, pedestrian re-identification from the perspective of drone aerial photography still has the following problems: UAV-ReID is different from traditional pedestrian re-identification. Drones are constantly moving, so even if the pictures taken by the same drone at the same time are very different due to different positions and angles; drones are more susceptible to occlusion, light and other noise than ground cameras; in addition, although the memory capacity of embedded devices carried by drones has become larger and larger, they still cannot directly run some current large networks like ground devices, which makes it impossible to directly transplant some current advanced pedestrian re-identification methods Re-ID into UAV-ReID.

[0005] Thanks to the deep separable convolutional layer proposed by MoblieNet, the network has a significant decrease in the number of parameters while maintaining the original accuracy as much as possible. In OSNet, building blocks are formed by stacking deep separable convolutional layers, and unified aggregation gates are used to manage these block structures to learn full-scale features. Although this method can change the weights of different branches through the gate structure, since it only stacks convolutional layers, some convolutional layers may contain invalid information, resulting in structural redundancy. Inspired by the field of automated machine learning, some methods have designed automated neural architecture search methods to enable the network to find the most efficient network architecture by itself. For example, CDNet uses the search space to combine the deep space to eliminate some invalid convolutional layers in the network. While ensuring accuracy, the number of parameters has also been reduced to a certain extent. Currently, lightweight models are mostly used to solve image classification and text retrieval problems, which is quite different from Re-ID. Summary of the invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a method and system for pedestrian re-identification from an aerial perspective of a drone. With depthwise separable convolution as the smallest unit, a parallel branch structure with a weighted attention module is constructed. On the one hand, the lightweight feature extraction network is realized so that it can be loaded onto a drone, overcoming the limitation of drone storage resources; on the other hand, through the weighted attention mechanism, the network can extract more fine-grained features of pedestrians, thereby better adapting to the feature extraction of pedestrians from the perspective of a drone.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0008] A pedestrian re-identification method from an aerial perspective applied to a drone comprises the following steps:

[0009] Obtain target pedestrian images and image datasets to be searched from an aerial perspective;

[0010] For the target pedestrian image and each image to be searched, a pre-trained UAV pedestrian re-identification network is used to extract features;

[0011] According to the similarity between the features of the target pedestrian image and each image to be searched, determining whether there is an image of the target pedestrian in the image data set to be searched;

[0012] The UAV pedestrian re-identification network includes a multi-scale convolution module and a weighted attention module. The multi-scale convolution module includes a backbone network and one or more parallel networks. The backbone network includes multiple groups of depth-separable residual convolution blocks connected in sequence, and a downsampling layer is provided between each two groups of depth-separable residual convolution blocks; and the output end of each downsampling layer of the backbone network is again connected to a parallel network via a downsampling layer; the parallel network includes one or more groups of depth-separable residual convolution blocks connected in sequence;

[0013] Information is exchanged at the third and fifth layers of the backbone network: the input of each subnetwork at the fourth layer is the sum of the outputs of the two subnetworks at the third layer, and the input of each subnetwork at the sixth layer is the sum of the outputs of the three subnetworks at the fifth layer. The conversion formula for the exchange process is:

[0014]

[0015] Where M represents the Mth layer of the subnetwork, 1 represents the first branch, and the values ​​of 1 and M are determined by the network structure;

[0016] The weighted attention module formula can be expressed as:

[0017]

[0018] Among them, S represents the number of branch networks; AN represents the attention network in the figure, and Fin represents the input feature, which is the output of different branches in LRBlock or the output of the sixth layer of the UAV pedestrian re-identification network (UAV-Re-ID). represents the Hadamard product of two characteristic matrices.

[0019] Furthermore, in the parallel stage of the backbone network and one or more parallel networks, for each group of depth-wise separable residual convolution blocks of different network layers, the output ends of the previous depth-wise separable residual convolution blocks are connected to the subsequent depth-wise separable residual convolution blocks of this group and other network layers.

[0020] Furthermore, the depth-separable residual convolution block includes a multi-scale convolution module and a weighted attention module, the multi-scale convolution module includes a trunk branch and a residual branch, and the trunk branch and the residual branch each include a plurality of depth-separable convolution units connected in sequence.

[0021] Furthermore, the depthwise separable convolution unit adopts a depthwise separable convolution unit with a receptive field size of 3×3.

[0022] Furthermore, the attention module includes a multi-layer attention network, each layer of the attention network includes a pooling layer, a first convolutional layer, a normalization layer, a first activation function, a second convolutional layer and a second activation function.

[0023] Furthermore, the output end of the weighted attention module is connected to the global average pooling layer, the linear layer, the batch normalization layer and the activation function in sequence. The output of the activation function is input to the triplet loss function on the one hand, and is input to the cross entropy loss function via the fully connected layer on the other hand. The loss function of the drone pedestrian re-identification network is the sum of the triplet loss function and the cross entropy loss function.

[0024] Furthermore, the UAV pedestrian re-identification network adopts a loss function to calculate gradients and updates network parameters through back propagation.

[0025] One or more embodiments provide a pedestrian re-identification system from an aerial perspective applied to a drone, including:

[0026] A data acquisition module is used to acquire target pedestrian images and image data sets to be searched from an aerial perspective;

[0027] A feature extraction module, used for extracting features from the target pedestrian image and each image to be searched using a pre-trained drone pedestrian re-identification network;

[0028] A pedestrian re-identification module is used to determine whether there is an image of the target pedestrian in the image data set to be searched based on the similarity between the target pedestrian image and the features of each image to be searched;

[0029] The UAV pedestrian re-identification network includes a multi-scale convolution module and a weighted attention module. The multi-scale convolution module includes a backbone network and one or more parallel networks. The backbone network includes multiple groups of depth-separable residual convolution blocks connected in sequence, and a downsampling layer is provided between each two groups of depth-separable residual convolution blocks; and the output end of each downsampling layer of the backbone network is again connected to a parallel network via a downsampling layer; the parallel network includes one or more groups of depth-separable residual convolution blocks connected in sequence;

[0030] Information is exchanged at the third and fifth layers of the backbone network: the input of each subnetwork at the fourth layer is the sum of the outputs of the two subnetworks at the third layer, and the input of each subnetwork at the sixth layer is the sum of the outputs of the three subnetworks at the fifth layer. The conversion formula for the exchange process is:

[0031]

[0032] Where M represents the Mth layer of the subnetwork, 1 represents the first branch, and the values ​​of 1 and M are determined by the network structure;

[0033] The weighted attention module formula can be expressed as:

[0034]

[0035] Among them, S represents the number of branch networks; AN represents the attention network in the figure, and F in Represents the input features, which are taken as the output of different branches in LRB1ock or the output of the sixth layer of the UAV-Re-ID network. represents the Hadamard product of two characteristic matrices.

[0036] One or more embodiments provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the pedestrian re-identification method when executing the program.

[0037] One or more embodiments provide a computer-readable storage medium having a computer program stored thereon, which implements the pedestrian re-identification method when executed by a processor.

[0038] One or more of the above technical solutions have the following beneficial effects:

[0039] Based on the deep separable convolutional block, a drone pedestrian re-identification network with parallel branches is constructed. Each branch takes the deep separable convolutional block as the basic unit. There is only one backbone network in the initial input. As the network depth increases, the number of network branches is continuously increased to transform a serial network into a parallel network. Using this method, new network branches can be continuously generated during network propagation to learn more discriminative features, and multi-scale features of pedestrians in the image can be obtained, which helps to enhance the model performance. At the same time, in order to reduce the number of model parameters, the sub-network will also use downsampling operations to reduce the network resolution during the learning process. Therefore, on the basis of effectively extracting multi-scale features of pedestrians, the network is also lightweight, solving the limitations brought by drone hardware.

[0040] Since the importance of features learned by each branch varies, a weighted attention module is added at the end of the branch network to dynamically adjust the proportion of each branch. By adding a weighted attention module after the branch network, fine-grained adjustment can be achieved, which allows the network to more accurately select discriminative feature areas and suppress certain areas where features are unavailable due to the influence of noise, making the network more resistant to noise. Therefore, for actual pedestrian images taken by drones, the unobstructed parts can be adaptively selected as feature areas, thereby ensuring recognition accuracy.

[0041] In addition, since the resolution of drone images is low and the area occupied by pedestrians in the images is smaller, perhaps only about 100 pixels, an information exchange process is added between parallel networks. The features of different branches are converted into the same dimension and then superimposed. This allows each branch to contain the information learned from other branches, thereby enriching the feature representation and obtaining richer features in limited image information, thereby improving recognition accuracy.

[0042] Experiments show that the pedestrian re-identification network used in the aerial perspective has better accuracy than other existing models while ensuring that it is as lightweight as possible. It can be applied to some micro embedded platforms, which eliminates the need to deploy the model in a central server in the traditional way. After the camera takes the picture, it needs to be transmitted to the server for judgment. The model is directly deployed on the front-end embedded platform, which has the advantages of fast response and low load. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0044] Figure 1A flowchart of a method for pedestrian re-identification from an aerial perspective applied to a drone according to one or more embodiments of the present invention;

[0045] Figure 2 This is a diagram showing the overall architecture of a drone pedestrian re-identification network in one or more embodiments of the present invention;

[0046] Figure 3 It is an overall architecture diagram of a depth-separable convolution block in one or more embodiments of the present invention;

[0047] Figure 4 A schematic diagram of a weighted attention module in one or more embodiments of the present invention;

[0048] Figure 5 This is a visualization effect diagram of extracting pedestrians based on the drone pedestrian re-identification network described in one or more embodiments of the present invention. DETAILED DESCRIPTION

[0049] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0050] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0051] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0052] Embodiment 1

[0053] In order to solve the problem of limited field of view of traditional ground cameras due to their inability to move, this embodiment provides a method for pedestrian re-identification from an aerial perspective applied to drone embedded devices, which can well solve the problems encountered in pedestrian tracking and pedestrian re-identification. In order to solve the storage pressure of embedded devices such as drones, this embodiment builds a deep learning network based on deep separable convolution. Assuming that there is a three-channel RGB picture with a length and width of 12 pixels, if the network uses 256 5×5×3 (height, width, number of channels) convolution kernels to extract features from the picture to generate a feature matrix, the resulting feature matrix is ​​8×8×256, and the amount of operations required to obtain such a feature matrix is ​​1228800 times. If deep separable convolution is used, first use three convolution kernels of size 5×5×1 to process the picture to obtain an 8×8×3 feature matrix, and then use 256 1×1×3 convolution kernels to further process the feature matrix to obtain an 8×8×256 feature matrix. The amount of operations required for this method is 53952 times. The dimensions of the final feature matrices obtained by the two methods are the same, but the amount of computation differs by more than twenty times. Therefore, in lightweight models, depthwise separable convolution is a better method than ordinary convolution, which greatly reduces the amount of computation while ensuring accuracy.

[0054] Based on this, this embodiment discloses a pedestrian re-identification method applied to an aerial perspective of a drone, such as Figure 1 As shown, including:

[0055] Step 1: Obtain a target pedestrian image taken from an aerial perspective and a library pedestrian image dataset; use the library pedestrian image dataset as a search object;

[0056] Step 2: For the target pedestrian image and the gallery pedestrian image, a pre-trained drone pedestrian re-identification network is used to extract features respectively;

[0057] Step 3: Based on the similarity between the features of the target pedestrian image and the gallery pedestrian image, determine whether there is an image of the target pedestrian in the gallery pedestrian image dataset.

[0058] In step 2, the UAV pedestrian re-identification network is trained based on the deep learning network architecture PBWA-Net. In this embodiment, the deep separable residual convolution block (LRBlock) is used as the basic unit of the deep learning network architecture PBWA-Net, and the deep separable residual convolution block uses the deep separable convolution block as the basic block.

[0059] Compared with OSNet, the deep learning network architecture PBWA-Net abandons the structure of a large number of parallel repeated stacked convolutions used in OSNet, and instead adopts a parallel network structure with multiple different lengths. At the same time, multiple networks exchange information multiple times, so even if the number of convolutional layers is not as many as OSNet, it can achieve performance that exceeds OSNet.

[0060] like Figure 2 As shown, the deep learning network architecture PBWA-Net includes an input layer, an initial convolution layer, a multi-scale convolution module and a weighted attention module connected in sequence. The multi-scale convolution module includes a backbone network and two parallel networks. The backbone network and the parallel network both include one or more groups of depth-separable residual convolution blocks. When there are multiple groups, a downsampling layer is provided between each two groups of depth-separable residual convolution blocks; wherein each group of depth-separable residual convolution blocks includes at least two depth-separable residual convolution blocks. The output end of each downsampling layer of the backbone network is again connected to a parallel network via a downsampling layer. Assuming that the backbone network has n groups of depth-separable residual convolution blocks, the parallel network connected to the first downsampling layer of the backbone network has n-1 groups of depth-separable residual convolution blocks, the parallel network connected to the second downsampling layer of the backbone network has n-2 groups of depth-separable residual convolution blocks, and so on.

[0061] Among them, the initial convolution layer is a 1×1 convolution layer, which is connected to the backbone network input end of the multi-scale convolution module; the weighted attention module is connected to the fully connected layer via the pooling layer, linear layer, normalization layer and activation function in sequence.

[0062] In this embodiment, the multi-scale convolution module includes a backbone network and two parallel networks, which are denoted as the first parallel network and the second parallel network. The backbone network includes three groups of depth-separable residual convolution blocks, and a downsampling layer is provided between each two groups of depth-separable residual convolution blocks; wherein each group of depth-separable residual convolution blocks includes two depth-separable residual convolution blocks. The output ends of the first downsampling layer and the second downsampling layer of the backbone network are respectively connected to the first parallel network and the second parallel network via a downsampling layer again. The first parallel network includes two groups of depth-separable residual convolution blocks, and the second parallel network includes one group of depth-separable residual convolution blocks.

[0063] After the first downsampling layer of the backbone network, the backbone network is parallel to the first parallel network, and after the second downsampling layer, the backbone network is parallel to the first parallel network and the second parallel network. In each parallel stage, for each group of depth-wise separable residual convolution blocks of different network layers, the output end of the previous depth-wise separable residual convolution block is connected to the subsequent depth-wise separable residual convolution blocks of this group and other network layers.

[0064] During the learning process, different subnetworks may contain information needed by other subnetworks. We exchange and fuse the information learned by the three subnetworks in the middle layer so that each branch subnetwork can obtain features containing more information. We exchanged information in the third and fifth layers of the backbone network: the input of each subnetwork in the fourth layer is the sum of the outputs of the two subnetworks in the third layer, and the input of each subnetwork in the sixth layer is the sum of the outputs of the three subnetworks in the fifth layer. The conversion formula of the exchange process is shown in formula (2). M represents the Mth layer of the subnetwork, 1 represents the first branch, and the values ​​of 1 and M are determined by the network structure. When the network is in the third layer, that is, 1=3, the network has two subnetworks, that is, M=2; when the network is in the fifth layer, that is, 1=5, the network has three subnetworks, that is, M=3.

[0065]

[0066] Since the drone is far away from the pedestrians, the image resolution is only about 6000 pixels, and the pedestrians only occupy about 100 pixels. Therefore, how to obtain as many discriminative features as possible from limited information is the focus of the research on pedestrian re-identification from the perspective of drones. Therefore, this embodiment uses three branches with different receptive fields to simultaneously extract features from the image. At the beginning of the network, there is only one sub-network. As the depth of the network continues to increase, the network uses downsampling to continuously divide new sub-networks, so that the serial network becomes a parallel network, and there are three sub-networks in total. Since the receptive field sizes of the LRBlocks in different branches of the parallel branches are different, different branches can have features of different scales. Moreover, in the process of learning the parallel branch structure, the three sub-networks perform multiple feature fusions in the middle layer of the network, so that the features obtain richer semantic information, ensuring that richer features can be obtained in limited image information, thereby improving recognition accuracy.

[0067] After learning is completed, the weighted attention module is also used for fusion. It is worth noting that there is also a weighted attention module in LRBLlock. Since the features processed by the two parts are different, the parameters of the two weighted attention modules are not shared, and independent weighted attention modules are used to adjust the features.

[0068] The last three branches of the network are combined to form a comprehensive feature that integrates features of different scales. The weight of each branch output feature in the final feature is dynamically determined by the weighted attention module. The weighted attention module can adjust the weights according to the importance of different branch output features.

[0069] like Figure 3As shown, this embodiment uses a depth-wise separable convolution layer as the smallest unit, stacks multiple depth-wise separable convolutions to form a convolution block, and inspired by ResNet, adds a residual branch containing a depth-wise separable convolution on the basis of the serial depth-wise separable convolution, and builds a depth-wise separable residual convolution block (LRBlock), hereinafter referred to as a basic block. The depth-wise separable residual convolution block includes an input layer, an initial convolution layer, a multi-scale convolution module, a weighted attention module, a final convolution layer and an output layer connected in sequence. The multi-scale convolution module includes a trunk branch and a residual branch. The trunk branch and the residual branch both include multiple basic blocks connected in sequence. The basic block uses Lite3×3 (depth-wise separable convolution with a receptive field size of 3×3). The output end of the initial convolution layer is respectively connected to the input end of the trunk branch and the residual branch of the multi-scale convolution module, and the number of basic blocks in the main branch and the residual branch is different.

[0070] The LRBlock calculation process is shown in formula (1). WAM represents the weighted attention module, Lite() represents the Lite3×3 basic block. k represents Lite3×3 repeated k times, k1 and k2 are different values. x represents the input after being processed by 1×1 convolution kernel.

[0071]

[0072] Unlike ResNet, the above residual structure does not simply pass the input features directly to the next layer, but adds a Lite3×3 basic block with unshared parameters to the residual branch. In this way, an LRBlock can use two branches with different receptive field sizes to simultaneously learn features of different scales, so that LRBlock can learn more information. Due to the different number of Lite 3×3 basic blocks in the main branch and the residual branch, the information extracted by the residual branch is different from that extracted by the main branch. Therefore, if the two branches are simply added together, the effective information may not be used at the same time and may even be contaminated. In order to better combine the effective information obtained from the two branches. We use a weighted attention module to fuse the information in the fusion layer.

[0073] At the same time, by adopting the depthwise separable convolution as the smallest unit of the network structure, since the convolution can ensure the accuracy as much as possible while the number of parameters is much smaller than that of ordinary convolution, the lightweight of the model is ensured.

[0074] The deep learning network architecture of this embodiment adds a weighted attention module at the end of the network to integrate the three sub-networks and allocate the proportion of each sub-network output to the total output in a targeted manner according to the different importance of the features learned by each sub-network. In this way, different weights are assigned to each sub-network. At the same time, the weighted attention module is also used in each LRBlock block to fuse the two branches. Figure 4 As shown, the essence of the weighted attention module is a multi-layer perceptron, and the weighted attention module includes a multi-layer attention network, each layer of the attention network includes a pooling layer, a first convolution layer, a normalization layer, a first activation function, a second convolution layer and a second activation function. Figure 4 Only two layers are shown in Figure 1. By inputting different features into the weighted attention module, the module can autonomously adjust each part of the feature, enhance effective features, and suppress meaningless features. Since the weighted attention module adjusts the features at a fine-grained level, it is more effective than the traditional coarse-grained attention mechanism.

[0075] The outputs of the three sub-networks of the deep learning network architecture are connected to a layer of attention network respectively; and the outputs of the two branches of the deep separable residual convolution block (LRBlock) are connected to a layer of attention network respectively. The parameters of the two weighted attention modules are not shared.

[0076] The weighted attention module formula can be expressed as:

[0077]

[0078] Among them, S represents the number of branch networks; AN represents the attention network in the figure, and F in Represents the input features, which are taken as the output of different branches in LRBlock or the output of the sixth layer of the UAV-Re-ID network. represents the Hadamard product of two characteristic matrices.

[0079] Through the weighted attention module (WAM), we can distinguish the information learned by sub-networks of different scales and select more discriminative features from them. Since the Hadamard product is used in the weighted attention module to process the input features, the attention weight matrix can act on different areas within the input features to obtain a more fine-grained representation. Since the Hadamard product is used internally to process the features, the Hadamard product is the product between each element between two features, which is a fine-grained matrix multiplication, so different areas of the feature matrix can be adjusted differently. This can increase the weight of the discriminative feature part while reducing the weight of the meaningless feature part.

[0080] For the entire deep learning network architecture PBWA-Net, we use cross entropy loss and triplet loss to train the network. The two loss functions focus on different aspects of the network. Cross entropy loss focuses on the gap between the predicted value and the true value, while triplet loss can widen the distance between the predicted value and the negative sample. Cross entropy loss makes the predicted value closer to the positive sample, while triplet loss makes the predicted value away from the negative sample. The two losses work together to enable the network to achieve better performance. In order to convert the features into the dimensions required by the loss function, we first use a global average pooling layer to reduce the dimension of the features. In order to prevent partial information loss due to dimensionality reduction, we then add a linear layer, a batch normalization layer, and an activation function (ReLU) to stabilize the process. Finally, since the two loss functions require different inputs, triplet loss requires a feature and cross entropy loss requires a probability, we use a fully connected layer (FC) to convert the features to meet the requirements of cross entropy loss. Triplet loss requires learned features, so the features do not need to be processed and can be directly input into the loss.

[0081] Therefore, the loss function of the network is expressed as formula (4)

[0082] Loss = L cro +L tri (2)

[0083]

[0084] Where L cro , L tri are the cross entropy loss and triplet loss respectively. out represents the features obtained by the output of the last layer activation function. i represents the i-th pedestrian. cro , P() is the true value probability matrix, when i is correctly classified, P(i) is equal to 1, otherwise P(i) is equal to 0. FC(F out )() is the predicted value probability matrix after transformation by the fully connected layer. L tri , F a OUT 、F p OUT 、F n OUT are the features corresponding to the anchor point of the sample, the positive sample of the anchor point, and the negative sample. α is a hyperparameter used to ensure that there is a minimum interval between samples. + means that if the sum is greater than 0 in [], the output is kept as it is, otherwise the loss is assigned to 0, || 2 2 represents the square of the 2-norm.

[0085] The above-mentioned pedestrian re-identification method, in order to solve the problem that the style of the same pedestrian picture is different due to different shooting positions and angles, designs a parallel branch network to learn more style-invariant features of pedestrians, so as to make the discrimination more accurate; in order to solve the influence of possible noise in the picture on the discrimination accuracy, a weighted attention module is added at the end of the branch network so that the network can find more discriminative areas by itself, and automatically suppress the problem that some regional features are unavailable due to noise. Finally, in order to solve the problem that large networks cannot run on drones, the network is lightweight by using the deep separable convolution as the smallest unit. Experiments show that the number of parameters of the model is much smaller than that of the current pedestrian re-identification network.

[0086] Embodiment 2

[0087] Based on the method provided in the first embodiment, this embodiment provides a pedestrian re-identification system from an aerial perspective applied to a drone, including:

[0088] The data acquisition module is used to obtain the target pedestrian image and the image dataset to be searched from an aerial perspective:

[0089] A feature extraction module, used for extracting features from the target pedestrian image and each image to be searched using a pre-trained drone pedestrian re-identification network;

[0090] A pedestrian re-identification module is used to determine whether there is an image of the target pedestrian in the image data set to be searched based on the similarity between the target pedestrian image and the features of each image to be searched;

[0091] Among them, the drone pedestrian re-identification network includes a multi-scale convolution module and a weighted attention module, the multi-scale convolution module includes a backbone network and one or more parallel networks, the backbone network includes multiple groups of deeply separable residual convolution blocks connected in sequence, and a downsampling layer is provided between every two groups of deeply separable residual convolution blocks; and the output end of each downsampling layer of the backbone network is again connected to a parallel network via a downsampling layer; the parallel network includes one or multiple groups of deeply separable residual convolution blocks connected in sequence.

[0092] Embodiment 3

[0093] The purpose of this embodiment is to provide an electronic device.

[0094] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first embodiment when executing the program.

[0095] Embodiment 4

[0096] The purpose of this embodiment is to provide a computer-readable storage medium.

[0097] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method described in the first embodiment.

[0098] Each step in the above embodiments 2 to 4 corresponds to the method embodiment 1, and the specific implementation method can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0099] In order to verify the accuracy of the above-mentioned drone pedestrian re-identification network, this embodiment selects a pedestrian re-identification dataset of a drone monitoring scene (PRAI-1581) and two pedestrian re-identification datasets from traditional ground perspectives (Market1501 and DukeMTMC) for experiments, and uses the mean average precision (mAP) curve and the cumulative matching feature (CMC) curve as the evaluation criteria for the model performance. At the same time, we also use the size of the network parameters as a criterion for measuring the quality of the model.

[0100] The PRAI-1581 dataset is a collection of 39,461 images of 1,581 pedestrians captured by two drones at different altitudes and locations. The resolution of the images is generally lower than that of images captured by fixed cameras, and pedestrians only occupy 30-150 pixels in the image. In addition to the low resolution, about 20% of the images are obscured by the environment, making them more difficult to identify than single-scale pedestrian images captured by traditional fixed cameras. The dataset is divided into a training set and a test set. The training set contains 19,523 images of 782 pedestrian identities. The test set contains 19,938 images of 799 pedestrian identities, including 4,680 query images and 15,258 gallery images.

[0101] The Market-1501 dataset is a classic dataset captured by ground cameras. The dataset consists of 32,668 images of 1,501 pedestrians captured by 6 independent cameras. In order to test the stability of the model, the dataset contains a part of irrelevant images to interfere with the model. The dataset is divided into a training set and a test set, where the training set contains 12,936 images of 751 pedestrians. The test set contains 19,732 images of 750 pedestrians.

[0102] DukeMTMC dataset uses 8 fixed cameras to take 36411 pictures of 1404 pedestrians. DukeMTMC uses 16522 pictures of 702 pedestrian identities as training set, 2228 images of 702 pedestrian identities as query set, and the remaining 17661 images as library. In some pictures of the dataset, pedestrians are blocked by the environment.

[0103] Table 1 Description of the data in the three datasets

[0104]

[0105] As shown in Table 2, by comparing some representative lightweight re-ID models, we can see that PBWA-Net has achieved advanced accuracy while ensuring lightweight. Compared with MobileNet, which only uses serial depthwise separable convolution, the accuracy of the present invention is significantly ahead of MobileNet on the three data sets. The Rank-1 score and mAP on PRAI-1581 are improved by 13.5% and 15.4% respectively, which shows that the parallel network structure can learn more discriminative information than the serial network structure. In GoogleNet, some sparse matrices in network operations are clustered into dense matrices to reduce the number of model parameters, but the network uses spatially separable convolutions, so the number of parameters is larger than that of the network using depthwise separable convolutions. SP+TL is the first pedestrian re-identification method specifically applied to UAV-ReID. They use subspace pooling to represent the input pedestrian image in the hope of learning a more discriminative feature. This method has improved the accuracy compared with previous methods. However, compared with the method of this invention, PBWA-Net is better than SP+TL in Rank-1score and mAP on PRAI-1581 by 8.3% and 9.0%, respectively. Among the two best performing lightweight models OSNet and CDNet, OSNet uses a unified aggregation gate to adjust the weights between features of different scales, and CDNet improves OSNet by using Combined Depth space to discard meaningless branches. The experiments show that the accuracy of PB-Net is almost the same as that of the other two methods in Market1501 and DukeMTMC datasets, while in PRAI-1581 dataset, we have improved Rank-1score and mAP by 3.7% and 6.5% respectively over OSNet, and by 1.5% and 1.2% respectively over CDNet. This shows that the present invention is more suitable for UAV-ReID and performs better in datasets taken by ground cameras.

[0106] Table 2 Performance of the model on PRAI-1581, Market-1501 and DukeMTMC datasets

[0107]

[0108] All models are trained from scratch. After one round of training, the network outputs the learned features and passes them into the loss function. The loss function calculates the gradient by calculating and outputting a loss value, and then performs back propagation to update the network parameters. After a certain number of rounds of training, the network performance tends to be stable. At this time, the network needs to be tested for the final time. The pictures used in the test process are the test set and the gallery set in the data set. By reading the test set pictures, people with the same identity are found from the gallery set pictures. Observe the values ​​of CMC and mAP and the number of parameters as the evaluation criteria for network performance. As shown in Table 2, in the table, * indicates the reproduced results, and - indicates that the results are unavailable. It can be seen that the PBWA-Net model is ahead of other models on the PRAI-1581 dataset and slightly stronger than other models on other datasets. Note: The experimental results are obtained by averaging the results of ten trials.

[0109] In order to verify that PBWA-Net can learn discriminative areas and has strong resistance to occlusion, four pedestrian images are randomly selected to input into the network and visualize the final output features, such as Figure 5 As shown in the figure, from left to right, each row of images are: original pedestrian image, grayscale feature map, heat map, and heat map superimposed with the original image. It can be seen that PBWA-Net mainly focuses on learning the pedestrian's body parts. For example, in the top row of pictures, the method pays more attention to the pattern of the pedestrian's top. For the case of occlusion, such as the bottom row of pedestrian pictures, the method can ignore the occluded parts and focus on learning the unoccluded parts. Through visualization, it can be seen that the parallel network structure can make correct judgments using limited information even when there is occlusion that makes most of the pedestrian's body unavailable. At the same time, it can be found that the network is more inclined to pay attention to the unique features of pedestrians (such as top patterns, items carried, etc.). Thanks to the weighted attention module, when occlusion occurs in the picture, it can also automatically pay attention to the unoccluded parts, thereby ignoring the meaningless occluded part features. Therefore, PBWA-Net can effectively capture robustness information and ignore noise interference caused by environmental factors.

[0110] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0111] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A pedestrian re-identification method from an aerial perspective applied to a drone, characterized in that: The following steps are involved: Obtain target pedestrian images and image datasets to be searched from an aerial perspective; For the target pedestrian image and each image to be searched, a pre-trained UAV pedestrian re-identification network is used to extract features; According to the similarity between the features of the target pedestrian image and each image to be searched, determining whether there is an image of the target pedestrian in the image data set to be searched; The UAV pedestrian re-identification network includes a multi-scale convolution module and a weighted attention module. The multi-scale convolution module includes a backbone network and one or more parallel networks. The backbone network includes multiple groups of depth-separable residual convolution blocks connected in sequence, and a downsampling layer is provided between each two groups of depth-separable residual convolution blocks; and the output end of each downsampling layer of the backbone network is again connected to a parallel network via a downsampling layer; the parallel network includes one or more groups of depth-separable residual convolution blocks connected in sequence; Information is exchanged at the third and fifth layers of the backbone network: the input of each subnetwork at the fourth layer is the sum of the outputs of the two subnetworks at the third layer, and the input of each subnetwork at the sixth layer is the sum of the outputs of the three subnetworks at the fifth layer. The conversion formula for the exchange process is: Where M represents the Mth layer of the subnetwork, 1 represents the first branch, and the values ​​of 1 and M are determined by the network structure; The weighted attention module formula can be expressed as: Among them, S represents the number of branch networks; AN represents the attention network in the figure, and F in Represents the input features, which are taken as the output of different branches in LRBlock or the output of the sixth layer of the drone pedestrian re-identification network. represents the Hadamard product of two characteristic matrices.

2. The method for pedestrian re-identification from an aerial perspective applied to a drone as claimed in claim 1, characterized in that: In the parallel stage of the backbone network and one or more parallel networks, for each group of depth-wise separable residual convolution blocks of different network layers, the output ends of the previous depth-wise separable residual convolution blocks are connected to the subsequent depth-wise separable residual convolution blocks of this group and other network layers.

3. The method for pedestrian re-identification from an aerial perspective applied to a drone as claimed in claim 1, characterized in that: The depth-separable residual convolution block includes a multi-scale convolution module and a weighted attention module. The multi-scale convolution module includes a trunk branch and a residual branch. The trunk branch and the residual branch each include a plurality of depth-separable convolution units connected in sequence.

4. The method for pedestrian re-identification from an aerial perspective applied to a drone as claimed in claim 3, characterized in that: The depthwise separable convolution unit uses a depthwise separable convolution unit with a receptive field size of 3×3.

5. The method for pedestrian re-identification from an aerial perspective applied to a drone as claimed in claim 1 or 3, characterized in that: The attention module includes a multi-layer attention network, each layer of the attention network includes a pooling layer, a first convolution layer, a normalization layer, a first activation function, a second convolution layer and a second activation function.

6. The method for pedestrian re-identification from an aerial perspective applied to a drone as claimed in claim 1, characterized in that: The output end of the weighted attention module is connected to the global average pooling layer, the linear layer, the batch normalization layer and the activation function in sequence. The output of the activation function is input into the triplet loss function on the one hand, and is input into the cross entropy loss function via the fully connected layer on the other hand. The loss function of the UAV pedestrian re-identification network is the sum of the triplet loss function and the cross entropy loss function.

7. The method for pedestrian re-identification from an aerial perspective applied to a drone as claimed in claim 1, characterized in that: The UAV pedestrian re-identification network adopts the loss function to calculate the gradient and updates the network parameters through back propagation.

8. A pedestrian re-identification system from an aerial perspective applied to a drone, characterized in that: include: A data acquisition module is used to acquire target pedestrian images and image data sets to be searched from an aerial perspective; A feature extraction module, used for extracting features from the target pedestrian image and each image to be searched using a pre-trained drone pedestrian re-identification network; A pedestrian re-identification module is used to determine whether there is an image of the target pedestrian in the image data set to be searched based on the similarity between the target pedestrian image and the features of each image to be searched; The UAV pedestrian re-identification network includes a multi-scale convolution module and a weighted attention module. The multi-scale convolution module includes a backbone network and one or more parallel networks. The backbone network includes multiple groups of depth-separable residual convolution blocks connected in sequence, and a downsampling layer is provided between each two groups of depth-separable residual convolution blocks; and the output end of each downsampling layer of the backbone network is again connected to a parallel network via a downsampling layer; the parallel network includes one or more groups of depth-separable residual convolution blocks connected in sequence; Information is exchanged at the third and fifth layers of the backbone network: the input of each subnetwork at the fourth layer is the sum of the outputs of the two subnetworks at the third layer, and the input of each subnetwork at the sixth layer is the sum of the outputs of the three subnetworks at the fifth layer. The conversion formula for the exchange process is: Where M represents the Mth layer of the subnetwork, 1 represents the first branch, and the values ​​of 1 and M are determined by the network structure; The weighted attention module formula can be expressed as: Among them, S represents the number of branch networks; AN represents the attention network in the figure, and F in Represents the input features, which are taken as the output of different branches in LRBlock or the output of the sixth layer of the drone pedestrian re-identification network. represents the Hadamard product of two characteristic matrices.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the pedestrian re-identification method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the pedestrian re-identification method as described in any one of claims 1 to 7 is implemented.

Images