Segmentation-guided attention group counting method for UAV aerial images

A segmentation-guided attention-based crowd counting method, which generates density maps and segmentation maps by training convolutional neural networks, solves the problem of inaccurate crowd counting in drone aerial images and achieves efficient and accurate crowd counting in complex backgrounds.

CN115019211BActive Publication Date: 2024-11-26BEIJING INST OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210740932.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-28
Publication Date
2024-11-26
Estimated Expiration
2042-06-28

AI Technical Summary

Technical Problem

Existing methods for counting crowds in drone aerial images are inaccurate in complex backgrounds, especially under occlusion and background noise interference, leading to misjudgments and inaccurate counting results.

Method used

A segmentation-guided attention-based crowd counting method for UAV aerial images is adopted. A density map and a segmentation map are generated by training a convolutional neural network. Multi-layer segmentation-guided attention modules and adders are used to perform feature weighted summation. The network is optimized by combining the loss functions of the density map and the segmentation map to generate accurate crowd counts.

Benefits of technology

It improves the accuracy and robustness of crowd counting in drone aerial images, reduces the impact of occlusion and background noise, and achieves efficient and accurate crowd counting.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115019211B_ABST
    Figure CN115019211B_ABST
Patent Text Reader

Abstract

The present invention provides a method for counting groups of drone aerial images by segmentation-guided attention. The method comprises: obtaining an image set including a training set and a test set; preprocessing the images in the training set; inputting the preprocessed images into a network, training the network until the network is fitted, and outputting a density map and a segmentation map of the trained network output image, wherein the network comprises a convolution block, an Inception cascade module, an upsampler, a shallow segmentation-guided attention module, a deep segmentation-guided attention module, and a first and a second adder; and according to the density map output by the network, summing the pixel values ​​of the density map to obtain the group count in the image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an image processing technology, and more specifically, to a segmentation-guided attention group counting method for unmanned aerial vehicle aerial images. Background Art

[0002] With the advancement of urbanization and the construction of smart cities, the population in cities is growing rapidly, and large-scale group activities are becoming more and more common. When the crowd density is too high, it is easy to cause crowd congestion and even stampedes, leading to serious safety accidents. In recent years, with the continuous expansion and increasing popularity of the drone market, using drones to analyze and predict crowd behavior from the air has gradually become one of the important means of security. This technology uses features extracted from video frame images to estimate the number of people in the current scene.

[0003] Traditional crowd density estimation algorithms can be divided into three categories: detection method, regression method and density estimation-based method. Detection-based methods usually use a sliding window to traverse the entire image, gradually use the sliding window to detect the crowd in the window, and add the detection results of all windows to get the counting result. However, when the crowd is severely occluded or the background is complex, the detection-based crowd counting algorithm cannot get accurate results. The regression-based method does not detect and locate each individual one by one, but completes the counting by mapping the relationship between the artificially designed features and the final counting results. Since the regression-based method only gives the final counting result and cannot accurately reflect the distribution of the crowd in the scene, the neglect of spatial information makes it insufficient to support practical applications. The density estimation-based method achieves counting by learning the mapping relationship between features and density maps, but due to insufficient extracted features and complex and diverse mapping relationships, the final effect is poor.

[0004] With the rapid development of deep learning, convolutional neural networks have been widely used in computer vision fields such as image classification and target detection due to their excellent learning ability of image features, and have also shown unique advantages in group counting.

[0005] Although deep learning has brought great progress to crowd counting, there are still some unresolved problems, such as the occlusion and interference of complex backgrounds on the crowd. Due to the visual characteristics of near and far objects, when drones take pictures in the air, adjacent people in the same scene will block each other, or even be blocked by other buildings or objects, resulting in inaccurate final counting results and affecting the final effect. Background pixel noise near the crowd will also interfere with the model's feature extraction, resulting in misjudgment and affecting the accuracy of counting.

[0006] Therefore, an efficient and accurate population counting method is needed. Summary of the invention

[0007] The present invention is proposed to solve the above-mentioned problems, and its purpose is to provide a segmentation-guided attention group counting method for drone aerial images.

[0008] In order to achieve the above-mentioned purpose of the invention, according to one aspect of the present invention, a segmentation-guided attention group counting method for unmanned aerial vehicle aerial images is provided, the method comprising: obtaining an image set including a training set and a test set; preprocessing the images in the training set; inputting the preprocessed images into a network, training the network until the network is fitted, and the trained network outputs a density map and a segmentation map of the image, the network comprising a convolution block, an Inception cascade module, an upsampler, a shallow segmentation-guided attention module, a deep segmentation-guided attention module, and a first and a second adder; and according to the density map output by the network, summing the pixel values ​​of the density map to obtain the group count in the image.

[0009] The steps to train the network include:

[0010] The convolution block processes the preprocessed image into feature F1, and the number of channels of F1 is 768;

[0011] The feature F1 is input into the Inception cascade module and the shallow segmentation guided attention module to obtain features F2, S1 and D1 respectively. The number of channels of feature F2 is 2048, the number of channels of S1 is 1, and the number of channels of feature D1 is 1;

[0012] The obtained feature F2 is upsampled by the upsampler, and the size of the upsampled feature becomes twice that of the feature F2. Then the upsampled feature F2 is input into the deep segmentation guided attention module to obtain features S2 and D2 respectively. The number of channels of feature S2 and feature D2 are 1 and 1 respectively.

[0013] The features S1 and S2 are weighted and summed by the first adder, and finally a predicted segmentation map with the same size as the input image is obtained;

[0014] The features D1 and D2 are weighted and summed by the second adder, and finally a predicted density map with the same size as the input image is obtained;

[0015] By adjusting the weights of features S1 and S2 as well as features D1 and D2 to minimize the loss function until the network is fit,

[0016] The loss function is:

[0017]

[0018] Represents the density map loss, as shown in formula (2),

[0019]

[0020] in Represents the generated i-th input image X i The corresponding prediction density map, θ represents the trainable parameters of the entire network, D i Represents image X i The true value density map, N is the number of training samples,

[0021] represents the segmentation map loss, as shown in formula (3),

[0022]

[0023]

[0024] in Represents the generated i-th input image X i The corresponding predicted segmentation map, W i represents the true value segmentation map, H i (j, k) constitute the cross entropy loss matrix H, (j, k) is the coordinate of each point in the predicted segmentation map, and ⊙ represents element-by-element multiplication.

[0025] The convolution block includes 5 convolution layers, the first to third InceptionA, InceptionB, and the first to fourth InceptionC.

[0026] The 5 convolutional layers are:

[0027] Conv:3×3,32;

[0028] Conv:3×3,32;

[0029] Conv:3×3,64;

[0030] Conv:3×3,80;

[0031] Conv:3×3,192;

[0032] Among them, Conv is convolution, 3×3,32 means the convolution kernel size is 3×3, and the number of channels of the output feature is 32; 3×3,64 means the convolution kernel size is 3×3, and the number of channels of the output feature is 64; 3×3,80 means the convolution kernel size is 3×3, and the number of channels of the output feature is 80; 3×3,192 means the convolution kernel size is 3×3, and the number of channels of the output feature is 192;

[0033] InceptionA: Duplicate the input features into 4 copies and send them to 4 branches respectively. From left to right, branch A1 performs 1×1 convolution on the input features; branch A2 first performs 1×1 convolution and then 5×5 convolution on the input features; branch A3 first performs 1×1 convolution on the input features, and then performs two 3×3 convolutions; branch A4 first performs pooling operation on the input features and then performs 1×1 convolution. Finally, all the outputs of the 4 branches are connected according to the channels as the final output features, and the number of channels is 288;

[0034] InceptionB: The input features are copied three times and sent to three branches respectively. From left to right, branch B1 performs 3×3 convolution on the input features; branch B2 first performs 1×1 convolution on the input features, and then performs two 3×3 convolutions; branch B3 performs pooling operation on the input features, and finally connects all the outputs of the three branches according to the channels as the final output features, with a channel number of 768;

[0035] InceptionC: The input features are copied 4 times and sent to 4 branches respectively. From left to right, branch C1 performs 1×1 convolution on the input features; branch C2 first performs 1×1 convolution on the input and then performs depthwise separable convolution, where the depthwise separable convolution consists of 1×7 and 7×1 convolutions; branch C3 first performs 1×1 convolution on the input features, and then performs two depthwise separable convolutions, where the depthwise separable convolution consists of 1×7 and 7×1 convolutions; branch C4 performs pooling operation on the input features and then performs 1×1 convolution; finally, all the outputs of the four branches are connected according to the channels as the final output features, and the number of channels is 768.

[0036] Inception cascade modules include InceptionD and InceptionE.

[0037] InceptionD: The input features are copied three times and sent to three branches respectively. From left to right, branch D1 first performs 1×1 convolution on the input features, and then performs 3×3 convolution; branch D2 first performs 1×1 convolution on the input features, and then performs depthwise separable convolution, where the depthwise separable convolution consists of 1×7 and 7×1 convolutions, and finally performs 3×3 convolution; branch D3 performs pooling operation on the input features, and finally connects all the outputs of the three branches according to the channels as the final output features, and the number of channels is 1280;

[0038] InceptionE: The input features are copied 4 times and sent to 4 branches respectively. From left to right, branch E1 first performs 1×1 and 3×3 convolutions on the input features, and then passes through two parallel convolutions, namely 1×3 and 3×1 convolutions; branch E2 first performs 1×1 convolution on the input features and then passes through two parallel convolutions, namely 1×3 and 3×1 convolutions; branch E3 first performs pooling operation on the input features and then performs 1×1 convolution; branch E4 performs 1×1 convolution on the input features; finally, all the outputs of the 4 branches are connected according to the channels as the final output feature F2, whose number of channels is 2048.

[0039] The shallow segmentation guided attention module includes an attention branch and a crowd counting branch.

[0040] The attention branches include:

[0041] Convolution layer Conv: 1×1, 1, Conv means convolution, 1×1, 1 means the convolution kernel size is 1×1, and the number of channels of the output feature is 1;

[0042] Sigmoid function;

[0043] Crowd counting branches include:

[0044] InceptionC, the number of channels of output features is 2048;

[0045] Multiplier, pixel-by-pixel multiplication;

[0046] Convolutional layer: 1×1, 1.

[0047] The deep segmentation-guided attention module also includes an attention branch and a crowd counting branch.

[0048] Preprocessing the images in the training set includes: limiting the height and width of the images to be divisible by 8, and adding the images in the training set after randomly flipping the images in the training set to the training set, so as to increase the number of images in the training set.

[0049] According to another aspect of the present invention, there is provided a computer program product encoded on a computer storage medium, comprising instructions which, when executed by one or more computers, will cause the one or more computers to perform the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flowchart of a segmentation-guided attention group counting method for drone aerial images according to an embodiment of the present disclosure is shown;

[0051] Figure 2 A network structure diagram according to an embodiment of the present disclosure is shown;

[0052] Figure 3 Shows the InceptionA~E structure diagram;

[0053] Figure 4 A schematic diagram showing a shallow segmentation guided attention module according to an embodiment of the present disclosure;

[0054] Figure 5 A schematic diagram showing a deep segmentation guided attention module according to an embodiment of the present disclosure; and

[0055] Figure 6 2 shows the test results of 4 images obtained according to the method disclosed in the present invention. DETAILED DESCRIPTION

[0056] Below, with reference to the accompanying drawings, embodiments of the present disclosure are described. In the accompanying drawings, the same reference numerals are used for identical or corresponding structural elements. In addition, in the description of the following embodiments, repeated descriptions of identical or corresponding structural elements may be omitted. However, even if the description of structural elements is omitted, it does not mean that those structural elements are not included in a certain embodiment.

[0057] Although the process steps, method steps, and algorithms are described in order in the flowcharts illustrated in this specification, these processes, methods, and algorithms can be configured to operate in any suitable order. In other words, the process steps, method steps, and algorithm steps described in various embodiments of the present disclosure do not need to be performed in the order described in the present disclosure.

[0058] In addition, even if some steps are described as being performed non-simultaneously, these steps may be performed simultaneously in other embodiments. In addition, the examples of the processes described by the drawings do not mean that the illustrated processes exclude other changes and modifications thereto, and do not mean that the illustrated processes or any one of their steps are essential to one or more of the various embodiments of the present disclosure, and do not mean that the illustrated processes are preferred.

[0059] Figure 1 A flowchart of a segmentation-guided attention group counting method for drone aerial images according to an embodiment of the present disclosure is shown.

[0060] In step S101, a data set is obtained. In the step of obtaining a data set, a public data set is first downloaded. The public data set includes a divided training set and a test set. The training set and the test set are independent of each other and are both subsets of the public data set.

[0061] Then, in step S102, the images in the training set are preprocessed. By preprocessing the images in the training set, the image data can be better adapted to the network of the present disclosure and the computational cost of the algorithm can be reduced to a certain extent.

[0062] In the preprocessing step of the image, the height and width of the image are first limited to be divisible by 8. This ensures the correctness of the output after the image enters the layer (Maxpool layer) for downsampling operation in the network of the present disclosure. Then the average image of the entire data set is subtracted from each image and normalized to ensure that each pixel value falls in [0,1].

[0063] In order to expand the number of images in the training set, it is necessary to perform a random flip operation on the images in the training set. The images in the training set that have been randomly flipped are added to the training set, thereby increasing the number of images in the training set.

[0064] After preprocessing the images in the training set in step S102, the process proceeds to step S103, where a batch of images in the training set are selected, the selected images are input into the network, and the network is trained until the network is fitted.

[0065] This method uses image cropping training method to crop the image X of the training set. i Randomly cropping into 8 128×128 pixel blocks makes the network input more diverse and helps alleviate the overfitting problem.

[0066] The cropped image X i enter Figure 2 Specifically, refer to Figure 2 The network structure diagram of the training network is described. Figure 2 As shown in the figure, the entire network mainly includes: convolution block, Inception cascade module, upsampler, shallow segmentation guided attention module, deep segmentation guided attention module and 2 adders.

[0067] Eight 128×128 pixel blocks are first input into the convolution block. Since the images in the training set are RGB images, the number of input channels is 3. The image passes through the convolution block to obtain feature F1, and the number of channels of F1 is 768.

[0068] The convolution block includes 5 convolution layers, the first to third InceptionA, InceptionB, and the first to fourth InceptionC.

[0069] The 5 convolutional layers are:

[0070] Conv:3×3,32;

[0071] Conv:3×3,32;

[0072] Conv:3×3,64;

[0073] Conv:3×3,80;

[0074] Conv:3×3,192;

[0075] Among them, in Conv:3×3,32, Conv is convolution, 3×3,32 means that the convolution kernel size is 3×3, and the number of channels of the output feature is 32; in Conv:3×3,64, Conv is convolution, 3×3,64 means that the convolution kernel size is 3×3, and the number of channels of the output feature is 64; in Conv:3×3,80, Conv is convolution, 3×3,80 means that the convolution kernel size is 3×3, and the number of channels of the output feature is 80; in Conv:3×3,192, Conv is convolution, 3×3,192 means that the convolution kernel size is 3×3, and the number of channels of the output feature is 192.

[0076] The structures of InceptionA, InceptionB and InceptionC are as follows Figure 3 As shown in the figure, the number of channels of the output feature map of InceptionA is 288; the number of channels of the output feature map of InceptionB is 768; the number of channels of the output feature map of InceptionC is 768.

[0077] InceptionA: The input features are copied 4 times and sent to 4 branches respectively. From left to right, branch A1 performs 1×1 convolution on the input features; branch A2 first performs 1×1 convolution and then 5×5 convolution on the input features; branch A3 first performs 1×1 convolution on the input features, and then performs two 3×3 convolutions; branch A4 first performs pooling operation on the input features and then performs 1×1 convolution. Finally, all the outputs of the 4 branches are connected according to the channels as the final output features, and the number of channels is 288.

[0078] InceptionB: The input features are copied three times and sent to three branches respectively. From left to right, branch B1 performs 3×3 convolution on the input features; branch B2 first performs 1×1 convolution on the input features, and then performs two 3×3 convolutions; branch B3 performs pooling operation on the input features, and finally connects all the outputs of the three branches according to the channels as the final output features, with a channel number of 768.

[0079] The first to fourth InceptionC structures are the same. Figure 3 They are collectively referred to as InceptionC.

[0080] InceptionC: The input features are copied 4 times and sent to 4 branches respectively. From left to right, branch C1 performs 1×1 convolution on the input features; branch C2 first performs 1×1 convolution on the input and then performs depthwise separable convolution, where the depthwise separable convolution consists of 1×7 and 7×1 convolutions; branch C3 first performs 1×1 convolution on the input features, and then performs two depthwise separable convolutions, where the depthwise separable convolution consists of 1×7 and 7×1 convolutions; branch C4 performs pooling operation on the input features and then performs 1×1 convolution; finally, all the outputs of the four branches are connected according to the channels as the final output features, and the number of channels is 768.

[0081] The feature F1 obtained by the convolution block is input into the Inception cascade module and the shallow segmentation guided attention module respectively to obtain features F2, S1 and D1 respectively. The number of channels of feature F2 is 2048, the number of channels of feature S1 is 1, and the number of channels of feature D1 is 1.

[0082] The Inception cascade module includes InceptionD and InceptionE. The number of output feature map channels of InceptionD is 1280; the number of output feature channels of InceptionE is 2048.

[0083] InceptionD: The input features are copied three times and sent to three branches respectively. From left to right, branch D1 first performs 1×1 convolution on the input features, and then performs 3×3 convolution; branch D2 first performs 1×1 convolution on the input features, and then performs depthwise separable convolution, where the depthwise separable convolution consists of 1×7 and 7×1 convolutions, and finally performs 3×3 convolution; branch D3 performs pooling on the input features, and finally connects all the outputs of the three branches according to the channels as the final output features, with 1280 channels.

[0084] InceptionE: The input features are copied 4 times and sent to 4 branches respectively. From left to right, branch E1 first performs 1×1 and 3×3 convolutions on the input features, and then passes through two parallel convolutions, namely 1×3 and 3×1 convolutions; branch E2 first performs 1×1 convolution on the input features and then passes through two parallel convolutions, namely 1×3 and 3×1 convolutions; branch E3 first performs pooling operation on the input features and then performs 1×1 convolution; branch E4 performs 1×1 convolution on the input features; finally, all the outputs of the 4 branches are connected according to the channels as the final output feature F2, whose number of channels is 2048.

[0085] Feature F1 is processed by the Inception cascade module to obtain feature F2, and the number of channels of F2 is 2048.

[0086] On the other side, the feature F1 obtained by the convolution block is input into the shallow segmentation-guided attention module.

[0087] Shallow segmentation guides the attention module such as Figure 4 As shown, the shallow segmentation-guided attention module includes an attention branch and a crowd counting branch.

[0088] The attention branches include:

[0089] Convolution layer Conv:1×1,1, in Conv:1×1,1, Conv means convolution, 1×1,1 means the convolution kernel size is 1×1, and the number of channels of the output feature is 1;

[0090] Sigmoid function;

[0091] Crowd counting branches include:

[0092] InceptionC, the number of channels of output features is 2048;

[0093] Multiplier, pixel-by-pixel multiplication;

[0094] Convolution layer Conv:1×1,1. In Conv:1×1,1, Conv means convolution, 1×1,1 means that the convolution kernel size is 1×1 and the number of channels of the output feature is 1.

[0095] In the shallow segmentation-guided attention module, the segmentation truth map is used to supervise the learning of the attention map in the attention branch, and the density truth map is used to supervise the learning of the density estimation map in the crowd counting branch. The conventional attention mechanism has high computational complexity and requires three inputs. In order to improve the computational efficiency while meeting the dynamic prediction characteristics of the segmentation task, this algorithm only uses one input feature to generate a predicted segmentation map, which is supervised and updated by the truth segmentation map.

[0096] The shallow segmentation guided attention module is processed as follows:

[0097] Feature F1 is input to the attention branch. The number of channels of feature F1 is 768. It passes through the convolution kernel (Conv: 1×1, 1) and then passes through the Sigmoid function to output feature S1. The value of feature S1 is between 0 and 1, and the number of channels is 1.

[0098] Feature F1 is also input into the crowd counting branch. The number of channels of feature F1 is 768. Feature V1 is obtained through InceptionC. The number of channels of feature V1 is 2048. The obtained feature V1 is multiplied pixel by pixel with the feature S1 output by the attention branch to obtain feature M1. The number of channels of feature M1 is 2048. M1 is passed through the convolution kernel (Conv: 1×1, 1) to obtain feature D1, which has a channel number of 1 and the same size as F1.

[0099] The obtained feature F2 is upsampled, and the size of the upsampled feature becomes twice that of the feature F2. Then the upsampled feature F2 is input into the deep segmentation guided attention module to obtain features S2 and D2 respectively. The number of channels of feature S2 and feature D2 are 1 and 1 respectively.

[0100] Deep segmentation guides the attention module such as Figure 5 As shown in Figure 1, the deep segmentation guided attention module has the same structure as the shallow segmentation guided attention module, and the only difference is the location in the network. The deep segmentation guided attention module also includes an attention branch and a crowd counting branch.

[0101] The attention branches include:

[0102] Convolution layer Conv:1×1,1, in Conv:1×1,1, Conv means convolution, 1×1,1 means the convolution kernel size is 1×1, and the number of channels of the output feature is 1;

[0103] Sigmoid function;

[0104] Crowd counting branches include:

[0105] InceptionC, the number of channels of output features is 2048;

[0106] Multiplier, pixel-by-pixel multiplication;

[0107] Convolution layer Conv:1×1,1. In Conv:1×1,1, Conv means convolution, 1×1,1 means that the convolution kernel size is 1×1 and the number of channels of the output feature is 1.

[0108] Similarly, in the deep segmentation-guided attention module, the segmentation truth map is used to supervise the learning of the attention map in the attention branch, and the density truth map is used to supervise the learning of the density estimation map in the crowd counting branch. The conventional attention mechanism has high computational complexity and requires three inputs. In order to improve the computational efficiency while meeting the dynamic prediction characteristics of the segmentation task, this algorithm only uses one input feature to generate the predicted segmentation map, which is supervised and updated by the truth segmentation map.

[0109] The processing of the deep segmentation-guided attention module is as follows:

[0110] Feature F2 is input to the attention branch as input. The number of channels of feature F2 is 2048. It passes through the convolution kernel (Conv: 1×1, 1) and then passes through the Sigmoid function to output feature S2. The value of feature S2 is between 0 and 1, and the number of channels is 1.

[0111] Feature F2 is input into the crowd counting branch. The number of channels of feature F2 is 2048. It is passed through InceptionC to obtain feature V2. The number of channels of feature V2 is 2048. The obtained feature V2 is multiplied pixel by pixel with the output S2 of the attention branch to obtain feature M2. The number of channels of feature M2 is 2048. M2 is passed through the convolution kernel (Conv: 1×1, 1) to obtain feature D2. The number of channels is 1, and the size is the same as F2.

[0112] The obtained features S1 and S2 are weighted and summed to finally obtain a predicted segmentation map with the same size as the input image, where the weights of features S1 and S2 are learned by the network, which will be described in detail below.

[0113] The obtained features D1 and D2 are weighted and summed to finally obtain a predicted density map with the same size as the input image, where the weights of features D1 and D2 are learned by the network, which will be described in detail below.

[0114] During the training process, the loss function consists of two parts: density map loss and segmentation map loss, as shown in formula (1). The network parameters are adjusted to the optimal value by minimizing the loss function.

[0115]

[0116] Using Euclidean distance as the density map loss function, Represents the density map loss, as shown in formula (2),

[0117]

[0118] in Represents the generated i-th input image X i The corresponding prediction density map, θ represents the trainable parameters of the entire network, D i Represents image X i The true value density map, N is the number of training samples, that is, the number of images used in each training. i and N is the data available from the training set.

[0119] Using cross entropy loss as the segmentation map loss function, represents the segmentation map loss, as shown in formula (3),

[0120]

[0121]

[0122] in Represents the generated i-th input image X i The corresponding predicted segmentation map, W i represents the true value segmentation map, H i(j, k) constitutes the cross entropy loss matrix H, (j, k) is the coordinate of each point in the segmentation map. ⊙ represents element-by-element multiplication.

[0123] The true value density map and the true value segmentation map are generated by point annotation. Specifically, the true value density map is centered on the coordinates of each head and draws a two-dimensional Gaussian distribution with a standard deviation of 4 and a kernel size of 15×15. The true value segmentation map is centered on the coordinates of each head and draws a 25×25 rectangular area with all values ​​1 in the rectangular area.

[0124] The weights of features D1 and D2 are obtained by minimizing the density map loss, and the weights of features S1 and S2 are obtained by minimizing the segmentation map loss.

[0125] After the weights are determined, the network has been trained.

[0126] In step S104, based on the density map output by the trained network, the density values ​​of all pixels in the input image are summed to obtain the population count in the image.

[0127] After the network training is completed, the images of the test set can be input into the network for testing to obtain the predicted density map, predicted segmentation map and crowd quantity.

[0128] When the test set is input into the multi-branch segmentation guided attention network to obtain the group counting result, the pixel values ​​of the density map are summed up according to the generated density map to obtain the number of people in the image, and the generated file is named "dataset name-image index number_pred_estimated number of people.png", such as "SHHA-50_pred_1000.596640625.png". The generated predicted segmentation map can be used to assist in judging the accuracy of the prediction results.

[0129] The results of testing the images in the test set using the network disclosed in this disclosure are as follows: Figure 6 shown. Figure 6 The test results of 4 images are shown in FIG. 1 , where the first column is the input image, the second column is the actual number of people, and the third column is the number of people estimated using the method disclosed in the present invention. It can be seen that the accuracy of group counting performed by the network disclosed in the present invention is very high.

[0130] The present invention proposes a multi-branch segmentation-guided attention module, which improves the network's ability to learn multi-scale features; an attention branch is added to the deep and shallow layers of the network, and foreground segmentation is used to enhance the network's attention to the crowd area.

[0131] Without increasing the cost of additional annotation, it provides supervision information for group counting and foreground segmentation of multi-task learning, integrates Euclidean distance loss and cross entropy loss as the loss function of the network, improves the utilization rate of point annotation, and promotes each other among multiple tasks, thus improving network performance.

[0132] The embodiments of the present disclosure can be written as computer programs and can be implemented in general-use digital computers that execute the programs using a computer readable recording medium.

[0133] In this state, the medium can continuously store programs that can be executed by the computer, or can temporarily store programs for execution or downloading. In addition, the medium can be a variety of recording devices or storage devices that combine single or multiple hardware, not limited to media that are directly connected to the computer system, and can exist in a distributed manner on the network. Examples of media include magnetic storage media (such as floppy disks or hard disks) configured to store program instructions, optical recording media (such as CD-ROMs or DVDs), magneto-optical media (such as floppy disks), and Rom, RAM, flash memory, etc. In addition, examples of other media may include: application stores for distributing applications, sites for providing or distributing various other software, and recording media or storage media managed at servers.

[0134] Although the present disclosure has been specifically shown and described with reference to preferred embodiments using specific terms, the embodiments and terms should be considered in a descriptive sense only and not for the purpose of limitation. Therefore, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope of the present disclosure as defined by the appended claims.

Claims

1. A segmentation-guided attention group counting method for drone aerial images, the method comprising: Obtain an image set including a training set and a test set; Preprocess the images in the training set; Inputting the preprocessed image into the network, training the network until the network is fitted, and the trained network outputs a density map and a segmentation map of the image, wherein the network includes a convolution block, an Inception cascade module, an upsampler, a shallow segmentation guided attention module, a deep segmentation guided attention module, and a first adder and a second adder; as well as According to the density map output by the network, the pixel values ​​of the density map are summed to get the population count in the image. The steps of training the network include: The convolution block processes the preprocessed image into feature F1, and the number of channels of F1 is 768; The feature F1 is input into the Inception cascade module and the shallow segmentation guided attention module to obtain features F2, S1 and D1 respectively. The number of channels of feature F2 is 2048, the number of channels of S1 is 1, and the number of channels of feature D1 is 1; The obtained feature F2 is upsampled by the upsampler, and the size of the upsampled feature becomes twice that of the feature F2. Then the upsampled feature F2 is input into the deep segmentation guided attention module to obtain features S2 and D2 respectively. The number of channels of feature S2 and feature D2 are 1 and 1 respectively. The features S1 and S2 are weighted and summed by the first adder, and finally a predicted segmentation map with the same size as the input image is obtained; The features D1 and D2 are weighted and summed by the second adder, and finally a predicted density map with the same size as the input image is obtained; By adjusting the weights of features S1 and S2 as well as features D1 and D2 to minimize the loss function until the network is fit, The loss function is: Among them, η is the weight coefficient, Represents the density map loss, as shown in formula (2), in Represents the generated i-th input image X i The corresponding prediction density map, θ represents the trainable parameters of the entire network, D i Represents image X i The true value density map, N is the number of training samples, represents the segmentation map loss, as shown in formula (3), in Represents the generated i-th input image X i The corresponding predicted segmentation map, W i represents the true value segmentation map, H i (j, k) constitute the cross entropy loss matrix H, (j, k) is the coordinate of each point in the predicted segmentation map, and ⊙ represents element-by-element multiplication.

2. The method according to claim 1, characterized in that The convolution block includes 5 convolution layers, the first InceptionA to the third InceptionA, InceptionB, and the first InceptionC to the fourth InceptionC. The 5 convolutional layers are: Conv:3×3,32; Conv:3×3,32; Conv:3×3,64; Conv:3×3,80; Conv:3×3,192; Among them, Conv is convolution, 3×3,32 means the convolution kernel size is 3×3, and the number of channels of the output feature is 32; 3×3,64 means the convolution kernel size is 3×3, and the number of channels of the output feature is 64; 3×3,80 means the convolution kernel size is 3×3, and the number of channels of the output feature is 80; 3×3,192 means the convolution kernel size is 3×3, and the number of channels of the output feature is 192; InceptionA: Duplicate the input features into 4 copies and send them to 4 branches respectively. From left to right, branch A1 performs 1×1 convolution on the input features; branch A2 first performs 1×1 convolution and then 5×5 convolution on the input features; branch A3 first performs 1×1 convolution on the input features, and then performs two 3×3 convolutions; branch A4 first performs pooling operation on the input features and then performs 1×1 convolution. Finally, all the outputs of the 4 branches are connected according to the channels as the final output features, and the number of channels is 288; InceptionB: The input features are copied three times and sent to three branches respectively. From left to right, branch B1 performs 3×3 convolution on the input features; branch B2 first performs 1×1 convolution on the input features, and then performs two 3×3 convolutions; branch B3 performs pooling operation on the input features, and finally connects all the outputs of the three branches according to the channels as the final output features, with a channel number of 768; InceptionC: The input features are copied 4 times and sent to 4 branches respectively. From left to right, branch C1 performs 1×1 convolution on the input features; branch C2 first performs 1×1 convolution on the input features and then performs depthwise separable convolution, where the depthwise separable convolution consists of 1×7 and 7×1 convolutions; branch C3 first performs 1×1 convolution on the input features, and then performs two depthwise separable convolutions, where the depthwise separable convolution consists of 1×7 and 7×1 convolutions; branch C4 performs pooling operation on the input features and then performs 1×1 convolution; finally, all the outputs of the 4 branches are connected according to the channels as the final output features, and the number of channels is 768.

3. The method according to claim 2, characterized in that Inception cascade modules include InceptionD and InceptionE. InceptionD: The input features are copied three times and sent to three branches respectively. From left to right, branch D1 first performs 1×1 convolution on the input features, and then performs 3×3 convolution; branch D2 first performs 1×1 convolution on the input features, and then performs depthwise separable convolution, where the depthwise separable convolution consists of 1×7 and 7×1 convolutions, and finally performs 3×3 convolution; branch D3 performs pooling operation on the input features, and finally connects all the outputs of the three branches according to the channels as the final output features, and the number of channels is 1280; InceptionE: The input features are copied 4 times and sent to 4 branches respectively. From left to right, branch E1 first performs 1×1 and 3×3 convolutions on the input features, and then passes through two parallel convolutions, namely 1×3 and 3×1 convolutions; branch E2 first performs 1×1 convolution on the input features and then passes through two parallel convolutions, namely 1×3 and 3×1 convolutions; branch E3 first performs pooling operation on the input features and then performs 1×1 convolution; branch E4 performs 1×1 convolution on the input features; finally, all the outputs of the 4 branches are connected according to the channels as the final output feature F2, whose number of channels is 2048.

4. The method according to claim 3, characterized in that The shallow segmentation guided attention module includes an attention branch and a crowd counting branch. The attention branches include: Convolution layer Conv: 1×1, 1, Conv means convolution, 1×1, 1 means the convolution kernel size is 1×1, and the number of channels of the output feature is 1; Sigmoid function; Crowd counting branches include: InceptionC, the number of channels of output features is 2048; Multiplier, pixel-by-pixel multiplication; Convolutional layer: 1×1, 1.

5. The method according to claim 4, characterized in that The deep segmentation-guided attention module also includes an attention branch and a crowd counting branch.

6. The method according to claim 1, characterized in that Preprocessing the images in the training set includes: limiting the height and width of the images to be divisible by 8, and adding the images in the training set after randomly flipping the images in the training set to the training set, so as to increase the number of images in the training set.

7. A computer program product encoded on a computer storage medium, comprising instructions which, when executed by one or more computers, will cause the one or more computers to perform the method of any one of claims 1 to 6.